Next Evals Oss

Next.js Evals

Agent evaluations for Next.js coding tasks, powered by [`@vercel/agent-eval`](https://www.npmjs.com/package/@vercel/agent-eval).

Each eval hands a coding agent a small Next.js app and a prompt, lets it work in an isolated sandbox, then runs withheld assertions against what it produced.

Quick start

pnpm bootstrap                 # install, sync eval fixtures, check credentials
cp .env.example .env.local     # then fill in the credentials it asked for
pnpm preflight                 # confirm they resolve
pnpm eval:smoke claude-opus-5  # one eval, one run, real sandbox

`pnpm bootstrap` is not required — it just runs the three setup steps below in order, because the middle one is easy to miss.

Setup

1. Install

pnpm install --frozen-lockfile

pnpm only: the lockfile and `packageManager` pin are pnpm, and CI installs frozen.

2. Sync the eval fixtures — required

pnpm sync-evals          # from vercel/next.js@canary
pnpm sync-evals     # ...or a branch, tag, or commit SHA

**The evals are not in this repo.** They live in [`vercel/next.js`](https://github.com/vercel/next.js/tree/canary/evals/evals) and `evals/` is git-ignored here, so a fresh clone has no fixtures and every command fails with `Evals directory not found`. `sync-evals` sparse-checkouts them.

Syncing from `canary` picks up whatever landed upstream since results were last recorded, so it usually reports evals as changed:

N result(s) have a changed eval and were left stale — run them to refresh.

That is expected, not a problem — it is the incremental workflow telling you what `pnpm status` will now list as work. To match CI instead, pass the SHA that [`.github/workflows/eval-cache-check.yml`](.github/workflows/eval-cache-check.yml) pins.

3. Provide credentials

Copy [`.env.example`](.env.example) to `.env.local` and fill in what you need. `pnpm preflight` reports exactly which variables are missing, and for which expe