DOCSEARCHbyAlgolia
Submit Docs

Agent Eval

repository·main·Indexed 42 days ago

https://github.com/vercel-labs/agent-eval

A framework for testing and measuring the performance of AI coding agents in isolated sandboxes. Version 2.3.0 allows developers to run controlled experiments, verify agent outputs via Vitest, and use agentic LLM judges for qualitative evaluation. Includes a Next.js web dashboard (@vercel/agent-eval-playground) for browsing experiment results and transcripts.

Tokens
29.3K
Snippets
87
Records
113
Agent score
53%

What's inside Agent Eval

  1. Understand the @vercel/agent-eval-playground architecture

    main

    The playground is designed as a lightweight, filesystem-driven Next.js application with the following characteristics:

    • Filesystem as Source of Truth: It uses no database; all data is read directly from the filesystem via Server Components (lib/data.ts).
    • Server-Side Data Fetching: It uses Server Components for all data fetching and contains no API routes. All data is read server-side, meaning there is no client-side fetching.
    • Client Interactivity: Client Components are used sparingly, only for interactive elements like ComparePage dropdowns or TranscriptViewer collapsibles.
    • Dynamic Data: All pages use force-dynamic to ensure that data can change between requests as new results are written to the filesystem.

    How it works internally: The CLI (bin.mjs) resolves the bundled next binary, sets RESULTS_DIR and EVALS_DIR as environment variables, and spawns next dev from the package directory.

  2. Enable Failure Classification

    main

    The framework can automatically classify failures into model (incorrect code), infra (API errors/rate limits), or timeout. This uses Claude Sonnet 4.5 via the Vercel AI Gateway.

    Requirements: Set AI_GATEWAY_API_KEY or VERCEL_OIDC_TOKEN in your environment.

    Behavior:

    • Enabled: Classifications are cached in classification.json. Non-model failures are removed by default so they can be retried. Use --ack-failures to keep them as final results.
    • Disabled: The classifier is skipped; all results are preserved.

  3. Understand the results directory structure

    main

    Results are stored in results/<experiment>/<timestamp>/. Each experiment run contains:

    • summary.json: High-level stats (pass rate, duration, fingerprint, and failure classification).
    • classification.json: Cached failure classification (if the eval failed).
    • run-N/result.json: Individual run result and observability summary.
    • run-N/transcript.json: Parsed/structured agent transcript.
    • run-N/transcript-raw.jsonl: Raw agent output for debugging.
    • run-N/outputs/: Test outputs and script logs.
    • run-N/project/: Agent-generated files (if copyFiles is enabled).

  4. How result reuse and fingerprinting work

    main

    The framework uses a SHA-256 fingerprint to avoid re-running unchanged evals. The fingerprint is a combination of the eval files and the experiment configuration.

    Fingerprint Components

    • Eval Files: The content of the test files.
    • Config Fields: agent, model, scripts, timeout, earlyExit, runs, webResearch, disableBundledSkills, sandboxImage, sandboxUser, and a pinned judge.

    Behavior

    • Adding new evals: Safe, no existing results are invalidated.
    • Changing config (e.g., model): Invalidates all evals in that experiment.
    • Editing an eval file: Only invalidates that specific eval.
    • --force flag: Use this to bypass fingerprinting and re-run everything (required if you change setup or editPrompt).

  5. Run @vercel/agent-eval-playground

    main

    The @vercel/agent-eval-playground is a standalone Next.js web dashboard used to browse agent-eval experiment results.

    To use it, run the command from your eval project root—specifically the directory where your results/ and evals/ directories are located.

    npx @vercel/agent-eval-playground

  6. Create an Eval Fixture

    main

    An evaluation (eval) tests a specific task. Each eval requires a directory containing a prompt, verification tests, and the environment setup.

    Directory Structure

    evals/
  <eval-name>/
    PROMPT.md           # The task description for the agent
    EVAL.ts             # Verification tests (use .tsx if JSX is needed)
    package.json        # Defines the framework/dependencies for the sandbox
    src/                # Starter code for the agent to work on

    File Roles

    • PROMPT.md: A markdown file describing the requirements (e.g., "Create a Button component with label and onClick props").
    • EVAL.ts / EVAL.tsx: A Vitest file used to verify the agent's output. Use .tsx if you need to use React Testing Library or render components.
    • package.json: Specifies the environment the agent will operate in.
    // Example EVAL.ts
import { test, expect } from 'vitest';
import { readFileSync, existsSync } from 'fs';
import { execSync } from 'child_process';

test('Button component exists', () => {
  expect(existsSync('src/components/Button.tsx')).toBe(true);
});

test('project builds', () => {
  execSync('npm run build', { stdio: 'pipe' });
});

  7. Perform response-only evaluations

    main

    For tasks where the primary goal is the agent's text output rather than file system changes, use validation: 'none'.

    In this mode, you still need PROMPT.md and package.json in your fixtures, but you do not need EVAL.ts or EVAL.tsx.

    const config: ExperimentConfig = {
  agent: 'vercel-ai-gateway/claude-code',
  model: 'sonnet',
  validation: 'none',
  runs: 10,
  earlyExit: false,
  brands: [
    { id: 'vercel', name: 'Vercel', aliases: ['Vercel Platform'], isYourBrand: true },
    { id: 'netlify', name: 'Netlify' },
    { id: 'railway', name: 'Railway' },
  ],
  onRunComplete: async ({ runData }) => {
    return runData;
  },
};

  8. Set up environment variables via .env

    main

    The framework uses dotenv to load environment variables. It looks for .env.local first, then falls back to .env.

    To initialize your environment, copy the provided example file and fill in your keys:

    cp .env.example .env

  9. Best practices for running experiments

    main

    To ensure reliable and cost-effective evaluations, follow these patterns:

    • Preview before running: Use the --dry flag to verify your configuration without making API calls or incurring costs.
    • Verify connectivity: Use the --smoke flag first to check that your API keys, model IDs, and sandbox are working correctly before a full run.
    • Ensure statistical significance: Avoid relying on single runs. Use multiple runs (e.g., runs: 10) and set earlyExit: false to gather meaningful data.
    • Isolate variables: When comparing experiments, change only one variable at a time (e.g., don't change both the model and the MCP tools in the same test).

  10. Quick Start with @vercel/agent-eval

    main

    To set up a new evaluation project for testing AI coding agents, follow these steps:

    1. Scaffold the project: Use npx @vercel/agent-eval init <name> to create a new directory structure.
    2. Install dependencies: Run npm install within the new directory.
    3. Configure environment variables: Copy .env.example to .env and populate it with your AI_GATEWAY_API_KEY and VERCEL_TOKEN.
    4. Preview (Optional): Run npx @vercel/agent-eval --dry to see which experiments will run without making actual API calls or incurring costs.
    5. Execute: Run npx @vercel/agent-eval to execute all experiments.
    # Create a new eval project
npx @vercel/agent-eval init my-agent-evals
cd my-agent-evals

# Install dependencies
npm install

# Add your API keys
cp .env.example .env
# Edit .env with your AI_GATEWAY_API_KEY and VERCEL_TOKEN

# Preview what will run (no API calls, no cost)
npx @vercel/agent-eval --dry

# Run all experiments
npx @vercel/agent-eval

  11. Split workspace files from test files

    main

    To prevent an agent from 'cheating' during an evaluation, you should separate files that the agent is allowed to see from files used for validation.

    Use splitTestFiles(files) to categorize SandboxFile[] into workspaceFiles (visible to the agent) and testFiles (hidden from the agent and uploaded only after completion).

    Files matching these patterns are treated as test files:

    • EVAL.ts / EVAL.tsx
    • PROMPT.md
    const { workspaceFiles, testFiles } = splitTestFiles(allFiles);

// 1. Upload workspaceFiles to sandbox
// 2. Run agent
// 3. Upload testFiles to sandbox for validation

  12. How earlyExit works in experiments

    main
    When earlyExit is enabled in your experiment configuration, the runner optimizes for speed by aborting unnecessary work. For any given fixture, once a single run (attempt) results in a passed status, all other in-flight and queued attempts for that specific fixture are immediately aborted. This is useful when you only need to know if an agent can pass a test, rather than its statistical pass rate.