vitest-evals

If a custom judge requires an LLM call, you must configure a judgeHarness at the matcher, judge, or suite level. Inside the judge's assess function, use ctx.runJudge(...) to execute the model call.

Key Details:

• ctx.runJudge automatically handles the current abort signal.
• Each call to runJudge is recorded as a separate run in the report metadata.
• Precedence: Matcher options > Judge defaults > Suite defaults.
• If ctx.runJudge is undefined, it means no judgeHarness was configured for the context.

import { createJudge } from "vitest-evals";

export const RubricJudge = createJudge({
  name: "RubricJudge",
  async assess(ctx) {
    if (!ctx.runJudge) {
      throw new Error("RubricJudge requires a configured judgeHarness.");
    }

    const verdict = await ctx.runJudge({
      prompt: `Grade this answer: ${JSON.stringify(ctx.output)}`,
      responseFormat: { type: "json" },
    });

    return parseRubricVerdict(verdict);
  },
});

When consuming or extending the vitest-evals ecosystem, the APIs follow a specific design philosophy focused on usability and type safety. Developers should keep the following mental models in mind:

Core Philosophy

• The Common Path First: APIs are designed around the most frequent use cases. Advanced capabilities are provided via 'escape hatches' rather than cluttering the primary interface.
• Small, Native Shapes: Expect primary interfaces to use plain objects, direct values, and familiar verbs (like run) rather than complex class hierarchies.
• Layered Abstractions: Low-level primitives are kept stable. Convenience helpers are added on top of these primitives to reduce boilerplate, but they do not replace the underlying primitives.

Role-Based Responsibility

• Harness vs. Judge: The system distinguishes between the entity that executes the test and the entity that evaluates it.
  • A Harness runs the system under test.
  • A Judge assesses a run.
• Judge Configuration: Assessment prompts, model choices, and parsers reside on the Judge. When authoring a custom judge, the preferred path is using createJudge("Name", assess).

Type Safety and Context

• Strong Typing for Capabilities: Instead of methods that fail at runtime if a feature isn't configured, the API uses strong TypeScript types to return more capable objects only when specific configurations are present.
• Context Management: To avoid repetitive parameters, the API often infers context from fixtures, registered runs, or the current test. However, explicit overrides are provided for values that cannot be reliably inferred.
• Output Handling: The API prefers typed run() output over mapping hooks. If a provider result needs to be projected into application output, use semantic selectors like output rather than generic normalization knobs.
• Generic Ordering: Public generics are ordered by frequency of use, typically following the pattern Input, Output, Metadata.

When using judges within Vitest matchers or custom logic, follow these rules:

Matcher Context

• expect(result).toSatisfyJudge(...) reuses the fixture-backed run context.
• expect(result.output).toSatisfyJudge(...) can reuse the exact run when the output object came from that run.
• Raw values outside an eval context require explicit input, session, run, harness, or custom judge params if the judge depends on them.
• Normalized sessions infer output from the latest assistant message content.
• Calling ctx.harness.run(...) inside a judge executes the app again; only do this when the judge intentionally needs a second run.

Usage and Reporting

• A judge harness is separate from the app harness. Return a full HarnessRun when the provider exposes usage.
• ctx.runJudge(...) returns the judge output, while vitest-evals records the complete normalized run under the score's judgeRuns field.
• Multiple runJudge(...) calls from one judge are retained separately.
• Stable token fields and costUsd contribute to separate application, judge, and combined report totals. Provider-specific pricing details stay under usage.metadata.

The vitest-evals library uses an explicit-run pattern for AI evaluations. This pattern binds a suite-level harness to a set of tests, allowing for structured LLM-based grading (judges).

Core Components

• describeEval: Binds a suite-level harness (which manages the agent/app execution) and judges (which grade the results). It also accepts a judgeThreshold.
• run(input): Inside a test, you call run to execute the harness with the provided input. This returns the result of the agent/app execution.
• Judges: Every judge is a named object with an assess(ctx) method. They receive a JudgeContext containing the input, output, normalized run/session, and tool calls.
• toSatisfyJudge: A Vitest matcher used to assert that a result meets a specific judge's criteria.

Example Usage

import { describeEval, FactualityJudge, toolCalls } from "vitest-evals";
import { piAiHarness, piAiJudgeHarness } from "@vitest-evals/harness-pi-ai";

// 1. Setup a dedicated judge harness (the model used for grading)
const judgeHarness = piAiJudgeHarness({
  model: getModel("anthropic", "claude-sonnet-4-5"),
  temperature: 0,
});

// 2. Create a judge instance
const factualityJudge = FactualityJudge({ judgeHarness });

// 3. Define the eval suite
describeEval(
  "my eval suite",
  {
    harness: piAiHarness({ agent: () => myAgent }),
    judges: [factualityJudge],
    judgeThreshold: 0.6,
  },
  (it) => {
    it("passes a test", async ({ input }, { run }) => {
      const result = await run(input);
      
      // Assertions on structure/tools
      expect(toolCalls(result)).toBeDefined();
      
      // Assertions via the judge
      await expect(result).toSatisfyJudge(factualityJudge, {
        expected: "The agent should be polite",
        threshold: 0.6,
      });
    });
  },
);

import { getModel } from "@mariozechner/pi-ai";
import { expect } from "vitest";
import { piAiHarness, piAiJudgeHarness } from "@vitest-evals/harness-pi-ai";
import {
  describeEval,
  FactualityJudge,
  toolCalls,
} from "vitest-evals";
import { createRefundAgent } from "../src/refundAgent";

const judgeHarness = piAiJudgeHarness({
  model: getModel("anthropic", "claude-sonnet-4-5"),
  temperature: 0,
});
const factualityJudge = FactualityJudge({ judgeHarness });

describeEval(
  "demo pi refund agent",
  {
    harness: piAiHarness({
      agent: () => createRefundAgent(),
    }),
    judges: [factualityJudge],
    judgeThreshold: 0.6,
  },
  (it) => {
    it.for([
      {
        name: "approves refundable invoice",
        input: "Refund invoice inv_123",
        expected: "The refund request is approved.",
        expectedStatus: "approved",
        expectedTools: ["lookupInvoice", "createRefund"],
      },
    ])("$name", async ({ input, ...expected }, { run }) => {
      const result = await run(input);

      expect(result.output).toMatchObject({
        status: expected.expectedStatus,
      });
      expect(toolCalls(result).map((call) => call.name)).toEqual(
        expected.expectedTools,
      );
      await expect(result).toSatisfyJudge(factualityJudge, {
        expected: expected.expected,
        threshold: 0.6,
      });
    });
  },
);