Evalite Documentation

Evalite is a TypeScript-native, local-first tool designed for testing LLM-powered applications. It is built on top of Vitest and provides a terminal UI for quick prototyping. Key features include:

• Open Source: No API keys are required for the core tool.
• Local-first: Runs entirely on your machine to ensure data privacy.
• Extensible: Supports tracing and custom scorers.
• Vitest-based: Leverages the Vitest TypeScript test runner.

Evalite is a TypeScript-native tool designed to simplify testing AI-powered applications. Key features include:

• TypeScript-native testing: Use .eval.ts files with an API that is familiar to TypeScript developers.
• Local Interactive UI: Run a local dev server to explore outputs, traces, and logs in an interactive interface.
• Vitest Integration: Built on top of Vitest, allowing you to use existing mocks, fixtures, and testing patterns.
• LLM Agnostic: No vendor lock-in; you can work with any LLM and compare outputs from different models.
• CI/CD Ready: Export evaluations as static HTML bundles and use score thresholds to determine build success or failure.

Evalite is a local-first, TypeScript-native tool designed for testing LLM-powered apps. Key features include:

• Local Execution: Runs entirely on your machine with a local server on localhost and live reload, ensuring data privacy and no vendor lock-in.
• Standard Tooling: Based on Vitest, allowing the use of familiar mocks and lifecycle hooks.
• Observability: Supports capturing traces and building custom scorers.
• File Format: Uses .eval.ts files for defining evaluation logic.

A standard Starlight project follows this directory structure:

• src/content/docs/: The primary location for documentation. Starlight automatically exposes .md or .mdx files in this directory as routes based on their filenames.
• src/assets/: Place images here to embed them in Markdown using relative links.
• public/: Place static assets like favicons here.
• astro.config.mjs: The Astro configuration file.
• package.json: Project dependencies and scripts.
• tsconfig.json: TypeScript configuration.

.
├── public/
├── src/
│   ├── assets/
│   ├── content/
│   │   ├── docs/
│   │   └── config.ts
│   └── env.d.ts
├── astro.config.mjs
├── package.json
└── tsconfig.json

To measure variance in non-deterministic evaluations, you can run each test case multiple times using trialCount.

• Global configuration: Set trialCount in evalite.config.ts to apply it to all evals.
• Per-eval override: Pass trialCount directly into the evalite() function call. Per-eval settings take precedence over the global config.

// Global configuration
import { defineConfig } from "evalite/config";

export default defineConfig({
  trialCount: 3, // Run each test case 3 times
});

// OR Per-eval override

evalite("Non-deterministic eval", {
  data: () => [{ input: "Alice", expected: "Alice" }],
  task: async (input) => {
    // Non-deterministic task
    return getRandomGreeting(input);
  },
  scorers: [
    /* ... */
  ],
  trialCount: 5, // Override config: run 5 times
});

The mode option determines how the runner behaves:

• run-once-and-exit: Runs all evaluations once and then terminates. This is the recommended mode for CI/CD pipelines.
• watch-for-file-changes: Watches for file changes and automatically re-runs evaluations. This mode also starts the Evalite UI server.

Note: When using watch-for-file-changes, it is strongly recommended to implement a caching layer for your LLM calls to avoid excessive API costs during rapid re-runs.

In Evalite, Evals are the equivalent of tests for AI-powered applications. While traditional tests provide a binary pass/fail metric, Evals provide a score (typically from 0-100) representing how well an AI application is performing a specific task.

Evalite uses .eval.ts files instead of standard .test.ts files to define these evaluations.

You can target specific evaluation files by passing the file path as an argument to the CLI. This works with the standard execution, watch, and serve modes.

evalite my-eval.eval.ts
evalite watch my-eval.eval.ts
evalite serve my-eval.eval.ts

Use the --threshold flag to define a minimum required score. If the evaluation score is lower than the specified threshold, the process will fail. This is particularly useful for CI/CD pipelines to ensure code quality.

evalite --threshold=50
evalite watch --threshold=70

To create a new project using the Starlight template, use the Astro installation command with the Starlight template flag.

npm create astro@latest -- --template starlight