DOCSEARCHbyAlgolia
Submit Docs

verifiers

repository·main·Indexed 26 days ago

https://github.com/primeintellect-ai/verifiers

A library for creating specialized environments used in the training and evaluation of Large Language Models (LLMs), specifically optimized for reinforcement learning workflows. It integrates with the Prime Intellect Environments Hub, the prime-rl training framework, and the Prime Intellect Hosted Training platform. The library provides tools for managing TaskSets, Harnesses, and sandboxes, including specialized support for SWE (Software Engineering) tasksets, BrowserEnv for browser automation, and MCPEnv for Model Context Protocol server integration.

Tokens
65.3K
Snippets
148
Records
346
Agent score
88%

What's inside verifiers

  1. Overview of verifiers

    main
    The verifiers library is designed for creating environments used to train and evaluate Large Language Models (LLMs). It is tightly integrated with the Prime Intellect Environments Hub, the prime-rl training framework, and the Prime Intellect Hosted Training platform.

  3. Understand the Episode and Trace relationship

    main
    In the verifiers framework, an Episode represents the global view of a single Env.run invocation, containing all agent interactions. A Trace represents an individual agent's local view of a single rollout. Every time an agent's run method is called within an environment, it automatically joins the current Episode as a Trace.

  4. Understand the verifiers framework concepts

    main

    The verifiers framework is designed for defining tasks, running agents and harnesses, scoring them, and using them for evaluations and reinforcement learning. Key components include:

    • Taskset: A collection and loader for work. It combines serializable TaskData (prompts, files, references, resource requirements) with task class behavior (lifecycle hooks, tools, metrics, and rewards). The load() method constructs these objects and declares types via Taskset[TaskT, ConfigT].
    • Harness: The program in which the model runs (e.g., Claude Code, Codex, or mini-swe-agent).
    • Agent: A reusable value composed of a harness × model × runtime policy. It is executed via agent.run(task) -> Trace.
    • Environment: The control flow between agents (e.g., how many agents run on a task, their order, and how they are judged). Using --env.id allows pairing a reusable interaction (like best-of-n or a judge) with any taskset.
    • Toolset: A set of tools defined by a taskset, installed as MCP servers into compatible harnesses.
    • Trace: A record of the message graph, rewards, metrics, errors, and ModelCall records (containing model, sampling, finish reason, usage, and timing) for every provider exchange.

  5. Use the Legacy Composable Task/Agent Architecture

    main

    The legacy experimental taskset/harness stack separates what to solve (the task) from how to solve it (the agent). It uses a TaskSet to define problems and a Harness to define agent execution. These are wired together using a ComposableEnv.

    Note: This is a legacy stack. New environments should use the verifiers.v1 Taskset / Harness format (vf.Env, vf.Taskset, and vf.Harness) instead.

  7. Understand the interception server mechanism

    main

    In the verifiers architecture, the harness does not call provider endpoints (like OpenAI or Anthropic) directly. Instead, model traffic is routed through an interception server via a local connection or Prime Tunnel.

    The interception server performs the following functions:

    • API Mapping: It forwards requests to the correct endpoint expected by the harness (e.g., mapping Codex requests to OpenAI Responses or Claude Code requests to the Anthropic Messages API).
    • Live Tracing: Builds traces live to collect trajectories as they occur.
    • Parameter Injection: Allows setting sampling parameters in harnesses that may not natively expose those settings.
    • Reward Hack Prevention: Intercepts and rewrites tool responses or server-side web search results to block reward hacks.

  8. Configure Hosted Training via prime lab setup

    main

    Hosted Training (available in the Lab platform) allows for automatic model training via prime-rl without managing infrastructure. It supports LoRA and any environment built with Verifiers.

    To download example configuration files for Hosted Training, evaluation, and GEPA, run the prime lab setup command. This populates your workspace with TOML configs in configs/.

    prime lab setup

  9. View model outputs and rollout details

    main

    Depending on how you are running your evaluations, use the following methods to inspect model outputs:

    Using the CLI

    If you used prime eval run, results are saved automatically. Use the interactive TUI to browse them:

    prime eval view

    TUI Controls:

    • Enter: Open rollout details
    • b: Go back
    • tab: Cycle panes
    • e / x: Expand or collapse history
    • pageup / pagedown: Scroll history
    • c: Copy Mode

    Using the Python API

    If using env.generate() or env.evaluate(), use the vf.print_prompt_completions_sample utility to inspect a sample of outputs.

    vf.print_prompt_completions_sample(outputs, n=3)

  10. Run evaluations via CLI or TOML

    main

    You can evaluate tasksets using the eval entrypoint. You can either pass the taskset ID directly via the CLI or use a .toml configuration file for complex setups. CLI arguments using dotted notation (e.g., --sampling.temperature 0.5) will overwrite values defined in a TOML file.

    To validate a configuration without running it: uv run eval @ config.toml --dry-run

    To run the evaluation: uv run eval @ config.toml

  11. Evaluate Tasksets with the Prime CLI

    main

    To run and evaluate verifiers tasksets, use the prime eval run command. You can specify the environment using a taskset ID or a Hub package.

    Taskset ID Resolution:

    • my-taskset: Resolves an importable local package.
    • owner/name: Installs a Hub package on demand.
    • owner/name@version: Pins a specific Hub version.

    Core Workflow:

    1. Dry Run: Validate configuration without making model calls: prime eval run <MY_ENV> --dry-run
    2. Gold Validation: If the taskset implements validate, run model-free validation: prime eval validate <MY_ENV> --runtime.type subprocess
    3. Smoke Test: Run a small sample to verify correctness: prime eval run <MY_ENV> -m <MODEL> -n <NUM_TASKS> -r <NUM_RETRIES>
    4. Scale: Only scale to full runs after verifying task loading, harness capability, runtime lifecycle, and scoring.
    prime eval run <MY_ENV>

  12. Install and set up verifiers locally

    main

    To run verifiers locally, you need to install uv, clone the repository, and synchronize the dependencies. Once set up, you can run tasksets using uv run eval <taskset-id> or scaffold new tasksets using uv run init <name>.

    curl -LsSf https://astral.sh/uv/install.sh | sh
git clone https://github.com/PrimeIntellect-ai/verifiers.git
cd verifiers
uv sync