Prime Intellect CLI & SDK

The EvalsClient is the primary synchronous interface for managing model evaluation lifecycles. The typical workflow involves:

1. Initialize: Create an APIClient and pass it to EvalsClient.
2. Create: Call create_evaluation to register a new evaluation session.
3. Push: Use push_samples to upload model outputs and scores.
4. Finalize: Call finalize_evaluation to close the evaluation and submit final metrics.

Commonly used methods include get_evaluation, list_evaluations, and get_samples for retrieving data.

from prime_evals import APIClient, EvalsClient

# Initialize client
api_client = APIClient(api_key="your-api-key")
client = EvalsClient(api_client)

# Create an evaluation
eval_response = client.create_evaluation(
    name="gsm8k-gpt4o-baseline",
    model_name="gpt-4o-mini",
    dataset="gsm8k",
    framework="verifiers",
    metadata={
        "version": "1.0",
        "num_examples": 10,
        "temperature": 0.7,
    }
)

eval_id = eval_response["evaluation_id"]

# Push samples
samples = [
    {
        "example_id": 0,
        "reward": 1.0,
        "correct": True,
        "answer": "18",
        "prompt": [{"role": "user", "content": "What is 9+9?"}],
        "completion": [{"role": "assistant", "content": "The answer is 18."}],
    }
]
client.push_samples(eval_id, samples)

# Finalize with metrics
metrics = {
    "avg_reward": 0.87,
    "avg_correctness": 0.82,
    "success_rate": 0.87,
}
client.finalize_evaluation(eval_id, metrics=metrics)

For tasks that should continue after the API call returns (e.g., training jobs), use start_background_job.

Key Concepts:

• Polling: Use get_background_job to poll for completion. Once status.completed is true, you can access status.stdout and status.exit_code.
• VM vs Container: get_background_jobs (plural) is VM-only and allows checking up to 100 jobs across sandboxes in one request. Container sandboxes use the standard get_background_job polling.
• Output Retrieval: For latency-sensitive polling, use get_background_job_statuses (status-only). Use get_background_job to fetch the full hydrated result (including stdout/stderr) after completion.
• Configuration: You can tune client-wide limits for background_job_output_concurrency, background_job_output_queue_size, and background_job_output_cache_bytes.

from prime_sandboxes import APIClient, SandboxClient, CreateSandboxRequest

sandbox_client = SandboxClient(APIClient())

# Create sandbox with extended timeout
sandbox = sandbox_client.create(CreateSandboxRequest(
    name="training-job",
    docker_image="python:3.11-slim",
    timeout_minutes=1440,  # 24 hours
    cpu_cores=4,
    memory_gb=16,
))
sandbox_client.wait_for_creation(sandbox.id)

# Start a long-running job in the background
job = sandbox_client.start_background_job(
    sandbox.id,
    "python train.py --epochs 100"
)

# Poll for completion
import time
while True:
    status = sandbox_client.get_background_job(sandbox.id, job)
    if status.completed:
        print(f"Job finished with exit code: {status.exit_code}")
        print(status.stdout)
        break
    print("Still running...")
    time.sleep(30)

# Download results
sandbox_client.download_file(sandbox.id, "/app/model.pt", "./model.pt")

Error Handling

• on_error mode: By default, on_error="warn" prevents platform errors from escaping your loop. Use on_error="raise" for CI/testing to surface failures from flush() or finish().
• Platform Errors: Errors raised by the platform belong to the prime_traces exception family.

Reliability and Degradation

• Streaming: Records are uploaded on a background thread. log_*() calls are queue puts and are safe inside coroutines.
• Transient Failures: A transient failure costs one batch. Three consecutive failures will cause the sink to retire (training runs pause for 5 minutes).
• Queue Overflow: If the queue is full, the SDK drops records rather than stalling the run. Monitor run.dropped_records and run.failed_records.

Lifecycle and Status

• Automatic Completion: Using with run: ensures the run is closed. Exceptions mark it failed, Ctrl-C marks it cancelled, and unexpected exits are reported as crashed via an atexit hook.
• Status Vocabulary: The platform uses completed or failed. cancelled and crashed are reported as failed with the specific reason in error_message.
• Draining: finish() attempts to upload queued data up to finish_timeout (default 300s). You can specify a custom timeout via finish(timeout=...).

The prime lab interface uses a three-pane shell model:

1. Section Nav: Switch between Environments, Training, Evaluations, and Settings.
2. Selector Pane: View scoped rows, grouped trees, or settings lists.
3. Inspector Pane: View details for the highlighted object.

Global Keyboard Shortcuts

• w: Open the Welcome screen.
• c: Open the Agent (full-screen assistant).
• s: Jump to Settings.
• /: Filter the active selector.
• Enter: Open, confirm, or enter the selected control.
• Esc: Go back from child screens or clear transient UI.

Pane Navigation

• Section Tree: Highlighting a section switches it immediately.
• Rows: Enter opens the row; Left returns to the nearest control/section nav; Right enters the next selector.
• Settings: Use Left/Right to switch categories, Up to stay, and Down/Enter to enter a category's row list.
• Evaluations (Trees): Left moves to parent; Right expands or enters the next child; Enter opens/toggles.

Use these commands to validate the prime lab environment, run tests, or render screens for development.

# View help for prime lab and doctor
env UV_CACHE_DIR=/private/tmp/uv-cache uv run --project packages/prime prime lab --help
env UV_CACHE_DIR=/private/tmp/uv-cache uv run --project packages/prime prime lab doctor --help

# Render all Lab screens for visual inspection
env UV_CACHE_DIR=/private/tmp/uv-cache uv run --project packages/prime python packages/prime/dev/render_lab_screens.py --screen all --width 120

# Run Lab view and package tests
env UV_CACHE_DIR=/private/tmp/uv-cache uv run --project packages/prime pytest packages/prime/tests/test_lab_view.py -q
env UV_CACHE_DIR=/private/tmp/uv-cache uv run --project packages/prime pytest packages/prime/tests -q

# Run linting (Ruff)
env UV_CACHE_DIR=/private/tmp/uv-cache uv run --project packages/prime ruff check packages/prime/src/prime_lab_app packages/prime/tests/test_lab_view.py
env UV_CACHE_DIR=/private/tmp/uv-cache uv run --project packages/prime ruff format --check packages/prime/src/prime_lab_app packages/prime/tests/test_lab_view.py

The Agent is an experimental full-screen chat assistant accessed via c or the Agent row. It can request native widgets to assist with complex tasks.

Agent Interaction

• Shift+Enter: Insert a newline.
• /: Open command actions.
• ?: Insert starter prompts.
• @: Insert Lab references.
• Up/Down: Traverse prompt history.

Agent Capabilities & Widgets

Agents can trigger native UI widgets for:

• choose: Making ambiguous decisions.
• search_environments: Looking up Hub environments.
• train_model: Resolving hosted-training launch fields.
• edit_config: Editing eval/RL/GEPA configurations.
• launch_run: Launching a selected configuration.
• preview_action, show_patch, inspect_rollouts: Reviewing side effects.

Note: Training launches created via Agent are written to .prime/lab/configs/rl/. The platform-generated name is used instead of any agent-proposed name.