DOCSEARCHbyAlgolia
Submit Docs

ARC-AGI Benchmarking Framework

repository·main·Indexed 43 days ago

https://github.com/arcprize/arc-agi-benchmarking

A benchmarking framework for testing AI models against the Abstraction and Reasoning Corpus (ARC-AGI) tasks. It features support for multiple model providers, automated batch execution, rate limiting, scoring scripts, and a multi-config launcher for concurrent matrix execution across different models and datasets. Version 0.1.0.

Tokens
14K
Snippets
42
Records
58
Agent score
56%

What's inside ARC-AGI Benchmarking

  1. Quickstart: Install and run a single-task dry run

    main

    To get started with ARC-AGI benchmarking, clone the repository and use uv to manage dependencies. You can perform a dry run using the random-baseline adapter to verify your setup without needing API keys.

    1. Clone and Install:

      git clone https://github.com/arcprize/arc-agi-benchmarking.git
cd arc-agi-benchmarking
uv sync

    2. Run a single-task dry run:

      uv run main.py \
  --data_dir data/sample/tasks \
  --config random-baseline \
  --task_id 66e6c45b \
  --save_submission_dir submissions/random-single
    git clone https://github.com/arcprize/arc-agi-benchmarking.git
cd arc-agi-benchmarking
uv sync

uv run main.py \
  --data_dir data/sample/tasks \
  --config random-baseline \
  --task_id 66e6c45b \
  --save_submission_dir submissions/random-single

  2. Score ARC-AGI task outputs

    main

    After generating submissions, use the scoring script to evaluate them against the task directory. You must provide the directory containing the original tasks (which includes solutions), the directory where your model outputs were saved, and a destination for the results.

    uv run src/arc_agi_benchmarking/scoring/scoring.py \
  --task_dir data/sample/tasks \
  --submission_dir submissions/random-baseline-sample \
  --results_dir results/random-baseline-sample

  3. How to add a new model adapter

    main

    To support a new provider, follow these steps:

    1. Implement the Adapter: Create a new file src/arc_agi_benchmarking/adapters/<provider>.py that implements the ProviderAdapter interface.
    2. Export the Adapter: Add the new adapter to src/arc_agi_benchmarking/adapters/__init__.py.
    3. Register in Factory: Add a branch in main.py (and any relevant factories) so the provider name is recognized by the runner.
    4. Update Configs: Add a new entry in src/arc_agi_benchmarking/models.yml pointing to your new provider: "<provider>".
    5. Test (Optional): Add tests for the new adapter and its parsing logic.

  4. Configure models in models.yml

    main

    Model configurations are defined in src/arc_agi_benchmarking/models.yml. Each entry specifies how the benchmark interacts with a specific model and provider.

    Key fields:

    • name: The identifier used on the CLI via --config.
    • model_name: The actual model ID used by the provider.
    • provider: The adapter name (e.g., openai, anthropic).
    • api_key_env: The name of the environment variable containing the API key (required for non-random providers).
    • pricing: An object containing input and output costs per 1M tokens and a date.
    • Provider kwargs: Any additional keys (like temperature or max_output_tokens) are passed directly to the provider's SDK.

    Example configuration:

    - name: "gpt-4o-2024-11-20"
  model_name: "gpt-4o-2024-11-20"
  provider: "openai"
  api_key_env: "OPENAI_API_KEY"
  max_output_tokens: 4096
  temperature: 0.0
  pricing:
    date: "2024-11-20"
    input: 5.00
    output: 15.00

  5. Use the Claude Agent SDK Adapter for ARC-AGI

    main

    The ClaudeagentsdkAdapter is a specialized provider adapter designed to leverage the Claude Agent SDK's agentic reasoning capabilities for ARC-AGI tasks. It uses an iterative reasoning approach by providing the model with a task-specific scratchpad directory.

    Key Features:

    • Agentic Reasoning: Automatically prepends instructions to the prompt to encourage the model to use a scratchpad for hypothesis testing and pattern analysis.
    • Scratchpad Support: Provides a file-based scratchpad in /tmp/arc_agi_scratchpad/ for each task, allowing the model to use Write, Read, and Edit tools.
    • Tool Sandboxing: Automatically configures a safe environment by allowing reasoning tools (TodoWrite, Write, Read, Edit, Task) while blocking risky tools like Bash, WebFetch, or WebSearch to prevent information leakage.
    • Automatic Session Management: Each call to make_prediction creates a new session with a unique, timestamped scratchpad directory.
    from arc_agi_benchmarking.adapters.claudeagentsdk import ClaudeagentsdkAdapter

# Note: Initialization requires a valid model configuration and ANTHROPIC_API_KEY
adapter = ClaudeagentsdkAdapter("claudeagentsdk-sonnet-4")

  6. How the --datasets argument works

    main

    When using the --datasets flag, you must provide a list of mappings in the format NAME=PATH.

    • NAME: A non-empty relative path without ... This name is used to construct the output directory structure for submissions and logs.
    • PATH: The actual filesystem path to the dataset directory.

    Constraints:

    • Dataset names must be unique.
    • Names cannot be absolute paths, ., or contain ...
    • You cannot combine --datasets with --run_name or --data_dir (the latter is forwarded to the child process and conflicts with the matrix logic).
    --datasets v1/public_eval=data/v1/public_eval v2/public_eval=data/v2/public_eval

  7. Use preflight cost validation in run_all.py

    main

    The run_all.py script includes a preflight validation step to prevent unexpected costs. If --cost-limit is provided, the script calculates an estimated cost based on the configuration. If the estimated_cost exceeds the specified limit, the script logs an error and exits with code 1 before any tasks are run.

    You can bypass this check using the --skip-preflight flag.

  8. View per-task logs and raw API events

    main

    The orchestrator generates structured logs for every task.

    1. Application Logs: General execution flow and ARCTester output.
    2. Raw API Logs: Detailed JSONL logs containing raw API events (e.g., OpenAI SDK logs).

    Logs are organized under the --logs-base-dir (default: logs). Each task has its own file located at: <logs_base_dir>/<task_id>.jsonl.

  10. Configure task execution behavior

    main

    When running the orchestrator, you can control how it handles task failures and progress:

    • Resuming Progress: By default, the orchestrator uses --resume logic to skip tasks that already have complete submissions in the --save_submission_dir. To re-run everything, use --no-resume in conjunction with --overwrite_submission.
    • Concurrency Control: Use --max-concurrency to limit the number of in-flight tasks per provider. This is useful when running multiple run_all.py instances that need to share a provider's capacity.
    • Error Resilience: The orchestrator implements a Circuit Breaker pattern. If a provider reaches a certain failure threshold (configured via --circuit-breaker-threshold), the orchestrator will stop attempting tasks for that provider to prevent further wasted resources/costs.

  11. Run ARC tasks concurrently with cli/run_all.py

    main

    The cli/run_all.py script is a CLI entrypoint used to orchestrate the concurrent execution of ARC tasks using a specific model configuration. It handles rate limiting, concurrency control, circuit breaking, and automatic resumption of incomplete tasks.

    Tasks can be specified via a .txt file containing task IDs (one per line) or automatically inferred from all .json files within a provided --data_dir.

    python cli/run_all.py --config my-model-config --data_dir data/tasks --save_submission_dir submissions

  12. Launch multiple benchmark configurations concurrently with run_configs.py

    main

    The run_configs.py CLI tool allows you to launch multiple ARC model configurations and dataset specifications concurrently. It acts as a wrapper around cli/run_all.py, managing concurrency, rate-limiting across providers, and organizing output directories for submissions and logs.

    Key Features

    • Concurrency Management: Control the maximum number of in-flight ARC tasks per provider using --max-concurrency.
    • Matrix Execution: Run a combination of multiple model configurations (--configs) against multiple datasets (--datasets).
    • Automatic Rate Limiting: Automatically calculates a rate_limit_divisor for each provider based on how many concurrent tasks are using that provider to prevent API exhaustion.
    • Argument Forwarding: Any CLI options not recognized by run_configs.py are automatically forwarded to the underlying cli/run_all.py processes.

    Usage Example

    uv run cli/run_configs.py --configs model-high model-low \
  --datasets v1/public_eval=data/v1/public_eval v2/public_eval=data/v2/public_eval \
  --save_submission_root submissions
    uv run cli/run_configs.py --configs model-high model-low --datasets v1/public_eval=data/v1/public_eval v2/public_eval=data/v2/public_eval --save_submission_root submissions