DOCSEARCHbyAlgolia
Submit Docs

Poetiq ARC-AGI Solver

repository·main·Indexed 23 days ago

https://github.com/poetiq-ai/poetiq-arc-agi-solver

A reasoning engine designed for the Abstraction and Reasoning Corpus (ARC-AGI) benchmarks. It provides a framework for reproducing performance on ARC-AGI-1 and ARC-AGI-2 tasks, featuring a solver that integrates with models via API keys (e.g., Gemini, OpenAI) and generates Kaggle-formatted predictions and token usage statistics.

Tokens
890
Snippets
2
Records
5
Agent score
76%

What's inside poetiq-arc-agi-solver

  1. Install and set up Poetiq for ARC-AGI reproduction

    main

    To reproduce Poetiq's ARC-AGI results, follow these steps to set up a Python virtual environment and install the necessary dependencies.

    Prerequisites

    • Python 3.11 or higher
    • API keys for the models you intend to test (e.g., GEMINI_API_KEY, OPENAI_API_KEY).

    Installation Steps

    1. Create and activate a virtual environment:
      python -m venv .venv
source .venv/bin/activate
    2. Install dependencies:
      pip install -r requirements.txt
    3. Configure environment variables: Create a .env file in the root directory and add your API keys:
      GEMINI_API_KEY=your_key_here
OPENAI_API_KEY=your_key_here
    python -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt

  2. Run the Poetiq solver

    main

    After setting up the environment and .env file, you can run the solver by following these steps:

    1. Configure the run: Open main.py and modify the constants to specify your desired problem set, the number of problems to solve, and other execution parameters.
    2. Select a configuration: By default, the code runs the Poetiq 3 configuration. You can choose different configurations by uncommenting other options in main.py or by modifying the settings in config.py.
    3. Execute: Run the script using the following command:
      python main.py
    python main.py

  3. Run the ARC-AGI solver via main.py

    main

    The main.py script serves as the primary entrypoint for running the ARC-AGI solver. It loads challenges from a JSON file, attempts to solve them using the solve function, and optionally scores the results if a solutions file is provided.

    Key configuration variables in main.py that you can modify to control the run:

    • NUM_PROBLEMS: Set to an integer to limit the number of problems solved (e.g., 5). Set to None to run all available problems.
    • SELECTED_PROBLEMS: A list of specific task IDs to solve (e.g., ['b7999b51']).
    • DATA_CHALLENGES: Path to the challenge input JSON file.
    • DATA_SOLUTIONS: Path to the optional solution JSON file for scoring.
    • OUTPUT_DIR: Directory where results will be saved.

    The script automatically generates a timestamped submission file (e.g., submission_2023-10-27_12-00-00.json) and a token usage file (tokens_TIMESTAMP.json) in the output directory.

  4. Configure solver execution via main.py variables

    main

    You can customize the solver's behavior by editing the following variables in main.py:

    VariableTypeDescription
    NUM_PROBLEMSOptional[int]Limits the number of problems to solve. None runs all.
    SELECTED_PROBLEMSlist[str]A list of specific task IDs to run (e.g., ['task_id_1', 'task_id_2']).
    DATA_CHALLENGESstrPath to the arc-agi_evaluation_challenges.json file.
    DATA_SOLUTIONSstrPath to the arc-agi_evaluation_solutions.json file. If missing, scoring is disabled.
    OUTPUT_DIRstrThe directory where submission_*.json, tokens_*.json, and config_*.json are written.

  5. Understand the solver output files

    main

    After a run, the following files are generated in the OUTPUT_DIR (defaulting to output/):

    1. submission_{TIMESTAMP}.json: A JSON object mapping task IDs to Kaggle-formatted predictions (two attempts per task).
    2. tokens_{TIMESTAMP}.json: A JSON object mapping task IDs to token usage statistics (prompt, completion, and total).
    3. config_{TIMESTAMP}.json: A dump of the CONFIG_LIST used during the run for reproducibility.

    Example structure of submission_*.json:

    {
  "task_id_1": [
    { "prediction": [...] },
    { "prediction": [...] }
  ]
}