DOCSEARCHbyAlgolia
Submit Docs

SIA (Self-Improving AI) Framework

repository·main·Indexed 24 days ago

https://github.com/hexo-ai/sia

SIA is a framework that autonomously improves AI models and task harnesses on specific benchmarks using a multi-agent loop consisting of Meta, Target, and Feedback agents. It supports multiple LLM providers via the sia-agent package (including Claude and OpenHands implementations) and includes a built-in web dashboard for visualizing execution trajectories, evaluation scores, and agent code across generations.

Tokens
13.9K
Snippets
40
Records
73
Agent score
84%

What's inside SIA

  2. Understand the Chess Hard Task benchmark

    main

    The Chess Hard Task is an LLM evaluation benchmark consisting of 50 challenging chess problems. It tests a model's ability to perform strategic analysis and mathematical calculation within a chess context.

    There are two primary problem types:

    1. Best 3 Moves: Given a FEN (Forsyth-Edwards Notation) position, the model must identify the three best subsequent moves using Standard Algebraic Notation (SAN).
    2. Knight Path: Given a 100x100 board and target squares, the model must calculate the minimum number of moves required for a knight to visit all target squares.

    Note that the provided chess_hard.json dataset contains the prompts but does not contain the ground truth answers.

  5. Understand the GPQA-style benchmark requirements

    main

    The GPQA-style benchmark evaluates an agent's ability to answer graduate-level multiple-choice questions in domains like biology, chemistry, and physics.

    Key Data Characteristics:

    • Dataset File: diamond_questions.json.
    • Question Structure: Each record contains an id, domain, subdomain, Question (stem), and options (a dictionary with keys A, B, C, and D).
    • Deterministic Shuffling: Options are pre-shuffled based on the question text. You must output the specific letter (A, B, C, or D) that corresponds to the correct option.

    Success Metric:

    • The primary goal is to maximize accuracy (correct answers / attempted questions).

  6. Understand the GPQA task format

    main

    Tasks in the GPQA (Graduate-level Google-Proof Q&A) benchmark follow a specific structure designed for high-difficulty STEM reasoning. When authoring or parsing these tasks, ensure they adhere to these requirements:

    • Difficulty: Graduate-level across STEM domains.
    • Structure: Multiple-choice with exactly 4 options (labeled A, B, C, D).
    • Answer Type: A single correct answer.
    • Reasoning Requirements: Tasks often require deep domain knowledge, complex calculations, conceptual synthesis, or multi-step reasoning.

    An agent interacting with these tasks must be able to:

    1. Parse the question text and all four options.
    2. Reason through the problem using domain-specific knowledge.
    3. Return a single letter answer (A, B, C, or D) in the specified format.
    4. Operate across various STEM domains (e.g., Physics, Chemistry, Biology).

  7. Understand the SIA directory layout

    main

    SIA organizes its source code, task data, and execution outputs into a specific structure. Understanding this layout is essential for locating task descriptions, inspecting agent code, or reviewing run results.

    Key Directories

    • sia/tasks/{task-id}/data/public/: Contains the task description (task.md) and public data files (*.csv).
    • sia/agent_impls/: Contains the runner backends (e.g., claude, openhands, pydantic-ai).
    • runs/run_{id}/: The root directory for a specific execution run.
      • venv/: An isolated Python environment created specifically for that run.
      • gen_{n}/: Artifacts for a specific generation (includes target_agent.py, agent_execution.json, and improvement.md).
    sia/
├── sia/
│   ├── orchestrator.py
│   ├── context_manager.py
│   ├── prompts.py
│   ├── agent_impls/
│   ├── prepare_mlebench_dataset.py
│   └── tasks/
│       └── {task-id}/
│           └── data/
│               └── public/
│                   ├── task.md
│                   └── *.csv
└── runs/
    └── run_{id}/
        ├── venv/
        └── gen_{n}/
            ├── target_agent.py
            ├── agent_execution.json
            └── improvement.md

  8. How the SIA agent loop works

    main

    SIA (Self-Improving AI) operates using a continuous loop involving three specialized agents. In each generation, the system inspects the previous attempt, rewrites the agent, and executes it again. This cycle continues until the --max_gen limit is reached.

    The Three Agents

    1. Meta-Agent: Reads the task description and generates the initial target_agent.py tailored to the specific task.
    2. Target Agent: The agent responsible for attempting the task. It records its actions and results in agent_execution.json.
    3. Feedback / Improvement Agent: Reviews the execution logs from the Target Agent, identifies areas for improvement, and rewrites the Target Agent for the subsequent generation.

    The Execution Lifecycle

    • Generation 1: The Meta-agent creates the first target_agent.py. The Target agent executes the task and logs to agent_execution.json. The Feedback agent then analyzes this run to prepare the next version.
    • Generation 2 through N: The current generation's Target agent executes the task, and the Feedback agent produces the next generation based on those results.

    Output Artifacts

    All artifacts are saved under runs/run_{run_id}/gen_{n}/. Each generation directory contains:

    • target_agent.py: The agent code for that generation.
    • agent_execution.json: The logs of the agent's actions and results.
    • improvement.md: Improvement notes (available from Generation 2 onwards).

  9. How profile and provider resolution works

    main

    SIA resolves profiles and providers using a specific hierarchy. If a value contains a / or ends in .json, it is treated as an explicit file path.

    If a bare name is provided, SIA resolves it in this order:

    1. The user directory defined by $SIA_PROFILES_DIR or $SIA_PROVIDERS_DIR (or ./profiles / ./providers locally).
    2. The bundled defaults shipped with the package.

    Usage Example:

    # Use a bundled profile name
sia run --task gpqa --target-agent-profile kimi-nebius-target

# Use an explicit path to a profile
sia run --task gpqa --target-agent-profile ./profiles/mine.json
    sia run --task gpqa --target-agent-profile ./profiles/mine.json

  10. How SIA (Self-Improving AI) works

    main

    SIA is a framework that autonomously improves the performance of an AI system (Model or Agent) on a specific benchmark task through an iterative loop. The system coordinates three distinct agent roles:

    1. Meta-Agent: Analyzes the task description and generates the initial Target Agent tailored for that task.
    2. Target / Task Specific Agent: The agent attempting to solve the task, recording its actions and results.
    3. Feedback/Improvement Agent: Reviews the Target Agent's performance logs, identifies weaknesses, and generates updates to improve the Target Agent.

    This loop allows the system to refine both the task harness and the agent's weights/logic over successive generations.

  11. Constraints for GPQA benchmark agents

    main

    When implementing an agent for the GPQA benchmark, adhere to these constraints:

    • Option Integrity: Answer only using the four provided option strings; do not invent new options.
    • Format Adherence: Follow the required output format exactly to allow automated mapping to shuffled options.
    • Independence: Treat all questions as independent. Do not attempt to solve all questions within a single loop or share context between questions.

  12. Test your evaluation script manually

    main

    Before running the full SIA orchestrator, you should verify that your evaluation script correctly identifies the submission file and generates the results.json file. Run the script manually using the --gen-dir flag pointing to a directory containing a valid submission.

    python tasks/<task_name>/data/public/evaluate.py --gen-dir runs/run_1/gen_1