ARC Lang

The ARC v2 solution uses a natural language evolution approach rather than Python code generation. It leverages Large Language Models (LLMs) to both generate transformation rules (instructions) and apply those rules to grids. The system operates through a multi-phase pipeline:

1. Instruction Generation: Creating initial rules based on training examples.
2. Instruction Following: Applying those rules to a test input.
3. Revision: Using feedback (errors and visual diffs) to refine instructions.
4. Scoring: Evaluating instruction quality via cross-validation and grid similarity.

This approach prioritizes expressiveness and flexibility, allowing models to describe nuanced patterns that might be difficult to capture in rigid code.

When instructions fail to produce correct outputs, the system triggers a revision phase. The REVISION_PROMPT provides the model with the incorrect outputs and compares them to the ground truth using Visual Diff Notation.

Visual Diff Notation Format:

• ✓ indicates a correct cell.
• actual→expected indicates an error.
• A bordered ASCII grid is used for clarity.

To ensure structured reasoning, the model is expected to respond using the ReviseInstructionsResponse format, forcing it to analyze failures before proposing fixes.

# Visual Diff Example
+-------+-------+-------+
|  ✓0   |  2→3  |  ✓5   |
+-------+-------+-------+
|  1→0  |  ✓4   |  ✓5   |
+-------+-------+-------+

# Expected Response Structure
class ReviseInstructionsResponse:
    reasoning_for_why_old_instructions_are_wrong: str
    revised_instructions: str

The system manages a population of instruction candidates through a specific lifecycle to maximize performance within token limits:

1. Initial Phase: Generate 30 candidates $\rightarrow$ Filter to top performers.
2. Individual Revision: Take top 5 $\rightarrow$ Generate 5 revised versions.
3. Pooled Revision: Take top 5 $\rightarrow$ Synthesize 5 versions from successful elements of different approaches.

Total limit: Maximum 40 instruction attempts per task.

To inspect why a model failed or to see its reasoning, use the following methods:

1. Visual Comparisons: Set VIZ=1 in your environment to automatically open matplotlib windows comparing predicted training grids against target grids.
2. Grid Logging: Set LOG_GRIDS=1 to dump the expected vs. actual grids directly into the logs.
3. Manual Diffing: Use generate_grid_diff (available in src/run.py) in a notebook to generate ASCII diffs between two grids.
4. Intermediate Inspection: Check the temp_solutions/ directory to see the model's current guesses for test grids during a run.

The instruction generation phase uses a prompt designed to prime the model for pattern recognition. It emphasizes role-playing as an expert, consistency across all training examples, and generalization to unseen test cases.

Grids are provided in multiple formats to optimize comprehension:

• ASCII Text Format: Space-separated integers.
• Base64 Images: Visual representations for vision-capable models.
• Structured Context: Explicit labeling of --Training Examples-- and --Test Input--.

--Training Examples--
Training Example 1
Input:
0 0 0 5 0
0 5 0 0 0
0 0 0 0 0
Output:
5 0 0 0 0
0 0 0 0 0
0 0 0 0 0
--End of Training Examples--
--Test Input--
0 0 0 0 0
0 0 0 0 0
0 0 0 0 0

ARC Lang requires Python 3.12+. You can install the dependencies using uv or pip.

uv sync

The main entry point is src/run.py. By default, it is configured to run the 2025 evaluation challenges using the grok_config_prod preset with a limit of 1 task for smoke-testing.

python src/run.py

To run a full sweep or use a different configuration, you can call run_from_json from your own Python script.

The system follows an iterative process to solve ARC challenges:

1. Generation: get_instructions_from_challenge creates initial instructions.
2. Scoring: get_instruction_scores evaluates multiple instruction attempts on the training set.
3. Revision: If scores aren't perfect, get_revised_instructions uses feedback from failed attempts to improve the instructions.
4. Pooling (Optional): get_pooling_instruction_from_scores can synthesize a single set of instructions from multiple high-scoring (but potentially flawed) attempts.
5. Final Guess: Once the best instructions are found, return_answer generates final output grids for the test set using the final_follow_model.

Common issues and solutions:

• KeyError on import: You have not defined the MAX_CONCURRENCY environment variable.
• Authentication Errors: Occur inside get_next_structure. Verify your API keys and provider quotas.
• Rate Limiting: If you see repeated retry_failed logs, the provider is rate-limiting you. Solution: Lower max_concurrent_tasks or MAX_CONCURRENCY.
• Matplotlib Errors on Headless Servers: Occur if VIZ=1 is set on a server without a display. Solution: Set VIZ=0.

You can programmatically execute a solve by calling run_from_json. This allows you to specify custom JSON paths, different RunConfig presets, and control the scope of the run via limit and offset.

await run_from_json(
    challenges_path=challenges_path,
    truth_solutions_path=None,      # disable automatic scoring
    config=mini_config,             # use the faster preset
    attempts_path=attempts_path,
    temp_attempts_dir=temp_attempts_path,
    limit=None,
    offset=0,
)