DOCSEARCHbyAlgolia
Submit Docs

autoresearch

repository·master·Indexed 13 days ago

https://github.com/karpathy/autoresearch

An autonomous AI research framework (v0.1.0) that enables LLM agents to iteratively optimize neural network architectures and hyperparameters. The system uses a fixed 5-minute training budget and evaluates performance based on validation bits per byte (val_bpb) to ensure comparable experiments across different model configurations.

Tokens
3.3K
Snippets
8
Records
12
Agent score
99%

What's inside autoresearch

  1. How the research loop and evaluation works

    master

    The autoresearch loop is governed by a fixed 5-minute time budget (wall clock time, excluding startup/compilation). This design ensures:

    1. Comparability: Experiments are directly comparable regardless of changes to model size, batch size, or architecture.
    2. Optimization: The agent finds the most optimal model for your specific compute platform within that time window.

    Evaluation Metric: Experiments are judged on val_bpb (validation bits per byte). A lower value is better. This metric is vocab-size-independent, allowing for fair comparison across different architectural changes.

  2. Understand the autoresearch file structure and roles

    master

    The project is designed around three core files with distinct roles to keep the agent's scope manageable:

    • prepare.py: Contains fixed constants, data preparation (downloads training data, trains BPE tokenizer), and runtime utilities (dataloader, evaluation). Do not modify this file.
    • train.py: The primary target for the agent. It contains the GPT model, optimizer (Muon + AdamW), and the training loop. The agent iterates on this file to change architecture, hyperparameters, optimizer, or batch size.
    • program.md: The instruction set for the agent. This is the file the human edits to guide the research direction and set up the autonomous research organization.

  3. Install and set up autoresearch

    master

    To use autoresearch, you need a single NVIDIA GPU (tested on H100), Python 3.10+, and the uv project manager. Follow these steps to install dependencies and prepare the environment:

    1. Install uv:
      curl -LsSf https://astral.sh/uv/install.sh | sh
    2. Install dependencies:
      uv sync
    3. Prepare data and tokenizer (one-time setup, ~2 min):
      uv run prepare.py
    4. Verify setup by running a single manual training experiment (~5 min):
      uv run train.py
    # 1. Install uv project manager (if you don't already have it)
curl -LsSf https://astral.sh/uv/install.sh | sh

# 2. Install dependencies
uv sync

# 3. Download data and train tokenizer (one-time, ~2 min)
uv run prepare.py

# 4. Manually run a single training experiment (~5 min)
uv run train.py

  4. Run and evaluate training experiments

    master

    Experiments are run using uv run train.py. Each run has a fixed wall-clock training time budget of 5 minutes (excluding startup/compilation).

    Constraints

    • Modifiable: You can only edit train.py. You may change model architecture, optimizer, hyperparameters, batch size, and model size.
    • Read-only: Do not modify prepare.py. It contains the fixed evaluation harness (evaluate_bpb), data loading, and training constants.
    • Dependencies: You cannot install new packages; use only what is defined in pyproject.toml.
    • VRAM: This is a soft constraint. Increases are acceptable for gains, but avoid dramatic blowups.

    Extracting Results

    After a run, redirect output to a log file and use grep to extract the key metrics:

    uv run train.py > run.log 2>&1
grep "^val_bpb:" run.log
    uv run train.py > run.log 2>&1
grep "^val_bpb:" run.log

  5. Set up a new autoresearch experiment

    master

    To start a fresh research run, follow these steps:

    1. Create a unique branch: Propose a tag based on the date (e.g., mar5) and create a new branch: git checkout -b autoresearch/<tag>.
    2. Verify data and tokenizer: Ensure ~/.cache/autoresearch/ contains data shards and a tokenizer. If missing, run:
      uv run prepare.py
    3. Initialize results tracking: Create a results.tsv file containing only the header row:
      commit	val_bpb	memory_gb	status	description
       *Note: Do not commit `results.tsv` to git; keep it untracked.*
4. **Establish baseline**: Run the training script without modifications first to record the baseline performance.

    git checkout -b autoresearch/mar5 uv run prepare.py

  6. How to run the autonomous research agent

    master

    The research process is driven by an AI agent (e.g., Claude or Codex) interacting with the repository.

    1. Human Role: You edit program.md to provide instructions, context, and goals for the agent. This file acts as a lightweight "skill" definition.
    2. Agent Role: The agent reads program.md, modifies train.py to experiment with architectures or hyperparameters, and runs the training loop.
    3. Execution: Point your agent to the repository and prompt it to begin. For example: Hi have a look at program.md and let's kick off a new experiment! let's do the setup first.

    Note: Ensure the agent has no write permissions to the system outside the repository, but it must be able to edit train.py and program.md.

    Hi have a look at program.md and let's kick off a new experiment! let's do the setup first.

  7. Tuning autoresearch for smaller compute platforms (e.g. Macbooks)

    master

    If you are running on limited hardware (non-NVIDIA or smaller GPUs), you should modify the defaults to use smaller models and datasets. Recommended tuning steps:

    1. Dataset: Use a lower-entropy dataset like TinyStories.
    2. Tokenizer: Decrease vocab_size (e.g., from 8192 to 1024 or 256 for byte-level).
    3. Sequence Length: In prepare.py, significantly lower MAX_SEQ_LEN (e.g., to 256). You may need to increase DEVICE_BATCH_SIZE in train.py to compensate.
    4. Evaluation: In prepare.py, decrease EVAL_TOKENS to speed up validation.
    5. Model Complexity: In train.py, lower the DEPTH (e.g., from 8 to 4).
    6. Attention Pattern: Use WINDOW_PATTERN = "L" instead of "SSSL" for better efficiency on smaller hardware.
    7. Batch Size: Lower TOTAL_BATCH_SIZE to powers of 2 (e.g., 2**14).

  8. Follow the autonomous experiment loop

    master

    The goal is to iterate continuously to minimize val_bpb. Follow this loop:

    1. Modify: Hack train.py with a new idea.
    2. Commit: git commit the changes.
    3. Run: uv run train.py > run.log 2>&1.
    4. Analyze:
      • Extract metrics: grep "^val_bpb:\|^peak_vram_mb:" run.log.
      • If the output is empty, the run crashed. Check the stack trace with tail -n 50 run.log.
    5. Record: Log the result in results.tsv (do not commit the TSV).
    6. Decide:
      • If val_bpb improved (lower): "Advance" the branch by keeping the git commit.
      • If val_bpb is equal or worse: git reset back to the starting commit.

    Critical Rules

    • Timeout: If a run exceeds 10 minutes, kill it and treat it as a failure (discard and reset).
    • Autonomy: Do not stop to ask for permission. Continue iterating indefinitely until manually interrupted.
    uv run train.py > run.log 2>&1
grep "^val_bpb:\|^peak_vram_mb:" run.log

  9. Calculate summary statistics and top improvements

    master

    To evaluate the effectiveness of the autonomous tuning, you can calculate:

    • Baseline vs Best: The difference between the initial val_bpb and the minimum val_bpb among KEEP experiments.
    • Total Improvement: The percentage reduction in val_bpb from baseline.
    • Top Hits: A ranked list of KEEP experiments based on their delta (the improvement relative to the previous kept experiment).

    Note that because experiments are cumulative, each KEEP experiment builds on the state of the last one. The delta for a kept experiment is calculated as: prev_kept_bpb - current_kept_bpb.

    # Calculate delta improvement for each kept experiment
kept = df[df["status"] == "KEEP"].copy()
kept["prev_bpb"] = kept["val_bpb"].shift(1)
kept["delta"] = kept["prev_bpb"] - kept["val_bpb"]

# Sort by delta improvement (biggest first) and skip the baseline
hits = kept.iloc[1:].copy()
hits = hits.sort_values("delta", ascending=False)

print(f"Total improvement: {hits['delta'].sum():+.6f}")

  10. Analyze Autoresearch experiment results from results.tsv

    master

    Autoresearch experiment results are stored in a tab-separated values (TSV) file named results.tsv. This file contains 5 columns: commit, val_bpb, memory_gb, status, and description.

    To analyze the results, you can load the TSV into a pandas DataFrame. The status column indicates the outcome of an experiment using the following values:

    • KEEP: An improvement that was successfully applied and became the new baseline.
    • DISCARD: An experiment that did not meet the criteria to be kept.
    • CRASH: An experiment that failed to run.

    You can calculate the 'Keep rate' by dividing the number of KEEP experiments by the sum of KEEP and DISCARD experiments.

    import pandas as pd

# Load the TSV (tab-separated, 5 columns: commit, val_bpb, memory_gb, status, description)
df = pd.read_csv("results.tsv", sep="\t")
df["val_bpb"] = pd.to_numeric(df["val_bpb"], errors="coerce")
df["memory_gb"] = pd.to_numeric(df["memory_gb"], errors="coerce")
df["status"] = df["status"].str.strip().str.upper()

  11. Visualize Autoresearch progress over time

    master

    You can generate a progress plot (progress.png) to track how the validation Bits Per Byte (val_bpb) evolves. The visualization highlights:

    • Discarded experiments: Faint background dots.
    • Kept experiments: Prominent green dots representing successful improvements.
    • Running best: A step line showing the 'frontier' (the best val_bpb achieved so far).

    The plot focuses on the 'interesting region'—points at or below the baseline val_bpb plus a small margin.

    To generate this plot, filter out CRASH statuses, identify the baseline from the first experiment, and use matplotlib to plot the KEEP and DISCARD points along with a cumulative minimum line for the kept experiments.

    import matplotlib.pyplot as plt

# Filter out crashes for plotting
valid = df[df["status"] != "CRASH"].copy()
valid = valid.reset_index(drop=True)

baseline_bpb = valid.loc[0, "val_bpb"]

# Only plot points at or below baseline (the interesting region)
below = valid[valid["val_bpb"] <= baseline_bpb + 0.0005]

# Plot discarded as faint background dots
disc = below[below["status"] == "DISCARD"]
plt.scatter(disc.index, disc["val_bpb"], c="#cccccc", s=12, alpha=0.5, label="Discarded")

# Plot kept experiments as prominent green dots
kept_v = below[below["status"] == "KEEP"]
plt.scatter(kept_v.index, kept_v["val_bpb"], c="#2ecc71", s=50, label="Kept", edgecolors="black")

# Running minimum step line
kept_mask = valid["status"] == "KEEP"
kept_idx = valid.index[kept_mask]
kept_bpb = valid.loc[kept_mask, "val_bpb"]
running_min = kept_bpb.cummin()
plt.step(kept_idx, running_min, where="post", color="#27ae60", linewidth=2, label="Running best")

plt.savefig("progress.png", dpi=150, bbox_inches="tight")

  12. Log experiment results to results.tsv

    master

    When an experiment completes, record the results in results.tsv. Use tabs as separators, not commas.

    TSV Schema

    ColumnDescription
    commitThe short 7-character git commit hash
    val_bpbThe achieved val_bpb (use 0.000000 for crashes)
    memory_gbPeak memory in GB, rounded to 0.1f (divide peak_vram_mb by 1024; use 0.0 for crashes)
    statusOne of: keep, discard, or crash
    descriptionA short text description of the experimental change

    Example Entry

    a1b2c3d	0.993200	44.2	keep	increase LR to 0.04