DOCSEARCHbyAlgolia
Submit Docs

Codex Autoresearch

repository·main·Indexed 21 days ago

https://github.com/thegreencedar/codex-autoresearch

A Codex plugin for automating iterative code improvement experiments through bounded, measured benchmark and optimization loops. It uses a ledger to record results, track progress, and prevent regressions. The system operates via a cyclic loop of setup, verification, execution, and logging, supporting parallel work through implementation and scout lanes. It includes a CLI engine and a read-only dashboard for monitoring metrics and packet diagnostics.

Tokens
53.3K
Snippets
152
Records
221
Agent score
74%

What's inside codex-autoresearch

  1. Understand data privacy and external data paths

    main

    Codex Autoresearch has no hosted backend, account, or product telemetry of its own. However, data may leave your machine through several distinct paths:

    1. Codex Session: Your prompt and repository context are sent to your configured model provider (governed by their specific terms).
    2. Approved Commands: Any command you approve during a session may independently:
      • Call an API
      • Download packages
      • Start a browser
      • Push to a Git remote
      • Read credentials available to the process
      • Upload or transform project data

    Users are responsible for managing the privacy policies, costs, and rate limits associated with their model providers and the external services called by approved commands.

  2. Understand Lane types and work separation

    main

    Codex Autoresearch splits broad work into specific lanes to manage scope and evidence. A parent session only combines evidence from these lanes after each lane reports its scope, evidence, recommendation, and merge criteria.

    Supported lanes:

    • scout: Mapping the problem without making source edits.
    • implementation: Changing one bounded surface in a worktree or writing a specific scope.
    • review: Testing for regressions and assessing overclaim risk.
    • finalization: Packaging, publishing, merging, and verifying work.

    Note: A big_idea lane may record advice without formal approval, but converting that advice into implementation or a measured packet requires a new, scoped approval.

  3. Interpret research evidence and metrics

    main

    Autoresearch generates various forms of evidence, including metrics, packet evidence, structured experiment notes, artifacts, and finalization previews.

    Important Limitation: A parsed metric or passing benchmark-lint is not a guarantee of correctness, security, privacy, compliance, deployment readiness, or commercial fitness.

    Even if benchmark-lint passes, the workflow may still be blocked by doctor, state, or finalization checks due to:

    • Dirty Git state
    • Runtime mismatches
    • Stale packets
    • Missing checks
    • Weak promotion evidence

  4. Understand Control-plane contracts and shared decisions

    main

    The Codex Autoresearch control plane (CLI, terminal reports, compact state, and dashboard) operates on a principle of unified authority. All surfaces must project the same resolvedDecision authority. If the CLI, dashboard, or reports disagree on the goal, blocker, or next action, it is considered a bug.

    Key decision contracts include:

    • Goal: Ensures the current prompt matches the durable goal, benchmark, and final claim.
    • Approval: Validates if there is current approval for a specific gate and scope.
    • Resources: Checks if processes, packets, or wall-clock slices are permitted.
    • Evidence: Determines the strongest claim supported by accepted evidence.
    • Lanes: Identifies which lane (scout, implementation, review, or finalization) owns the next task.
    • Finalization: Tracks if work is previewed, local, pushed, in CI, merged, or ready for cleanup.
    • Readout: Defines the single next action for a person or Codex.

    To ensure readability during long sessions, the system enforces output ceilings:

    • state --compact: Max 10 KiB / 200 lines.
    • Default state: Max 20 KiB / 260 lines.
    • doctor / --report: Max 8 KiB / 100 lines.

    Use the --json-full flag only when you require complete state or full diagnostics.

  5. Understand the limitations of Codex hooks

    main

    Codex hooks are intended for reminders and context injection only. They are not a substitute for the following critical safety and validation mechanisms:

    • Benchmark and schema validation
    • Command approval gates
    • Last-run freshness checks
    • Git scope and pending-transaction receipts
    • Dashboard freshness labels
    • Human approval for irreversible work (e.g., branch creation)

  6. Decide whether to continue a session

    main

    Before running a new packet, evaluate if continuing the session is beneficial or authorized.

    • Check authorization: Do not rely solely on continuation.shouldContinue (which indicates the session is active). Instead, check loopContract.canRunNextPacket or the canRunNextPacket field in the compact state to see if the next packet is actually authorized.
    • Check constraints: If continuation.forbidFinalAnswer is true, the agent must not complete the goal yet.
    • Get recommendations: Use recommend-next with the --operator-checklist flag to get a human-readable recommendation on the next step.

    Sessions may enter a 'repair' state if a packet is stale, the budget is exhausted, the benchmark has changed, or checks have failed.

    # Get the current recommendation for the next step
node scripts/autoresearch.mjs recommend-next --cwd <project> --compact --operator-checklist

  7. How the packet write path and decision loop works

    main

    The Autoresearch workflow follows a structured cycle for executing tasks and committing results:

    1. Execution: Run the next command to trigger benchmarks and checks.
    2. Persistence: The system writes the last-run packet and an evidence bundle.
    3. Inspection: A human or Codex inspects the metrics, checks, and diffs.
    4. Logging: Use log --from-last to review the recent history.
    5. Decision Gate:
      • Keep: If the decision is safe, the system performs scoped commit paths and appends the ledger and continuation.
      • Discard / Crash / Checks Failed: The system performs scoped experiment cleanup and reverts changes.
      • Measure: If only measuring, the system updates the ledger without Git mutations.

    Note: If a Git mutation is interrupted, the next write operation will be blocked until the ledger and worktree are reconciled via pending transaction receipts.

    # Example command to inspect recent session history
log --from-last

  8. Understand the relationship between Codex and Autoresearch

    main

    Codex and Autoresearch have distinct boundaries regarding state ownership:

    • Codex: Owns the task-level Goal state.
    • Autoresearch: Owns the benchmark contract, ledger, continuation, and completion audit.

    To bridge these, the codex-goal-brief utility converts the current Autoresearch state into an objective or completion-audit packet. It is important to note that codex-goal-brief does not update the Codex Goal state itself and does not access private Codex databases.

  9. Understand Codex Autoresearch core terminology

    main

    Use these terms to navigate the CLI commands and dashboard labels:

    • Packet: A single benchmark run including its associated evidence. The next command generates reusable packets used by log --from-last.
    • Primary metric: The specific METRIC name=value used to determine if an attempt improved the system.
    • Checks: Independent commands used to test correctness. A successful metric does not count as a 'keep' if checks fail.
    • Structured experiment note (asi): A decision record containing the hypothesis, evidence, rollback reason, next action, and optional metadata (lane/risk).
    • Continuation: The decision returned after logging, such as continue, stop, repair, change segment, or finalize.
    • Segment: A comparable chapter of a session. You should start a new segment when the benchmark, metric, direction, or phase changes.
    • Quality gap: A qualitative work item. A checked box is provisional until a gap-decide record provides validation. Setting quality_gap=0 closes only that specific accepted round.
    • Finalization: The process of turning accepted 'keeps' into reviewable branches. This involves a review step before any mutation occurs.

  10. Protect the control and command boundary

    main

    When configuring and running commands, adhere to these security and boundary constraints:

    • Benchmark Paths: Keep protectedBenchmarkPaths small enough to be fingerprinted and reviewed. If a change alters the meaning of a benchmark, move it to a new segment.
    • Fixed Controls: If a configuration contains fixedControl, reuse its existing artifact. Do not execute a matching forbidden command unless the user provides explicit approval via the --allow-fixed-control-rerun flag.
    • Command Sandboxing: Note that benchmark and checks commands are not sandboxed.
    • Security Best Practices:
      • Review all commands.
      • Keep secrets out of command lines and output.
      • Treat persisted redaction as best-effort only.
    # Explicitly allow re-running a fixed control command
<command> --allow-fixed-control-rerun

  11. Handle sensitive data and external services

    main

    When using Codex Autoresearch, you are responsible for data security and external service management:

    • Data Privacy: Do not include secrets, credentials, private customer data, regulated data, or confidential business data in commands, outputs, descriptions, experiment notes, or artifacts. Note that redaction is best-effort only and not guaranteed.
    • External Services: If your research workflow involves external services, you are responsible for managing their data flow, account terms, costs, rate limits, and compliance requirements.

  12. How the Autoresearch loop works

    main

    Autoresearch operates in a cyclic loop to experiment, measure, and record results. The lifecycle follows these stages:

    1. setup: Records the goal, metric, benchmark, checks, budget, and file scope.
    2. doctor: Verifies that the benchmark is reliable before starting experiments.
    3. next: Executes a single benchmark packet (one experiment).
    4. log: Records the outcome as a baseline, keep, discard, or failure.
    5. state: Analyzes the ledger to determine the next logical step.
    6. finalize-preview: Displays results that can be converted into review branches without immediately changing your current Git branch.

    Data Storage

    • Git Repositories: Transient packet state is stored in .git/autoresearch/.
    • Non-Git Repositories: Falls back to local worktree files.
    • Session Records: Durable records are stored within the target project.