DOCSEARCHbyAlgolia
Submit Docs

autocontext

repository·main·Indexed 22 days ago

https://github.com/greyhaven-ai/autocontext

A recursive self-improving harness and control plane for iterative strategy evolution. autocontext helps AI agents succeed by running evaluations, capturing lessons, and generating playbooks, datasets, and training artifacts. It includes a CLI (autoctx) for managing scenarios, simulations, and training data export, with built-in templates for content generation, prompt optimization, and RAG accuracy.

Tokens
154K
Snippets
276
Records
807
Agent score
76%

What's inside autocontext

  1. Overview of the traces/ directory

    main

    The traces/ directory provides the infrastructure for public trace schemas, data-plane curation, dataset adapters/discovery, distillation pipelines, redaction, and sharing/publishing workflows. It is organized into several functional domains that map to specific Python implementations.

    Key Domains:

    • Production Traces (production_traces/): Handles contracts, taxonomy, emission, and redaction/validation. Key files include public-schema*.ts, redaction*.ts, and trace-ingest-workflow.ts.
    • Sharing (sharing/): Manages attestation, bundling, manifests, publishers, and review processes. Key files include publishers*.ts, publishing-workflow.ts, and export-*-workflow.ts.
    • Training & Autoresearch (training/autoresearch/): Manages data selection, augmentation, and distillation. Key files include distillation-*.ts and dataset-*.ts.

    Note on Directory Structure: This directory is distinct from ts/src/production-traces/. While ts/src/production-traces/ is a direct TypeScript mirror of the Python production_traces/ (covering contract, ingest, redaction, dataset, and CLI layers), the traces/ directory covers a broader surface area including sharing and training workflows.

  2. Identify active autocontext packages by language

    main

    The autocontext project is split into language-specific shipping packages. Depending on your development environment, you should use one of the following active packages:

    • Python: autocontext
    • TypeScript: autoctx
    • Pi: pi-autocontext

    Note that core/control split packages are currently deferred and are not part of the active shipping surface.

  3. Overview of RAG Accuracy evaluation dimensions

    main

    The RAG Accuracy template uses an LLM judge to evaluate optimized RAG pipeline configurations across five weighted dimensions:

    1. Retrieval Relevance (weight: 0.30): Measures if parameters maximize relevant chunk retrieval.
    2. Answer Grounding (weight: 0.25): Measures if the configuration supports well-grounded answers.
    3. Citation Accuracy (weight: 0.20): Measures if the configuration facilitates source attribution.
    4. Hallucination Detection (weight: 0.15): Measures the presence of anti-hallucination mechanisms.
    5. Parameter Justification (weight: 0.10): Measures if configuration choices are well-justified.

  4. Understand the Background Session Domain model

    main

    The BackgroundSession is a read-model aggregate that provides an operator-facing view of background work. It is not a replacement for existing entities like Run, Task, Mission, or RuntimeSessionEventLog, but rather an assembly of data from them.

    Identity Derivation Order:

    1. An explicit background session ID (if provided by a store).
    2. A run-scoped runtime session ID (run:<run_id>:runtime).
    3. A queue task ID for queued work that hasn't created a runtime session yet.

    Aggregation Composition: A BackgroundSession aggregates:

    • One runtime-session event log.
    • One queue task row.
    • One run row/status object.
    • Zero or more outcome artifacts.
    • Zero or more child session summaries.
    • Optional trigger metadata.

    Note: The read model must not mutate source records. Command handling and worker execution are separate concerns.

  5. Understand Run Utilization Report window semantics

    main

    The report uses a time window defined by telemetry timestamps.

    • window.started_at: The earliest known telemetry timestamp from run events and role usage rows.
    • window.completed_at: The latest known telemetry timestamp from run events and role usage rows.
    • duration_seconds: The total window duration in seconds. This value is null if no telemetry contains timestamps.

  6. Understand the Scenario Environment Contract

    main

    The Scenario Environment Contract defines a standardized lifecycle for Autocontext scenarios. It ensures that scenarios are deterministic, verifiable, and reproducible by moving reset and verification logic from optional harness details to the public scenario boundary.

    A serious scenario must implement a contract that describes how it:

    1. Resets: Returns to a clean, seeded state.
    2. Runs: Executes a candidate strategy or action trace.
    3. Verifies: Checks validity using validators, probes, or rubrics.
    4. Scores: Emits a scalar score for search and reporting.
    5. Preserves: Maintains replay/evidence (transcripts, metrics, or errors).
    6. Cleans up: Removes mutable resources.

    The contract is defined using a canonical JSON shape. While Python and TypeScript implementations may differ internally, the persisted wire format must maintain stable hook names and keys.

  7. Identify the source of truth for lessons

    main

    Lessons in autocontext are derived directly from live markdown artifacts rather than a centralized JSON store. When implementing or querying lessons, use the following locations as the authoritative sources:

    1. Playbook Lessons: Content found in knowledge/<scenario>/playbook.md located between the <!-- LESSONS_START --> and <!-- LESSONS_END --> markers.
    2. Skill-specific Lessons: Content found in skills/<scenario>-ops/SKILL.md under the ## Operational Lessons heading (used when a lesson exists only within a specific skill).

    Note: lessons.json is deprecated. Do not write lesson text to lessons.json; instead, mutate the markdown files directly.

  8. Implement Workspace and Session Event Stores for Edge Agents

    main

    For edge-based agents, workspace and session data must be managed through provider-neutral contracts rather than local filesystems.

    Workspace Storage

    Use the AgentAppFetchWorkspaceStore contract.

    • Pure Handlers: Use the in-memory reference implementation for request-local workspaces.
    • Persistent Artifacts: Pass a workspaceStore to createAgentAppFetchHandler() to allow artifacts to survive beyond a single request. The store must provide read-your-writes behavior.
    • Adapter: createAgentAppFetchWorkspaceEnv() can adapt a store to the RuntimeWorkspaceEnv surface.

    Session Event Logging

    Use the AgentAppFetchSessionEventStore contract for runtime-backed apps. The contract requires:

    • Appending batches keyed by runtime-session id.
    • Reading session metadata and timelines by id.
    • Idempotent writes using eventId.
    • Replay ordered by per-session sequence.
    • Read-your-writes behavior after an append resolves.

  9. Shared features between Python and TypeScript

    main

    Despite functional differences, both the Python and TypeScript packages share the following core characteristics:

    • Family Identifiers: They use the same 11 family names and type markers.
    • Classification: They use the same family classification logic.
    • Persistence Layout: Custom scenarios are persisted using the same layout: knowledge/_custom_scenarios/<name>/.
    • Database Compatibility: They use the same migration SQL, making them cross-compatible.
    • MCP Entrypoints: Both provide solve-on-demand MCP (Model Context Protocol) entrypoints, though they differ in runtime depth and naming conventions.

  10. Use Autocontext MCP for structured tool calls

    main

    If your environment has Autocontext MCP configured, you can use it for structured tool calls that provide typed schemas and managed tool discovery. This is an optional alternative to the CLI.

    To check available MCP commands:

    uv run autoctx mcp-serve --help

    Use MCP when it provides value beyond the CLI, such as stable schemas or when host policies disallow shell access.

  11. Understand CandidateEvidence artifacts

    main

    A CandidateEvidence artifact is a structured record emitted by a candidate (a proposed change to a mechanism like a prompt playbook, code path, or tool wrapper) before it is evaluated. It contains the hypothesis, concrete changes, expected fix/regression cases, cost expectations, and cross-language parity status.

    The schema is shared between Python and TypeScript to ensure contract consistency. The schema is defined in JSON and generates the CandidateEvidence interface in TypeScript and the CandidateEvidence Pydantic model in Python.

    {
  "schema_version": 1,
  "candidate_id": "cand-full-002",
  "parent_frontier_id": "frontier-001",
  "mechanism_name": "require-target-theorem-present",
  "mechanism_type": "mixed",
  "target_surface": "evaluator",
  "hypothesis": "Requiring the target theorem to be present catches false-pass truncations.",
  "changes": "Add a post-compile check that the named theorem exists in the output, plus a playbook note describing the failure mode.",
  "changed_artifacts": ["src/autocontext/loop/loop_driver.py", "knowledge/lean/playbook.md"],
  "fix_cases": ["divergence-empty-file", "truncated-claude-p-output"],
  "regression_cases": ["convective_eq_fderiv", "divergence-integrability"],
  "observed": "Dry-run over the divergence seed now fails the compile-only oracle as expected.",
  "validation_plan": "Re-run the divergence seed and confirm the empty-file case now fails, then replay the 60 banked sub-lemmas for regressions.",
  "cost_expectation": {
    "extra_tokens": 1200,
    "extra_calls": 1,
    "extra_seconds": 4.5
  },
  "leakage_scope": ["public-fixtures", "banked-sub-lemmas"],
  "parity": {
    "python": "implemented",
    "typescript": "implemented",
    "schema_hash": "9f8e7d6c5b4a"
  }
}