DOCSEARCHbyAlgolia
Submit Docs

Headroom Context Compression

repository·main·Indexed 13 days ago

https://github.com/headroomlabs-ai/headroom

A context compression layer for AI agents that reduces token usage by 60-95% for JSON and 15-20% for code. It compresses tool outputs, logs, RAG chunks, and conversation history before they reach the LLM while remaining reversible via a local cache. Features include the SmartCrusher for intelligent tool output compression, a tiered signal detection framework for content classification, and a proxy server for routing LLM clients.

Tokens
275.4K
Snippets
830
Records
1.2K
Agent score
98%

What's inside Headroom

  1. Overview of Phase B: Live-Zone-Only Compression Engine

    main

    Phase B transitions Headroom to a 'Live-Zone-Only' compression engine. This phase involves retiring legacy components like ICM (Incremental Context Management), RollingWindow, ProgressiveSummarizer, and ToolCrusher, replacing them with a more streamlined architecture centered around a Live-zone block dispatcher written in Rust and specialized type-aware compressors.

    Key architectural components introduced or finalized in Phase B include:

    • Type-aware compressors: Specialized compressors for different data types, including SmartCrusher, LogCompressor, SearchCompressor, DiffCompressor, and CodeCompressor.
    • Live-zone block dispatcher: A Rust-based dispatcher for managing live-zone blocks.
    • Token validation gate: A mechanism that uses per-type byte thresholds and provides fallbacks.
    • TOIN (Token Observation & Inference Network): Operates in observation-only mode with per-tenant aggregation keys.
    • CCR (Contextual Compression Registry): A persistent backend with always-on tool registration.
    • Memory Management: Memory routes directly to the live-zone tail without causing system mutation.

  2. Deployment variants of Headroom

    main

    Headroom can be deployed in several specialized configurations based on plugin combinations:

    • Headroom No-Proxy Edition: Includes headroom-recall (P1) and headroom-admission (P2). It requires zero API interposition, making it suitable for users who cannot reroute model traffic.
    • Headroom Admission Control (Enterprise): Includes headroom-admission (P2), headroom-policy (P3), and headroom-sandbox (P4). It is positioned as an AI-agent DLP (Data Loss Prevention) and governance solution.
    • Headroom Fleet: Includes headroom-attribution (P5) and enumerateAdapterDirs for organization-wide rollout state and cost reporting.

  3. Phase E: Cache Stabilization Features

    main

    Phase E (Phase 3 Cache Stabilization) introduces several deterministic and automated features to improve cache hit rates and system stability. Once implemented, the following behaviors are active:

    • Deterministic Tool Sorting: Tool arrays are sorted alphabetically to ensure idempotent and deterministic behavior.
    • Recursive JSON Schema Sorting: JSON Schema keys are sorted recursively to maintain consistency.
    • Automated Cache Control: cache_control is automatically placed on Pay-As-You-Go (PAYG) plans (via 4 specific markers).
    • Automated Prompt Cache Keys: prompt_cache_key is automatically injected for OpenAI on PAYG plans.
    • Volatile Content Detection: A detector identifies volatile content and issues a customer warning without performing a rewrite.
    • Cache-Bust Drift Telemetry: Telemetry is provided per session to monitor cache-bust drift.

  4. Understand the documentation project structure

    main

    The documentation is built using Next.js and Fumadocs. Key files and routes include:

    • lib/source.ts: Contains the content source adapter. Use the loader() interface to access content.
    • lib/layout.shared.tsx: Contains shared layout options.
    • app/(home): Route group for the landing page and other primary pages.
    • app/docs: The main documentation layout and pages.
    • app/api/search/route.ts: The Route Handler used for search functionality.

    Configuration for content, such as frontmatter schemas, is managed via the source.config.ts file.

  5. Understand the Headroom Auth-Mode Policy Matrix

    main

    Headroom applies different compression and header policies based on the user's authentication mode: PAYG (Pay-As-You-Go), OAuth, or Subscription.

    Key differences include:

    • Compression Strategy: PAYG allows aggressive and lossy compression (e.g., LLMLingua), while OAuth and Subscription modes are restricted to lossless-only compression.
    • Cache Control: Automatic cache_control placement and prompt_cache_key injection are only enabled for PAYG users.
    • Header Handling: X-Forwarded-* headers are added for PAYG and OAuth users, but not for Subscription users. accept-encoding is stripped for PAYG and OAuth but preserved for Subscription users.
    • Tool Definitions: Alpha-sorting of tool definitions and JSON Schema key sorting is applied across all modes.
    • Memory Injection: Memory injection is applied to the live-zone tail for all modes, but is gated for OAuth and Subscription users.

  6. TOIN aggregation key structure

    main

    The TOIN (Token Observation & Intelligence Network) telemetry system uses a composite aggregation key to provide recommendations tailored to specific authentication and model contexts.

    Instead of aggregating solely by structure_hash, the system now uses a tuple of (auth_mode, model_family, structure_hash). This allows the system to generate separate recommendations in recommendations.toml for different user types (e.g., different compression strategies for Payg vs Subscription).

    New Aggregation Key Format:

    • auth_mode: (e.g., Payg, OAuth, Subscription)
    • model_family: The family of the model being used.
    • structure_hash: The structural hash of the request/response.

  7. Identify what Headroom does NOT modify

    main

    To ensure safety and compatibility, Headroom explicitly avoids modifying the following:

    • Message History: It never drops messages from history.
    • Core Context: It does not modify system, tools, or any old turns.
    • Tool Inputs: It does not modify tool_use.input JSON key order, tool_calls.function.arguments string contents, phase fields, V4A patches, or local_shell_call.action.command argv arrays.
    • Sensitive Data: It does not modify encrypted or redacted content.
    • Media: It does not compress images, base64 blobs, or audio.
    • Specific Endpoints: It does not compress on /v1/responses/compact or /v1/conversations (these are passthrough only).
    • Headers: It does not add User-Agent headers; the customer's UA passes through verbatim.

  8. How Headroom context management works

    main

    Headroom uses a 'live-zone-only' compression strategy. Instead of dropping messages or using score-based pruning, it identifies the live zone (the newest user message and the latest tool result/output) and compresses only those blocks.

    Key characteristics:

    • Type-aware & Reversible: Compression is reversible via CCR, allowing the LLM to retrieve original content using a marker like compressed, retrieve: <hash>.
    • Cache Preservation: The cache hot zone (system prompt, tool definitions, and older turns) is never mutated. This ensures provider prompt caching remains stable and hit rates stay high.
    • No Message Dropping: Headroom never drops messages from the conversation history; it only compresses the newest content.
    Conversation with a large latest tool result
  -> Identify the live zone (newest user message + latest tool output)
  -> Compress the live zone type-aware, cache original in CCR (hash=def456)
  -> Insert marker: "compressed, retrieve: def456"
  -> Older turns, tools, and system prompt are forwarded byte-for-byte

  9. How the Anthropic SDK wrapper works

    main

    The withHeadroom() function returns a proxy around the Anthropic client that intercepts messages.create() calls. The workflow is:

    1. Conversion: Converts Anthropic-format messages to OpenAI format.
    2. Compression: Sends the converted messages to the Headroom proxy's /v1/compress endpoint.
    3. Reversion: Converts the compressed messages back to Anthropic format.
    4. Forwarding: Forwards the request to Anthropic.

    Note on Format Conversion: While the TypeScript adapter normalizes to OpenAI types internally, the POST /v1/compress endpoint itself is capable of handling Anthropic content blocks natively without conversion. If you are calling the proxy endpoint directly (e.g., via a gateway), you should send Anthropic-shaped messages as-is.

  10. Understand project-scoped `.headroom/` directories

    main

    Certain Headroom components use project-local .headroom/ directories resolved relative to the current working directory (CWD) rather than the global HEADROOM_WORKSPACE_DIR. This is intended to keep project-specific memory and artifacts within the project folder.

    Affected components:

    • headroom/proxy/server.py (Project-scoped memory DB default)
    • headroom/memory/mcp_server.py (Project-scoped memory DB default)
    • headroom/cli/wrap.py (Project-scoped memory and hook artifacts)

    If you require a single, centrally located memory store instead of project-local stores, you must explicitly pass --memory-db-path <path> or set the path via the plugin API.

  11. Manage memory with CCR (Compressed Context Retrieval)

    main

    CCR allows you to keep a compact memory digest while maintaining the ability to recover the original, detailed content from a local store. This is ideal for large, non-instruction-bearing data.

    Core Components:

    • headroom_retrieve: Use this for on-demand recovery of the original, uncompressed content from the local store.
    • HEADROOM_CCR_TTL_SECONDS: An environment variable used to set the lifetime of the local CCR store.
    • compression_strategy: A metadata field used to identify how the original content was produced. Do not infer routing from payload shape; always use the compression_strategy metadata.

    Workflow: When a stored original expires (based on the TTL), you must either regenerate the digest or re-read the source content. CCR is designed to be reversible, meaning the backing detail remains recoverable as long as it exists in the store.

  12. Enable session-sticky memory tool injection

    main

    To maintain consistency in agentic workflows, once the Headroom memory subsystem injects a tool into the body["tools"] array for a specific session, that injection becomes session-sticky.

    Behavior:

    • Every subsequent request in that session will receive the same injected tool (identical name and byte-equal definition).
    • Once a tool is injected, toggling memory off mid-session is forbidden; the tool remains present for the duration of the session to ensure tool-use stability.