DOCSEARCHbyAlgolia
Submit Docs

lossless-claw

repository·main·Indexed 26 days ago

https://github.com/martian-engineering/lossless-claw

A Lossless Context Management (LCM) plugin for OpenClaw that implements a DAG-based conversation summarization system. It preserves full conversation history in a SQLite database and uses threshold compaction to manage LLM token limits. Features include recall tools (lcm_grep, lcm_describe, lcm_expand), a dedicated shell CLI, and an interactive TUI (lcm-tui) for database inspection and maintenance.

Tokens
50.3K
Snippets
84
Records
272
Agent score
87%

What's inside lossless-claw

  1. Overview of lossless-claw

    main

    lossless-claw is a Lossless Context Management (LCM) plugin for OpenClaw. It replaces standard sliding-window compaction with a DAG-based (Directed Acyclic Graph) summarization system.

    Key features:

    • Persists every message in a SQLite database.
    • Summarizes chunks of older messages using your configured LLM.
    • Condenses summaries into higher-level nodes to form a DAG.
    • Assembles context by combining summaries with recent raw messages.
    • Provides recall tools (lcm_grep, lcm_describe, lcm_expand) for agents to search historical details.

  2. CLI Operational Boundaries and Safety Guarantees

    main

    The lcm CLI is designed as a read-only diagnostic and configuration tool for LCM (Lossless Context Management) databases. To ensure data integrity, the following safety rules are enforced:

    • Read-Only Access: The CLI opens LCM databases in read-only mode using PRAGMA query_only = ON. It is strictly prohibited from performing database mutations such as migrations, vacuum, optimize, checkpoint, cleanup, repair, delete, compact, rotate, rewrite, transplant, or backfill.
    • No Side Effects: The CLI will never create a missing database as a side effect of an inspection command.
    • Config Safety: When using configuration commands, the CLI performs atomic replacements and backs up the existing configuration before a successful replacement. Config edits are targeted and validated to ensure they only affect Lossless-specific keys, preventing the exposure or accidental modification of unrelated OpenClaw secrets or configurations.

  3. Understand Transcript Reconciliation logic in lossless-claw

    main

    Transcript reconciliation is the process of synchronizing the LCM SQLite store with the OpenClaw session JSONL file.

    Currently, the system relies on a heuristic-based approach because message identity is considered 'lossy'. Messages are identified by role + "\0" + content (messageIdentity), which prevents the system from distinguishing between a replayed message and a new, identical message (e.g., heartbeats or empty tool results).

    To prevent data corruption like 'replay floods' (silent duplication) or 'frozen conversations' (failure to compact), the system uses a stack of incident-calibrated thresholds (guards) to prove non-duplication before insertion.

  4. Understand Focus Brief architecture and constraints

    main

    Focus briefs are task-oriented artifacts generated by a delegated subagent using recall tools (lcm_grep, lcm_describe, and lcm_expand).

    Key architectural properties:

    • Isolation: A focus brief is not a summary DAG node and is not written to context_items. It must not be consumed by normal compaction processes.
    • Persistence: Briefs are persisted outside the summary DAG, making them inspectable and supersedable without replacing canonical storage.
    • Lifecycle: Commands like focus, refocus, and unfocus act as prompt-prefix lifecycle operations. Because they break prompt caching, they are paired with forced full-sweep compactions to ensure the subagent operates on the freshest summary frontier.

  5. Understand Tool Result Externalization and Transcript GC

    main

    The lossless-claw plugin manages context growth by preventing large tool outputs from bloating active session transcripts. It uses three primary mechanisms:

    1. Tool-result externalization: Oversized tool outputs are moved to large_files storage. Instead of the raw blob, a compact [LCM Tool Output: ...] placeholder is stored in the message content. Retrieval is maintained via file_... references.
    2. Transcript GC (Garbage Collection): Once tool results are externalized and condensed into summaries, the plugin can request the OpenClaw runtime to rewrite the active transcript, replacing giant inline blobs with compact placeholders.
    3. Incremental bootstrap: To reduce restart costs, the system uses fast paths for unchanged files or append-only transcripts (tail-import), falling back to full streaming reconciliation only when necessary.

  6. Understand operation serialization in lossless-claw

    main
    All mutating operations, specifically ingest and compact, are serialized on a per-session basis using a promise queue. This design ensures that concurrent afterTurn or compact calls for the same conversation do not cause race conditions, while allowing operations on different conversations to proceed in parallel without blocking.

  7. Understand Transcript Reconciliation by Entry ID

    main

    The lossless-claw project implements a multi-phase design to ensure idempotent and exact reconciliation of conversation transcripts. The core mechanism promotes the transcript entry ID from a heuristic to the primary message identity. This allows the system to handle replayed transcripts, host-side history edits (rewrites), and session rotations without creating duplicate entries in the database.

    Key design goals include:

    • Idempotency: Using transcript_entry_id as a unique key to prevent duplicate ingestion.
    • Exact Alignment: Aligning runtime batches against the 'covered frontier' (the point where the database and transcript are proven to match).
    • Epoch Management: Using declared session headers to detect rotations or rewrites instead of inferring them via heuristics.
    • Stale-ID Adoption: Handling host-side 'copy-on-write' history edits by re-stamping existing database rows with new IDs instead of importing duplicates.

  8. Understand the Lossless-Claw (LCM) Architecture

    main

    The Lossless Context Management (LCM) plugin is designed as a standalone package (@martian-engineering/lossless-claw) that integrates with OpenClaw. It uses a Dependency Injection pattern to interact with OpenClaw core services. Instead of direct imports from OpenClaw internals, the LCM engine receives its dependencies via the LcmDependencies interface, which is constructed from the OpenClawPluginApi at the plugin entry point (index.ts).

    Key components include:

    • LcmContextEngine: The core engine implementing the ContextEngine interface.
    • CompactionEngine: Handles context compaction.
    • ContextAssembler: Assembles context for LLM usage.
    • RetrievalEngine: Manages context retrieval.
    • Expansion Logic: Handles context expansion and policy.
    • Summarization: Provides summarization with escalation via summarize.ts.
    • Storage: Uses better-sqlite3 for local persistence (defaulting to ~/.openclaw/lcm.db).

  9. Understand LCM Stub-Tier Stratification

    main

    LCM Stub-Tier Stratification is an architectural approach to reduce per-turn assembled context costs in Large Context Management (LCM) systems. It separates the semantic conversation thread from large tool-result payloads.

    How it works:

    • Thread: The messages.content field stores text messages and tool calls (requests) as usual, but replaces large tool result bodies with a small STUB.
    • Payloads: The actual large tool result bodies are moved to a content-addressable blob tier (lcm_blobs).
    • Stubs: A stub follows this format: [tool_result blob_ref=blob_abc123 size=26585t kind=vm_status].
    • Assembly: The assemble() process inlines full tool results only if they fall within a specific 'inline window'. Results outside this window are represented by their stubs, significantly reducing the token count sent to the model.

    Key Benefits:

    • Reduces per-turn assembled token costs by approximately 50%.
    • Preserves all data (lossless) via content-addressable storage.
    • Maintains backward compatibility with existing v4.1 tools.

  10. Understand the lossless-claw Data Model

    main

    The system manages context through three primary entities:

    1. Conversations and Messages: Every session is a conversation. Messages include seq (sequence), role (user, assistant, system, or tool), content, tokenCount, and createdAt. Structured content is preserved in message_parts (e.g., tool calls, reasoning, file content).
    2. Summary DAG: Summaries form a Directed Acyclic Graph (DAG) with two types:
      • Leaf summaries (kind: "leaf"): Created from raw message chunks (800–1200 tokens).
      • Condensed summaries (`kind: "condensed"): Created by merging summaries at the same depth (1500–2000 tokens).
    3. Context Items: An ordered list of references (messages or summaries) that defines exactly what the model sees in its context window.

  11. Manage Lossless Claw configuration with lcm config

    main

    Use lcm config to inspect and modify the Lossless plugin configuration. Configuration paths are relative to plugins.entries.lossless-claw.config.

    • lcm config show: Returns the raw Lossless plugin config, effective LcmConfig, and diagnostics.
    • lcm config get <path>: Retrieves a specific configuration value.
    • lcm config set <path> <json-value>: Updates a configuration value. This command performs atomic writes with timestamped backups.

    Constraints for lcm config set:

    • Values must be valid JSON (no JSON5 or comments).
    • Paths must exist in the Lossless manifest schema.
    • Symlink config paths are rejected.
    lcm config get freshTailCount
lcm config set freshTailCount 96
lcm config set summaryModel '"openai/gpt-5.4-mini"'
lcm config set ignoreSessionPatterns '["agent:*:cron:**"]'

  12. Use the Lossless Claw CLI (lcm)

    main

    The lcm executable provides structured, read-only access to Lossless Claw's persisted data (conversations, messages, summaries, context state, etc.). By default, it outputs JSON, making it ideal for programmatic consumption by agents.

    To improve human readability, use the following flags:

    • --pretty: Indented JSON output.
    • --format table: Compact terminal table output.

    Note: Most commands run in SQLite read-only mode. The only command that modifies state is lcm config set.