OpenToken Token-Compression Engine

OpenToken acts as a transparent filter between a tool's runtime and the LLM context window. It uses a pipeline architecture to reduce token usage by 50-80% without losing semantic meaning.

The Pipeline Process

1. Secrets Redaction: Runs first to ensure sensitive data is removed.
2. Binary Detection: Uses a NUL byte guard to prevent corruption of binary streams.
3. ANSI Stripping: Removes terminal color codes and control sequences.
4. Family Detection & Routing: The engine detects the command context (e.g., git, npm, cargo) and routes the output to a specialized pipeline of 10-20 stages.
5. Conservative Safety Filter: Every stage ends with a filter that estimates BPE token counts. If a stage increases the token count, the original output is returned. This ensures a 0-risk principle where compression never regresses quality.

Compression Techniques

• Noise Reduction: Pre-call rewrites (adding --quiet or -q to commands), JSON cleanup, and path shortening.
• Structural Compression: Diff folding, log folding (collapsing identical lines), and line noise normalization (replacing timestamps/PIDs with placeholders).
• Advanced Compression: LTSC (Lossless Token Sequence Compression), LZW token substitution, and reversible compression.

The auto-escalation system automatically adjusts text compression levels based on how much of the LLM's context window is filled. As the fillPct (fill percentage) increases, the system moves through progressively more aggressive compression modes to save space.

Compression Levels

• off: No compression applied.
• lean: Drops filler words and replaces long words with short synonyms.
• ultra: Applies lean compression, removes low-information sentences, replaces phrases with symbols (e.g., →, ≠), and compresses lists into semicolon-separated strings.
• ceiling: The most aggressive mode. If the text exceeds 30 lines, it truncates the middle, keeping only the first 10 and last 5 lines.

Escalation Thresholds

• 50% fill $\rightarrow$ lean
• 75% fill $\rightarrow$ ultra
• 85% fill $\rightarrow$ ceiling

// Example of how levels escalate based on context usage
// 50% fill -> lean
// 75% fill -> ultra
// 85% fill -> ceiling

TOON (Token-Oriented Object Notation) is a format optimized for structured data reduction. It uses two primary patterns depending on the data type:

1. Tabular Arrays (Arrays of Objects)

When converting an array of objects, TOON uses a header-based approach to avoid repeating keys for every entry.

Format: {first_key}[{count}]{key1,key2,...}: {val1,val2,...} {val1,val2,...}

• first_key: The first key found in the data.
• [count]: The total number of items in the array.
• {key1,key2,...}: A comma-separated list of all unique keys found in the array.
• :: The header terminator.
• Rows: Each subsequent line contains the values for that object, separated by commas.

2. Nested Objects and Primitives

For standard objects, TOON uses a key-value mapping:

• Primitives: key: value
• Nested Objects: key: followed by indented lines.
• Primitive Arrays: key[length]: val1,val2,...

Example of a nested TOON structure:

user:
  id: 1
  profile:
    name: Alice
    tags[2]: admin,staff

The deduplication engine reduces noise by collapsing repeated information into single reference lines. It operates using several key mechanisms:

1. Session Isolation: All state is keyed by a sessionID using a SessionStore. This ensures that deduplication logic in one session does not corrupt or interfere with another.
2. Detection Methods:
   • Exact Match: Uses content hashing to find identical strings.
   • Fuzzy Similarity: Uses Jaccard similarity (threshold of 0.85) to detect content that is substantially similar but not identical. This is only applied to content longer than 100 characters.
3. Adaptive Windowing: To manage memory and performance, the engine maintains a sliding window of recent calls. The window size is adaptive based on content length:
   • Small content: MIN_WINDOW (8 calls)
   • Medium content (1000-5000 chars): 32 calls
   • Large content (>5000 chars): MAX_WINDOW (64 calls)
4. Cross-Tool Support: The engine tracks the tool name associated with each call, allowing it to provide context in the duplicate reference (e.g., [Duplicate of call #1 (tool_name) ...]).

The progressive disclosure system manages large tool outputs to maintain efficient context windows. It operates based on two thresholds:

• MAX_INLINE_LINES: 80 lines
• MAX_INLINE_BYTES: 8KB

How it works

1. Inline Mode: If the content is below the thresholds, the full content is returned directly.
2. Offload Mode: If the content exceeds thresholds, the system:
   • Generates a summary based on the tool type (e.g., counting errors for bash, symbols for read, or matches for grep).
   • Writes the full content to a temporary file in ~/.config/opentoken/offload/.
   • Returns a summary and a pointer (an entryId) to the user.
3. Fallback Mode: If writing to a file fails, the system returns the summary plus the first 50 lines and the last 20 lines of the content.

Summary Formats by Tool

• bash: Includes error counts and file change counts.
• read: Includes total lines and symbol counts (e.g., functions, classes, interfaces).
• grep: Includes match counts and the number of unique files matched.
• glob: Includes the total number of files found.