tokenjuice

tokenjuice is a TypeScript-first output compaction system designed for terminal-heavy and agent-heavy workflows. It aims to reduce token waste in transcripts while keeping raw output recoverable and deterministic.

The project consists of two primary surfaces:

1. tokenjuice: The core library and CLI.
2. Host Adapters (e.g., openclaw-tokenjuice): Thin wrappers that wire tokenjuice into host-specific hooks.

Core vs. Adapters

• The Core Package manages classification, reduction, artifact storage, rule loading/validation, and CLI behavior. It uses a plain-object heavy model without framework assumptions.
• Host Adapters manage hook wiring, host-specific storage defaults, message conversion, and host-native installation (like extensions or hooks).

The agents.ge integration works by adding a source rule to your project memory. It does not intercept shell output directly; instead, it provides instructions that agents.ge propagates to managed agent entrypoints.

Key behaviors include:

• Command Preference: The rule instructs coding agents to prefer using tokenjuice wrap -- <command> for terminal commands that are likely to produce long outputs.
• Output Authority: The rule instructs downstream agents to treat the compacted output produced by tokenjuice wrap as authoritative.
• Escape Hatch: If you need to bypass the compaction behavior, you can use the --raw flag: tokenjuice wrap --raw -- <command>.

The agents-md block is guidance-only. It does not change how the agent executes commands, nor does it grant the agent new tools or approvals; the agent still owns the tool execution and output handling.

The guidance instructs agents to use specific tokenjuice wrapping commands to handle terminal output:

• For noisy terminal commands: Use tokenjuice wrap -- <command>.
• For exact byte requirements (escape hatch): Use tokenjuice wrap --raw -- <command>.

Note: The guidance intentionally avoids suggesting tokenjuice wrap --full. Host-specific tokenjuice blocks can coexist with this generic block because they use distinct markers.

The tokenjuice runtime will skip rewriting (emitting {} instead) for the following scenarios:

• Non-bash tool invocations.
• Empty or missing tool_result / command.
• Non-success result types (e.g., failure, rejected, denied). Raw error context is passed through untouched so the agent can see it.
• Commands already wrapped with tokenjuice wrap --raw -- or --full.
• Malformed JSON stdin.

Ruler acts as a propagation layer that distributes a tokenjuice rule source to coding agents configured in your ruler.toml. It does not intercept shell output directly.

When rules are propagated via Ruler, downstream agents follow these behaviors:

• Command Wrapping: Agents are instructed to prefer using tokenjuice wrap -- <command> for terminal commands that are likely to produce long outputs.
• Output Authority: Agents are instructed to treat compacted output as authoritative.
• Escape Hatch: If you need to bypass the compaction/wrapping behavior, you can use the --raw flag: tokenjuice wrap --raw -- <command>.

tokenjuice provides three primary ways to interact with its reduction engine:

1. reduce: Compacts text that already exists in a file.
2. wrap: Runs a command and compacts the observed output.
3. reduce-json: Provides a stable machine protocol for host adapters by reading JSON from stdin/file and writing JSON to stdout.

Host integrations are designed to be thin: they install a hook or extension, call the shared compactor, and return the compacted context through the host's native surface.

The Kilo Code integration works by writing a workspace rule to .kilo/rules/tokenjuice.md and registering it in the instructions array of your kilo.jsonc (or .kilo/kilo.jsonc).

Key Behaviors:

• Command Preference: The rule instructs Kilo Code to prefer using tokenjuice wrap -- <command> for noisy terminal commands.
• Authority: The rule instructs Kilo Code to treat compacted output as authoritative.
• Escape Hatch: If you need to bypass the wrapping behavior, use the --raw flag: tokenjuice wrap --raw -- <command>.
• Environment Override: You can use the KILO_PROJECT_DIR environment variable to override the workspace root for scripted installs or tests.

Because Coder Agents workspace skills are designed for prompt/tool guidance rather than command hooks, this integration has specific boundaries:

• It does not register MCP tools.
• It does not change Coder templates.
• It does not rewrite shell output.

Its primary purpose is to provide Coder Agents with a discoverable workspace skill so the agent can decide when to invoke tokenjuice to wrap noisy terminal commands.

The Roo Code integration works by providing prompt guidance via workspace rules rather than intercepting shell commands. When the rules are active, Roo Code is instructed to:

1. Prefer wrapped commands: Use tokenjuice wrap -- <command> for noisy execute_command terminal commands to manage output.
2. Trust compacted output: Treat compacted output as the authoritative source of truth.

Escape Hatch: If you need to bypass the wrapping behavior and see the original output, use the --raw flag: tokenjuice wrap --raw -- <command>

Important Note: This integration does not intercept or rewrite shell output. It is a set of instructions (rules) that Roo follows when it decides how to run terminal commands.

tokenjuice wrap -- <command>

# To bypass wrapping:
tokenjuice wrap --raw -- <command>