cavemem

The cavemem compression engine is a deterministic, offline system that compresses prose without invoking an LLM. It operates on a specific contract to ensure data integrity:

1. Determinism: compress(x) always produces the same output for the same input and intensity level.
2. Technical Token Preservation: The tokenizer identifies and protects technical tokens (code, URLs, paths, commands, version numbers, dates, numeric literals, and identifiers) from any transformations. These are preserved byte-for-byte.
3. Substantive Round-tripping: While prose content is lossy (removing filler and hedging words), expand(compress(x)) is guaranteed to preserve every technical token exactly as it appeared in the original input.

To minimize token usage (saving approximately 10×), follow the recommended three-layer pattern when interacting with cavemem via MCP:

1. Discovery: Use search (or list_sessions followed by timeline) to retrieve a compact index of observations.
2. Review: Inspect the returned IDs and metadata.
3. Retrieval: Use get_observations with the specific filtered set of IDs to fetch full content.

This approach prevents fetching full observation bodies upfront, which significantly reduces the context window load for agents.

Cavemem includes several mechanisms to protect sensitive data:

• Content Stripping: Any content wrapped in <private>...</private> tags is stripped before being written to memory.
• Path Exclusion: Files matching patterns in privacy.excludePatterns are never captured, even if they appear in tool fields like file_path, path, or notebook_path.
• Secret Redaction: When privacy.redactSecrets is true, the system scrubs secret-shaped substrings (API keys, tokens, passwords) and replaces them with [REDACTED].
• Worker Security: The worker binds only to 127.0.0.1, validates Host/Origin headers on every request, and requires a local bearer token located at <dataDir>/worker-token (with 0600 permissions) for all /api/* requests.

cavemem provides cross-agent persistent memory by intercepting session events. The workflow is:

1. Session Event: An IDE hook triggers (e.g., UserPromptSubmit).
2. Redaction: Content inside <private>...</private> tags is stripped.
3. Compression: Observations are compressed using the 'caveman grammar', which reduces prose tokens by ~75% while preserving code, paths, identifiers, and version numbers byte-for-byte.
4. Storage: Data is written to a local SQLite database with FTS5 keyword search and a local vector index.
5. Retrieval: Agents query history via MCP tools using progressive disclosure (compact snippets first, full bodies on demand).

Compression Example:

• Input: "The auth middleware throws a 401 when the session token expires; we should add a refresh path."
• Stored: "auth mw throws 401 @ session token expires. add refresh path."

Cavemem operates through two primary paths: a Write path for ingesting data from an IDE and a Read path for retrieving it via MCP or a web browser.

Write Path (Ingestion)

1. An IDE sends input via hooks.
2. The CLI executes hook run which invokes runHook(name, input).
3. The system runs redactPrivate to strip content marked with <private>.
4. The @cavemem/compress package transforms prose while preserving technical tokens.
5. Storage.insertObservation commits the data to SQLite, triggering FTS5 updates.
6. If enabled, embeddings are computed asynchronously by the @cavemem/worker.

Read Path (Retrieval)

• For AI Models (via MCP): Uses compact search. Calling get_observations(expand: true) returns the full, readable text.
• For Humans (via Browser): The @cavemem/worker serves expanded text over HTTP at 127.0.0.1:37777.

The cavemem worker's HTTP API (used by the viewer) is protected with several security layers:

• Host/Origin Validation: Rejects requests that do not originate from 127.0.0.1 or localhost to prevent DNS-rebinding and CSRF attacks.
• Token Authentication: All /api/* endpoints require an Authorization: Bearer <token> or X-Cavemem-Token header. The token is a 32-byte random value stored in <dataDir>/worker-token.
• Viewer Access: The plain HTML viewer pages (/, /sessions/:id) remain token-free for ease of use. The worker automatically injects the required token into the browser session via window.__CAVEMEM_TOKEN__ so the UI can call the API without manual configuration.

To disable the worker's idle shutdown, set embedding.idleShutdownMs to 0 in your settings.

To document changes for the next release, use the changeset command. This will generate a file that must be committed along with your Pull Request.

pnpm changeset

To add new abbreviation rules or linguistic transformations to the engine, follow these steps:

1. Edit packages/compress/src/lexicon.json to add the new mapping.
2. Add a new fixture in packages/compress/test/fixtures/ that demonstrates the new rule and verifies the round-trip (compressing and then expanding).
3. Run the tests for the compression package using:

pnpm --filter @cavemem/compress test

4. If the new rules significantly change the compression ratio, update the benchmark numbers in the evals/ directory.

To prevent development runs from modifying your global state in ~/.cavemem, you can use the CAVEMEM_HOME environment variable to point cavemem to a local scratch directory. This variable overrides the location where settings.json, data.db, and other state files are stored (as resolved by @cavemem/config's resolveCavememHome).

Set the variable to a local path and run the development command:

export CAVEMEM_HOME=$PWD/.cavemem-dev
pnpm dev