tokensave

tokensave is a semantic code intelligence tool designed for AI coding agents (like Claude Code, Cursor, or Copilot). It provides a pre-indexed semantic knowledge graph of your codebase, allowing agents to query symbols, relationships, and source code directly via an MCP (Model Context Protocol) server.

Instead of agents wasting tokens on repetitive grep, glob, and file read operations to explore a codebase, they can use tokensave to perform instant, structured lookups. This reduces token usage, minimizes tool calls, and keeps all processing 100% local using a libSQL graph database.

This document provides a comparative analysis between Token Savior (a Python-based project with 105 tools) and tokensave (a Rust-based project with 38 tools). While both projects aim to reduce token consumption via structural code navigation through MCP, Token Savior includes advanced features built around the code graph that are currently absent or less mature in tokensave.

Key functional areas where Token Savior leads include:

• Persistent Memory: Storing observations (decisions, bugfixes, etc.) in SQLite.
• Context Optimization: Using knapsack optimization to fit symbols into a token budget.
• Advanced Analysis: Program slicing, breaking change detection, and edit verification.
• Predictive Navigation: Markov prefetching and Co-Activation Tensors (TCA) to predict next tool calls or related symbols.

In Multiproject mode, tokensave treats direct subdirectories of the root as distinct projects. This allows a single .tokensave database to scope data to specific sub-projects within a larger workspace.

Example Directory Structure:

Code/              <- root, initialized with `tokensave init --multiproject` here
  shared.rs        <- project: [root]
  App1/
    src/main.rs    <- project: App1
  App2/
    src/lib.rs     <- project: App2

This structure allows you to filter queries by project name, enabling you to isolate analysis to a specific application within a monorepo.

Code/              <- root, init --multiproject here
  shared.rs        <- project: [root]
  App1/
    src/main.rs    <- project: App1
  App2/
    src/lib.rs     <- project: App2

Tokensave supports two ways to index prose explanations alongside code so agents can read summaries instead of large files:

1. Sidecar Files: A Markdown file named Filename.ext.readme.md placed next to Filename.ext. No configuration required.
2. Docs Directory: A central directory (default tokensave-docs/) where Markdown files use YAML front matter to declare which files they cover via applies_to globs.

Example Docs Directory entry:

---
applies_to:
  - "**/*.es8.cs"
  - "src/legacy/**/*.cs"
---
These files target the ES8 runtime; prefer the ES7 variants for new work.

Retrieving Docs:

• Use the tokensave_doc tool to get the doc path, content, covered files, and a doc_stale signal (true if the code was updated after the doc).
• tokensave_entities reports has_doc and doc_path so agents can detect documentation availability before reading.

While most tokensave tools are read-only and safe to call in parallel, the following categories of tools mutate state and should not be parallelized:

• Edit tools (modify source files):
  • tokensave_str_replace
  • tokensave_multi_str_replace
  • tokensave_insert_at
  • tokensave_insert_at_symbol
  • tokensave_replace_symbol
  • tokensave_ast_grep_rewrite
• Session and memory tools (write to .tokensave/):
  • tokensave_session_start
  • tokensave_session_end
  • tokensave_record_decision
  • tokensave_record_code_area
• Test tools (runs a cargo test subprocess):
  • tokensave_run_affected_tests

Plugins run in-process with full trust and no sandboxing. You can control plugin behavior via configuration:

• Disable all plugins: Set plugins.enabled = false in your config (useful for CI/CD).
• Allowlist plugins: Use plugins.allow = ["tokensave-elixir"] to restrict loading to specific plugins.
• Override built-ins: Plugins that declare an extension already claimed by a built-in extractor take precedence (opt-in override) unless disabled in config.

CodeGraph supports semantic search through vector embeddings and cosine similarity.

Key Components:

• TextEmbedder: A wrapper around the ort crate for ONNX inference. It uses the nomic-embed-text-v1.5 model. Note that it requires adding search_query: or search_document: prefixes to text per the model's requirements.
• store_vector(db, node_id, embedding, model): Stores an embedding as a BLOB in the database associated with a specific node_id.
• get_vector(db, node_id): Retrieves and decodes the stored embedding.
• brute_force_search(db, query, limit): Performs a similarity search by loading all vectors and computing cosine similarity against the query vector, returning the top-k results.
• create_node_text(node): Generates a searchable text representation of a code node (including name, kind, and docstrings) to be embedded.

Tokensave uses FTS5 with BM25 scoring rather than embedding models. While FTS5 is highly efficient, it relies on literal matches in names, signatures, or docstrings. To bridge the gap between conceptual queries (e.g., "authentication") and specific code (e.g., login()), agents should provide a keywords array to the tokensave_context MCP tool.

Each keyword acts as an independent FTS5 query, and the results are merged and deduplicated by node ID. This approach avoids the high indexing cost, model dependencies, and latency associated with local embeddings.