TreeSearch: Structure-Aware Document Retrieval

The memory system follows a three-stage operational loop to manage information from raw logs to structured knowledge:

1. Retain: Normalize daily narrative logs into self-contained, tagged "facts". Instead of tiny snippets, store narrative bullets that preserve context. Use a ## Retain section in daily Markdown files (memory/YYYY-MM-DD.md) with specific type prefixes:

   • W (World): Objective facts.
   • B (Biographical/Experience): What the agent did.
   • O (Opinion): Subjective preferences or judgments, optionally with confidence O(c=0.95).
   • S (Observation/Summary): Generated summaries.
   • Entities: Mention entities using the @ symbol (e.g., @Peter) to link to entity pages.

2. Recall: Query the derived index (SQLite) to retrieve facts. Recall should support lexical (FTS5), entity-based, temporal, and opinion-based queries. Results should be agent-friendly, returning kind, timestamp, entities, content, and a source (e.g., memory/2025-11-27.md#L12).

3. Reflect: A scheduled job that updates the structured bank/ directory. It updates entity summaries in bank/entities/*.md, evolves opinions in bank/opinions.md by adjusting confidence based on new evidence, and proposes edits to the core memory.md file.

## Retain
- W @Peter: Currently in Marrakech (Nov 27–Dec 1, 2025) for Andy’s birthday.
- B @warelay: I fixed the Baileys WS crash by wrapping connection.update handlers in try/catch (see memory/2025-11-27.md).
- O(c=0.95) @Peter: Prefers concise replies (<1500 chars) on WhatsApp; long content goes into files.

Tree mode is optimized for academic papers and technical documents with deep heading hierarchies. It uses a process of anchor retrieval → tree walk → path aggregation to provide context-aware results.

When using search_mode="tree", the results include an additional paths field that shows the hierarchical relationship between nodes.

from treesearch import TreeSearch

# Use tree mode for hierarchical documents
ts = TreeSearch("papers/", "docs/")
results = ts.search("experimental method", search_mode="tree")

# Standard node results
for doc in results["documents"]:
    for node in doc["nodes"]:
        print(f"[{node['score']:.2f}] {node['title']}")

# Hierarchical path results
for path in results["paths"]:
    chain = " > ".join(p["title"] for p in path["path"])
    print(f"[{path['score']:.2f}] {chain}")
    print(f"  {path['snippet'][:200]}")

Choosing a Search Mode

ts = TreeSearch("papers/", "docs/")
results = ts.search("实验方法", search_mode="tree")

for path in results["paths"]:
    chain = " > ".join(p["title"] for p in path["path"])
    print(f"[{path['score']:.2f}] {chain}")
    print(f"  {path['snippet'][:200]}")

TreeSearch provides three primary search strategies to balance accuracy and speed depending on your document structure:

1. fts5 (Fastest): A full-text search mode optimized for flat documents (e.g., PDFs, plain text, or code). It is extremely fast but lacks structural awareness.
2. tree (Most Accurate for Structured Docs): A structure-aware mode designed for hierarchical documents (e.g., Markdown with deep heading levels). It uses a 7-stage reranking pipeline to leverage parent-child relationships and section context.
3. auto (Recommended/Default): The optimal production strategy. It automatically routes queries based on document characteristics:
   • Markdown/JSON with sufficient depth $\ge 2$ and $\ge 30%$ hierarchical structure $\rightarrow$ tree mode.
   • Code/PDF/TXT or shallow/flat structures $\rightarrow$ fts5 mode.

Decision Logic for auto mode:

• Layer 1: Maps source_type (e.g., markdown $\rightarrow$ tree candidate; pdf $\rightarrow$ flat candidate).
• Layer 2: Verifies actual tree depth (must be $\ge 2$).
• Layer 3: Checks hierarchy ratio (if $\ge 30%$ of documents have hierarchy $\rightarrow$ tree, else flat).

The final ranking in Tree mode is determined by the _build_flat_nodes() function, which processes nodes through four distinct stages to balance text similarity with document structure:

1. FTS5 Base Scores: All nodes with FTS5 scores serve as the foundation.
2. Generic Section Demotion: Nodes identified as generic sections (e.g., Abstract, Introduction, Conclusion) at low depths (0-1) are penalized (e.g., by a 0.60x factor) to prevent them from dominating results due to text length.
3. Title-Prefix Propagation: Nodes with near-zero FTS5 scores can inherit a portion (e.g., 30%) of a high-scoring parent node's score if their titles match the parent's title as a prefix.
4. Walk Boost: Nodes discovered during the Tree Walk process receive a structural bonus (e.g., 15%) to their combined score.
5. Term Density Boost: For queries with multiple terms, nodes that show high coverage (e.g., ≥60% of query terms) receive an additional density-based bonus.

Finally, nodes are sorted by their calculated score in descending order.

Stage 1: FTS5 Base Scores
    ↓ All nodes with FTS5 scores as base
Stage 1b: Generic Section Demotion
    ↓ Abstract/Introduction/Conclusion etc. nodes ×0.60
Stage 2: Title-Prefix Propagation
    ↓ FTS5=0 child nodes inherit 30% score from high-score parent
Stage 3: Walk Boost
    ↓ Walked nodes get 15% structural bonus
Stage 4: Term Density Boost
    ↓ Multi-term queries: high term coverage nodes get 10% bonus
    ↓
Final: Sort by score descending → flat_nodes

TreeSearch uses different retrieval strategies depending on the document structure:

• Flat Mode (Default): Uses SQLite FTS5 with structure-aware columns (title, summary, body, code, front_matter). It provides fast, keyword-based results using column weighting. No LLM is required.
• Tree Mode: Designed for hierarchical documents (like papers or books). It uses a best-first search: FTS5 finds 'anchor nodes', then performs a tree walk (checking parents, children, and siblings) using heuristic scoring (term overlap, IDF weighting, etc.) to find the most relevant path through the hierarchy.
• Auto Mode (Recommended): Automatically routes queries to either Flat or Tree mode based on the document structure. This provides the best balance of speed and accuracy without manual configuration.
• Source-Type Routing: For code files, the system automatically combines GrepFilter (ripgrep-accelerated) with FTS5 for precise symbol matching, controlled by the PREFILTER_ROUTING logic.