haiku.rag Documentation

Haiku RAG provides an agentic RAG pipeline with the following core features:

• Ingestion: Supports PDFs, DOCX, HTML, images, and 40+ formats via Docling. You can add files, URLs, or directories using haiku-rag add-src, or use the haiku-ingester service for continuous ingestion from filesystem, HTTP, S3, or WebDAV.
• Search: Features hybrid retrieval (vector + full-text with reciprocal rank fusion), optional cross-encoder reranking, and structure-aware context expansion. It supports cross-modal retrieval (image-as-query) if configured with a multimodal embedder.
• Answering: Provides RAG with citations (page numbers, section headings, visual grounding). Includes a vision capability where models receive figure bytes, and an Analysis capability using a sandboxed Python interpreter for computation across documents.
• Integration: Accessible via Python API, CLI, MCP server, or as composable Pydantic AI capabilities.
• Operation: Uses embedded LanceDB by default (no servers required), but can run on S3, GCS, Azure, or LanceDB Cloud. Supports time-travel queries via LanceDB versioning.

To improve retrieval quality, use expand_context(search_results) to include surrounding content from the documents found in a search.

How it works

• Structured Documents: Expansion includes the entire section containing the match. If a section is too large or small, it grows outward item-by-item, skipping noise like footnotes or headers.
• Unstructured Documents: Grows outward item-by-item from the match.
• Smart Merging: If expanded results overlap within the same document, they are merged into a single continuous result with the highest relevance score.
• Exemptions: Picture and table matches return their enclosing section as-is and do not cross boundaries. Custom chunks without doc_item_refs are not expanded.

Configuration: Control the maximum expansion size via search.max_context_chars (default: 5000).

# Get initial search results
search_results = await client.search("machine learning", limit=3)

# Expand with section-bounded context
expanded_results = await client.expand_context(search_results)

for result in expanded_results:
    print(f"Expanded content: {result.content}")

A tag is a logical snapshot of the database state, composed of a single LanceDB tag on each of the five tables. Tags allow you to name specific database states (e.g., release-1) for deployment or after ingestion runs.

Important considerations:

• Completeness: A tag is considered 'complete' only if it exists on every table. If it is missing from some tables, haiku-rag tag list will mark it as a 'partial' tag. Partial tags can be listed or deleted but cannot be restored.
• Concurrency: Tag creation coordinates writers within a single process only. To avoid capturing a 'mixed state' (where a writer in another process commits between table snapshots), stop all other writers before creating a tag.
• Cleanup/Vacuum: Tagged versions are protected from vacuum. The vacuum process retains the oldest tagged version and every version newer than it. To allow the database to clean up old versions, delete tags that are no longer needed.
• Storage: Tagged versions survive vacuum operations.

# Tag the current state
haiku-rag tag create release-1

# List tags (partial tags will be marked)
haiku-rag tag list

# Delete a tag to allow vacuum to clean up old versions
haiku-rag tag delete release-1

To enable cross-modal retrieval (where text and pictures share a single vector space), set embeddings.model.multimodal: true.

Important Notes:

• Capability is determined by this flag, not the provider name.
• Setting this on a provider that doesn't support it (anything other than vllm, voyageai, or cohere) will raise an error at startup.
• If you change this setting, you must rebuild or re-ingest your data to add or drop picture chunks.

Supported Multimodal Providers:

vLLM

Use provider: vllm for multimodal models. This allows text inputs via the standard input field and image inputs via messages-with-image_url.

embeddings:
  model:
    provider: vllm
    name: Qwen/Qwen3-VL-Embedding-8B
    vector_dim: 4096
    base_url: http://localhost:8000/v1
    multimodal: true

VoyageAI

Requires VOYAGE_API_KEY.

embeddings:
  model:
    provider: voyageai
    name: voyage-multimodal-3
    vector_dim: 1024
    multimodal: true

Cohere

Requires CO_API_KEY.

embeddings:
  model:
    provider: cohere
    name: embed-v4.0
    vector_dim: 1536
    multimodal: true

The retrieval pipeline follows this flow: chunking → embedding → hybrid search (vector + FTS) → reranking → context expansion → LLM generation.

Retrieval tuning (from chunking through reranking) is the highest-leverage stage. If the LLM does not receive the correct chunks, changes to prompts or models will not improve performance.

Key Tuning Levers

1. Chunking

• chunk_size: Controls granularity. Smaller chunks improve precision but reduce context; larger chunks provide more context but may dilute relevance.
• chunker_type: Choose between hybrid (default) and hierarchical. hierarchical chunking preserves heading structures and is better for deeply nested or structured content.

2. Embedding Model

• Larger models improve retrieval quality but increase indexing time and storage costs. This setting has a significant impact on retrieval quality.

3. Reranking

• When configured, a cross-encoder reranker re-scores 10x the requested candidates. This improves precision at the cost of increased latency.

4. Search Settings

• limit: Controls how many results reach the LLM. Increasing this improves recall but increases token usage.
• Context Expansion: Automatically expands search results to include surrounding content. For structured documents, expansion stays within section boundaries and filters noise (footnotes, headers). For unstructured documents, it grows outward until max_context_chars is reached.

Haiku RAG uses several metrics to evaluate performance across different datasets. When reviewing benchmark results, understand the following key metrics:

• MAP (Mean Average Precision): Used for evaluating Retrieval performance. It measures how well the system surfaces relevant documents in the search results.
• QA Accuracy: Measures the correctness of the answer generated by the capability model. Note that for financial datasets like T²-RAGBench (FinQA), this is a deterministic numeric match (using NumberMatchEvaluator with a relative tolerance of 0.01) rather than an LLM-judged score.
• Mean cited_map: Measures Citation Retrieval performance. It evaluates how effectively the model uses the cite tool to register URIs that match the gold expected_uris.

Benchmarks are categorized by the type of embedding approach used:

1. Multimodal embedder: Uses models like Qwen/Qwen3-VL-Embedding-8B where picture bytes and text share a vector space. No VLM is run during ingestion.
2. Text embedder + VLM picture descriptions: Uses a text embedder (e.g., qwen3-embedding:4b) and a VLM (e.g., ollama/ministral-3) to describe images at ingestion. These descriptions are then woven into the text chunks for retrieval.