EmbedAnything Documentation

EmbedAnything is a high-performance, modular, and lightweight embedding pipeline built in Rust. It is designed for local, multimodal embedding generation (text, images, audio, PDFs, etc.) and supports streaming embeddings directly to vector databases to reduce latency and memory usage.

Key capabilities include:

• No PyTorch Dependency: Low memory footprint and easy cloud deployment.
• Multimodality: Supports PDFs, TXT, MD, JPG, and WAV.
• Vector Streaming: Separates file processing, indexing, and inference on different threads using Rust MPSC for high throughput.
• Diverse Embedding Types: Supports dense, sparse (Splade), ONNX, model2vec, and late-interaction embeddings.
• Built-in Chunking: Includes semantic and late-chunking strategies.

EmbedAnything supports processing and embedding for various data types. Currently supported modalities include:

• Audio files
• Markdowns
• Websites
• Images
• Videos (via frame sampling; requires enabling the video feature)

Future support for Graph data is planned.

EmbedAnything is a multimodal framework supporting various data types:

• Text: Supports HTML, PDFs, and Markdown. Common models include Jina and AllMiniLM.
• Images: Supports image embedding using CLIP.
• Audio: Supports audio embedding using Whisper. It can also link Whisper's decoded text with a text embedding model to provide metadata including timestamps.
• Late-Interaction Models: Supports models like ColPali, which allow embedding entire PDF pages as a whole, removing the need for manual OCR or chunking.

Version 0.3 introduced several features designed to optimize RAG (Retrieval-Augmented Generation) workflows and indexing efficiency:

• Semantic Chunking: An optimized chunking strategy for better RAG performance.
• Vector Streaming: Enables memory-efficient indexing in vector databases.
• Chunkwise Streaming: Supports streaming at the chunk level rather than just the file level, allowing for more flexible data processing.
• Zero-Shot Applications: Support for zero-shot application demos.

Modular Control Protocol (MCP) is a standardized communication framework that allows different components of a system to interact efficiently. In the context of Lumo, it follows a client-server architecture:

• Hosts: LLM applications (like Claude Desktop) that initiate connections.
• Clients: Components that maintain one-to-one connections with servers inside the host application.
• Servers: Systems that provide Resources (data for context), Tools (executable code/side effects), and Prompts (reusable interaction templates) to clients.

MCP vs. Traditional Function-Calling: While traditional function-calling focuses on translating natural language into JSON function calls, MCP provides a standardized protocol for resources and tool calls. This standardization makes agentic systems highly scalable compared to platform-specific function-calling implementations.

Vector Streaming is a feature designed to handle massive datasets (e.g., 10GB+ files) without exhausting system RAM.

It works by creating an asynchronous pipeline using Rust's concurrency patterns (specifically the MPSC - Multi-producer Single Consumer module). The process follows these steps:

1. Asynchronous Chunking: Content is extracted and split into chunks.
2. Buffered Embedding: Chunks are passed into an embedding thread via a buffer.
3. Streaming Output: Once a buffer is full, the chunks are embedded and sent to the main thread to be pushed to a vector database.
4. Memory Efficiency: Only the chunks and embeddings currently in the buffer are stored in memory; they are cleared once moved to the vector database.

This prevents bottlenecks by allowing text extraction, splitting, embedding, and database ingestion to happen simultaneously.

In a Fusion Deep Research agent, the "done" flag is a boolean control mechanism used in the agent's decision-making output to determine the research lifecycle.

• "done": false: Indicates that knowledge gaps still exist and more research iterations (new queries) are needed.
• "done": true: Indicates that sufficient information has been collected based on the evaluation criteria, and the process should stop.

When the agent sets "done": false, it should provide new queries to explore different perspectives of the topic.

{
  "querys": [
    "Key architectural differences between SSM models and transformer models",
    "Computational efficiency comparison between SSM models and transformer models",
    "When to use SSM models versus transformer models"
  ],
  "done": false
}

EmbedAnything supports late-interaction models to improve retrieval accuracy:

• ColPali: For running ColPali on local machines, use the quantized version available on Hugging Face (starlight-ai/colpali-v1.2-merged-onnx). It can be executed using either the Candle or ONNX runtimes.
• ColBERT: Enables fast, scalable BERT-based search over large text collections with millisecond latency.

The TextEmbedConfig object is the central configuration for the embedding process in EmbedAnything. It consists of three essential components:

1. The embedding model: Defines how text is converted into vectors.
2. Splitting strategy with chunk size: Determines how documents are divided into smaller pieces.
3. Batch size: Controls the performance of vector streaming by determining how many chunks are processed simultaneously.

Optimizing these three components allows you to balance retrieval precision, contextual richness, and hardware performance.