CocoIndex Documentation

website·Indexed 34 days ago

https://cocoindex.io/docs/

An ultra-performant framework for building data processing pipelines for AI workloads, featuring built-in incremental processing, state-driven programming, and memoization. The documentation covers the Python SDK, core concepts like @coco.fn, and a wide array of connectors for sources and targets including Amazon S3, Azure Blob Storage, BigQuery, Kafka, LanceDB, Neo4j, and Postgres.

Tokens
114.3K
Snippets
511
Records
630
Agent score
71%

What's inside CocoIndex

  1. Overview of In-memory LiveMap

    LiveMap[K, V] (cocoindex.resources.live_map) is an in-memory, keyed collection that acts as a connector between a producing component and a consuming component. It is designed for live mode, where the consumer reacts in real-time as the producer updates the map. The map is per-session and does not persist across restarts. It is ideal for scenarios where one part of a pipeline extracts entities and another part builds indexes or enrichments for those entities based on their IDs.
  2. Overview of CocoIndex and its programming model

    CocoIndex is a high-performance framework designed for AI data processing pipelines. It uses a declarative, state-driven programming model where developers specify the desired target state as a function of source data. The engine automatically handles change detection and incremental updates, similar to how React manages UI renders or spreadsheets manage cell formulas. This allows developers to write simple batch-style Python code without manually implementing delta logic, state handling, or complex orchestration DAGs.
  3. Overview of CocoIndex built-in operation modules

    CocoIndex provides a set of built-in operations under the cocoindex.ops package. These modules are independently importable and can be composed within a pipeline. Available modules include:

    • text: Provides code language detection, regex-based SeparatorSplitter, and syntax-aware RecursiveSplitter (using tree-sitter) that returns position-tracked Chunk objects.
    • sentence_transformers: Provides local text embeddings with model caching, thread-safe GPU access, and optional normalization.
    • litellm: Provides embeddings and audio transcription via LiteLLM's unified API, supporting 100+ providers such as OpenAI, Azure, Vertex AI, Bedrock, and Cohere.
    • entity_resolution: Deduplicates entity names using FAISS embedding similarity and a pluggable LLM pair-resolver with PINNED or PREFERRED canonical policies.
    from cocoindex.ops import text, litellm, sentence_transformers, entity_resolution
  4. Understand CocoIndex Connector roles

    A connector in CocoIndex links a flow to an external system and can serve one or both of the following roles:

    • As source: Imports rows into a flow. It supports change capture to ensure only modified data is reprocessed.
    • As target: Exports target states from a flow to an external system, maintaining synchronization via incremental upserts and deletions.
  5. Implement multi-vector image search with ColPali and Qdrant

    ColPali allows for finer-grained image retrieval than CLIP by emitting a 'bag' of vectors (one per image patch) instead of a single embedding. This is particularly effective for dense or text-heavy images. To implement this, use a Qdrant collection configured with a MultiVectorSchema and the max_sim multivector comparator, which enables late-interaction scoring where each query vector matches its best-matching patch.
  6. Build a semantic index from Postgres using CocoIndex

    CocoIndex can be used to create a semantic index over structured Postgres data. The pipeline reads rows from a source table, transforms them (e.g., deriving fields and generating embeddings), and writes them to a target table using pgvector. The system uses a Rust engine to handle incremental processing, meaning only changed rows are re-processed and re-upserted.
  7. Aggregate multiple file summaries into a project-level overview

    When dealing with multi-file projects, use an aggregation function to synthesize individual file extractions into a single CodebaseInfo object. For single-file projects, the function can return the file info directly to avoid unnecessary LLM calls. For multi-file projects, the LLM is used to create a unified project purpose and a consolidated Mermaid graph of components.
    @coco.fn
    async def aggregate_project_info(
        project_name: str,
        file_infos: list[CodebaseInfo],
    ) -> CodebaseInfo:
        """Aggregate multiple file extractions into a project-level summary."""
        if not file_infos:
            return CodebaseInfo(name=project_name, summary="Empty project with no Python files.")
    
        if len(file_infos) == 1:
            info = file_infos[0]
            return CodebaseInfo(
                name=project_name,
                summary=info.summary,
                public_classes=info.public_classes,
                public_functions=info.public_functions,
                mermaid_graphs=info.mermaid_graphs,
            )
    
        files_text = "\n\n".join(
            f"### {info.name}\n"
            f"Summary: {info.summary}\n"
            f"Classes: {', '.join(c.name for c in info.public_classes) or 'None'}\n"
            f"Functions: {', '.join(f.name for f in info.public_functions) or 'None'}"
            for info in file_infos
        )
    
        all_graphs = [g for info in file_infos for g in info.mermaid_graphs]
    
        prompt = f"""Aggregate the following Python files into a project-level summary.
    
    Project name: {project_name}
    
    Files:
    {files_text}
    
    Create a unified CodebaseInfo that:
    1. Summarizes the overall project purpose
    2. Lists the most important public classes across all files
    3. Lists the most important public functions across all files
    4. For mermaid_graphs: create a single unified graph showing how the CocoIndex
       components connect across the project
    """
    
        result = await _instructor_client.chat.completions.create(
            model=LLM_MODEL,
            response_model=CodebaseInfo,
            messages=[{"role": "user", "content": prompt}],
        )
        result = CodebaseInfo.model_validate(result.model_dump())
    
        if not result.mermaid_graphs and all_graphs:
            result.mermaid_graphs = all_graphs
    
        return result
  8. Resolve duplicate entities in a knowledge graph pipeline

    When building a knowledge graph from documentation, LLMs may identify the same concept using different names (e.g., "CocoIndex" vs "Cocoindex"). To handle this, implement an entity-resolution pass using embeddings and an LLM to collapse near-duplicates. This resolution step should be positioned in the pipeline between the per-file extraction phase (Phase 1) and the concept graph construction phase (Phase 2).
  9. Build a semantic codebase index with CocoIndex

    CocoIndex allows developers to create a live semantic index of a codebase for AI agents and code search. It uses a Rust engine for incremental processing and change tracking, ensuring only modified files are re-embedded. Key features include syntax-aware chunking via Tree-sitter (preserving functions and classes), live updates via filesystem watching, and a consistent embedding model across indexing and query paths.
  10. Understand Processing Components and Component Paths

    A Processing Component is the unit of execution for a single source item (e.g., a file or database row). It handles transformation logic and declares the target states produced for that item.

    CocoIndex uses a hierarchical component path (a stable identifier) to match components across runs, detect changes, and sync target states. If a path disappears, CocoIndex cleans up the target states owned by that path and its children. Paths are created using coco.component_subpath() with stable identifiers like filenames or primary keys.

    # Examples of creating stable component paths
    coco.component_subpath("hello.pdf")           # Using a filename
    coco.component_subpath("user", 12345)        # Using a category and ID
  11. Build a product recommendation graph with CocoIndex and Neo4j

    This pipeline transforms product JSON listings into a Neo4j graph to enable recommendation queries. The system uses an LLM to extract taxonomy labels (what a product is) and complementary labels (what pairs with it). The graph consists of Product nodes and shared Taxonomy nodes, connected by PRODUCT_TAXONOMY and PRODUCT_COMPLEMENTARY_TAXONOMY edges. Recommendations are generated by finding products whose 'is-a' taxonomy matches another product's 'complementary' taxonomy.
  12. Perform semantic image search using CLIP and Qdrant

    By using a shared embedding space for images and text (such as CLIP), you can perform searches based on visual meaning rather than metadata tags. In this architecture, a text query is embedded using the same CLIP model as the images, and a cosine similarity search is performed in Qdrant to find the most visually similar images.