DOCSEARCHbyAlgolia
Submit Docs

CocoIndex Documentation

repository·main·Indexed 27 days ago

https://github.com/cocoindex-io/cocoindex

An open-source Python framework for building incremental, production-grade data pipelines for AI agents. CocoIndex transforms diverse data sources such as code, PDFs, and Slack into fresh context for RAG and LLM applications by processing only data changes (deltas).

Tokens
191.1K
Snippets
567
Records
757
Agent score
93%

What's inside CocoIndex

  1. Overview of CocoIndex operation modules

    main

    CocoIndex includes several specialized operation modules:

    • Text operations: Provides code language detection, regex-based SeparatorSplitter, and syntax-aware RecursiveSplitter (using tree-sitter) that returns position-tracked Chunk objects.
    • Sentence Transformers: Provides local text embeddings using sentence-transformers with model caching, thread-safe GPU access, and optional normalization.
    • LiteLLM: Provides embeddings and audio transcription via a unified API for 100+ providers (e.g., OpenAI, Azure, Vertex AI, Bedrock, Cohere).
    • Entity resolution: Deduplicates entity names using FAISS embedding similarity combined with a pluggable LLM pair-resolver, supporting PINNED and PREFERRED canonical policies.

  2. Overview of CocoIndex Connectors

    main

    A connector links a CocoIndex flow to an external system. Connectors operate in one or both of the following roles:

    • As source: Imports rows into a flow. It includes change capture so that only modified data is reprocessed.
    • As target: Exports target states from a flow. It keeps the external system in sync using incremental upserts and deletions.

    Many built-in connectors support both roles.

  3. Overview of CocoIndex core concepts

    main

    CocoIndex is an incremental indexing framework designed for AI agents and RAG (Retrieval-Augmented Generation) pipelines.

    Key features include:

    • Incremental Processing: Only the delta (Δ) is reprocessed on every change, significantly reducing compute and embedding costs.
    • Declarative ETL: Define what should be in your target, and the engine ensures it stays in sync.
    • Data Lineage: Every target record (vector, row, or graph node) traces back to its exact source byte for auditability.
    • Scalability: Built with a Rust core, it supports parallel chunking and scales from single repositories to petabyte-scale stores.

  4. Understand the CocoIndex mental model

    main

    CocoIndex uses a declarative, state-driven programming model for data processing pipelines. Instead of writing imperative logic to handle data deltas (inserts, updates, deletes), you specify the Target State—what the output should look like based on the current Source State.

    Key concepts include:

    • Data transformations: Reading source state and performing operations (e.g., PDF to Markdown).
    • Target states: The desired output in external systems (e.g., Postgres, vector databases). The relationship is defined as TargetState = Transform(SourceState).
    • Incremental processing: CocoIndex automatically calculates the delta between the current target and the new target state, applying only necessary changes.
    • Function memoization: Automatically skips computations if both the input data and the transformation code remain unchanged.

  6. Understand the CocoIndex Python SDK package organization

    main

    The CocoIndex SDK is organized into a core package and several specialized sub-packages:

    • cocoindex: Contains all core APIs. These are async-first; sync variants are identified by a _blocking suffix.
    • cocoindex.connectors: Provides connectors for various data sources and targets (e.g., localfs, postgres).
    • cocoindex.resources: Contains shared data types, vector schema annotations, and ID generation utilities (e.g., Chunk, FileLike).
    • cocoindex.ops: Contains built-in operations for data processing tasks like text splitting or embedding.

    Import sub-modules directly for specific functionality:

    from cocoindex.connectors import localfs, postgres
from cocoindex.ops.text import RecursiveSplitter
from cocoindex.ops.sentence_transformers import SentenceTransformerEmbedder
from cocoindex.resources.file import FileLike, PatternFilePathMatcher
from cocoindex.resources.chunk import Chunk

  7. Understand the CocoIndex programming model

    main

    CocoIndex uses a declarative, state-driven programming model. Instead of writing logic to handle incremental updates (deltas), you specify the target state as a function of your source data.

    CocoIndex automatically handles:

    • Change detection in source data.
    • Dependency tracking.
    • Applying only the necessary updates (insertions, updates, deletions) to the target.

    This allows you to write simple, batch-style Python code that CocoIndex executes incrementally in both batch and live modes.

  8. Understand CocoIndex deadlock prevention in nested mounts

    main
    CocoIndex includes built-in deadlock prevention for nested mount scenarios. When a parent processing component mounts a child component, the parent automatically releases its concurrency slot. This allows the child component to acquire a slot and make progress, preventing the system from hanging even when max_inflight_components is set to a low value like 1.

  9. Incremental updates for Amazon S3 sources

    main

    CocoIndex performs incremental updates by calculating the difference between the S3 bucket state and the target database. When you run cocoindex update, the engine:

    • Adds new objects: Only the new files are chunked and embedded.
    • Edits existing objects: Files are re-chunked; unchanged chunks are skipped (via @coco.fn(memo=True)), while new or modified chunks are upserted.
    • Deletes objects: Rows in the target table whose source S3 objects no longer exist are automatically removed.

    Note: Since S3 is catch-up only, you must manually trigger cocoindex update <module> to detect changes in the bucket.

  10. Understand Processing Components and Component Paths

    main

    A Processing Component is the unit of execution for a single source item (like a file, row, or entity). It runs transformation logic and declares the target states produced.

    Component paths are stable, hierarchical identifiers used to match a component to its previous run, detect changes, and sync target states. Paths form a tree structure. You can create child paths using coco.component_subpath() with stable identifiers like strings, file names, or IDs.

    Example path tree:

    (root)                         ← app_main component
└── process_file
    ├── "hello.pdf"            ← process_file component
    └── "world.pdf"            ← process_file component

    If an item's path is no longer present in a run, CocoIndex automatically cleans up the target states owned by that path and its sub-paths.

    # Creating subpaths
coco.component_subpath(filename)           # e.g., coco.component_subpath("hello.pdf")
coco.component_subpath("user", user_id)    # e.g., coco.component_subpath("user", 12345)

  11. Understand CocoIndex telemetry data collection

    main

    CocoIndex sends anonymous JSON events to a Scarf gateway during release builds to understand platform and runtime usage.

    Collected Events: Telemetry is triggered at four lifecycle points:

    • init: Once per process upon import.
    • app_create: When a cocoindex.App is constructed.
    • app_update: Each time app.update() is called.
    • app_drop: Each time app.drop() is called.

    Event Schema: Each event contains exactly three fields:

    { "event": "app_update", "platform": "aarch64-macos", "lang": "python3.11" }

    Privacy Guarantees: CocoIndex never collects:

    • Your data, flow definitions, or function code.
    • Connector configurations, credentials, URLs, or database names.
    • App names, component names, file paths, or user-provided identifiers.
    • Row counts, processing times, or memory usage.
    • Persistent machine or user identifiers.