DOCSEARCHbyAlgolia
Submit Docs

DataChain Documentation

repository·main·Indexed 25 days ago

https://github.com/datachain-ai/datachain

A Python library that acts as a context layer for unstructured data, transforming files from cloud storage (S3, GCS, Azure) or local filesystems into versioned, typed, and queryable datasets using Pydantic schemas. It features a Compute Engine for heavy workloads, a Dataset DB for high-speed metadata queries and similarity search, and a Knowledge Base for AI agent integration. Supports incremental updates, checkpoint-recoverable pipelines, and remote job execution via DataChain Studio.

Tokens
95K
Snippets
202
Records
564
Agent score
83%

What's inside datachain

  1. Overview of DataChain

    main

    DataChain is a Python library designed as a context layer for unstructured data. It transforms files stored in S3, GCS, and Azure into versioned, typed datasets that can be queried at warehouse speed.

    Key components include:

    • Compute Engine: Enables distributed Python processing over files.
    • Dataset DB: Provides warehouse-speed queries over Pydantic-typed records.
    • Knowledge Base: Maintains markdown summaries for agent workflows.
    • Agent Harness: Provides skills and MCP (Model Context Protocol) support to integrate with tools like Claude Code, Cursor, and Codex, allowing them to understand your data.

  3. Understand the DataChain Chain concept

    main

    A Chain is a composable sequence of Python functions and data operations. It uses Lazy Evaluation, meaning no computation occurs until a Terminal Operation (like save(), show(), or to_pandas()) is called.

    DataChain optimizes execution by:

    1. Compiling data operations (filters, joins, aggregates) into SQL to run at warehouse speed via the Dataset DB.
    2. Running Python functions (ML models, LLM calls, multimodal extraction) on the Compute Engine.
    3. Using Atomicity to ensure that results are only committed to the Dataset DB if the entire chain succeeds, preventing partial data pollution.

  4. Understand DataChain Dataset Organization

    main

    DataChain organizes datasets using a hierarchical structure consisting of a namespace, a project, and a dataset name. The full identifier follows the pattern:

    <namespace>.<project>.<dataset>

    For example, dev.analytics.metrics refers to the metrics dataset within the analytics project of the dev namespace.

  5. Explore DataChain Studio core features

    main

    DataChain Studio provides several key capabilities for managing data pipelines:

    • Jobs: Run and monitor your data processing jobs.
    • Git Connections: Connect your Git repositories to integrate your code and workflows with the Studio.

    For programmatic interaction with the platform, refer to the API Reference. To receive event notifications from your workflows, configure Webhooks.

  8. Explore DataChain API Modules

    main

    DataChain's API is organized into several specialized modules depending on your task:

    • Core Operations: Use the datachain module for chain operations and dataset management.
    • Data Types: Access specific modules for handling different file formats (File, TextFile, ImageFile, VideoFile, AudioFile, TarVFile, ArrowRow) and spatial/geometric data (BBox, Pose, Segment).
    • Python Operations: Use custom Python functions and class-based operations.
    • Built-in Functions: Use the func module for standard data manipulation and analysis.
    • PyTorch Integration: Use the torch module for PyTorch-specific data loading utilities.
    • ML Toolkit: Use the toolkit module for common Data Science and Machine Learning operations.

  9. Understand the difference between Skill (OSS) and MCP (Studio)

    main

    DataChain provides two ways for an agent to reach your data:

    Skill (Open Source)

    • Installation: pip install datachain.
    • Data Access: Reads from local dc-knowledge/ (compiled markdown) and .datachain/db (SQLite dataset registry).
    • Compute: Runs locally on your machine using parallel threads and async prefetch via the in-process Compute Engine.
    • Best for: Solo developers or small teams syncing dc-knowledge/ and .datachain/db via Git.

    MCP (Studio)

    • Installation: Requires DataChain Studio setup (contact datachain.ai).
    • Data Access: Uses a centralized data warehouse and shared Knowledge Base via the Model Context Protocol (MCP).
    • Compute: Dispatches map(), gen(), and agg() operations to attached BYOC (Bring Your Own Cloud) clusters (CPU/GPU).
    • Best for: Large teams requiring access control, billion-row datasets, distributed compute, or headless production agent services.

  10. Understand the DataChain Data Harness concept

    main

    DataChain acts as a 'data harness' designed to complement code-centric harnesses (like Claude Code, Cursor, or Codex). While code harnesses provide context for repositories, the DataChain harness provides persistent, typed context for unstructured data.

    Key components of the Data harness include:

    • Working Unit: Typed datasets stored in the Dataset DB.
    • Raw Inputs: Files in object storage or tables in databases.
    • Persistent Context: The Dataset DB and the Knowledge Base (dc-knowledge/).
    • Tools: DataChain operations such as read_storage, map, and save.
    • Output: New dataset versions stored in .datachain/db.

    This architecture aims to optimize 'recall economics' by prioritizing cheaper summaries and datasets over expensive raw-data paths.

  11. Understand the Knowledge Base concept

    main
    The Knowledge Base is a compilation layer in DataChain that transforms persistent, typed datasets from the Dataset DB into agent-readable knowledge. Instead of using traditional RAG (Retrieval-Augmented Generation) to pull raw chunks, the system compiles datasets into structured pages that agents consume as premises before acting. This ensures agents have the necessary schema, lineage, and context to avoid hallucinations (like recomputing existing data or joining on incompatible columns).

  12. Understand the CAST Methodology for DataChain Knowledge

    main

    CAST is the core doctrine for the datachain-knowledge skill. It dictates how data should be structured into reusable layers to prevent regressions where immediate answers are provided at the expense of long-term reusability.

    Core Principles:

    • Substrate Building: The primary goal is to grow a reusable substrate for unstructured data. Answering a specific question is a byproduct, not the main job.
    • General-Purpose Rows: Every row in the Container, Asset, or Sense layers must hold the full output of the operation. Do not project or filter data to fit a specific question during the build phase. Filters, aggregations, and thresholds should be applied downstream during the Task phase.
    • Avoid Question Leakage: If you find yourself applying filters like if conf > 0.5 or if label == X before calling .save(), you are leaking the current question into the substrate. Push these operations to the Task query instead.