repository·main·Indexed 26 days ago
https://github.com/feyninc/chonkieA lightweight, high-performance Python library for efficient text chunking in RAG (Retrieval-Augmented Generation) pipelines. Version 1.7.0 supports various strategies including token-based, semantic, agentic, and code chunking. It features a Pipeline API for chaining chunking and refinement steps, a self-hosted REST API server, and integrated support for multiple tokenizers, embedding providers, LLM genies, and vector database handshakes.
Chefs are specialized Python classes designed to automate data preprocessing and transformation. They prepare raw data (like text, markdown, or images) into structured formats suitable for downstream tasks like chunking or indexing.
Note: Chefs are currently available only in Python.
The Refinery module in Chonkie is used to enhance chunks by adding additional context. This process improves the quality of embeddings and keyword indexing.
There are two primary types of refineries available:
Chonkie provides several chunking strategies tailored for different use cases. Choose a chunker based on your data type and performance requirements:
Chonkie uses a concept called Handshakes to facilitate easy integration with various vector databases. Handshakes allow you to embed your text chunks and write them directly to your chosen database with minimal code.
Supported vector database handshakes include:
Chonkie is designed for RAG (Retrieval-Augmented Generation) applications with a focus on speed, simplicity, and lightweight resource usage.
To ensure high-quality chunks, Chonkie divides the process into three distinct stages:
When using Chonkie, aim for chunks that are:
Chonkie achieves high performance through several internal mechanisms:
AutoTikTokenizer (a wrapper around tiktoken) with estimate-validate feedback loops to optimize chunk sizes efficiently.Model2Vec static embeddings by default for ultra-fast, lightweight embedding lookups.To use JinaEmbeddings for semantic chunking, you must install the chonkie package with the jina extra to include the necessary jina library dependency.
pip install "chonkie[jina]"The Chonkie CLI allows you to configure chunking behavior using two methods: dedicated explicit options and key-value parameter pairs.
Use these dedicated flags for common settings:
--chunk-size: Set the maximum tokens per chunk.--chunk-overlap: Set the overlap between chunks.--threshold: Set the semantic similarity threshold.For component-specific or additional parameters, use the *_params options (e.g., --chunker-params, --chef-params, --refiner-params, --handshaker-params) with key=value syntax.
Type Conversion Rules:
true/false $\rightarrow$ booleannone/null $\rightarrow$ NonePrecedence: Explicit options (like --chunk-size) will override values provided within *_params if both are present.
# Single parameter
--chunker-params tokenizer=gpt2
# Multiple parameters (repeat the option)
--chunker-params tokenizer=gpt2 --chunker-params min_characters_per_chunk=50
# Boolean parameters
--chunker-params verbose=true
# Numeric parameters
--chunker-params chunk_size=512
--chunker-params threshold=0.8Fetchers are the first stage in the CHOMP pipeline (CHef -> CHunker -> Refinery -> Porter/Handshake). They retrieve data from sources and pass it to subsequent stages. You can use the .fetch_from() method within a Pipeline object to ingest data.
from chonkie.pipeline import Pipeline
# Fetch a single file
doc = (Pipeline()
.fetch_from("file", path="document.txt")
.process_with("text")
.chunk_with("recursive", chunk_size=512)
.run())
# Fetch all files in a directory with specific extensions
docs = (Pipeline()
.fetch_from("file", dir="./docs", ext=[".txt", ".md"])
.process_with("text")
.chunk_with("recursive", chunk_size=512)
.run())To use the ElasticHandshake class for exporting Chonkie chunks to Elasticsearch, you must install the elastic extra for the chonkie package.
pip install chonkie[elastic]To use the TurbopufferHandshake class to export Chonkie chunks into a Turbopuffer vector database, you must install the turbopuffer extra for chonkie via pip.
pip install chonkie[turbopuffer]