chonkiejs

Chonkie is organized into several packages depending on your requirements:

ChonkieJS is distributed via two main packages:

• @chonkiejs/core: The primary library containing all chunking implementations.
• @chonkiejs/token: Provides support for HuggingFace tokenizers.

ChonkieJS provides several chunking strategies tailored to different text processing requirements. Depending on your data type and performance needs, you can choose from the following:

• RecursiveChunker: Hierarchical splitting using customizable rules. Recommended for general-purpose chunking.
• TokenChunker: Fixed-size chunking based on token counts. Best for maintaining consistent chunk sizes.
• SentenceChunker: Splits text at sentence boundaries to preserve semantic completeness.
• SemanticChunker: Groups content based on semantic similarity (requires custom embeddings).
• CodeChunker: AST-aware chunking specifically designed for source code files.
• TableChunker: Chunks markdown or HTML tables by row or token count.
• FastChunker: High-performance, byte-based chunking utilizing SIMD acceleration.

All chunkers in ChonkieJS follow a consistent asynchronous interface. They are initialized via an async create() factory method and provide methods for both single-text and batch processing.

To use a chunker, call .chunk(text) for a single string or .chunkBatch(texts) for an array of strings. Both methods return a Promise<Chunk[]>.

// Chunk a single text
const chunks = await chunker.chunk(text);

// Chunk multiple texts
const batchResults = await chunker.chunkBatch(texts);

The integration follows a dynamic import pattern to keep the core library lightweight:

1. When you call Tokenizer.create('model-name') in @chonkiejs/core, the core attempts to dynamically import @chonkiejs/token.
2. If installed: The core uses the HuggingFaceTokenizer to perform the requested operations.
3. If not installed: The core will throw a helpful error message instructing you to install the package.

All chunkers in chonkiejs follow a factory pattern for initialization. Because chunkers may require asynchronous setup (e.g., loading models or tokenizers), you must use the static create() method instead of the new keyword.

Example for any chunker (e.g., RecursiveChunker):

const chunker = await RecursiveChunker.create(options);

// All chunkers: RecursiveChunker, TokenChunker, SentenceChunker, SemanticChunker, CodeChunker, TableChunker, FastChunker
const chunker = await RecursiveChunker.create(options);

Chonkie provides several specialized chunking strategies depending on your data type and performance requirements:

The tokenCount property on the output chunks behaves differently depending on the mode used:

• In Row mode: tokenCount represents the number of data rows in that chunk.
• In Token mode: tokenCount represents the tokenizer-based token count.

You can use the namespace parameter to partition different datasets within a single Pinecone index. This allows you to maintain separate logical collections (e.g., 'docs' vs 'faq') using the same index instance.

When a namespace is configured, the count() method returns the record count specifically for that namespace. If no namespace is set, count() returns the total count for the entire index.

const docsHandshake = new PineconeHandshake({
  index,
  namespace: 'docs',
  embeddings: embed,
});

const faqHandshake = new PineconeHandshake({
  index,
  namespace: 'faq',
  embeddings: embed,
});

Similarity is computed using the cosine distance operator (<=>) in PostgreSQL. The returned score is calculated as:

score = 1 - (embedding <=> query_vector)

Results are returned ordered by cosine distance in ascending order (most similar results first).