text-splitter

The splitter aims to maximize chunk size while respecting semantic boundaries. It follows these steps:

1. Identify Levels: It splits text by increasing semantic levels (e.g., from characters up to headings or syntax tree depth).
2. Select Highest Level: It checks the first item of each level and selects the highest level whose first item still fits within the chunk size.
3. Merge: It merges neighboring sections of that level (or higher) into a chunk to maximize length without crossing semantic boundaries.

Semantic Levels by Splitter Type

TextSplitter (Ascending order)

1. Characters
2. Unicode Grapheme Cluster Boundaries
3. Unicode Word Boundaries
4. Unicode Sentence Boundaries
5. Newline sequences (e.g., \n\n is a higher level than \n)

MarkdownSplitter (Ascending order)

1. Characters
2. Unicode Grapheme Cluster Boundaries
3. Unicode Word Boundaries
4. Unicode Sentence Boundaries
5. Soft line breaks (single newline)
6. Inline elements (text, emphasis, links, etc.)
7. Block elements (paragraphs, code blocks, metadata, etc.)
8. Thematic breaks (horizontal rules)
9. Headings by level

CodeSplitter (Ascending order)

1. Characters
2. Unicode Grapheme Cluster Boundaries
3. Unicode Word Boundaries
4. Unicode Sentence Boundaries
5. Syntax tree depth (e.g., a function is a higher level than a statement inside it)

Note: Splitting never occurs below the character level to ensure valid Unicode strings.

To use a Hugging Face tokenizer for token-based splitting, you must enable the tokenizers feature in text-splitter and the http feature in tokenizers.

Setup:

cargo add text-splitter --features tokenizers
cargo add tokenizers --features http

Usage: Note: If your tokenizer has truncation enabled, disable it before passing it to the splitter to prevent chunk sizes from being capped by the tokenizer's limit.

use text_splitter::{ChunkConfig, TextSplitter};
use tokenizers::Tokenizer;

let mut tokenizer = Tokenizer::from_pretrained("bert-base-cased", None).unwrap();
// Disable truncation to avoid capping chunk sizes
tokenizer.with_truncation(None).unwrap();

let max_tokens = 1000;
let splitter = TextSplitter::new(ChunkConfig::new(max_tokens).with_sizer(tokenizer));

let chunks = splitter.chunks("your document text");

use text_splitter::{ChunkConfig, TextSplitter};
// Can also use anything else that implements the ChunkSizer
// trait from the text_splitter crate.
use tokenizers::Tokenizer;

let mut tokenizer = Tokenizer::from_pretrained("bert-base-cased", None).unwrap();
// If your tokenizer has truncation enabled, disable it before passing it to
// the splitter. Otherwise chunk sizes can be capped by the tokenizer's
// truncation limit.
tokenizer.with_truncation(None).unwrap();

let max_tokens = 1000;
let splitter = TextSplitter::new(ChunkConfig::new(max_tokens).with_sizer(tokenizer));

let chunks = splitter.chunks("your document text");

To split code using tree-sitter parsers, enable the code feature and add the appropriate tree-sitter-<language> crate.

Setup:

cargo add text-splitter --features code
cargo add tree-sitter-<language>

Usage:

use text_splitter::CodeSplitter;

let max_characters = 1000;
// Requires a tree-sitter language parser
let splitter = CodeSplitter::new(tree_sitter_rust::LANGUAGE, max_characters).expect("Invalid tree-sitter language");

let chunks = splitter.chunks("your code file");

use text_splitter::CodeSplitter;
// Maximum number of characters in a chunk. Can also use a range.
let max_characters = 1000;
// Default implementation uses character count for chunk size.
// Can also use all of the same tokenizer implementations as `TextSplitter`.
let splitter = CodeSplitter::new(tree_sitter_rust::LANGUAGE, max_characters).expect("Invalid tree-sitter language");

let chunks = splitter.chunks("your code file");

The ChunkCapacity struct defines the target and maximum size for text chunks. It allows for two distinct modes of operation:

1. Fixed Size: Set desired and max to the same value. This is useful when you need to strictly adhere to a limit, such as a model's context window.
2. Loose Targeting: Set max to a value larger than desired. The splitter will attempt to stay close to desired but is permitted to grow up to max if it allows the chunk to stay at a higher semantic level (e.g., not breaking a paragraph mid-sentence).

You can create a ChunkCapacity from a single usize (where desired == max) or from various Rust range types.

// Fixed size (desired = 512, max = 512)
let capacity = ChunkCapacity::new(512);

// Loose targeting (desired = 512, max = 1024)
let capacity = ChunkCapacity::new(512).with_max(1024).unwrap();

// From a range (desired = 5, max = 10)
let capacity = ChunkCapacity::from(5..11);

When using custom semantic levels for text splitting, it is possible that a semantic unit (like a paragraph or a sentence) is larger than the target chunk size. To prevent the splitter from getting stuck, it uses a FallbackLevel to break down text into smaller, standard Unicode segments.

Available fallback levels:

• Char: Splits by individual characters. This is the smallest possible unit that ensures valid UTF-8 strings.
• GraphemeCluster: Splits by Unicode grapheme clusters.
• Word: Splits by Unicode words.
• Sentence: Splits by Unicode sentences.

You can use the sections method on a FallbackLevel to iterate over the text segments, returning an iterator of (index, &str) tuples representing the start position and the content of each segment.

// Example of how FallbackLevel segments text
use text_splitter::FallbackLevel;

let text = "Hello, world!";
let segments: Vec<(usize, &str)> = FallbackLevel::Word.sections(text).collect();
// segments will contain the indices and substrings for each word