Advanced Chunker

Use the semantic_chunker.visualization module to inspect the semantic relationships and clustering results. You can generate attention heatmaps, semantic graphs, and cluster previews using debug information obtained from get_debug_info().

from semantic_chunker.visualization import plot_attention_matrix, plot_semantic_graph, preview_clusters
from semantic_chunker.core import SemanticChunker

chunker = SemanticChunker(max_tokens=512)
debug = chunker.get_debug_info(chunks)

preview_clusters(debug["original_chunks"], debug["clusters"])
plot_attention_matrix(debug["similarity_matrix"], debug["clusters"])
plot_semantic_graph(debug["original_chunks"], debug["semantic_pairs"], debug["clusters"])

Install the package using pip to enable semantic-aware chunking and clustering for RAG and LLM pipelines.

pip install advanced-chunker

Use the SemanticChunker class to merge semantically similar text chunks. The input should be a list of dictionaries, where each dictionary contains a text key. The chunk() method returns a list of merged chunks, each containing the combined text and a Metadata list of the original constituent chunks.

from semantic_chunker.core import SemanticChunker

chunks = [
    {"text": "Artificial intelligence is a growing field."},
    {"text": "Machine learning is a subset of AI."},
    {"text": "Photosynthesis occurs in plants."},
    {"text": "Deep learning uses neural networks."},
    {"text": "Plants convert sunlight into energy."},
]

chunker = SemanticChunker(max_tokens=512)
merged_chunks = chunker.chunk(chunks)

for i, merged in enumerate(merged_chunks):
    print(f"Chunk {i}:")
    print(merged["text"])
    print()

To use semantic chunking within a LangChain pipeline, use the SemanticChunkerSplitter. It accepts Document objects and returns split documents based on semantic clusters.

from langchain_core.documents import Document
from semantic_chunker.integrations.langchain import SemanticChunkerSplitter

splitter = SemanticChunkerSplitter(cluster_threshold=0.4, return_merged=True)

docs = [Document(page_content="Some long technical text here", metadata={"source": "report.pdf"})]
split = splitter.split_documents(docs)

The following arguments are available for the chunker chunk command:

The CLI allows for automated semantic chunking via the chunk command. You can specify input files, similarity thresholds, and export settings.

chunker chunk \
  --chunks path/to/chunks.json \
  --threshold 0.5 \
  --similarity-threshold 0.4 \
  --max-tokens 512 \
  --preview \
  --visualize \
  --export \
  --export-path output/merged \
  --export-format json

The following flags are available for the chunk command in the Semantic Chunker CLI:

--chunks (required): Path to input JSON list of chunks
--threshold (float, default: 0.5): Cluster threshold
--similarity-threshold (float, default: 0.4): Min similarity for clustering
--max-tokens (int, default: 512): Max tokens per merged chunk
--preview: Show preview of clustered chunks
--visualize: Show similarity matrix plot
--export: Export merged chunks
--export-path (string, default: "output/merged"): Path for export file (no extension)
--export-format (choices: ["json", "md", "csv"], default: "json"): Export file format

The chunk command allows you to merge semantically similar chunks from an input JSON file. It uses clustering and similarity thresholds to group related text segments and can optionally preview clusters, visualize similarity matrices, or export the results to various formats.

Input Format: The --chunks argument requires a path to a JSON file containing a list of chunks.

Output Formats: Supported formats via --export-format are json, md, and csv.

# Basic usage to merge chunks
python -m semantic_chunker.cli chunk --chunks input_chunks.json

# Advanced usage with visualization and export
python -m semantic_chunker.cli chunk \
    --chunks input_chunks.json \
    --threshold 0.6 \
    --similarity-threshold 0.5 \
    --max-tokens 1024 \
    --preview \
    --visualize \
    --export \
    --export-path output/my_result \
    --export-format md