DOCSEARCHbyAlgolia
Submit Docs

chunking_evaluation

repository·main·Indexed 19 days ago

https://github.com/brandonstarxel/chunking_evaluation

A toolkit for text chunking and evaluation designed to compare chunking strategies and evaluate retrieval quality in AI applications. It features the ClusterSemanticChunker, tools for generating synthetic datasets via SyntheticEvaluation, and a framework for implementing custom chunkers by inheriting from BaseChunker.

Tokens
1.7K
Snippets
5
Records
6
Agent score
18%

What's inside chunking_evaluation

  1. Implement and evaluate a custom chunker

    main

    To evaluate your own chunking logic, inherit from BaseChunker and implement the split_text method. You can then use GeneralEvaluation to run the evaluation against a provided embedding function.

    from chunking_evaluation import BaseChunker, GeneralEvaluation
from chromadb.utils import embedding_functions

# Define a custom chunking class
class CustomChunker(BaseChunker):
    def split_text(self, text):
        # Custom chunking logic
        return [text[i:i+1200] for i in range(0, len(text), 1200)]

# Instantiate the custom chunker and evaluation
chunker = CustomChunker()
evaluation = GeneralEvaluation()

# Choose embedding function
default_ef = embedding_functions.OpenAIEmbeddingFunction(
    api_key="OPENAI_API_KEY",
    model_name="text-embedding-3-large"
)

# Evaluate the chunker
results = evaluation.run(chunker, default_ef)
print(results)

  3. Generate a synthetic dataset for domain-specific evaluation

    main

    Use SyntheticEvaluation to create a custom dataset from your own corpora. This involves initializing with corpora paths, generating queries/excerpts, and applying filters to ensure data quality.

    from chunking_evaluation import SyntheticEvaluation

# 1. Initialize the Environment
corpora_paths = [
    'path/to/chatlogs.txt',
    'path/to/finance.txt',
]
queries_csv_path = 'generated_queries_excerpts.csv'

evaluation = SyntheticEvaluation(corpora_paths, queries_csv_path, openai_api_key="OPENAI_API_KEY")

# 2. Generate Queries and Excerpts
# Use approximate_excerpts=True if standard generation fails
evaluation.generate_queries_and_excerpts()

# 3. Apply Filters
evaluation.filter_poor_excerpts(threshold=0.36)
evaluation.filter_duplicates(threshold=0.6)

# 4. Run the Evaluation
from chunking_evaluation import BaseChunker

class CustomChunker(BaseChunker):
    def split_text(self, text):
        return [text[i:i+1200] for i in range(0, len(text), 1200)]

chunker = CustomChunker()
results = evaluation.run(chunker)
print("Evaluation Results:", results)

  4. Evaluate a custom embedding function

    main

    You can evaluate how a specific embedding function interacts with a chunker by passing a custom EmbeddingFunction (following the chromadb interface) to evaluation.run().

    from chromadb import Documents, EmbeddingFunction, Embeddings

class MyEmbeddingFunction(EmbeddingFunction):
    def __call__(self, input: Documents) -> Embeddings:
        # embed the documents somehow
        return embeddings

# Instantiate instance of ef
default_ef = MyEmbeddingFunction()

# Evaluate the embedding function with a chunker
results = evaluation.run(chunker, default_ef)

  5. Use and evaluate ClusterSemanticChunker

    main

    The ClusterSemanticChunker is a novel chunking method provided by the package. It requires an embedding function during instantiation. You can evaluate its performance using GeneralEvaluation.

    from chunking_evaluation import BaseChunker, GeneralEvaluation
from chunking_evaluation.chunking import ClusterSemanticChunker
from chromadb.utils import embedding_functions

# Instantiate evaluation
evaluation = GeneralEvaluation()

# Choose embedding function
default_ef = embedding_functions.OpenAIEmbeddingFunction(
    api_key="OPENAI_API_KEY",
    model_name="text-embedding-3-large"
)

# Instantiate chunker and run the evaluation
chunker = ClusterSemanticChunker(default_ef, max_chunk_size=400)
results = evaluation.run(chunker, default_ef)

print(results)

  6. SyntheticEvaluation API reference

    main

    The SyntheticEvaluation class is used to build domain-specific evaluation datasets from raw text corpora.

    Methods:

    • generate_queries_and_excerpts(approximate_excerpts: bool = False): Generates queries and excerpts and saves them to the specified CSV path. Set approximate_excerpts=True if standard generation is unsuccessful.
    • filter_poor_excerpts(threshold: float): Removes queries associated with poor quality excerpts based on the provided threshold.
    • filter_duplicates(threshold: float): Removes duplicate entries based on the provided threshold.
    • run(chunker: BaseChunker): Runs the evaluation using the provided chunker on the processed synthetic dataset.