tantivy-py Documentation

A DocAddress object contains the necessary information to uniquely identify a specific document within the context of a Searcher.

It consists of two parts:

• segment_ord: An integer identifying the specific segment hosting the document. This ordinal is only meaningful when used in the context of a Searcher.
• doc: The segment-local DocId (an integer) for the document.

In tantivy-py, the Schema object defines the structure of your index. The schema is strictly typed. Because of this strictness, you should not attempt to instantiate Schema directly; instead, use the SchemaBuilder class to construct your schema definition.

# Note: Actual usage requires SchemaBuilder as mentioned in the documentation
# Example pattern based on documentation description:
from tantivy import SchemaBuilder

schema_builder = SchemaBuilder()
# ... add fields using schema_builder ...
schema = schema_builder.build()

The Tokenizer class provides access to all of Tantivy's built-in tokenizers via static methods. Each method returns a wrapper around a specific Tantivy tokenizer.

These tokenizer objects are primarily intended to be passed to a TextAnalyzerBuilder using the tokenizer= parameter to define how text should be broken down into tokens during indexing or searching.

tokenizer = Tokenizer.regex(r"\w+")
# Typically used as: 
# builder = TextAnalyzerBuilder(tokenizer=tokenizer)

In Tantivy-py, there is a distinction between a Tokenizer and a Text Analyzer, though the Index API uses the term 'tokenizer' to refer to both:

• Tokenizer: A component that segments raw text into individual tokens.
• Text Analyzer: A complete pipeline that starts with one Tokenizer and applies zero or more Filter objects (like lowercase or stopword removal) to the tokens.

Important API Note: When using SchemaBuilder.add_text_field(..., tokenizer_name=...) or Index.register_tokenizer(...), the parameter expects the name of a Text Analyzer, not just a raw tokenizer.

Tantivy requires a strict schema where you must specify in advance whether a field is indexed, stored, or configured as a fast field. You use the SchemaBuilder class to define fields one by one and then finalize the process by calling .build().

Key Concepts:

• Stored: If True, the field's content can be retrieved from a Searcher later.
• Indexed: If True, the field is indexed for searching.
• Fast Fields: A column-oriented storage format designed for fast random access of document fields given a document ID.
• Build Lifecycle: Once .build() is called, the SchemaBuilder instance can no longer be used.

>>> builder = tantivy.SchemaBuilder()
>>> title = builder.add_text_field("title", stored=True)
>>> body = builder.add_text_field("body")
>>> schema = builder.build()

The Filter class provides access to all of Tantivy's built-in TokenFilters. These filter objects are designed to be passed to the filter() method of a TextAnalyzerBuilder instance to customize how text is processed during indexing or searching.

Common use cases include removing stopwords, stemming words, or normalizing text (e.g., converting to lowercase or ASCII folding).

# Example of creating a filter
filter = Filter.alpha_num()

When you add documents to a tantivy index, the data is stored in multiple sections called segments. To maintain index performance, these segments are merged together in background threads.

Currently, tantivy-py uses the LogMergePolicy as the default merge policy, which is suitable for most use cases. Because merging happens in background threads, you must ensure these processes complete before finishing your indexing task to avoid data inconsistencies or incomplete segments.

After adding documents and calling writer.commit(), you must call writer.wait_merging_threads() to allow background segment merging to finish.

Warning: Calling wait_merging_threads() will consume the writer object, making the identifier no longer usable for further operations. This should be the final step in your indexing workflow.

schema = Schema(...) # Define your schema
index = Index(schema)
writer = index.writer()

for ... in data:
    document = Document(...) # Create your document
    writer.add_document(document)

writer.commit()
# Final step: wait for background threads to finish
writer.wait_merging_threads()