Whoosh-Reloaded

Whoosh is a fast, pure-Python search engine library designed for indexing free-form or structured text and performing searches based on simple or complex criteria.

Key characteristics include:

• Pure Python: Runs anywhere Python runs without requiring a compiler or Java.
• Customizable: Most core behaviors—including text indexing, term information storage, query parsing, and scoring algorithms—are extensible and replaceable.
• Ranking: Uses Okapi BM25F as the default ranking function, but supports custom scoring.
• Storage: Creates relatively small indexes and allows storing arbitrary Python objects alongside indexed documents.
• Requirement: All indexed text must be unicode.

Stemming is a heuristic process that reduces words to their base form (e.g., "rendering" becomes "render"). This allows users to find documents containing different word forms of the same root.

Whoosh provides several algorithms including Porter, Porter2, Paice Husk, and Lovins.

Pros:

• Increases recall (matches variations).
• Reduces index size by collapsing multiple forms into one term.
• Faster than using variations.

Cons:

• Can incorrectly change word meanings.
• Stemmed forms are often not real words, making them unsuitable for spell-checking.

>>> from whoosh.lang.porter import stem
>>> stem("rendering")
'render'

Formatters define how the highlighted terms are visually represented in the output (e.g., using HTML tags or case changes).

• UppercaseFormatter: Converts highlighted terms to uppercase.
• HtmlFormatter: Wraps highlighted terms in HTML tags (e.g., <b> or <em>).
• GenshiFormatter: Uses the Genshi templating engine to format the output.

The whoosh.spelling module provides Corrector objects designed to identify and suggest corrections for spelling errors.

Available corrector types include:

• Corrector: The base class for spelling correction logic.
• ReaderCorrector: A corrector that utilizes a reader to perform corrections.
• MultiCorrector: A corrector that can combine multiple correction strategies.

The whoosh.codec.base module provides the base classes required to implement custom indexing and reading logic. To create a custom storage format, you must implement or extend the following core classes:

• Codec: The top-level entry point for managing the index format.
• PerDocumentWriter: Handles the writing of data on a per-document basis.
• FieldWriter: Responsible for writing individual field data.
• PostingsWriter: Manages the writing of postings (the list of document IDs for a term).
• TermsReader: Provides the ability to read terms from the index.
• PerDocumentReader: Handles reading data for specific documents.
• Segment: Represents a portion of the index managed by the codec.

The whoosh.scoring module provides the framework for calculating how relevant a document is to a query. It is organized into three main categories of classes:

1. Base classes: The foundational interfaces for defining how weights are applied and how scores are calculated.
2. Scoring algorithm classes: Concrete implementations of standard information retrieval algorithms (e.g., BM25F, TF-IDF).
3. Scoring utility classes: Tools used to combine, modify, or transform weights (e.g., combining multiple weighting models or reversing them).

The whoosh.spelling module provides QueryCorrector objects to handle error correction directly within the query processing pipeline.

Available query corrector types include:

• QueryCorrector: The base class for query-level correction.
• SimpleQueryCorrector: A simplified implementation of query correction.

When a correction is made, it is represented by a Correction object.

When performing facet operations, Whoosh handles documents that lack the required field or terms as follows:

• Sorting: Documents with missing values will always sort to the end of the results.
• Grouping: Missing documents will be placed into a group with the key None.

The whoosh.filedb.filestore module provides the storage layer for the file database. It is built around the Storage base class, which defines the interface for managing data.

Users can choose between two primary implementations:

1. FileStorage: Persists data to the file system.
2. RamStorage: Keeps data in memory (volatile).

Use FileStorage for persistent search indexes and RamStorage for testing or temporary, high-speed operations.

A schema defines the set of all possible fields that can exist in a document within an index. While the schema defines the available fields, individual documents can contain only a subset of them.

Fields have two primary capabilities:

1. Indexed: The field is searchable.
2. Stored: The field value is saved in the index so it can be retrieved and displayed in search results.

Some fields are both indexed and stored, while others might only be one or the other.