Charabia

Charabia is a multilingual library used to tokenize queries and documents. Its primary role is to take a sentence or phrase and split it into smaller language units called tokens.

It provides a simple API to perform three main operations on text of a specific language:

1. Segmentation: Splitting text into smaller units.
2. Normalization: Transforming text (e.g., lowercase, removing marks, handling specific language rules).
3. Tokenization: A combination of segmentation and normalization.

Charabia automatically detects the Script/Language of the input text to choose the appropriate specialized pipeline.

The dictionary data is vendored into the dictionaries/source/ directory. To update the local dictionary with the latest version from the upstream source, run the sync_dictionaries binary.

bin/sync_dictionaries

The PersianNormalizer applies specific transformations to Persian text to ensure consistent search behavior. It targets tokens identified as having the Arabic script and the Pes (Persian) language.

Key transformations include:

• Yeh Normalization: Converts various forms of Yeh ('ي', 'ی', 'ى', 'ۀ') to the standard Persian 'ی'.
• Kaf Normalization: Converts Arabic 'ك' and Persian 'ک' variants to the standard Persian 'ک'.
• Digit Normalization: Converts Persian digits ('۰'-'۹') to ASCII digits ('0'-'9').
• Punctuation Normalization: Converts Persian/Arabic comma ('،') to ASCII ',' and question mark ('؟') to ASCII '?'.
• Rial Sign: Normalizes the Rial sign ('﷼') to the string "RIAL".
• ZWNJ Removal: Removes the Zero Width Non-Joiner (ZWNJ) character ('\u{200C}').

Normalization in Charabia is split into two categories of normalizers:

1. Non-lossy Normalizers (NORMALIZERS): These are always applied. They include CompatibilityDecompositionNormalizer, ControlCharNormalizer, Classifier, and PersianNormalizer.
2. Lossy Normalizers (LOSSY_NORMALIZERS): These are only applied if NormalizerOption.lossy is set to true. They include LowercaseNormalizer, QuoteNormalizer, AeOeNormalizer, ChineseNormalizer, JapaneseNormalizer, GreekNormalizer, ArabicNormalizer, NonspacingMarkNormalizer, VietnameseNormalizer, and TurkishNormalizer.

When you call .normalize(), the system iterates through these lists and calls should_normalize() on each normalizer to determine if it is applicable to the current Token (based on its Script and Language).

Charabia uses TokenKind to classify tokens during processing. This classification determines how a token is treated during indexing and search.

TokenKind

• Word: A standard searchable token.
• StopWord: A token that can be ignored to optimize size and performance, or indexed as a Word.
• Separator(SeparatorKind): A token that shouldn't be indexed but is used to determine word proximity.
• Unknown: The default kind.

SeparatorKind

Separators are further categorized into two types:

• Hard: Separates two tokens that are not in the same context (e.g., different phrases).
• Soft: Separates two tokens that are in the same context (e.g., within the same phrase).

The TurkishNormalizer is a specialized normalizer designed to handle specific characters in the Turkish alphabet. Its primary function is to normalize the character 'ı' (dotless i) to 'i' (dotted i).

Key behaviors:

• Character Mapping: It maps 'ı' $\rightarrow$ 'i'.
• Scope: It only targets the 'ı' character because other peculiar Turkish characters are already handled by the standard Script::Latin normalizer.
• Activation: The normalizer only applies to tokens where the script is Script::Latin and the lemma contains the character 'ı'.

The kvariants crate provides a way to map specific ideograph characters to their canonical variants using the KVARIANTS constant. You can use the .get() method on KVARIANTS with a character to retrieve a mapping. If a mapping exists, you can access the destination_ideograph field to get the canonical version.

use kvariants::KVARIANTS;

let c = '澚';

let kvariant = match KVARIANTS.get(&c) {
    Some(kvariant) => kvariant.destination_ideograph,
    None => c,
};

assert_eq!(kvariant, '澳');

To tokenize text, use the tokenize() method provided by the Tokenize trait. This returns an iterator of tokens. Each token can be inspected to see its normalized form (lemma) and its classification (e.g., whether it is a word or a separator).

Example:

use charabia::Tokenize;

let orig = "Thé quick ("brown") fox can't jump 32.3 feet, right? Brr, it's 29.3°F!";

// tokenize the text.
let mut tokens = orig.tokenize();

let token = tokens.next().unwrap();
// the lemma into the token is normalized: `Thé` became `the`.
assert_eq!(token.lemma(), "the");
// token is classified as a word
assert!(token.is_word());

let token = tokens.next().unwrap();
assert_eq!(token.lemma(), " ");
// token is classified as a separator
assert!(token.is_separator());

To split text into segments without full normalization, use the segment_str() method provided by the Segment trait. This returns an iterator over the segments of the string.

use charabia::Segment;

let orig = "The quick ("brown") fox can't jump 32.3 feet, right? Brr, it's 29.3°F!";

// segment the text.
let mut segments = orig.segment_str();

assert_eq!(segments.next(), Some("The"));
assert_eq!(segments.next(), Some(" "));
assert_eq!(segments.next(), Some("quick"));

The NormalizerOption struct controls how the normalization pipeline behaves. It is used by both Token and &str normalization methods.

Key fields:

• create_char_map: If true, the normalizer will attempt to track the mapping between original characters and their normalized versions. This is useful for mapping normalized text back to original offsets.
• lossy: If true, the pipeline will include LOSSY_NORMALIZERS (such as LowercaseNormalizer, QuoteNormalizer, ArabicNormalizer, etc.) in addition to the standard NORMALIZERS.
• classifier: A ClassifierOption used to configure the Classifier normalizer, specifically for stop_words and separators.

#[derive(Debug, Clone, Default)]
pub struct NormalizerOption<'tb> {
    pub create_char_map: bool,
    pub classifier: ClassifierOption<'tb>,
    pub lossy: bool,
}