DOCSEARCHbyAlgolia
Submit Docs

whichlang Documentation

repository·main·Indexed 19 days ago

https://github.com/quickwit-oss/whichlang

A lightweight, high-performance Rust library for fast and precise language detection, optimized for high-throughput workloads like log and trace analysis. It supports 16 languages including English, Mandarin, and Russian, utilizing a multiclass logistic regression model with zero dependencies to achieve high accuracy and processing speeds over 100 MB/s.

Tokens
1.8K
Snippets
5
Records
9
Agent score
65%

What's inside whichlang

  1. Overview of Whichlang

    main

    Whichlang is a high-performance, high-precision language detection library written in Rust. It is designed for high-throughput environments (like search engines) where speed and accuracy are critical.

    Key Features

    • Zero Dependencies: Lightweight and easy to integrate.
    • High Throughput: Capable of processing over 100 MB/s for both short and long strings.
    • High Accuracy: Achieves approximately 99.5% accuracy on validation datasets (accuracy scales with input size).
    • Supported Languages: Arabic, Dutch, English, French, German, Hindi, Italian, Japanese, Korean, Mandarin, Portuguese, Russian, Spanish, Swedish, Turkish, and Vietnamese.

  2. How Whichlang works

    main

    Whichlang implements a multiclass logistic regression model. The model uses the following features:

    • 2, 3, and 4-grams of letters on ASCII.
    • Codepoint / 128.
    • A projection of codepoints over a specific class.

    It utilizes the 'hashing trick' to project these features into a fixed space of size 4_096. The weights used for the logistic regression are pre-calculated and stored in weight.rs.

  3. Export trained weights to Rust (`src/weights.rs`)

    main

    Once a model is trained in Python, you must export the coefficients and intercepts to a Rust file so the whichlang library can use them. The export script generates a src/weights.rs file containing:

    1. A Lang enum representing the supported languages.
    2. A three_letter_code implementation for the Lang enum.
    3. A LANGUAGES constant array.
    4. A WEIGHTS constant array (flattened coefficients).
    5. An INTERCEPTS constant array.

    The weights are extracted from model.coef_ and model.intercept_ and written as f32 values.

    # Example of the structure generated in src/weights.rs
f.write("#[derive(Clone, Copy, Debug, Eq, PartialEq)]\n")
f.write("pub enum Lang {\n")
# ... loops through model.classes_ ...
f.write("}\n\n")

f.write("pub const WEIGHTS: [f32; %d] = [\n\t" % (LANG * DIM))
# ... writes flattened coefs ...
f.write("];\n\n")

f.write("pub const INTERCEPTS: [f32; %d] = [\n\t" % LANG)
# ... writes intercepts ...
f.write("];\n\n")

  4. Train a language detection model using Logistic Regression

    main

    To train a model for whichlang, you can use sklearn.linear_model.LogisticRegression. The process involves loading a dataset (typically a CSV), converting features into a sparse CSR matrix, normalizing them, and fitting the model.

    Key parameters for the LogisticRegression model used in this workflow:

    • max_iter: Set to 500 to ensure convergence.
    • penalty: Use 'l2'.
    • multi_class: Use 'multinomial'.
    • C: Regularization strength (e.g., 64).
    • class_weight: Set to 'balanced' to handle imbalanced language distributions.

    After training, you can evaluate accuracy on both training and test sets using (model.predict(X) == y).mean().

    for C in [64]:
    model = sklearn.linear_model.LogisticRegression(
        max_iter=500, 
        penalty='l2', 
        multi_class='multinomial', 
        C=C, 
        verbose=1, 
        class_weight='balanced'
    )
    model.fit(X_train, y_train)
    print(f"Train Accuracy: {(model.predict(X_train) == y_train).mean()}")
    print(f"Test Accuracy: {(model.predict(X_test) == y_test).mean()}")

  5. Evaluate model performance with a confusion matrix

    main

    To understand which languages are being misidentified, use sklearn.metrics.confusion_matrix. You can pass specific labels to the confusion matrix to focus on certain language groups (e.g., European languages vs. Asian languages).

    from sklearn import metrics

# Evaluate specific language groups
print(sklearn.metrics.confusion_matrix(
    y_test, 
    model.predict(X_test), 
    labels=['deu', 'eng', 'fra', 'ita', 'nld', 'por', 'rus', 'spa']
))

# Evaluate another group
print(sklearn.metrics.confusion_matrix(
    y_test, 
    model.predict(X_test), 
    labels=['kor', 'jpn', 'cmn']
))

  6. Iterate over text features with `emit_tokens`

    main

    The emit_tokens function allows you to process the linguistic features of a string manually. It takes a text slice and a closure (listener) that is called for every Feature found in the text.

    Features include:

    • Feature::AsciiNGram(u32): N-grams (bigrams, trigrams, etc.) for ASCII text.
    • Feature::Unicode(char): Individual Unicode characters.
    • Feature::UnicodeClass(char): A classification of the Unicode character.

    This is useful if you want to implement custom scoring or analysis based on the same tokens used by the detector.

    use whichlang::{emit_tokens, Feature};

emit_tokens("hello", |token| {
    match token {
        Feature::AsciiNGram(ngram) => println!("Found ASCII N-gram: {:x}", ngram),
        Feature::Unicode(c) => println!("Found Unicode char: {}", c),
        Feature::UnicodeClass(c) => println!("Found Unicode class for: {}", c),
    }
});

  7. Detect the language of a string with `detect_language`

    main

    Use detect_language to identify the language of a given text slice. It returns a Lang enum representing the detected language. If the input text contains no detectable features (e.g., an empty string), it defaults to Lang::Eng (English).

    use whichlang::{detect_language, Lang};

let text = "Bonjour joyeux contribuable";
let lang = detect_language(text);

assert_eq!(lang, Lang::Fra);