lingua-py Documentation

Lingua-py is a language detection library designed to identify the language of a given text snippet. It is particularly effective for short text fragments like single words, phrases, or social media messages where other libraries (like CLD2 or CLD3) often struggle.

Key features:

• High Accuracy on Short Text: Works well on single words and short phrases.
• Offline Capability: Once downloaded, it requires no external API calls or internet connection.
• Performance: Since version 2.0.0, it uses compiled Python bindings to a native Rust implementation, providing high performance and a small memory footprint.
• No Neural Networks: It uses a combination of rule-based and statistical Naive Bayes methods without relying on heavy neural networks or large word dictionaries.

Because Lingua uses Rust bindings via PyO3, the Language enum does not behave exactly like native Python enums.

• Iteration: You cannot iterate directly over Language. Instead, use sorted(Language.all()).
• Dynamic Access: You cannot use string subscripting (e.g., Language["GERMAN"]). Instead, use Language.from_str("string") which is case-insensitive.

# Iterating through all members
for language in sorted(Language.all()):
    print(language)

# Getting an enum member dynamically
assert Language.from_str("GERMAN") == Language.GERMAN
assert Language.from_str("german") == Language.GERMAN
assert Language.from_str("GeRmAn") == Language.GERMAN

Lingua uses a two-step detection process:

1. Rule-based engine: Determines the alphabet and searches for unique characters to filter out impossible languages.
2. Probabilistic n-gram model: Uses n-grams of sizes 1 to 5 to classify the remaining candidates.

Best Practice: To improve runtime performance and reduce memory consumption, restrict the set of languages to be considered in the classification process using the API methods. If you know certain languages will never appear in your input text, do not include them in the classification process. While the rule-based engine is effective at filtering, manual restriction via the API is always preferable.

If you want to reproduce the accuracy results for the library, you can generate test reports using the provided scripts. This requires Poetry.

To generate reports for all classifiers and languages:

poetry install --no-root --only script
poetry run python3 scripts/accuracy_reporter.py

To generate reports for a specific subset of detectors and languages:

poetry run python3 scripts/accuracy_reporter.py --detectors cld2 lingua-high-accuracy --languages bulgarian german

Reports are written to the /accuracy-reports directory.

To build the project from the source repository, ensure you have Python >= 3.12. Follow these steps to set up a virtual environment and install the Python wheel for your platform:

1. Clone the repository.
2. Create and activate a virtual environment.
3. Install the package using the --find-links flag pointing to the lingua directory.

git clone https://github.com/pemistahl/lingua-py.git
cd lingua-py
python3 -m venv .venv
source .venv/bin/activate
pip install --find-links=lingua lingua-language-detector

To detect the language of a text, use LanguageDetectorBuilder to create a LanguageDetector instance specifying the target languages. Call detect_language_of(text) to get the most likely Language enum member. The library is thread-safe, allowing a single instance to be shared across multiple threads.

>>> from lingua import Language, LanguageDetectorBuilder
>>> languages = [Language.ENGLISH, Language.FRENCH, Language.GERMAN, Language.SPANISH]
>>> detector = LanguageDetectorBuilder.from_languages(*languages).build()
>>> language = detector.detect_language_of("languages are awesome")
>>> language
Language.ENGLISH
>>> language.iso_code_639_1
IsoCode639_1.EN
>>> language.iso_code_639_1.name
'EN'
>>> language.iso_code_639_3
IsoCode639_3.ENG
>>> language.iso_code_639_3.name
'ENG'

You can install the library directly from PyPI using pip:

pip install lingua-language-detector

Lingua uses lazy-loading by default, loading language models only when needed. For web services where you want to avoid latency during the first request, you can enable eager-loading (preloading all models) using .with_preloaded_language_models().

LanguageDetectorBuilder.from_all_languages().with_preloaded_language_models().build()

Lingua offers a low accuracy mode for resource-constrained systems or long texts. Use .with_low_accuracy_mode() to load only a small subset of models. Note that accuracy for texts shorter than 120 characters drops significantly in this mode.

LanguageDetectorBuilder.from_all_languages().with_low_accuracy_mode().build()

By default, Lingua returns the most likely language. To avoid incorrect results for ambiguous words (e.g., "prologue" which is both English and French), you can use .with_minimum_relative_distance(threshold) during construction. If the detection is not sufficiently reliable based on this threshold, the detector returns None.

>>> from lingua import Language, LanguageDetectorBuilder
>>> languages = [Language.ENGLISH, Language.FRENCH, Language.GERMAN, Language.SPANISH]
>>> detector = LanguageDetectorBuilder.from_languages(*languages)\n.with_minimum_relative_distance(0.9)\n.build()
>>> print(detector.detect_language_of("languages are awesome"))
None