DOCSEARCHbyAlgolia
Submit Docs

lingua-rs

repository·main·Indexed 22 days ago

https://github.com/pemistahl/lingua-rs

A high-accuracy language detection library for Rust and Python, version 1.8.0. Lingua is specifically designed to handle short text fragments, such as single words or phrases, and functions completely offline using rule-based and statistical Naive Bayes methods. It supports 75 languages and utilizes finite-state transducer (FST) storage formats in its language models to reduce memory consumption.

Tokens
10.6K
Snippets
30
Records
64
Agent score
76%

What's inside lingua

  1. What is Lingua and when to use it

    main

    Lingua is a language detection library designed to identify the language of a given text. It is particularly useful as a preprocessing step for natural language processing (NLP) tasks like text classification, spell checking, or routing communications (e.g., emails) to the correct department.

    Unlike many other libraries that require lengthy text fragments to be accurate, Lingua is optimized for high accuracy even on very short text snippets, such as single words or short phrases. It uses a combination of rule-based and statistical Naive Bayes methods without relying on neural networks, external APIs, or large word dictionaries, allowing it to function completely offline.

  2. What is Lingua?

    main

    Lingua is a language detection library designed to identify the language of a given text. It is particularly effective for short text snippets, such as single words or phrases, where other libraries often struggle.

    Key features include:

    • High Accuracy on Short Text: Performs well on single words, word pairs, and short sentences.
    • Offline Capability: Works completely offline without requiring external APIs or services.
    • Hybrid Approach: Uses both rule-based and statistical Naive Bayes methods (without neural networks or word dictionaries).
    • Performance: Since version 2.0.0, it uses compiled Python bindings to a native Rust implementation, providing high performance and a small memory footprint.

  4. Use the Zulu language model with Lingua

    main
    This package provides the Zulu language model for Lingua. To enable Zulu language detection in your Lingua project, you must include this language model package as a dependency. The model uses a finite-state transducer (FST) storage format to minimize memory consumption and is compressed using the Brotli algorithm to reduce file size.

  5. Use the Welsh language model with Lingua

    main

    This package provides the language model for Welsh (cy) used by Lingua. To perform language detection for Welsh, include this language model in your Lingua setup.

    Note that since version 1.3.0, models are stored as finite-state transducers (FSTs), which significantly reduces memory consumption compared to previous versions.

  6. Use the Yoruba language model with Lingua

    main
    The lingua-yo language model provides support for the Yoruba language within the Lingua ecosystem. To use Yoruba in your language detection tasks, you must include this language model package alongside the core lingua library. This model uses a finite-state transducer (FST) storage format to minimize memory consumption.

  11. Use the Bosnian language model with Lingua

    main

    The lingua-bs package provides the language model specifically for the Bosnian language. This model is intended to be used as a plugin or dependency for the main Lingua library to enable high-accuracy Bosnian language detection.

    Note that as of version 1.3.0, the model uses a finite-state transducer (FST) storage format, which significantly reduces memory consumption compared to previous versions.

    /* This package is a language model dependency for the main Lingua library. */