DOCSEARCHbyAlgolia
Submit Docs

GlotLID Language Identification

repository·main·Indexed 42 days ago

https://github.com/cisnlp/glotlid

An open-source language identification (LID) model supporting over 2000 language-script labels. It provides high-accuracy identification, sentence embeddings, and the ability to restrict predictions to specific language subsets via the CustomLID class. Available in versions V1, V2, and V3, with V3 supporting 2114 labels including 'zxx' and 'und' series for non-linguistic or undetermined content.

Tokens
20.7K
Snippets
10
Records
16
Agent score
3%

What's inside GlotLID

  1. Interpret 'zxx' and 'und' labels in GlotLID V3

    main

    GlotLID V3 introduces specific labels to handle non-linguistic or unknown content:

    • zxx (Non-linguistic content): Used for content that is not a natural language, often resulting from OCR errors, poorly formatted PDFs, or web crawling artifacts.
      • Examples: zxx_Latn, zxx_Arab, zxx_Zzzz.
    • und (Undetermined): Used for content containing Unicode characters that the model has not seen before. These labels are generated based on Unicode categories/ranges (via GlotScript) to denote undetermined writing systems.
      • Examples: und_Xsux, und_Wcho.

    Key Difference: Unlike some other models (like NLLB), GlotLID uses zxx specifically for noise/formatting issues and und for unknown Unicode/writing systems.

  2. Limit language identification to a specific subset

    main

    If you want to restrict predictions to a smaller set of languages, you can use the CustomLID class. This avoids the need to train a new model. There are two modes for limiting labels:

    1. mode='before' (Default): Limits the labels before the softmax operation. This ensures the predicted probabilities are normalized specifically to your chosen subset.
    2. mode='after': Limits the labels after the softmax operation. This means the probabilities are calculated relative to the entire 2000+ language set and then filtered to your subset.

    Note: Ensure your target labels match the format in model.labels (e.g., __label__eng_Latn).

    from huggingface_hub import hf_hub_download
# Note: CustomLID implementation is provided in the project assets/inference/customlid.py
from customlid import CustomLID 

model_path = hf_hub_download(repo_id="cis-lmu/glotlid", filename="model.bin")

# Define your subset
limited_languages = ['__label__eng_Latn', '__label__arb_Arab', '__label__rus_Cyrl']

# Initialize with 'before' mode for normalized probabilities within the subset
model = CustomLID(model_path, languages=limited_languages, mode='before')

# Predict top 3 from the subset
labels, probs = model.predict("Hello, world!", 3)

  3. Understand GlotLID V3 label changes and macro language handling

    main

    GlotLID V3 uses more exclusive labels compared to previous versions. The primary goal was to eliminate macro languages in favor of individual language labels, though some exceptions remain:

    • Merged Labels: pes_Arab (Dari) and prs_Arab (Iranian Persian) are merged under the macro label fas_Arab.
    • Sardinian Labels: srd_Latn represents sro_Latn and src_Latn, but does not include sdc_Latn, which has its own separate label.
    • Macro Language Exceptions: kpe_Latn and doi_Deva are supported as macro languages without individual language labels.
    • Renamed Labels: tgl_Latn has been changed to fil_Latn.
    • Deleted Macro Languages: Labels like zho_Hani, aze_Latn, and est_Latn were removed because the model now supports enough constituent individual languages to represent them.

    When interpreting model outputs, be aware that these labels follow the Glottolog tree and specific linguistic mapping decisions.

  4. How to use GlotLID for language identification

    main

    GlotLID is designed for language identification with support for over 2000 labels. The typical workflow involves:

    1. Installing dependencies (huggingface_hub and fasttext-numpy2-wheel).
    2. Downloading the model using huggingface_hub.
    3. Loading the model into Python.
    4. Calling the prediction function on text.

    Best Practices & Limitations:

    • Text Length: The model is primarily trained on longer sentences. Avoid using it on very short sentences, as accuracy may decrease.
    • Confidence Thresholds: Confidence thresholds vary by language due to linguistic proximity (e.g., a 0.6 confidence might be high for very similar languages like dyu and bam).
    • Language Subsets: If you know your input text is limited to a specific subset of languages, you can limit the prediction to those languages to improve efficiency and accuracy.

  5. Load the GlotLID model

    main

    You can download the model from Hugging Face and load it using fasttext. By default, model.bin points to the latest version (currently V3). If you need a specific version, you can specify model_v1.bin, model_v2.bin, or model_v3.bin.

    import fasttext
from huggingface_hub import hf_hub_download

# Download model and get the path
# cache_dir can be specified to control where the model is stored
model_path = hf_hub_download(repo_id="cis-lmu/glotlid", filename="model.bin", cache_dir=None)

# Load the model
model = fasttext.load_model(model_path)

  6. Install GlotLID dependencies

    main

    To use GlotLID, you need to install huggingface_hub and a compatible version of fasttext. For Python versions $\ge$ 3.10, it is recommended to install fasttext-numpy2-wheel instead of the standard fasttext package to ensure compatibility with newer NumPy versions.

    # For Python $\ge$ 3.10
pip install fasttext-numpy2-wheel
# For older Python versions
# pip install fasttext

pip install huggingface_hub

  7. Troubleshoot incorrect language predictions

    main

    If GlotLID predicts incorrect tags for normal, long text, consider the following:

    1. Script Verification: Use GlotScript to verify if the predicted lang_script actually exists in the sentence. If the script is not supported by the model, you may need to implement a fallback function that returns 'und_mainscript'.
    2. Sentence Length: Ensure the text is not too short, as the model is optimized for longer sentences.
    3. False Positives: Note that the False Positive Rate (FPR) for high-resource languages is higher than for low-resource languages, though it remains lower than models that only recognize high-resource languages.

  8. Predict language labels with GlotLID

    main

    Once the model is loaded, you can predict language labels for input text. The prediction returns a tuple containing the predicted labels (prefixed with __label__) and an array of corresponding confidence scores.

    Example Output Formats:

    • Single prediction: (('__label__eng_Latn',), array([0.99850202]))
    • Multiple predictions: (('__label__eng_Latn', '__label__sun_Latn', '__label__ind_Latn'), array([9.98502016e-01, 7.95848144e-04, 3.11827025e-04]))

  9. List supported language labels

    main

    To see all supported language-script labels (e.g., __label__eng_Latn), access the labels attribute of the loaded model.

    # Returns a list of all supported labels
print(model.labels)

  10. Get sentence vectors

    main

    GlotLID allows you to extract sentence embeddings (vectors) for a given input string using the get_sentence_vector() method.

    # Get sentence vector of input sentence
embedding = model.get_sentence_vector(sent)

  11. Perform Language Identification

    main

    Use the model.predict() method to identify the language of a given text. You can request the top $k$ predictions by passing an integer as the second argument.

    # Single prediction
# Returns: (('__label__lang_Script',), array([probability]))
model.predict("Hello, world!")

# Top k predictions (e.g., k=3)
model.predict("Hello, world!", 3)

  12. Supported labels and performance for GlotLID V3

    main

    GlotLID V3 supports a wide range of language labels, often following the format language_script (e.g., eng_Latn, bzh_Latn, bzi_Thai). The performance metrics provided in the documentation include precision, recall, and F1 scores for various language labels.

    Note: The provided segment contains a partial list of labels from index 268 to 357. Each entry includes the label name and associated performance statistics.

    |  268 | byr_Latn |   1        |          1        |       1        |           0           |       21525 |              0.999909 |
|  269 | byv_Latn |   0.995025 |          1        |       0.990099 |           0        |        2042 |              0.993148 |
|  270 | byx_Latn |   1        |          1        |       1        |           0           |       20598 |              0.999779 |
|  271 | bzd_Latn |   0.995484 | 0.998993 |       0.992    |           5.30467e-07 |       29765 |              0.998612 |
|  272 | bzh_Latn |   0.9995   |          0.999001 |       1        |           5.30467e-07 |       28883 |              0.99899  |
|  273 | bzi_Thai |   1        |          1        |       1        |           0           |       12964 |              0.999872 |
|  274 | bzj_Latn |   0.99349  |          0.994985 |       0.992    |           2.65234e-06 |      127628 |              0.99656  |
|  275 | caa_Latn |   1        |          1        |       1        |           0           |       20968 |              0.999806 |
|  276 | cab_Latn |   0.997996 |          1        |       0.996    |           0           |      236766 |              0.998776 |
|  277 | cac_Latn |   1        |          1        |       1        |           0           |       64274 |              0.999767 |
|  278 | caf_Latn |   0.996509 |          0.99403  |       0.999    |           3.1828e-06  |       20945 |              0.997042 |
|  279 | cag_Latn |   0.998999 |          1        |       0.998    |           0           |       14201 |              0.997919 |
|  280 | cak_Latn |   0.996997 |          0.997996 |       0.996    |           1.06093e-06 |       281166 |              0.99768  |
|  281 | cao_Latn |   1        |          1        |       1        |           0           |       20844 |              0.999055 |
|  282 | cap_Latn |   1        |          1        |       1        |           0           |       33919 |              0.9997   |
|  283 | caq_Latn |   0.998999 |          1        |       0.998    |           0           |       37222 |              0.998567 |
|  284 | car_Latn |   0.9995   |          1        |       0.999    |           0           |       22338 |              0.99844  |
|  285 | cas_Latn |   0.999364 |          1        |       0.998729 |           0           |       7878 |              0.999033 |
|  286 | cat_Latn |   0.976972 |          0.957733 |       0.997    |           2.33406e-05 |     2103768 |              0.995484 |
|  287 | cav_Latn |   1        |          1        |       1        |           0           |       20643 |              0.999487 |
|  288 | cax_Latn |   1        |          1        |       1        |           0           |       33951 |              0.999237 |
|  289 | cbc_Latn |   1        |          1        |       1        |           0           |       20629 |              0.999609 |
|  290 | cbi_Latn |   1        |          1        |       1        |           0           |       12913 |              0.999375 |
|  291 | cbk_Latn |   0.987393 |          0.995931 |       0.979    |           2.12187e-06 |      120570 |              0.988911 |
|  292 | cbr_Latn |   1        |          1        |       1        |           0           |       28570 |              0.999785 |
|  293 | cbs_Latn |   0.9995   |          1        |       0.999    |           0           |       32132 |              0.998788 |
|  294 | cbt_Latn |   0.9995   |          0.999001 |       1        |           5.30467e-07 |       12804 |              0.99954  |
|  295 | cbu_Latn |   1        |          1        |       1        |           0           |       19980 |              0.999413 |
|  296 | cbv_Latn |   1        |          1        |       1        |           0           |       32889 |              0.999813 |
|  297 | cce_Latn |   0.991    |          0.991    |       0.991    |           4.77421e-06 |      118765 |              0.994289 |
|  298 | cco_Latn |   1        |          1        |       1        |           0           |       20811 |              0.999953 |
|  299 | ccp_Latn |   0.998107 |          0.998737 |       0.997478 |           5.30409e-07 |        7957 |              0.995679 |
|  300 | cdf_Latn |   0.99937  |          1        |       0.998741 |           0           |       7957 |              0.997739 |
|  301 | ceb_Latn |   0.989578 |          0.982266 |       0.997    |           9.54841e-06 |     2266139 |              0.997099 |
|  302 | ceg_Latn |   1        |          1        |       1        |           0           |       15911 |              0.999097 |
|  303 | cek_Latn |   0.997003 |          0.996008 |       0.998    |           2.12187e-06 |       51879 |              0.9984   |
|  304 | ces_Latn |   0.991546 |          0.986152 |       0.997    |           7.42654e-06 |     1863677 |              0.996721 |
|  305 | cfm_Latn |   0.993497 |          0.993994 |       0.993    |           3.1828e-06  |       94985 |              0.993514 |
|  306 | cgc_Latn |   0.997998 |          0.998998 |       0.997    |           5.30467e-07 |       21016 |              0.996486 |
|  307 | cgg_Latn |   0.990495 |          0.990991 |       0.99     |           4.77421e-06 |       40675 |              0.985496 |
|  308 | cha_Latn |   0.998499 |          0.998999 |       0.998    |           5.30467e-07 |       22719 |              0.995706 |
|  309 | chd_Latn |   0.9995   |          1        |       0.999    |           0           |       21221 |              0.997678 |
|  310 | che_Cyrl |   0.998002 |          0.997006 |       0.999    |           1.5914e-06  |       66169 |              0.996684 |
|  311 | chf_Latn |   0.9995   |          1        |       0.999    |           0           |       21189 |              0.998989 |
|  312 | chj_Latn |   1        |          1        |       1        |           0           |        5043 |              0.999838 |
|  313 | chk_Latn |   0.998999 |          1        |       0.998    |           0           |      375518 |              0.99865  |
|  314 | cho_Latn |   1        |          1        |       1        |           0           |        1213 |              0.997342 |
|  315 | chq_Latn |   1        |          1        |       1        |           0           |       12900 |              0.999886 |
|  316 | chr_Cher |   0.998759 |          0.998759 |       0.998759 |           5.30413e-07 |       13015 |              0.999483 |
|  317 | chr_Latn |   1        |          1        |       1        |           0           |        7963 |              0.998625 |
|  318 | chu_Cyrl |   0.957949 |          0.983158 |       0.934    |           8.48748e-06 |       30487 |              0.924445 |
|  319 | chv_Cyrl |   0.993497 |          0.993994 |       0.993    |           3.1828e-06  |       56256 |              0.994409 |
|  320 | chw_Latn |   0.992511 |          0.991027 |       0.994    |           4.77421e-06 |      117497 |              0.992601 |
|  321 | chz_Latn |   1        |          1        |       1        |           0           |       20973 |              0.999928 |
|  322 | cjk_Latn |   0.999    |          0.999    |       0.999    |           5.30467e-07 |      193070 |              0.995815 |
|  323 | cjo_Latn |   0.996004 |          0.99501  |       0.997    |           2.65234e-06 |       20445 |              0.998048 |
|  324 | cjp_Latn |   1        |          1        |       1        |           0           |       15904 |              0.999748 |
|  325 | cjs_Cyrl |   0.993548 |          1        |       0.987179 |           0           |        1553 |              0.99233  |
|  326 | cjv_Latn |   1        |          1        |       1        |           0           |       28558 |              0.999257 |
|  327 | ckb_Arab |   0.969072 |          0.951784 |       0.987    |           2.65234e-05 |      199107 |              0.983412 |
|  328 | ckm_Latn |   0.887591 |          0.921212 |       0.856338 |           1.37874e-05 |        3538 |              0.924876 |
|  329 | cko_Latn |   0.9995   |          0.999001 |       1        |           5.30467e-07 |       20851 |              0.999059 |
|  330 | ckt_Cyrl |   0.990403 |          1        |       0.980989 |           0           |       2670 |              0.991611 |
|  331 | cle_Latn |   1        |          1        |       1        |           0           |       20976 |              0.99992  |
|  332 | clu_Latn |   0.998    |          0.998    |       0.998    |           1.06093e-06 |       12877 |              0.996239 |
|  333 | cly_Latn |   0.895258 |          0.934712 |       0.859    |           3.1828e-05  |       15898 |              0.946527 |
|  334 | cme_Latn |   0.9995   |          0.999001 |       1        |           5.30467e-07 |       20867 |              0.999753 |
|  335 | cmn_Hani |   0.826432 |          0.71028  |       0.988    |           0.000213778 |     1138828 |              0.978307 |
|  336 | cmo_Khmr |   1        |          1        |       1        |           0           |       13015 |              0.999941 |
|  337 | cmo_Latn |   0.9995   |          0.999001 |       1        |           5.30467e-07 |       21164 |              0.999544 |
|  338 | cmr_Latn |   1        |          1        |       1        |           0           |        7870 |              0.999751 |
|  339 | cnh_Latn |   0.995511 |          0.993035 |       0.998    |           3.71327e-06 |       419551 |              0.997566 |
|  340 | cni_Latn |   0.996502 |          0.996004 |       0.997    |           2.12187e-06 |       38457 |              0.99569  |
|  341 | cnk_Latn |   0.998999 |          1        |       0.998    |           0           |       13021 |              0.995218 |
|  342 | cnl_Latn |   1        |          1        |       1        |           0           |       20946 |              0.999529 |
|  343 | cnr_Latn |   0.762238 |          0.893443 |       0.664634 |           6.89302e-06 |        1631 |              0.815741 |
|  344 | cnt_Latn |   1        |          1        |       1        |           0           |       20951 |              0.999898 |
|  345 | cnw_Latn |   0.99937  |          1        |       0.998741 |           0           |        7930 |              0.996223 |
|  346 | coe_Latn |   1        |          1        |       1        |           0           |       20452 |              0.999813 |
|  347 | cof_Latn |   0.9995   |          1        |       0.999    |           0           |       19901 |              0.998039 |
|  348 | cok_Latn |   0.996505 |          0.995015 |       0.998    |           2.65234e-06 |       33636 |              0.994007 |
|  349 | con_Latn |   1        |          1        |       1        |           0           |       20819 |              0.998934 |
|  350 | cop_Copt |   1        |          1        |       1        |           0           |       32357 |              0.99987  |
|  351 | cor_Latn |   0.998    |          0.998    |       0.998    |           1.06093e-06 |       48753 |              0.997631 |
|  352 | cos_Latn |   0.980769 |          0.992828 |       0.969    |           3.77421e-06 |       11087 |              0.971323 |
|  353 | cot_Latn |   1        |          1        |       1        |           0           |       20863 |              0.998614 |
|  354 | cou_Latn |   1        |          1        |       1        |           0           |        7936 |              0.999426 |
|  355 | cpa_Latn |   1        |          1        |       1        |           0           |       20974 |              0.99989  |
|  356 | cpb_Latn |   0.994    |          0.994    |       0.994    |           3.1828e-06  |       20970 |              0.992854 |
|  357 | cpc_Latn |   0.997003 |          0.996008 |       0.998    |           2.12187e-06 |       20971 |              0.996957 |