DOCSEARCHbyAlgolia
Submit Docs

langdetect Documentation

repository·master·Indexed 23 days ago

https://github.com/mimino666/langdetect

A Python port of Google's Java language-detection library for identifying the language of text strings and retrieving language probabilities. Supports Python 2.7 and 3.4+, providing ISO 639-1 codes for 55 languages. Includes functions like detect() and detect_langs(), and allows for deterministic results via DetectorFactory.seed.

Tokens
665
Snippets
4
Records
6
Agent score
34%

What's inside langdetect

  1. Enforce deterministic results

    master

    The language detection algorithm is non-deterministic. For short or ambiguous text, results may vary between runs. To ensure consistent, reproducible results, set the DetectorFactory.seed before performing any detection.

    from langdetect import DetectorFactory
DetectorFactory.seed = 0

  3. Add a new language via language profiles

    master

    To add a new language, you must create a language profile. This is typically done using the langdetect.jar tool by generating profiles from Wikipedia abstract database files or plain text files.

    Generate profile from Wikipedia abstract databases

    Use the --genprofile flag with the -d option to specify the directory containing the abstract databases. The tool supports gzip compressed files.

    Note for Chinese databases: Wikipedia filenames like zhwiki-(version)-abstract-zh-cn.xml must be renamed to zh-cnwiki-(version)-abstract.xml (and similarly for zh-tw) for the tool to process them correctly.

    Command: java -jar langdetect.jar --genprofile -d [directory path] [language codes]

    Generate profile from plain text

    Use the --genprofile-text flag with the -l option to specify the language code and the path to the text file.

    Command: java -jar langdetect.jar --genprofile-text -l [language code] [text file path]

  4. Detect the language of a text

    master

    Use the detect() function to return the ISO 639-1 language code of the provided text.

    >>> from langdetect import detect
>>> detect("War doesn't show who's right, just who's left.")
'en'
>>> detect("Ein, zwei, drei, vier")
'de'

  5. Get language probabilities with detect_langs

    master

    Use detect_langs() to find out the probabilities for the top candidate languages. This returns a list of language codes paired with their respective probabilities.

    >>> from langdetect import detect_langs
>>> detect_langs("Otec matka syn.")
[sk:0.572770823327, pl:0.292872522702, cs:0.134356653968]

  6. Supported ISO 639-1 language codes

    master

    The library supports 55 languages out of the box using the following ISO 639-1 codes:

    af, ar, bg, bn, ca, cs, cy, da, de, el, en, es, et, fa, fi, fr, gu, he, hi, hr, hu, id, it, ja, kn, ko, lt, lv, mk, ml, mr, ne, nl, no, pa, pl, pt, ro, ru, sk, sl, so, sq, sv, sw, ta, te, th, tl, tr, uk, ur, vi, zh-cn, zh-tw