langcodes: IETF Language Tag Tools

To use methods related to language names in different languages or demographic data (like speaking/writing populations), you must install the optional language_data package.

You can install it via pip:

pip install langcodes[data]

If you are managing dependencies in a setup.py file, use langcodes[data] as the requirement.

When using parse_tag(), the returned tuples use the following subtag type names to categorize components:

• language: The primary language code.
• extlang: Extended language codes (used for macrolanguages like Chinese or Arabic).
• script: The writing system (e.g., Latn, Hant).
• territory: The geographic region (e.g., US, 419).
• variant: A specific variation of the language.
• extension: BCP 47 extensions (e.g., u-co-backward).
• private: Private-use subtags starting with x-.
• grandfathered: Legacy tags from RFC 3066.

In the context of language distance calculation, a TagTriple is a specific type alias used to represent a fully qualified language specification. It is defined as a Tuple[str, str, str] consisting of:

1. Language: The language code (e.g., "en", "es", "ar").
2. Script: The script code (e.g., "Latn", "Cyrl", "Hant").
3. Territory: The territory/region code (e.g., "US", "GB", "419").

These triples are typically generated by 'maximizing' a standard language tag to ensure all three components are present for the distance algorithm to process.

The langcodes.find() function performs the reverse of .display_name(), converting a natural language name (like "French") into a Language object (like fr).

• Language Support: Works with hundreds of names used on Wiktionary across many languages.
• Ambiguity: If a name is ambiguous (e.g., "Tonga" could refer to to or tog), langcodes.find() may raise a LookupError. To resolve this, provide the language= parameter to specify the language in which the name is written.
• Limitation: It currently ignores parenthetical expressions (e.g., find('English (Canada)') returns the base en language).

>>> import langcodes
>>> langcodes.find('french')
Language.make(language='fr')

>>> # Resolving ambiguity
>>> langcodes.find('tonga', language='id')
Language.make(language='to')

The tag_distance function returns a numeric value from 0 to 134 representing the distance between a desired language and a supported language. This distance is based on CLDR data and indicates how similar or intelligible two languages are.

Distance Value Reference

See the tag_distance docstring for more detailed examples.

While BCP 47 prefers 2-letter codes, some applications require 3-letter (alpha3) codes. Use Language.to_alpha3() to retrieve them.

• By default, it returns the 'terminology' code.
• Passing variant='B' returns the 'bibliographic' code.
• The method always returns a 3-letter string.
• If a language code is unknown and has no alpha3 mapping, it raises a LookupError.

from langcodes import *

# Get default (terminology) alpha3 code
Language.get('fr').to_alpha3()    # 'fra'
Language.get('de').to_alpha3()    # 'deu'

# Get bibliographic alpha3 code
Language.get('fr').to_alpha3(variant='B') # 'fre'

# Handling errors for unknown codes
try:
    Language.get('un').to_alpha3()
except LookupError:
    print("Unknown language")

A language code is considered valid if every subtag (language, script, territory, and variants) is assigned a meaning by IANA.

• Use Language.get(tag).is_valid() to check a parsed object.
• Use the top-level tag_is_valid(tag) function for a more convenient check that returns False even for tags that fail to parse.

Note: und (undetermined) and private use codes (starting with x-) are considered valid. Non-ASCII characters in tags are considered invalid because they fail to parse.

from langcodes import *

# Checking validity of parsed objects
Language.get('ja').is_valid() # True
Language.get('jp').is_valid() # False

# Using the top-level convenience function
tag_is_valid('C')      # False
tag_is_valid('en-000') # False
tag_is_valid('und')    # True
tag_is_valid('x-other') # True

# Invalid due to non-ASCII characters
tag_is_valid('zh-普通话') # False

Using the optional language_data package, you can retrieve Unicode estimates for language demographics.

• .speaking_population(): Estimates the number of speakers. You can limit this to a specific territory by providing a territory code (e.g., es-BR). Script codes are ignored as they don't affect speaking population.
• .writing_population(): Estimates the number of people who write the language. This is often derived from literacy rates and may be an overestimate.

Both methods support territory-specific filtering.

>>> Language.get('es').speaking_population()
493528077

>>> Language.get('es-BR').speaking_population()
76218

>>> Language.get('zh-Hans').writing_population()
36863340