epitran

Pre- and post-processors use a grammar of context-sensitive string rewrite rules. These files support symbol definitions, rewrite rules, comments, and blank lines.

Symbol Definitions

Define reusable substrings using the syntax ::symbol_name:: = regex_pattern. Symbols must be defined before they are used.

::vowels:: = a|e|i|o|u

Rewrite Rule Syntax

Rules follow the pattern: target -> replacement / context.

• ->: "is rewritten as"
• /: "in the context of"
• _: Indicates the position of the rewrite within the context.
• #: Represents a word boundary (beginning or end of string).
• 0: Represents the empty string (used for insertions or deletions).

Examples

• Contextual rewrite: e -> ə / _ # (Rewrite /e/ to /ə/ at the end of a word).
• Deletion: ə -> 0 / k _ l (Delete /ə/ between /k/ and /l/).
• Context-free: ch -> x / _ (Rewrite ch to x regardless of context).
• Regex integration: Since Epitran uses the regex package, you can use standard regular expression notation in rules: c -> s / _ [ie] or c -> s / _ (i|e).
• Using Symbols: u -> w / _ (::vowels::) (Rewrite /u/ to /w/ before a defined vowel symbol).

Metathesis (Swapping)

To swap two characters (e.g., "AB" to "BA"), use named capture groups sw1 and sw2 with the ?P<name> syntax:

(?P<sw1>[char_set])(?P<sw2>.) -> 0 / _

Rule Execution

Rules are applied in the order they appear in the file. Earlier rules can "feed" (provide input for) or "bleed" (remove input for) later rules.

The Backoff class allows for graceful fallback between different language-script modes. It processes text on a token-by-token basis. If a token contains mixed scripts that cannot be handled by any provided mode, it returns an empty string.

Constructor: Backoff(lang_script_codes, cedict_file=None)

• lang_script_codes: A list of language-script codes (e.g., ['hin-Deva', 'eng-Latn']).
• cedict_file: Path to CC-CEDict (if needed for Chinese modes).

Limitations:

• Does not support parameterized pre/post-processors.
• Does not support non-standard ligatures.
• Does not support punctuation normalization.

Public Methods:

• transliterate(text): Returns a Unicode string of IPA phonemes.
• trans_list(text): Returns a list of IPA Unicode strings (one per phoneme).
• xsampa_list(text): Returns a list of X-SAMPA (ASCII) strings (one per phoneme).

from epitran.backoff import Backoff

# Fallback order: Hindi -> English -> Mandarin
backoff = Backoff(['hin-Deva', 'eng-Latn', 'cmn-Hans'], cedict_file='cedict_1_0_ts_utf-8_mdbg.txt')

print(backoff.transliterate('हिन्दी'))  # 'ɦindiː'
print(backoff.trans_list('हिन्दी'))     # ['ɦ', 'i', 'n', 'd', 'iː']
print(backoff.xsampa_list('हिन्दी'))    # ['h\', 'i', 'n', 'd', 'i:']

The DictFirst class provides an alternative to Backoff. It checks if an input token exists in a provided dictionary for 'Language A'. If it does, it uses Language A's transliteration; otherwise, it falls back to 'Language B'.

Constructor: DictFirst(lang_a_code, lang_b_code, dictionary_file_path)

• lang_a_code: The primary language-script code.
• lang_b_code: The fallback language-script code.
• dictionary_file_path: Path to a UTF-8 encoded text file containing one word per line.

Public Method:

• transliterate(token): Returns the transliteration for Language A if the token is in the dictionary, otherwise returns the Language B transliteration.

import dictfirst
# If word is in sample-dict.txt, use tpi-Latn; else use eng-Latn
df = dictfirst.DictFirst('tpi-Latn', 'eng-Latn', '../sample-dict.txt')

print(df.transliterate('pela')) # 'pela' (if in dict)
print(df.transliterate('pelo')) # 'pɛlow' (fallback)

The epitran.vector module provides the VectorsWithIPASpace class, which converts words into a structured representation containing phonetic forms and phonological feature vectors. This is useful for tasks requiring machine-readable representations of IPA features.

Constructor Arguments

• code: The language-script code for the language being processed.
• spaces: A list of codes for the punctuation/symbol/IPA spaces where characters/segments are expected to reside.

The word_to_segs method

VectorsWithIPASpace.word_to_segs(word, normpunc=False)

• word: A Unicode string to process.
• normpunc: If set to True, punctuation in the input word is normalized to its ASCII equivalents.

Returns a list of tuples representing the segments of the word.

import epitran.vector
# Initialize with language-script code and the relevant IPA space
vwis = epitran.vector.VectorsWithIPASpace('uzb-Latn', ['uzb-Latn'])
# Convert word to segments
segments = vwis.word_to_segs('darë')

The word_to_tuples method returns a list of tuples providing a granular breakdown of a word's phonetic structure.

Note: This method is not implemented for all language-script pairs. Also, if pre-processors are active, the orthographic_form returned may not match the original input string, as pre-processors may modify the text to facilitate mapping.

Tuple Structure: Each tuple in the list follows this schema:

• character_category (String): Unicode General Category (e.g., 'L' for letters).
• is_upper (Integer): Case indicator.
• orthographic_form (Unicode String): The processed orthographic character.
• phonetic_form (Unicode String): The resulting IPA character.
• segments (List of Tuples): A list of sub-segments, where each sub-segment is (segment_string, vector_of_integers).

import epitran
epi = epitran.Epitran('tur-Latn')
# Returns a list of detailed phonetic tuples
segments = epi.word_to_tuples('Düğün')

The Epitran class is the primary interface for transliterating orthographic text to IPA. To initialize it, you must provide an ISO 639-3 language code followed by a hyphen and a four-letter script code (e.g., 'uig-Arab' for Uyghur in Perso-Arabic).

Constructor Arguments:

• code (required): ISO 639-3 language code + hyphen + 4-letter script code.
• preproc (bool): Enables pre-processors. Defaults to True.
• postproc (bool): Enables post-processors. Defaults to True.
• ligatures (bool): Enables non-standard IPA ligatures like "ʤ" and "ʨ". Defaults to False.
• cedict_file (str): Path to a CC-CEDict dictionary file (required for Mandarin Chinese cmn-Hans or cmn-Hant).
• tones (bool): Enables IPA tones (e.g., ˩˨˧˦˥). Defaults to False (removes tones).

import epitran
# Uyghur in Perso-Arabic script
epi = epitran.Epitran('uig-Arab')

# Mandarin Chinese with CC-CEDict
epi = epitran.Epitran('cmn-Hans', cedict_file='cedict_1_0_ts_utf-8_mdbg.txt')