DOCSEARCHbyAlgolia
Submit Docs

PyCantonese

repository·main·Indexed 19 days ago

https://github.com/jacksonllee/pycantonese

A specialized Python library for Cantonese linguistics and NLP. Version 5.0.0 provides tools for word segmentation, part-of-speech (POS) tagging, Jyutping romanization, and corpus access. It integrates data from sources such as HKCanCor, CantoMap, Common Voice, CTCPC, and rime-cantonese to power its linguistic models and dictionaries.

Tokens
22.2K
Snippets
82
Records
105
Agent score
62%

What's inside pycantonese

  1. Overview of PyCantonese features

    main

    PyCantonese is a Python library designed for Cantonese linguistics and natural language processing (NLP).

    Core features include:

    • Corpus Access: Accessing and searching corpus data.
    • Romanization: Parsing and conversion tools for Jyutping romanization.
    • Text Processing: Parsing Cantonese text, word segmentation, and part-of-speech (POS) tagging.
    • Linguistic Tools: Stop words and other NLP utilities.

    Note on Dependencies: Since version 4.0.0 (March 2026), PyCantonese depends on Rustling for efficient CHAT data handling, word segmentation, and part-of-speech tagging.

  2. Understand the Hong Kong Cantonese Corpus (HKCanCor) in PyCantonese

    main

    The Hong Kong Cantonese Corpus (HKCanCor) is a collection of conversational Cantonese data originally compiled by Kang Kwong Luke. In PyCantonese, this corpus is used for:

    1. Jyutping data: Providing romanization information.
    2. Model training: Powering word segmentation and part-of-speech (POS) tagging models.

    Note that the version of HKCanCor included in PyCantonese has been modified from its original source to use the CHAT format (developed for CHILDES). The trained models are stored as zst-compressed FlatBuffers binaries.

    If you use this corpus in your research, please cite:

    K. K. Luke and May L.Y. Wong (2015) The Hong Kong Cantonese Corpus: Design and Uses. Journal of Chinese Linguistics Monograph Series, 25, 312–333.

  3. What is rime-cantonese data in PyCantonese

    main

    The rime-cantonese data is a component incorporated into PyCantonese to support two primary NLP tasks:

    1. Word Segmentation: Providing a dictionary for segmenting Cantonese text.
    2. Characters-to-Jyutping Conversion: Mapping characters and words to their Jyutping pronunciations.

    The data is stored as JSON files and zst-compressed FlatBuffers binaries. It is licensed under CC BY 4.0.

  4. Perform complex searches

    main
    The pycantonese.CHAT.search method is designed to target a single match word per query. If your search requirements involve relationships between multiple words or more complex patterns, you must implement custom logic to iterate through the corpus data and filter based on the available annotations.

  5. How G2P conversion works internally

    main

    The pycantonese.g2p function is a pipeline that composes two primary steps:

    1. pycantonese.characters_to_jyutping: Converts characters to Jyutping romanization (including word segmentation).
    2. pycantonese.jyutping_to_ipa: Converts the Jyutping output into IPA symbols.

    For lower-level control, such as accessing intermediate Jyutping or managing tones manually, you can call these two functions directly.

  6. How PyCantonese benchmarks work

    main
    Benchmark scripts are designed for process isolation. When run in normal mode (without --json), the script invokes itself as a subprocess with PYTHONPATH configured so that the _baseline/ directory is found first. This prevents import conflicts between the current version and the v3.4.0 baseline.

  7. How the CHAT format and Rustling integration work

    main

    PyCantonese adopts the CHAT corpus format, which is widely used for linguistic annotations in Cantonese research.

    To handle parsing, PyCantonese utilizes the Rustling library. The integration includes Cantonese-specific enhancements for:

    • Jyutping romanization
    • Chinese characters
    • General corpus search functionality

    If you are processing data outside of PyCantonese, you can use Rustling to parse your files, perform necessary processing, and then instantiate a pycantonese.CHAT object using the CHAT.from_strs method.

  8. The CHAT data format in HKCanCor

    main

    PyCantonese uses the CHAT format for its HKCanCor data files (rendered as .cha files). This format is designed for conversational data and consists of two main components:

    • Headers: Lines beginning with @ containing metadata (e.g., @UTF8, @Begin, @Languages, @Participants, @ID, @Date, @Tape Number).
    • Transcriptions: Lines beginning with * for the utterance, accompanied by annotations in the %mor (morphology) tiers.

    Key Format Differences from Original Source

    When working with these files, be aware of how the data has been transformed from the original HKCanCor source:

    FeatureCHAT Format Implementation
    LanguagesSpecified in @Languages and @ID headers (typically yue, eng).
    Participant CodesExpanded to three characters (e.g., XXA, XXB) to comply with CHAT requirements.
    Participant AgeIf an age range was provided (e.g., 25-30), the lower number is used (e.g., 25).
    Word RepresentationChinese characters are on the utterance line (*). Part-of-speech tags and Jyutping romanization are moved to the %mor tier (e.g., e|wai3).
    PunctuationConverted to ASCII. Chinese-style delimiters (parentheses, brackets) are collapsed into double quotes ("). Colons are removed to prevent parsing errors.

    Example CHAT File Structure

    @UTF8
@Begin
@Languages:	yue , eng
@Participants:	XXA A Adult , XXB B Adult
@ID:	yue , eng|HKCanCor|XXA|34;|female|||Adult||origin:HK|
@ID:	yue , eng|HKCanCor|XXB|37;|female|||Adult||origin:HK|
@Date:	30-APR-1997
@Tape Number:	001
*XXA:	喂 遲 啲 去 唔 去 旅行 啊 ?
%mor:	e|wai3 a|ci4 u|di1 v|heoi3 d|m4 v|heoi3 vn|leoi5hang4 y|aa3	?
*XXA:	你 老公 有冇 平 機票 啊 ?
%mor:	r|nei5 n|lou5gung1 v1|jau5mou5 a|peng4 n|gei1piu3 y|aa3 ?
*XXB:	平 機票 要 淡季 先 有得 平 𡃉 喎 .
%mor:	a|peng4 n|gei1piu3 vu|jiu3 an|daam6gwai3 d|sin1 vu|jau5dak1	a|peng4 y|gaa3 y|wo3 .

  9. EAF-to-CHAT conversion logic

    main

    The download.py script performs several transformations to convert ELAN .eaf files into the CHAT format used in the extracted/ directory:

    1. Tier pairing: Matches <prefix>-word and <prefix>-jyutping tiers by their prefix (where each prefix represents a speaker).
    2. Speaker codes: Converts tier prefixes into 3-character CHAT participant codes padded with leading Xs (e.g., E becomes XXE, E1 becomes XE1).
    3. Time alignment: Aligns word and Jyutping annotations using (start_time, end_time) timestamps and sorts all speaker utterances chronologically.
    4. Jyutping normalization:
      • # (pause markers): Splits a single annotation into multiple utterances.
      • Pipe-separated alternatives (e.g., gam3|gam2): Keeps only the first variant.
      • &-prefixed particles (e.g., &le1): Strips the & to keep the Jyutping (le1).
      • *N tone-change notation (e.g., gin3dou3*2): Replaces the tone digit before the * with the digit after it (gin3dou2).
      • hao4_(Mandarin): Replaces with the word 好 and Jyutping hou2.

  10. Work with Corpus Tokens and Utterances

    main

    When working with corpus data, PyCantonese uses two primary data structures:

    • pycantonese.corpus.Token: Represents an individual token within a corpus.
    • pycantonese.corpus.Utterance: Represents a complete utterance (a sequence of tokens) within a corpus.

  11. Understand the rime-cantonese data files

    main

    PyCantonese utilizes two main processed JSON files derived from the rime-cantonese sources:

    • chars_to_jyutping.json: Contains mappings of words and characters to Jyutping. This is used for character-to-Jyutping conversion and serves as the word dictionary for supervised word segmentation training.
    • phrase_fragments.json: Contains an array of multi-word phrase fragments. This is used exclusively for unsupervised EM (Expectation-Maximization) refinement of the word segmenter. These are intentionally excluded from supervised training to prevent corruption of word boundary labels, as they are stored as unsegmented strings.

  12. Understand the CantoMap extracted CHAT format

    main

    The extracted/ directory contains CHAT (.cha) files derived from CantoMap's ELAN (.eaf) files.

    File Naming: Filenames are derived from the source path by replacing directory separators with __. For example: ConversationData/Subjects-1_2/160725_009_1_2_A1.eaf becomes Subjects-1_2__160725_009_1_2_A1.cha.

    File Structure: Each .cha file contains utterances with two primary tiers:

    1. Main tier: The Cantonese text.
    2. %mor tier: The morphology tier containing part-of-speech (POS) tags assigned using PyCantonese's POS tagger with the HKCanCor tagset (formatted as tag|jyutping).

    Example format:

    *XXE:    好 咁樣 .
%mor:    a|hou2 d|gam2joeng2 .
*XXG:    起點 喺 引依湖 .
%mor:    n|hei2dim2 v|hai2 n|jan5ji1wu4 .