wiktextract

Each extracted word is represented as a dictionary. The primary keys available in a word entry include:

• word: The word form.
• pos: Part-of-speech (e.g., "noun", "verb", "adj"). See wiktextract.PARTS_OF_SPEECH for the full list.
• lang: Language name (e.g., "English").
• lang_code: Wiktionary language code (e.g., "en").
• senses: A list of dictionaries representing word senses.
• forms: A list of dictionaries for inflected or alternative forms (e.g., plural, comparative), containing form and tags keys.
• sounds: A list of dictionaries for pronunciation, hyphenation, and rhyming information.
• categories: List of non-disambiguated categories.
• topics: List of non-disambiguated topics.
• translations: Non-disambiguated translation entries.
• etymology_text: Cleaned etymology text.
• etymology_templates: List of templates and arguments from the etymology section.
• etymology_number: The etymology number (as a string) for words with multiple etymologies.
• descendants: List of descendant words.
• synonyms, antonyms, hypernyms, holonyms, meronyms, derived, related, coordinate_terms: Lists of non-disambiguated linkages.
• wikidata: Wikidata identifier.
• wiktionary: Wikipedia page title.
• head_templates: Part-of-speech specific head tags.
• inflection_templates: Conjugation and declension templates.

A word entry can contain multiple glosses under the senses key. Each sense is a dictionary containing:

• glosses: Cleaned gloss strings.
• raw_glosses: Less cleaned gloss strings (includes parenthesized tags/topics).
• tags: List of qualifiers (e.g., "archaic", "colloquial").
• categories / topics: Sense-disambiguated category and topic names.
• alt_of: List of words this sense is an alternative form of.
• form_of: List of words this sense is an inflected form of.
• translations: Sense-disambiguated translation entries.
• synonyms, antonyms, hypernyms, holonyms, meronyms, coordinate_terms, derived, related: Sense-disambiguated linkages.
• senseid: Textual identifiers for the sense.
• wikidata: List of QIDs (e.g., Q123).
• wikipedia: List of Wikipedia page titles.
• examples: List of usage examples. Each example dictionary contains text, and optionally ref, english (translation), type (example or quotation), roman, or note.

To extract data from a Wiktionary template, you typically expand the template into its HTML representation and then traverse the resulting nodes.

Key steps in the extraction pattern:

1. Expand the template: Use wxr.wtp.parse(wxr.wtp.node_to_wikitext(node), expand_all=True) to get the full HTML structure.
2. Find specific elements: Use methods like find_html or find_html_recursively to locate specific tags (e.g., <span> with a specific class like Jpan or tr).
3. Clean the content: Use clean_node(wxr, context, node) to extract the text content without the underlying markup.
4. Calculate offsets: Use utility functions like calculate_bold_offsets to track the positions of bolded text within the cleaned strings.

def extract_quote_template(
    wxr: WiktextractContext, sense: Sense, t_node: TemplateNode
) -> str:
    # ...
    expanded_node = wxr.wtp.parse(
        wxr.wtp.node_to_wikitext(node), expand_all=True
    )
    for span_tag in expanded_node.find_html_recursively("span"):
        span_class = span_tag.attrs.get("class", "")
        if "cited-source" == span_class:
            example_data.ref = clean_node(wxr, None, span_tag)
    # ...
    return example.ref

When building a new Wiktionary extractor, you can implement logic to extract audio URLs and hyphenation data by following these steps:

1. Define Pydantic models: Extend your WordEntry model (inheriting from EnglishBaseModel) to include new fields like sounds: list[Sound] and hyphenations: list[Hyphenation].
2. Implement extraction logic: Create functions that traverse the LevelNode and identify specific TemplateNode types (e.g., audio or hyphenation).
3. Handle template expansion: Use wxr.wtp.parse(..., expand_all=True) on the template's wikitext to access the expanded HTML content, which is often necessary to find metadata like pronunciation labels or specific language spans.
4. Register the section parser: Update the parse_section function in your page.py to check for specific section titles (e.g., "Pronunciation") and call your new extraction functions.

Note: For audio, use set_sound_file_url_fields to automatically populate URL fields like mp3_url and ogg_url from a filename.

# Example Pydantic models
class Sound(EnglishBaseModel):
    audio: str = ""
    ogg_url: str = ""
    mp3_url: str = ""
    tags: list[str] = []

class Hyphenation(EnglishBaseModel):
    parts: list[str] = []

class WordEntry(EnglishBaseModel):
    sounds: list[Sound] = []
    hyphenations: list[Hyphenation] = []

# Example section registration in page.py
def parse_section(wxr, page_data, base_data, level_node):
    title_text = clean_node(wxr, None, level_node.largs)
    if title_text == "Pronunciation":
        extract_sound_section(wxr, base_data, level_node)