DOCSEARCHbyAlgolia
Submit Docs

Chinese Xinhua Database

repository·master·Indexed 9 days ago

https://github.com/pwxcoo/chinese-xinhua

A collection of JSON-based databases containing Chinese linguistic data, including 31,648 idioms (成语), 264,334 words (词语), 16,142 characters (汉字), and 14,032 xiehouyu (歇后语/riddles). The repository provides structured data files and Python scripts for scraping, cleaning, and processing these datasets for use in linguistic applications.

Tokens
1.5K
Snippets
6
Records
9
Agent score
94%

What's inside Chinese Xinhua

  1. Overview of chinese-xinhua database

    master

    The chinese-xinhua repository provides a comprehensive database of Chinese linguistic data. It is designed for developers who need access to idioms, words, characters, and歇后语 (xiehouyu/riddles) without having to perform manual web scraping.

    The database includes:

    • 14,032 歇后语 (xiehouyu/riddles)
    • 16,142 汉字 (characters)
    • 264,334 词语 (words)
    • 31,648 成语 (idioms)

  2. Overview of data scraping and processing scripts

    master
    The scripts/ directory contains Python and Jupyter Notebook scripts used to scrape, clean, and organize the Chinese language data within this repository. These scripts are primarily used for data acquisition and maintenance of the JSON datasets.

  3. Data structure and file locations

    master

    All data is stored in the data/ directory as JSON files. You can access the following datasets directly from the repository:

    • data/idiom.json: Idioms (成语)
    • data/ci.json: Words (词语)
    • data/word.json: Characters (汉字)
    • data/xiehouyu.json: Xiehouyu/Riddles (歇后语)
    chinese-xinhua/
|
+- data/ <-- Data folder
|  |
|  +- idiom.json <-- Idioms
|  |
|  +- word.json <-- Characters
|  |
|  +- xiehouyu.json <-- Xiehouyu
|  |
|  +- ci.json <-- Words

  4. Scrape and process Chinese language data using Python scripts

    master

    The repository provides specific scripts for different types of linguistic data. Use the corresponding script based on the data category you wish to scrape or process:

    • Idioms (成语): Use chengyu.py to scrape idioms and addAbbreviation.py to add abbreviations to existing idiom data.
    • Words (词语): Use ci.py to scrape words.
    • Characters (汉字): Use word.py to scrape Chinese characters.
    • Xiehouyu (歇后语): Use xiehouyu.py to scrape Xiehouyu (two-part allegorical sayings).
    • Data Cleaning: Use clean.ipynb (Jupyter Notebook) to remove duplicate idioms from the dataset.

  5. Remove duplicate entries from idiom.json

    master

    To clean the idiom.json dataset and remove duplicate entries based on the word key, you can use a dictionary-based deduplication strategy. This process ensures that for any given word, only one instance is kept in the final output. The logic compares the string representation of the idiom objects to decide which one to retain, then saves the unique values to a new file.

    import json

# Load the original data
with open("archived/idiom.json") as fp:
    idioms = json.load(fp)

# Deduplicate using a dictionary where 'word' is the key
check = dict()
for idiom in idioms:
    # If the word exists, keep the one with the 'greater' string representation
    if idiom["word"] in check and str(idiom) > str(check[idiom["word"]]):
        check[idiom["word"]] = idiom
    else:
        check[idiom["word"]] = idiom

# Save the cleaned list to a new file
with open("data/idiom-clean.json", "w+", encoding="utf-8") as fp:
    json.dump(list(check.values()), fp, ensure_ascii=False)

  7. Schema for idiom.json (成语)

    master

    The idiom.json file contains an array of idiom objects. Each object includes the following fields:

    • word: The idiom itself.
    • pinyin: The pronunciation in Pinyin.
    • explanation: A detailed definition and usage context.
    • derivation: The historical or literary origin of the idiom.
    • example: Usage examples in sentences.
    • abbreviation: A short form or abbreviation of the idiom.
    [
    {
        "derivation": "语出《法华经·法师功德品》下至阿鼻地狱。”",
        "example": "但也有少数意志薄弱的……逐步上当，终至堕入～。★《上饶集中营·炼狱杂记》",
        "explanation": "阿鼻梵语的译音，意译为无间”，即痛苦无有间断之意。常用来比喻黑暗的社会和严酷的牢狱。又比喻无法摆脱的极其痛苦的境地。",
        "pinyin": "ā bí dì yù",
        "word": "阿鼻地狱",
        "abbreviation": "abdy"
    }
]

  8. Schema for word.json (汉字)

    master

    The word.json file contains an array of character objects. Each object includes:

    • word: The character.
    • oldword: The traditional or alternative form.
    • pinyin: The pronunciation.
    • strokes: The total stroke count.
    • radicals: The radical component.
    • explanation: Detailed linguistic information including pronunciations, meanings, and usage.
    • more: Additional metadata such as Zhengma (郑码) encoding, stroke order, and further semantic details.
    [
    {
        "word": "嗄",
        "oldword": "嗄",
        "strokes": "13",
        "pinyin": "á",
        "radicals": "口",
        "explanation": "...",
        "more": "..."
    }
]

  9. Schema for xiehouyu.json (歇后语)

    master

    The xiehouyu.json file contains an array of riddle objects. Each object includes:

    • riddle: The first part of the xiehouyu (the setup).
    • answer: The second part (the punchline/meaning).
    [
    {
        "riddle": "飞机上聊天",
        "answer": "高谈阔论"
    }
]