DOCSEARCHbyAlgolia
Submit Docs

HanLP Multilingual NLP Library

repository·master·Indexed 12 days ago

https://github.com/hankcs/hanlp

A multilingual NLP library built on PyTorch and TensorFlow 2.x supporting 130 languages. It provides deep learning techniques for tokenization, POS tagging, NER, and dependency parsing via native Python APIs or RESTful interfaces. Includes support for various annotation schemes such as CTB, PTB, NPCMJ, and Stanford Dependencies.

Tokens
51.3K
Snippets
145
Records
222
Agent score
99%

What's inside HanLP

  1. Overview of HanLP

    master
    HanLP is a multilingual Natural Language Processing (NLP) library designed for researchers and industrial applications. It is built on PyTorch and TensorFlow 2.x and supports state-of-the-art deep learning techniques. The library provides pretrained models for multiple languages, including English, Chinese, and Japanese, and is designed to be efficient, user-friendly, and extendable.

  2. Use the Multi-Task Learning (MTL) framework in HanLP

    master

    HanLP provides a Multi-Task Learning (MTL) framework designed to handle multiple NLP tasks simultaneously. This framework allows for more efficient model training and inference by sharing representations across different tasks.

    To use the MTL framework, you can explore the core MTL components and specific task implementations provided in the HanLP documentation.

  3. Use the hanlp_trie package for Trie structures

    master
    The hanlp_trie package provides implementations for Trie (prefix tree) data structures, consisting of Node and Trie classes. These are typically used for efficient string searching, prefix matching, and dictionary-based operations in NLP tasks.

  4. Perform Named Entity Recognition (NER) in HanLP

    master
    HanLP provides Named Entity Recognition (NER) capabilities to identify and categorize entities such as persons, locations, and organizations within text. The library supports specific models for Chinese (including datasets like PKU and MSRA) and multilingual models (such as OntoNotes).

  6. Annotate Numerical Expressions (NUMEX)

    master

    Numerical expressions (NUMEX) cover various mathematical and quantitative representations:

    • Percentage (per): Percentages and related terms like 'five成' or 'six折'. Note that fractions like 'one millionth' should be marked as fra instead of per.
      • Example: [per*百分之二十五*], [per*六折*].
    • Money (mon): Currency amounts. Do not include place names within the currency tag. If different currency forms are used for the same amount, annotate them separately.
      • Example: [mon*43.6亿美元*], [mon*四万五千元人民币*].
    • Frequency (fre): How often something occurs (e.g., 'twice', 'many times'). Note: 'once' (一次) is generally not annotated.
      • Example: [fre*两次*], [fre*无数次*].
    • Integer (int): Combinations of number words and quantifiers (e.g., 'three kinds', 'ten people').
      • Example: [int*三种*], [int*五十户*].
    • Fraction (fra): Fractions and multipliers (e.g., 'half', 'three times', 'two to three times').
      • Example: [fra*一半*], [fra*4倍半*].
    • Decimal (dec): Decimal numbers.
      • Example: [dec*3.14*], [dec*三点一四*].
    • Ordinal (ord): Ordinal numbers indicating order (e.g., 'first', 'third', 'No. 1174').
      • Example: [ord*第一任*], [ord*第十六楼*].
    • Ratio (rat): Ratios and proportions.
      • Example: [rat*一比廿五*], [rat*1∶0*].
    /* 大约 */ [per*5%]
[mon*四亿元台币*]
[fre*数度*]
[int*卅七件*]
[fra*四分之三*]
[dec*3.14*]
[ord*第一期*]
[rat*一比廿五*]

  7. Understand MSR Chinese Text Annotation Formats

    master

    The MSR (Microsoft Research) Chinese Text Annotation Specification (v5.0) defines two primary formats for marking entities in text:

    1. Format-1 (Annotator-oriented): Uses slashes / for word boundaries and square brackets [] with a tag for entities. Example: /十月九日/上午/ $\rightarrow$ /[dat十月九日]/[tim上午]/

    2. Format-2 (XML-based): Uses XML tags for entities. Example: /十月九日/上午/ $\rightarrow$ <w><TIMEX TYPE="DATE">十月九日</TIMEX></w><w><TIMEX TYPE="TIME">上午</TIMEX></w>

    *Note: TIMEX is a time expression, with DATE and TIME as sub-classes. For most practical purposes in this specification, Format-1 is used.

    format-1: /[dat十月九日]/[tim上午]/
format-2: <w><TIMEX TYPE="DATE">十月九日</TIMEX></w><w><TIMEX TYPE="TIME">上午</TIMEX></w>

  8. Segment CAS words based on grammatical function

    master

    Many common Chinese words require different segmentation rules depending on whether they function as nouns/adverbs or as part of a number-quantifier phrase.

    WordDo NOT split when...DO split when...
    一起 (together)Used as a noun or adverbUsed as a "Number + Quantifier" phrase (e.g., 一/起/车祸)
    一点 (a bit)Used as an adjective (meaning "slight")Used as a number-quantifier phrase
    一道 (a path/way)Used as an adverbUsed as a "Number + Quantifier" phrase (e.g., 一/道/风景线)
    一面 (a side/aspect)Used as a noun or adverbUsed as a "Number + Quantifier" phrase
    一口 (a mouthful)Used as an adjective or adverbUsed as a "Number + Quantifier" phrase
    一片 (a piece/area)Used as an adjectiveUsed as a number-quantifier phrase
    上下 (up/down)Used as a noun (e.g., 上下/两册)Used as a verb (e.g., 上/下/车)
    从前 (formerly)Used as a time noun从 is used as a preposition for location (e.g., 从/前/不久)
    人才 (talent)Used as a noun(N/A)
    走向 (direction/trend)Used as a nounUsed as a verb + preposition (e.g., 走/向/世界)
    才能 (ability)Used as a nounUsed as an adverb + modal verb (e.g., 才/能/有)

  9. Segmentation rules for Adverbs (AD)

    master

    Adverbs (AD) are generally separated from the phrase (XP) they modify. Key rules include:

    • Reduplication: Reduplicated VA (adjective) or AD can become an AD (e.g., 常常[always]/AD).
    • DT+M/N: Strings like 这样[this way] are tagged as AD when modifying a VP/S.
    • Frozen PPs: Certain prepositional phrases function as adverbs and are tagged as AD (e.g., 迄今[until now], 为何[why]).
    • PN+LC: If a Proper Noun + Locative Complement loses its NP function and acts like an adverb, tag as AD (e.g., 此外[in addition]).
    • Other functional shifts: If a string loses its original head function and behaves like an adverb, tag as AD (e.g., 进一步[a step further]).
    近[almost]/AD 三十[thirty]/CD
这样[this way]/AD 做[do]/W

  10. Annotate Dynastic Names as Date (dat) or Location (L)

    master

    The annotation for dynasty names (e.g., 清 (Qing), 元 (Yuan), 秦 (Qin)) depends on the context:

    • Default to dat: If the dynasty refers to a historical period (e.g., [dat*清*], [dat*秦*]).
    • Use L (Location): If the dynasty name is used to refer to the country/state (e.g., [L*秦*]兵 - Qin soldiers, [L*清*]政府 - Qing government).
    • Use P (Person): If the name refers to an emperor (e.g., [P*康熙*]).
    /*最近*/*发现*/*一*/*张*/*在*/*农家*/*珍藏*/*的*/[dat*清代*]/[P*康熙*]...
/*只*/*带*/[dur*三天*]/*粮食*/*，*/*渡过*/[L*漳河*]/*去*/*与*/*强大*/*的*/[dat*秦*]/*兵*/*作战*/*。

  11. Use HanLP Single-Task Learning models

    master
    In addition to Multi-Task Learning, HanLP provides specialized Single-Task Learning models for specific core NLP tasks (such as tagging or parsing). These are often more accurate for specific languages than general multilingual MTL models. Refer to the pretrained model documentation for a full list of available single-task models.

  12. Segmentation rules for V+Coverb (V+X) patterns

    master

    When a verb is followed by a monosyllabic coverb (X), the segmentation depends on the specific coverb:

    • Treat as one word if X is in this list:
      • 给 (give), 为 (become), 成 (become), 作 (treat as), 到 (arrive), 出 (out), 自 (from), 向 (toward), 入 (in), 以 (with).
      • Example: 送给[give/send to]/VV.
    • Treat as two words if X is in this list:
      • 在 (at), 似 (like).
      • Example: 生[to be born]/W 在[at]/P.
    • Conditional (V+于):
      • If 于 can be replaced by 在, tag as two words (V+P).
      • Otherwise, tag as one word.
      • Example (One word): 等于[equal to]/VV.
      • Example (Two words): 生[to be born]/W 于[at]/P.
    给[give]: 送给[give/send to]/VV，交给[hand in]/VV
在[at]: 生[to be born]/W 在[at]/P