jieba: Chinese Text Segmentation for Python

To use the paddle mode (which uses the PaddlePaddle deep learning framework for segmentation and POS tagging), you must first install paddlepaddle-tiny and then call jieba.enable_paddle().

Requirements:

• jieba version 0.40 or higher.
• paddlepaddle-tiny==1.6.1.

pip install paddlepaddle-tiny==1.6.1

import jieba
jieba.enable_paddle()
# Now you can use use_paddle=True in jieba.cut()

You can install jieba using standard Python package managers. It is compatible with both Python 2 and 3.

pip install jieba
# or
pip3 install jieba

If a specific phrase (e.g., '台中') is being incorrectly split into individual characters (e.g., '台 中'), it is because the probability of the phrase occurring is lower than the product of its parts ($P( ext{phrase}) < P( ext{part1}) imes P( ext{part2})$).

You can force the segmenter to treat the phrase as a single word by increasing its frequency using jieba.add_word() or jieba.suggest_freq() with the True flag.

# Method 1: Add the word to the dictionary
jieba.add_word('台中')

# Method 2: Suggest a higher frequency for the phrase
jieba.suggest_freq('台中', True)

You can install jieba using several methods:

Automatic installation (Recommended):

pip install jieba

Semi-automatic installation: Download the package from PyPI, extract it, and run:

python setup.py install

Manual installation: Place the jieba directory directly into your current working directory or your Python site-packages directory.

If a phrase is being incorrectly joined together (e.g., '今天天气' instead of '今天 天气'), you can force the segmenter to split it by decreasing its frequency or removing it.

Use jieba.suggest_freq() with the True flag to suggest a split, or use jieba.del_word() to remove the phrase from the dictionary entirely.

# Method 1: Suggest splitting the phrase into specific parts
jieba.suggest_freq(('今天', '天气'), True)

# Method 2: Delete the phrase from the dictionary
jieba.del_word('今天天气')

Use jieba.posseg to segment text and identify the Part-of-Speech for each word. The tagging follows the ictclas standard.

API:

• jieba.posseg.cut(string, ...): Returns an iterator of (word, flag) tuples.
• Supports use_paddle=True for deep learning-based tagging.

Common Tags:

• n: Noun, nr: Person, ns: Place, nt: Organization, v: Verb, a: Adjective, r: Pronoun, etc.
• Paddle mode also includes uppercase tags like PER (Person), LOC (Location), and ORG (Organization).

import jieba.posseg as pseg

words = pseg.cut("我爱北京天安门")
for word, flag in words:
    print(f"{word} {flag}")
# Output example: 我 r, 爱 v, 北京 ns, 天安门 ns

For large datasets, you can speed up segmentation by using multiple processes. This is based on Python's multiprocessing module and is not supported on Windows.

API:

• jieba.enable_parallel(processes): Enables parallel mode with the specified number of processes.
• jieba.disable_parallel(): Disables parallel mode.

import jieba

jieba.enable_parallel(4)  # Use 4 processes
# ... perform segmentation ...
jieba.disable_parallel()

You can improve segmentation accuracy by providing your own dictionary.

Loading a User Dictionary

Use jieba.load_userdict(file_name) to load a file. The file must be UTF-8 encoded. Each line should follow this format: word [frequency] [tag] (space-separated). Frequency and tag are optional.

Example format:

创新办 3 i
云计算 5
台中

Dynamic Dictionary Modification

• jieba.add_word(word, freq=None, tag=None): Add a word dynamically.
• jieba.del_word(word): Remove a word.
• jieba.suggest_freq(segment, tune=True): Adjust the frequency of a word to force or prevent it from being segmented in a specific way.

import jieba

# Load a file
jieba.load_userdict("my_dict.txt")

# Dynamic updates
jieba.add_word("new_word")
jieba.suggest_freq("台中", True)

The jieba.tokenize() method returns the start and end indices of each word in the original string.

API:

• jieba.tokenize(string, mode='default')
• mode='default': Standard segmentation.
• mode='search': Search engine mode segmentation.

Returns an iterable of tuples: (word, start_index, end_index).

import jieba

# Default mode
result = jieba.tokenize(u'永和服装饰品有限公司')
for word, start, end in result:
    print(f"word {word}\t\t start: {start} \t\t end:{end}")

You can improve accuracy by providing your own dictionary or modifying the existing one dynamically.

Load a dictionary file

Use jieba.load_userdict(file_name) where file_name is a path or a file-like object. The format should be: word [frequency] [POS tag] (one per line, space-separated).

Example format:

创新办 3 i
云计算 5
台中

Dynamic modification

• jieba.add_word(word, freq=None, tag=None): Add a word to the dictionary.
• jieba.del_word(word): Remove a word.
• jieba.suggest_freq(segment, tune=True): Adjust the frequency of a single word to force or prevent specific segmentation.

import jieba

# Load a file
jieba.load_userdict("my_dict.txt")

# Add a word
jieba.add_word("云计算", freq=5, tag="n")

# Adjust frequency to fix segmentation
jieba.suggest_freq(('中', '将'), True)
# Or for a single word
jieba.suggest_freq('台中', True)