repository·master·Indexed 24 days ago
https://github.com/thunlp/thulac-pythonA high-efficiency Chinese lexical analysis toolkit developed by Tsinghua University. It provides high-accuracy word segmentation and Part-of-Speech (POS) tagging via a Python API and command-line interface. Features include support for user dictionaries, Traditional to Simplified Chinese conversion, and a high-performance fast interface using libthulac.so.
You can also install THULAC by downloading it from GitHub. Note that if you use this method, you must manually download the model files and place them in the thulac directory for the library to function.
thulac folder in your working directory.thulac directory.The easiest way to install THULAC is via pip, which includes the necessary model files automatically.
sudo pip install thulacAfter installation, you can use it by importing the thulac module.
The thulac() constructor allows you to customize the analyzer's behavior using the following parameters:
| Parameter | Default | Description |
|---|---|---|
user_dict | None | Path to a UTF8 encoded user dictionary (one word per line). Words in this dictionary are tagged with uw. |
model_path | models/ | Directory where the model files are located. |
T2S | False | Whether to convert Traditional Chinese to Simplified Chinese. |
seg_only | False | If True, only performs word segmentation and skips POS tagging. |
filt | False | If True, uses a filter to remove meaningless words (e.g., "可以"). |
deli | '_' | The separator between a word and its POS tag. |
rm_space | False | Whether to remove spaces from the original text before segmentation. |
You can initialize the THULAC analyzer and perform word segmentation (and POS tagging) on strings or files.
By default, THULAC performs both word segmentation and Part-of-Speech (POS) tagging.
import thulac
thu1 = thulac.thulac() # Initialize in default mode
text = thu1.cut("我爱北京天安门", text=True) # Segment a sentence
print(text)To perform only word segmentation without POS tagging, set seg_only=True during initialization.
thu1 = thulac.thulac(seg_only=True)
# Segment a file and save to another
thu1.cut_f("input.txt", "output.txt")import thulac
thu1 = thulac.thulac() # 默认模式
text = thu1.cut("我爱北京天安门", text=True) #进行一句话分词
print(text)For high-performance requirements, a .so version is available. After downloading and placing libthulac.so in the same directory as your models folder, you can use the fast_ prefixed versions of the standard functions. These functions accept the same parameters as the standard API.
cut $\rightarrow$ fast_cutcut_f $\rightarrow$ fast_cut_fcut(text, text=False)Segments a single string.
text (bool): If True, returns the original text along with the segments. If False (default), returns a 2D array of [[word, tag], ...] pairs. In seg_only mode, the tag will be an empty string.cut_f(input_file, output_file)Segments the content of an input file and writes the results to an output file.
THULAC uses the following Part-of-Speech (POS) tags:
| Tag | Meaning |
|---|---|
n | Noun |
np | Person name |
ns | Place name |
ni | Organization name |
nz | Other proper noun |
m | Number |
q | Quantifier |
mq | Quantity word |
t | Time word |
f | Directional word |
s | Location word |
v | Verb |
a | Adjective |
d | Adverb |
h | Pre-component |
k | Post-component |
i | Idiom |
j | Abbreviation |
r | Pronoun |
c | Conjunction |
p | Preposition |
u | Particle |
y | Modal particle |
e | Exclamation |
o | Onomatopoeia |
g | Morpheme |
w | Punctuation |
x | Other |
If you installed THULAC via pip, you can run it directly from the terminal.
Read from input.txt and output results to output.txt:
python -m thulac input.txt output.txtAdd the seg_only flag to skip POS tagging:
python -m thulac input.txt output.txt seg_onlypython -m thulac input.txt output.txt
# or for segmentation only:
python -m thulac input.txt output.txt seg_only