pkuseg-python

To ensure fair comparison with other segmentation tools (like jieba or THULAC) that do not provide domain-specific pre-trained models, you can train pkuseg on your own datasets.

When training, pkuseg uses an interface that does not require a dictionary. To evaluate the performance of your trained model, use the test method. For consistent benchmarking, ensure all segmentation tools are trained and tested on the same data splits.

If you are not using a supported OS/architecture for pip installation, or prefer to build from source, you can download the code from GitHub and run the setup script.

Note: GitHub source code does not include pre-trained models. You must manually download models from the releases page and specify the model path using the model_name parameter when initializing.

python setup.py build_ext -i

If you have downloaded the code directly from GitHub, you must build the extension manually.

Note: The GitHub repository does not include pre-trained models. You must download them from the official releases and specify the path using the model_name parameter when initializing the toolkit.

python setup.py build_ext -i

Install the pkuseg toolkit using PyPI. The default installation includes the mixed-domain model.

Requirements:

• Python 3
• Supported Python versions: 3.5, 3.6, 3.7 on Linux, Mac, and Windows 64-bit.

To install the default version:

pip3 install pkuseg

To update to the latest version:

pip3 install -U pkuseg

To install using the Tsinghua mirror for faster speeds:

pip3 install -i https://pypi.tuna.tsinghua.edu.cn/simple pkuseg

When using the multiprocessing capabilities of pkuseg (via the nthread parameter), you must wrap your execution code within an if __name__ == '__main__': block to prevent errors during process spawning.

This is required for both pkuseg.test() and pkuseg.train() functions when nthread is greater than 1.

import pkuseg

if __name__ == '__main__':
    # Segmenting a file using 20 threads
    pkuseg.test('input.txt', 'output.txt', nthread=20)
    
    # Training a model using 20 threads
    pkuseg.train('msr_training.utf8', 'msr_test_gold.utf8', './models', nthread=20)

Use pkuseg.train with the init_model parameter to continue training from an existing model (fine-tuning).

Parameters:

• train_file: Path to training data.
• test_file: Path to testing data.
• model_dir: Directory where the trained model will be saved.
• train_iter: Number of training iterations (epochs).
• init_model: Path to the pre-trained model directory to initialize training.

To use a model you have already downloaded (e.g., a CTB8 model), pass the local directory path to the model_name parameter.

import pkuseg

# Load a model from a local directory
seg = pkuseg.pkuseg(model_name='./ctb8')  
text = seg.cut('我爱北京天安门')            # Perform segmentation
print(text)

If you are unsure of the specific domain of your text, use the default model. This is the recommended approach for general-purpose segmentation.

import pkuseg

seg = pkuseg.pkuseg()           # Load model with default configuration
text = seg.cut('我爱北京天安门')  # Perform segmentation
print(text)

import pkuseg

seg = pkuseg.pkuseg()           # 以默认配置加载模型
text = seg.cut('我爱北京天安门')  # 进行分词
print(text)

Use pkuseg.train(train_file, test_file, model_dir) to train a model with random initialization.

Requirements:

• Files must be UTF-8 encoded.
• Training and test sets must have words separated by one or more spaces.
• The final model from the last epoch will be saved in model_dir.