pycorrector

The toolkit supports several deep learning paradigms for text correction:

• End-to-End RNN Models: Uses RNN sequence models (like RNN with Attention) to fit text tasks without manual feature extraction.
• CRF (Conditional Random Fields): Calculates the conditional probability of the globally optimal output nodes, which is effective for detecting specific error types based on the context of the entire sentence.
• Seq2Seq Models: Utilizes an Encoder-Decoder architecture to solve sequence transformation problems, similar to machine translation.
• Pre-trained Models (BERT/ELECTRA/ERNIE/MacBERT): Leverages the powerful language representation of pre-trained models. These can be adapted for correction using the MASK feature and fine-tuning to achieve state-of-the-art results.

The rule-based correction approach in pycorrector follows a two-step process: detection and correction.

1. Error Detection: The system uses Jieba Chinese segmentation to tokenize the text. It detects errors at both the character level and the word level to account for segmentation errors caused by typos. These results are integrated into a candidate set of suspected error locations.
2. Error Correction: For each suspected error location, the system uses dictionaries based on Pinyin similarity (sound-alike), stroke/Wubi similarity (shape-alike), and edit distance. It then uses a language model to calculate the perplexity of the sentence, ranking the candidates to find the optimal correction.

You can install pycorrector automatically via pip or manually from the source repository.

Automatic installation:

pip install pycorrector

Manual installation:

git clone https://github.com/shibing624/pycorrector.git
cd pycorrector
python setup.py install

Required dependencies: To use language model features, you must install kenlm:

pip install https://github.com/kpu/kenlm/archive/master.zip

Otherwise, install all other requirements using:

pip install -r requirements.txt

Language models are critical for the error correction process. Pycorrector provides default models, but you can train specialized ones for better performance in specific domains.

• Default Models: The project uses a large-scale Chinese language model (zh_giga.no_cna_cmn.prune01244.klm) and a lightweight version trained on the People's Daily 2014 corpus (people2014corpus_chars.klm).
• Custom Training: You can train a general-purpose language model using corpora like Chinese Wikipedia (use pycorrector.utils.text_utils for Traditional-to-Simplified conversion) or a domain-specific model using professional corpora.
• Tools: The project supports training via kenlm.

You can train error correction models using your own business-specific error samples. There are two primary ways to prepare this data:

1. Manual Annotation: Annotate existing error samples by identifying the wrong_ids (character positions) and the correct_text.
2. Synthetic Data Generation: If you lack real error samples, use scripts to generate original_text by replacing characters in a correct sentence with similar-sounding (phonetic) or similar-looking (shape) characters at specific wrong_ids positions.

To simulate OCR (Optical Character Recognition) errors for English text correction testing, you can use the nlpaug library. The following pattern demonstrates how to create an augmentation function that introduces character and word-level errors into a list of strings using nac.OcrAug.

import nlpaug.augmenter.char as nac

def ocr_augment_chars(text, **kwargs):
    aug = nac.OcrAug(**kwargs)
    augmented_data = aug.augment(text)
    return augmented_data

text_list = ['i am s student.', 'i am a teacher.']
ocr_text = ocr_augment_chars(text_list, aug_char_p=0.4, aug_word_p=0.6)

Use the detect method of the Corrector class to identify error locations in a string.

Returns a list of lists, where each inner list follows the format: [error_word, begin_pos, end_pos, error_type]. Note that pos indices are 0-based.

from pycorrector import Corrector
m = Corrector()
idx_errors = m.detect('少先队员因该为老人让坐')
print(idx_errors)
# Output example: [['因该', 4, 6, 'word'], ['坐', 10, 11, 'char']]

The MacBertCorrector uses a modified MacBERT architecture for Chinese spelling correction. You can initialize it by passing a HuggingFace model path.

from pycorrector import MacBertCorrector
m = MacBertCorrector("shibing624/macbert4csc-base-chinese")
print(m.correct_batch(['今天新情很好', '你找到你最喜欢的工作，我也很高心。']))

To improve precision (by whitelisting known correct terms) or recall (by adding known errors), you can load a custom confusion set using the custom_confusion_path_or_dict parameter in the Corrector constructor.

The text file format should be space-separated: [Incorrect_Term] [Correct_Term] (e.g., iPhone差 iPhoneX).

from pycorrector import Corrector

error_sentences = [
    '买iphonex，要多少钱',
    '共同实际控制人萧华、霍荣铨、张旗康',
]
m = Corrector()
print(m.correct_batch(error_sentences))

# Load custom confusion set
m = Corrector(custom_confusion_path_or_dict='./my_custom_confusion.txt')
print(m.correct_batch(error_sentences))

The MuCGECBartCorrector is a Bart-based model.

Installation Requirements:

pip install pycorrector modelscope==1.16.0 fairseq==0.12.2

Usage: Note that the model is automatically downloaded to ~/.cache/modelscope/hub/ on its first run. It has been tested on python=3.8.19.

from pycorrector.mucgec_bart.mucgec_bart_corrector import MuCGECBartCorrector

if __name__ == '__main__':
    m = MuCGECBartCorrector()
    result = m.correct_batch(['这洋的话，下一年的福气来到自己身上。', 
                               '在拥挤时间，为了让人们尊守交通规律，派至少两个警察或者交通管理者。', 
                               '随着中国经济突飞猛近，建造工业与日俱增', 
                               "北京是中国的都。", 
                               "他说：”我最爱的运动是打蓝球“", 
                               "我每天大约喝5次水左右。", 
                               "今天，我非常开开心。"])
    print(result)

Use pycorrector.correct() to perform Chinese text error correction. It returns the corrected sentence and a detailed list of corrections.

Output format for details: The detail list contains tuples in the format: [(original_word, corrected_word, start_index, end_index), ...].

import pycorrector

corrected_sent, detail = pycorrector.correct('少先队员因该为老人让坐')
print(corrected_sent, detail)
# Output example:
# 少先队员应该为老人让座 [[('因该', '应该', 4, 6)], [('坐', '座', 10, 11)]]

You can provide a custom text file containing proper names or idioms to the Corrector constructor via the proper_name_path parameter to improve correction accuracy for specific terms.

from pycorrector import Corrector
m = Corrector(proper_name_path='./my_custom_proper.txt')
x = ['报应接中迩来', '这块名表带带相传',]
for i in x:
    print(i, ' -> ', m.correct(i))