CppJieba Documentation

CppJieba supports advanced NLP tasks including keyword extraction and Part-of-Speech (POS) tagging.

• Keyword Extraction: Returns words along with their offsets and weights.
• POS Tagging: Returns words paired with their part-of-speech tags (e.g., n for noun, v for verb, r for pronoun). You can define custom tags by adding them to your user dictionary (e.g., word tag).

CppJieba provides several segmentation modes to suit different use cases:

• MPSegment (Precise Mode): Standard segmentation based on the dictionary.
• HMMSegment: Uses Hidden Markov Model to identify out-of-vocabulary (OOV) words.
• MixSegment: A hybrid approach combining MP and HMM; generally considered the most effective as it captures both dictionary words and OOV words.
• FullSegment (All Mode): Extracts all possible words found in the dictionary.
• QuerySegment (Search Engine Mode): Uses MixSegment first, then applies FullSegment to the resulting longer words to optimize for search engines.

To install CppJieba, clone the repository, create a build directory, and use CMake to build the project. You will need a C++ compiler (g++ 4.1+ or clang++) and CMake (2.6+).

git clone https://github.com/yanyiwu/cppjieba.git
cd cppjieba
mkdir build
cd build
cmake ..
make

CppJieba supports custom user dictionaries to handle specific terms. You can provide multiple dictionary files by separating their paths with a pipe | or a semicolon ;.

User dictionaries support three line formats (one entry per line, columns separated by spaces):

1. word (uses default frequency and empty POS tag)
2. word pos (uses default frequency)
3. word freq pos (explicit frequency and POS tag)

Note: The main dictionary (dict/jieba.dict.utf8) uses a fixed three-column format: word freq pos. If a user dictionary does not provide a frequency, it defaults to the median weight from the main dictionary.

To use the KeywordExtractor (which utilizes the TF-IDF algorithm), you must provide the following dictionary files:

• idf.utf8 (or .gbk): Provides the Inverse Document Frequency (IDF) information.
• stop_words.utf8 (or .gbk): Provides the stop words list to filter out common words.

The main dictionary files used by MPSegment (Max Probability) must follow a fixed three-column format per line:

词语 词频 词性 (Word Frequency Part-of-Speech)

• Each line must contain exactly three columns.
• 词频 (Frequency) is converted to logarithmic weight upon reading.
• 词性 (Part-of-Speech) is stored as a string; the library does not perform enumeration validation on these strings.

You can provide custom dictionaries to influence segmentation. The user.dict supports three specific line formats:

1. 词语 (Word only): Uses default weight and empty part-of-speech.
2. 词语 词性 (Word Part-of-Speech): Specifies the part-of-speech.
3. 词语 词频 词性 (Word Frequency Part-of-Speech): Specifies weight and part-of-speech.

Constraints:

• Multiple user dictionary paths can be passed separated by | or ;.
• Does not support comment syntax, extra columns, or words containing spaces.
• Default weights for new words are derived from the main dictionary (typically the median weight). Weights can be adjusted via cppjieba::DictTrie using WordWeightMin, WordWeightMedian, or WordWeightMax.

The project includes a benchmark target to compare local performance. Running this command builds and executes test/benchmark.cpp, providing metrics for dictionary loading (DictTrieLoad), HMM model loading (HMMModelLoad), segmentation throughput (MPCut, MixCut), and dictionary lookup throughput (DictFind).

make benchmark