gojieba

CppJieba supports several segmentation modes:

• MPSegment (Precise Mode): Standard segmentation.
• HMMSegment: Uses Hidden Markov Model to identify out-of-vocabulary (OOV) words.
• MixSegment: Combines MP and HMM modes. This is generally the most effective as it accurately segments dictionary words while also identifying OOV words (e.g., "杭研").
• FullSegment (Full Mode): Segments all possible words found in the dictionary.
• QuerySegment (Search Engine Mode): First uses MixSegment, then applies FullSegment to the resulting long words to optimize for search engines.

CppJieba requires a C++ compiler (g++ 4.1+ or clang++) and CMake (2.6+). Follow these steps to build from source:

1. Clone the repository.
2. Create a build directory.
3. Run CMake and make.

git clone https://github.com/yanyiwu/cppjieba.git
cd cppjieba
mkdir build
cd build
cmake ..
make

Install GoJieba using go get. Note that this library requires cgo and a C++ compiler. Pure Go cross-compilation (CGO_ENABLED=0) is not supported.

To cross-compile (e.g., to Linux amd64), you must enable CGO_ENABLED=1 and provide the appropriate target C/C++ toolchain via CC and CXX environment variables.

go get github.com/yanyiwu/gojieba

# Example cross-compilation to Linux amd64
CGO_ENABLED=1 \
CC=x86_64-linux-gnu-gcc \
CXX=x86_64-linux-gnu-g++ \
GOOS=linux \
GOARCH=amd64 \
go build

User dictionaries allow you to define custom words. You can provide multiple user dictionary file paths separated by | or ;.

Supported line formats for user.dict.utf8:

1. 词语 (Word only: uses default weight, Part-of-Speech is empty)
2. 词语 词性 (Word and Part-of-Speech)
3. 词语 词频 词性 (Word, Frequency, and Part-of-Speech)

Constraints:

• Does not support comment syntax.
• Does not support extra columns.
• Does not support words containing spaces.

Weighting Strategy: By default, weights are derived from the main dictionary's statistics using the median weight. You can adjust this strategy during cppjieba::DictTrie construction using WordWeightMin, WordWeightMedian, or WordWeightMax.

You can provide custom user dictionaries to influence segmentation results. Multiple dictionary files can be passed by separating them with | or ;.

Supported formats (one entry per line):

• Word (only word, default frequency and empty POS)
• Word POS (word and part-of-speech, default frequency)
• Word Frequency POS (word, frequency, and part-of-speech)

Note: The main dictionary (dict/jieba.dict.utf8) uses a fixed three-column format: Word Frequency POS. If frequency is not provided in a user dictionary, it defaults to the median weight from the main dictionary.

Use gojieba.NewJieba(...string) to create a new instance. If no arguments are provided, it uses the default dictionary. Always call defer x.Free() to release the underlying C++ resources. For specialized extraction, use gojieba.NewExtractor(...string).

x := gojieba.NewJieba()
defer x.Free()

CppJieba provides capabilities for keyword extraction and Part-of-Speech (POS) tagging.

Keyword Extraction returns a list of words with their associated weights. Example output: ["CEO:11.7392", "升职:10.8562"]

POS Tagging assigns a tag to each word. You can define custom tags in your user dictionary (e.g., 蓝翔 nz) to ensure specific words are tagged correctly.

You can add words to the dictionary at runtime to improve segmentation accuracy for specific terms.

• AddWord(word string): Adds a word with default weight.
• AddWordEx(word string, weight int, tag string): Adds a word with a specific weight and optional part-of-speech tag. Use this if AddWord fails due to low weight.

x.AddWord("比特币")
// Or with specific weight and tag
// x.AddWordEx("比特币", 100000, "")

GoJieba provides utilities for keyword extraction and Part-of-Speech (POS) tagging:

• POS Tagging: Tag(s string) []string returns words with their tags (e.g., word,tag).
• Keyword Extraction: ExtractWithWeight(s string, topN int) []string extracts the top N keywords based on weight.
• Tokenization: Tokenize(s string, mode int, use_hmm bool) []WordInfo returns detailed token information including start and end offsets. Use gojieba.DefaultMode or gojieba.SearchMode for the mode parameter.

GoJieba supports several segmentation modes:

• Full Mode (CutAll): Uses the maximum probability pattern.
• Precise Mode (Cut): Uses the HMM new word discovery pattern. Requires a use_hmm boolean parameter.
• Search Engine Mode (CutForSearch): Optimized for search engines.

Methods:

• CutAll(s string) []string
• Cut(s string, use_hmm bool) []string
• CutForSearch(s string, use_hmm bool) []string

Keyword extraction in CppJieba uses the TF-IDF algorithm and requires two specific dictionary files:

1. IDF Dictionary (idf.utf8): Provides the Inverse Document Frequency information required for the TF-IDF calculation.
2. Stop Words Dictionary (stop_words.utf8): A list of words to be ignored during extraction.