jieba.NET Documentation

jieba.NET provides several core capabilities for Chinese text processing:

• Three Segmentation Modes: Precise, Full, and Search Engine modes.
• Traditional Chinese Support: Supports segmentation for traditional Chinese characters.
• Customization: Ability to add custom dictionaries and custom words.
• Keyword Extraction: Extracting key terms from text.
• Part-of-Speech (POS) Tagging: Identifying the grammatical role of words.
• Tokenization: Returning the start and end positions of words within the original text.
• Parallel Segmentation: Support for parallel processing.
• KeywordProcessor: A specialized tool (inspired by FlashText) for flexible keyword extraction, supporting features like case-insensitivity and handling words with spaces.
• Lucene.NET Integration: Integration capabilities for search indexing.

jieba.NET provides three distinct segmentation modes to suit different use cases:

1. Precise Mode (精确模式): Attempts to segment the sentence as accurately as possible. Best suited for text analysis.
2. Full Mode (全模式): Scans all possible words in a sentence. It is very fast but cannot resolve ambiguity because it does not use maximum probability paths or HMM.
3. Search Engine Mode (搜索引擎模式): Based on Precise Mode, but further segments long words to improve recall. Best suited for search engine indexing.

jieba.NET requires dictionary and data files located in a Resources directory.

Option 1: Default Configuration

Copy the Resources directory from the packages\jieba.NET folder into your application's output directory. The library will use this directory by default.

Option 2: App.config or Web.config

Specify a custom path using the JiebaConfigFileDir key in your configuration file. The path can be absolute or relative to the application's BaseDirectory.

Option 3: Programmatic Configuration

If you cannot use config files, you can set the base directory in code before calling any segmentation methods. It is recommended to use an absolute path for reliability.

<appSettings>
    <add key="JiebaConfigFileDir" value="C:\\jiebanet\\config" />
</appSettings>

JiebaNet.Segmenter.ConfigManager.ConfigFileBaseDir = @"C:\\jiebanet\\config";

jieba.NET is a C# implementation of the jieba Chinese segmentation tool. It supports .NET 4.0, .NET 4.5, and .NET Standard 2.0. You can install it using the NuGet Package Manager.

PM> Install-Package jieba.NET

Use the JiebaSegmenter class to segment Chinese text into words using different modes:

• Full Mode: Uses Cut(text, cutAll: true) to find all possible words.
• Precise Mode: Uses Cut(text) (default) for the most accurate segmentation.
• Search Engine Mode: Uses CutForSearch(text) to segment text in a way that is optimized for search engines (returning more granular tokens).
• HMM Model: Both Cut and CutForSearch support an hmm parameter to enable/disable the Hidden Markov Model for recognizing new words (out-of-vocabulary words).

var segmenter = new JiebaSegmenter();

// Full mode
var segments = segmenter.Cut("我来到北京清华大学", cutAll: true);

// Precise mode (default)
segments = segmenter.Cut("我来到北京清华大学");

// Search engine mode
segments = segmenter.CutForSearch("小明硕士毕业于中国科学院计算所");

The Counter<T> class (inspired by Python's Counter) allows you to count the occurrences of words in a collection. You can modify counts using Add, Subtract, and Union, and retrieve the most frequent items using MostCommon(n).

var s = "算法是一个表示为有限长列表的有效方法。";
var freqs = new Counter<string>(segmenter.Cut(s));

foreach (var pair in freqs.MostCommon(5))
{
    Console.WriteLine($"{pair.Key}: {pair.Value}");
}

You can improve segmentation accuracy by providing your own dictionary or modifying the existing one:

• Load a file: Use LoadUserDict(path) to load a custom dictionary file. The format is one entry per line: word [frequency] [tag], separated by spaces. If frequency is omitted, the segmenter calculates an appropriate one.
• Add/Update words: Use AddWord(word, freq: 0, tag: null) to add a new word or adjust the frequency of an existing one. If freq is not a positive integer, an automatic frequency is used to ensure the word is segmented.
• Remove words: Use DeleteWord(word) to prevent a specific word from being segmented.

segmenter.LoadUserDict("user_dict_file_path");

// Add or adjust frequency
segmenter.AddWord("机器学习", freq: 3);

// Remove a word
segmenter.DeleteWord("某个词");

For high-performance requirements, use the parallel versions of the segmentation methods:

• JiebaSegmenter.CutInParallel()
• JiebaSegmenter.CutForSearchInParallel()
• PosSegmenter.CutInParallel()

Use the PosSegmenter class from JiebaNet.Segmenter.PosSeg to segment text and assign a part-of-speech tag to each word. The tags are compatible with the ICTCLAS standard.

var s = "一团硕大无朋的高能离子云";
var tokens = posSeg.Cut(s);

foreach (var token in tokens)
{
    // token.Word is the word, token.Flag is the POS tag
    Console.WriteLine($"{token.Word}/{token.Flag}");
}

The Tokenize method returns the start and end indices of each word in the original string.

• Default Mode: Standard segmentation.
• Search Mode: Uses TokenizerMode.Search to provide more granular tokens (similar to search engine mode) along with their positions.

var s = "永和服装饰品有限公司";

// Default Tokenize
var tokens = segmenter.Tokenize(s);

// Search Mode Tokenize
var searchTokens = segmenter.Tokenize(s, TokenizerMode.Search);

foreach (var token in tokens)
{
    Console.WriteLine($"word {token.Word} start: {token.StartIndex} end: {token.EndIndex}");
}

The KeywordProcessor is used to find specific, pre-defined keywords within a text. Unlike the TfidfExtractor which finds important words, KeywordProcessor matches known words from a dictionary.

• AddKeywords(IEnumerable<string>): Adds a list of keywords to the processor.
• ExtractKeywords(string text, bool raw: false): Finds keywords in the text. If raw is set to true, it returns the original word as it appeared in the input text; otherwise, it returns the keyword as it was defined in the dictionary.

kp.AddKeywords(new []{ ".NET Core", "Java", "C语言" });

// Returns dictionary version: ".NET Core"
var keywords = kp.ExtractKeywords("学习.NET core", raw: false);

// Returns original text version: ".NET core"
var rawKeywords = kp.ExtractKeywords("学习.NET core", raw: true);