sudachi.rs

SudachiPy provides pre-built Python wheels for several platforms. Use these wheels for faster installation and performance (especially on Linux) compared to building from source.

Supported Architectures:

• Linux: amd64 (x86_64) and aarch64 (arm64). Linux wheels are built with Profile-Guided Optimizations (PGO) for better performance and use manylinux containers for broad distribution compatibility.
• Windows: x86_64. Note that Windows builds do not use PGO.
• macOS: universal2 (Intel and ARM), x86_64, and arm64.

Support Levels:

• Linux: Fully supported; bugs are actively fixed.
• Windows: Best-effort support; bug fixes may take time.
• macOS: Not officially supported; builds are provided on a best-effort basis and are tested less rigorously. Contributions for macOS support are welcomed.

When using Sudachi.rs as a library, you should expect that any issues arising from user-provided text will be returned as Errors rather than causing a panic or crash. The library is designed to be safe against arbitrary user-inputted text.

Important distinction:

• User Input: Should always return an Error. Crashes caused by user input are considered bugs.
• Invalid Dictionaries: Sudachi.rs may panic or exhibit undefined behavior if provided with an invalid binary dictionary. Binary dictionaries are treated as trusted components.
• Incorrect Usage: The library may panic if used incorrectly by the developer (e.g., violating the API contract), but it should not panic based on specific input values.

Sudachi.rs follows a specific threat model regarding security:

1. User Text: It is safe to process any user-inputted text; crashes related to text analysis are treated as bugs.
2. Binary Dictionaries: Maliciously crafted binary dictionaries are not considered a security threat that the library protects against. While binary dictionaries are mapped read-only (preventing memory modification), they could theoretically allow data leaking.

Best Practice for Unsafe Dictionaries: If you must use a dictionary provided by an untrusted user, do not use a pre-compiled binary dictionary. Instead, accept the dictionary in its CSV form and compile it into a binary dictionary yourself using the Sudachi.rs compilation tools. This ensures the data is validated during the compilation process.

The sudachipy.Morpheme.split method also supports memory reuse, but it introduces a dependency on the parent MorphemeList. When you call .split() on a Morpheme, the resulting MorphemeList contains references to the data in the parent MorphemeList.

Warning: If you reuse the parent MorphemeList as an out parameter in a subsequent tokenize() call, the data in the original list is replaced, and any MorphemeList objects derived from it (via .split()) will be invalidated. Accessing them after the parent has been reused may raise an exception.

ml1 = tok.tokenize("外国人参政権")
subl1 = ml1[0].split(SplitMode.A)

# This next line invalidates subl1 because ml1 is being reused
tok.tokenize("something", out=ml1)

# Accessing subl1 now is unsafe and may raise an exception
subl1[0].surface()

SudachiPy supports multi-granular tokenization via the SplitMode enum. This allows you to control how text is segmented.

• SplitMode.C (Default): Least granular (longest words).
• SplitMode.B: Intermediate granularity.
• SplitMode.A: Most granular (shortest words).

Example: For the input "国家公務員":

• Mode C: ['国家公務員']
• Mode B: ['国家', '公務員']
• Mode A: ['国家', '公務', '員']

from sudachipy import Dictionary, SplitMode

tokenizer_obj = Dictionary().create()

# Mode C (Default)
print([m.surface() for m in tokenizer_obj.tokenize("国家公務員", SplitMode.C)])

# Mode B
print([m.surface() for m in tokenizer_obj.tokenize("国家公務員", SplitMode.B)])

# Mode A
print([m.surface() for m in tokenizer_obj.tokenize("国家公務員", SplitMode.A)])

Sudachi requires a dictionary to function. You can either download a zip file (small, core, or full) from SudachiDict, extract it, and place the system_*.dic file in a known location (the default config expects resources/system.dic), or use the provided script to automate the process.

To download the latest core dictionary:

./fetch_dictionary.sh

To download a specific version and type (e.g., small):

./fetch_dictionary.sh 20241021 small

If you prefer using Sudachi via Python, you can install the sudachipy package using pip.

pip install --upgrade 'sudachipy>=0.6.11'

To use SudachiPy, you must install both the library and a dictionary package. The sudachidict_core edition is the default.

# Install the library
pip install sudachipy

# Install the default core dictionary
pip install sudachidict_core

pip install sudachipy sudachidict_core

When using SudachiPy, importing directly from sudachipy.morpheme is deprecated. To ensure compatibility with future versions, you should import the Morpheme class directly from the sudachipy package.

Deprecated approach:

from sudachipy.morpheme import Morpheme

Recommended approach:

from sudachipy import Morpheme