Sudachi Japanese Morphological Analyzer

Works Applications provides a suite of software and linguistic resources for Natural Language Processing (NLP), primarily focused on Japanese.

Software

• Sudachi: A Japanese morphological analyzer.
• SudachiPy: The Python implementation of Sudachi.
• elasticsearch-sudachi: A plugin for integrating Sudachi with Elasticsearch.
• Kintoki: A dependency parser.
• jdartsclone: A Trie data structure implementation using double arrays.

Linguistic Resources

• SudachiDict: Japanese morphological dictionaries.
• SudachiDict Synonym: A Japanese synonym dictionary.
• chiVe: Pre-trained Japanese word vectors.
• chiTra: Pre-trained Japanese language models.

chiTra is a library designed to use large-scale pre-trained language models in conjunction with the Sudachi tokenizer (specifically via SudachiPy). It is used for advanced NLP tasks requiring high-capacity models like NWJC.

Available versions:

• v1.0: Uses NWJC (148GB) and is compatible with Sudachi 0.6.2 and SudachiDict 20211220-core.
• v1.1: Uses NWJC with additional cleaning (79GB) and is compatible with Sudachi 0.6.6 and SudachiDict 20220729-core.

When defining the headword (Index 0) for the TRIE, you must register it in its normalized form.

Sudachi performs character normalization (lowercase conversion and Unicode NFKC normalization) before looking up the headword. If you register a word using a form that does not appear after normalization (e.g., using uppercase Latin characters), it will never match because Sudachi will look for the lowercase version internally.

Note: Sudachi does not automatically apply normalization to the user dictionary headwords during loading; the creator must manually ensure the headwords are already in their normalized state.

Sudachi performs character normalization to handle variations in text. The DefaultInputTextPlugin follows this order:

1. Lowercase conversion via Character.toLowerCase().
2. Unicode normalization using NFKC.

Customizing normalization via rewrite.def: You can define custom rules in a rewrite.def file:

• Ignore: Skip specific characters (e.g., # single code point: this character is skipped in character normalization\n髙).
• Replace: Define mapping rules (e.g., # rewrite rule: <target> <replacement>\nA' Ā).

Note: If normalization increases the character count, Sudachi may output morphemes with length 0 relative to the original input.

Sudachi supports three splitting modes that allow you to control the granularity of tokenization. This is useful for balancing precision and recall in search applications (e.g., using A and C together).

• Mode A: Shortest units (equivalent to UniDic short units).
• Mode B: Intermediate units.
• Mode C: Longest units (equivalent to named entity/proper noun units).

Example Comparison (Core Dictionary):

• A: 選挙/管理/委員/会 (Election/Management/Committee)
• B: 選挙/管理/委員会
• C: 選挙管理委員会 (Election Management Committee)

Example Comparison (Full Dictionary):

• A: さっぽろ/テレビ/塔
• B: さっぽろ/テレビ塔
• C: さっぽろテレビ塔

Sudachi uses OOV handlers to process words not present in the dictionary. A valid Sudachi configuration must include at least one OOV handler, or dictionary initialization will fail.

Crucial Requirement: The last OOV handler in your oovPlugins list must be capable of producing a node for a boundary where no other words were found. The SimpleOovProviderPlugin is typically used for this purpose.

You can reorder or override OOV plugins by specifying their class names in the oovPlugins array within your configuration file.

{
  "oovPlugins": [
    {
      "class": "com.worksap.nlp.sudachi.SimpleOovProviderPlugin"
    },
    {
      "class" : "com.worksap.nlp.sudachi.MeCabOovProviderPlugin"
    }
  ]
}

Use benchmark_multithread.sh to execute analysis using multiple threads simultaneously. Each thread is assigned an individual tokenizer instance generated from a single dictionary instance.

• The analysis output is redirected to /dev/null.
• Metadata including the target file and start/end timestamps is appended to data/benchmark.log.

Usage:

./benchmark_multithread.sh corpus_file [num_thread [dict_type]]

• corpus_file: The path to the text file you want to analyze.
• num_thread: The number of threads to create (default: 3).
• dict_type: The dictionary type to use (default: small).

./benchmark_multithread.sh my_corpus.txt 8 full

Use benchmark_setup.sh to build Sudachi and the Sudachi dictionaries. This script performs the following actions:

1. Builds Sudachi and extracts the resulting sudachi-executable-[VERSION].zip into ../build/distributions/sudachi/.
2. Builds the following dictionary files in the data/ directory: system_small.dic, system_core.dic, and system_full.dic.

Note: You must place the downloaded Sudachi dictionary data in data/dictdata/ before running this script.

Usage:

./benchmark_setup.sh [dict_version]

• dict_version: The version of the Sudachi dictionary to use (default: 20240716).

./benchmark_setup.sh 20240716

SudachiDict is a Japanese morphological analysis dictionary. You can download pre-built dictionary files from the official S3 website hosted via the AWS Open Data Sponsorship Program.

http://sudachi.s3-website-ap-northeast-1.amazonaws.com/sudachidict/