repository·master·Indexed 19 days ago
https://github.com/ku-nlp/jumanppA high-performance Japanese morphological analyzer utilizing a Recurrent Neural Network Language Model (RNNLM) for semantic plausibility. The repository includes the jumanpp CLI tool, details on lattice structures and beam search modes, dictionary specification guidelines, and Pathie, a C++ library for platform-independent Unicode pathname manipulation.
JUMAN++ handles the nominalization of verb continuative forms (連用形) differently than the original JUMAN.
In JUMAN, a verb used as a noun (e.g., "響き" from "響く") is output as the original verb's continuative form. In JUMAN++, these are output as nouns. To indicate that this nominalization was performed during morphological analysis, JUMAN++ adds the semantic information 連用形名詞化:形態素解析 to the output string.
When building pipelines that use both JUMAN and JUMAN++ outputs, you must account for this change in part-of-speech (POS) tagging.
# Example: JUMAN++ outputting a nominalized verb as a noun
% echo "音の響きを大切にする" | jumanpp
音 おん 音 名詞 6 普通名詞 1 * 0 0 "代表表記:音/おん 漢字読み:音 カテゴリ:抽象物"
@ 音 おと 音 名詞 6 普通名詞 1 * 0 0 "代表表記:音/おと 漢字読み:訓 カテゴリ:抽象物"
の の の 助詞 9 格助詞 1 * 0 0 NIL
響き ひびき 響き 名詞 6 普通名詞 1 * 0 0 "代表表記:響き/ひびき 連用形名詞化:形態素解析"
を を を 助詞 9 格助詞 1 * 0 0 NIL
大切に たいせつに 大切だ 形容詞 3 * 0 ナ形容詞 21 ダ列基本連用形 7 "代表表記:大切だ/たいせつだ 反義:形容 詞:粗末だ/そまつだ"
する する する 接尾辞 14 動詞性接尾辞 7 サ変動詞 16 基本形 2 "代表表記:する/する"
EOSJuman++ performs node deduplication during dictionary build time to optimize analysis. A node is uniquely identified by its aliasing set, which consists of all fields used in the training loss that have a non-zero weight.
If multiple dictionary entries share the same values for all fields in the aliasing set, they are merged into a single node. For these merged nodes, additional features are only computed for the first entry in the set.
Important Behavior for Fields:
Best Practice to avoid data loss: To ensure all dictionary information is preserved, avoid using fields with zero weight in the training loss if those fields are needed for analysis. Ideally, every field used in ngram features should also be included in the training loss with a non-zero weight.
Jumandic (Kyoto University Kurohashi-Kawahara lab standard) uses a 4-layered POS tagging system:
Tags can be represented in two ways:
接尾辞-形容詞性述語接尾辞-イ形容詞アウオ段-基本形. These are stable across releases.14-5-18-2. These are not stable and may change between releases.Note on Asterisks (*): In string representations, an asterisk indicates that a specific layer is not applicable to that POS (e.g., 名詞-人名-*-*). In the ID representation, this is encoded as 0.
The Beam search mechanism in Juman++ operates in two distinct modes:
Beam elements consist of:
The Analysis Spec completely defines how Juman++ performs analysis and training. It can be defined using a C++ DSL or a text file. The spec is parsed from top to bottom, and statements cannot refer to statements defined lower in the file.
A complete spec consists of five main parts:
Note: Comments in the text file spec start with the # symbol.
Juman++ uses a specialized binary storage format based on varints (variable-length integers) to optimize dictionary size.
To save space, actual string values are deduplicated and stored in a data domain for each field. The values are sorted in decreasing frequency order. A domain pointer is the offset from the beginning of this domain data.
varint length followed by uint8* data.list_elements structure, which contains a varint length and a sequence of varint offset_difference values (diff-encoded domain pointers).Juman++ represents sentences using a lattice structure where connections exist at codepoint boundaries, including before the first codepoint and after the last one (representing the EOS|sentence|BOS structure). For a sentence with $C$ codepoints, there are $C + 1$ possible connection boundaries.
A Node is defined by a codepoint span and a pointer to either a dictionary entry or information from UNK (unknown) node creators. The underlying data structure for a node seed is:
struct NodeSeed {
u16 begin; // index of first codepoint
u16 end;
i32 pointer; // negative ones are offset to unk nodes
}The lattice container groups node objects by the beginning of their codepoint span. Each boundary contains:
Connections are structured as beam-size x num(end) x num(begin) and each connection contains:
Because default compilers and CMake on CentOS/RedHat EL may be outdated, use Software Collections (SCL) to install a modern toolset. Using devtoolset allows you to build binaries that remain runnable on machines that do not have the devtoolset installed.
Follow these steps to install the necessary tools and build the project from the build directory:
# Install Software Collections and devtoolset (need to do only once)
sudo yum install centos-release-scl
sudo yum install devtoolset-7 llvm-toolset-7-cmake
scl enable devtoolset-7 llvm-toolset-7 bash
# Following lines should be typed from the build directory
cmake .. -DCMAKE_BUILD_TYPE=Release
make -j
make testTo install JUMAN++, download the distribution archive, extract it, and build it using configure, make, and make install. The installation includes the JUMAN++ binary, the standard dictionary, the standard model (trained parameters), and the language model.
gcc (4.9 or later)Boost C++ Libraries (1.57 or later)gperftoollibunwind (required to run gperftool in 64-bit environments)% wget http://lotus.kuee.kyoto-u.ac.jp/nl-resource/jumanpp/jumanpp-1.01.tar.xz
% tar xJvf jumanpp-1.01.tar.xz
% cd jumanpp-1.01
% ./configure
% make
% sudo make installJUMAN++ performs morphological analysis by reading UTF-8 encoded raw text from standard input. Lines starting with # are treated as comments and are not analyzed.
Example usage:
echo "text to analyze" | jumanpp% echo "私もあさって日曜最終日" | jumanpp