AzooKeyKanaKanjiConverter

This document outlines known design decisions in AzooKeyKanaKanjiConverter that may present challenges for developers forking the project or building new Japanese input software. These include specific implementations of cost calculation, character identification, and part-of-speech (POS) tagging that may require migration if changed.

Key areas of technical debt include:

• Cost Calculation: Uses Float16 for log-likelihood values.
• Character Identification: Uses a custom CharID system for LOUDS search acceleration.
• Part-of-Speech (POS) Tagging: Based on ipadic with specific custom extensions.

The core conversion engine uses the Viterbi algorithm.

To optimize for real-time typing (where users frequently add, delete, or replace single characters), the converter selects algorithms optimized for these specific delta operations. A key feature is the calculation of additional costs based on content word bigrams. This ensures that words that frequently co-occur in natural language receive higher evaluation scores, while unlikely word combinations are penalized.

To accelerate LOUDS search, labels are mapped to a custom CharID rather than using raw character codes.

Implications for developers:

• The current CharID assignment is not strictly uniform, meaning some characters may be included or excluded unexpectedly.
• Any redesign of the CharID system would require a full data migration.

Typo correction is implemented as a replacement operation on ComposingText.input. For example, a sequence like ts might be interpreted as t with a penalty, or ta might be interpreted as da with a penalty. These rules are predefined in the source code.

To prevent the combinatorial explosion of possible corrections (e.g., applying rules to every character in a long string), the converter uses two optimization techniques:

1. Penalty Caps: Each replacement accumulates a penalty. If the total penalty exceeds a predefined limit, that path is discarded.
2. Parallel Dictionary Search: Since version v0.9, typo correction and dictionary searching happen in parallel. By using a Trie-based dictionary, the system can prune correction paths early if the resulting prefix does not exist in the dictionary.

The ComposingText API is used to manage the state of text being composed during Japanese IME input. It handles the distinction between "direct input" (e.g., typing kana directly) and "romanized input" (e.g., typing 'kyou' to get 'きょう').

ComposingText maintains two primary representations:

1. input: The raw sequence of InputElement objects representing what the user actually typed (including raw Roman characters and input styles).
2. convertTarget: The "apparent string" (the kana/kanji visible to the user). This is what the implementation must ensure matches the user's visual experience.

Key properties for synchronization:

• convertTarget: The string currently visible to the user.
• convertTargetCursorPosition: The cursor position relative to the convertTarget string. Implementations must ensure this matches the actual visual cursor position.

Introduced in v0.9, the Dual-Indexed Lattice solves the problem of matching partial Romaji inputs to dictionary entries that require specific kana representations (like small characters/Sokuon).

In traditional lattice structures, nodes were indexed solely by the ComposingText.input sequence. This failed for cases like ittai (いっtai) because no single substring of the Romaji input perfectly matched the dictionary entry for イッ (the small ッ).

The Dual-Indexed Lattice uses two types of indices simultaneously:

1. Surface String Level Index: A string-based index (e.g., イッタイ) used for standard conversion.
2. Internal String Level Index: A history-based index (e.g., ittai) used for typo correction.

By managing the relationship between these two indices, the converter can correctly handle complex kana transitions that a single-index approach would miss.

The dictionary data cost is implemented as log-likelihood using the Float16 type.

Implications for developers:

• Performance: Summation calculations may be slower compared to integer types.
• Compatibility: There are difficulties due to macOS not natively supporting Float16 in certain contexts.
• Migration: If the cost type is changed to UInt16 or UInt8 in future versions, users will be required to recompile:
  • Emojis, Kaomoji, and User Dictionaries
  • Learning data

The ordering of conversion candidates is determined by Converter.swift using the following priority logic:

1. First 5 candidates: Exact matches, predictive conversion, or Romaji-to-English conversions (at least one exact match must be included in the top 3).
2. Next 5 candidates: Conversions based on phrase/segment units (文節単位変換).
3. Remaining candidates:
   • All-Hiragana, all-Katakana, or all-Uppercase conversions.
   • Prefix matches sorted by length and evaluation score.
   • Special conversions (Unicode, Western/Japanese calendar years, email addresses, decorative characters) are typically inserted around the 5th position.

The project uses an ipadic based POS system with three custom extensions to handle specific linguistic contexts in kana-kanji conversion:

Implications for developers:

• The system is heavily reliant on the ipadic structure. While Unidic might be a more robust alternative, switching would require massive refactoring of the dictionary generation modules and existing codebases.