DOCSEARCHbyAlgolia
Submit Docs

analysis-sudachi Plugin

repository·develop·Indexed 42 days ago

https://github.com/worksapplications/elasticsearch-sudachi

An Elasticsearch and OpenSearch plugin providing Japanese morphological analysis using the Sudachi engine. It includes the sudachi_tokenizer and various token filters for part-of-speech filtering, stopword removal, lemmatization (sudachi_baseform), normalization (sudachi_normalizedform), and reading form conversion (sudachi_readingform).

Tokens
12.7K
Snippets
30
Records
48
Agent score
51%

What's inside analysis-sudachi

  1. Understand the branch management policy for ES/OS API differences

    develop

    Because Elasticsearch (ES) and OpenSearch (OS) have different APIs depending on their versions, this project manages code differences using a branch-per-series strategy.

    Core Principles:

    • develop branch: Holds all common functionality. Changes should originate here and be cherry-picked/backported to specific series branches.
    • Series branches: One branch exists for each combination of engine and major version (e.g., es-9, os-3).
    • Branching logic: Minimize branching. Only split branches when significant code differences occur between minor versions within a series.
    • Code location: Avoid using src/main/ext/ for new version differences; implement changes directly in the target series branch instead.

  2. How to reference words in a user dictionary

    develop

    To link entries within the user dictionary or to the system dictionary, use a Word reference.

    A word reference is an 8-tuple separated by commas: (surface, POS_level_1, POS_level_2, POS_level_3, POS_level_4, conjugation_type, conjugation_form, reading).

    All 8 fields must match an equivalent entry in either the system dictionary or the current user dictionary. Resolution order:

    1. System dictionary
    2. User dictionary

  3. Control synonym expansion direction using `=>`

    develop

    When using the Solr synonyms format, you can control the direction of expansion using the => notation. This allows you to specify which terms should expand into others and which should remain static.

    For example, if you want アイスクリーム to expand into multiple terms but prevent アイス from expanding, you can format your entries like this:

    # Expand アイスクリーム into multiple terms
アイスクリーム,ice cream,ice=>アイスクリーム,ice cream,ice,アイス

# Prevent expansion for アイス
アイス=>アイス

  4. Understand the branch management policy for ES/OS variants

    develop

    The project manages code differences between Elasticsearch (ES) and OpenSearch (OS) versions by using separate branches for each major API series. The develop branch contains shared functionality, while specific engine/version branches contain the divergent code required for those targets.

    Core Principles

    • develop branch: The default branch for shared features. Most changes should originate here.
    • Series branches: Maintained for specific engine + major version combinations (e.g., es-9, os-3).
    • Propagation: Changes made to develop must be cherry-picked or backported to the relevant series branches.

    Current Branch Structure

    • develop: Shared-feature branch (default).
    • es-9: Elasticsearch v9.*
    • es-8.18-plus: Elasticsearch v8.18.*
    • es-8.10-8.17: Elasticsearch v8.10.* to v8.17.*
    • os-3: OpenSearch v3.*
    • os-2.6-plus: OpenSearch v2.6.*

  5. Configure a Sudachi synonym analyzer in Elasticsearch

    develop

    To use converted Sudachi synonyms, set up a custom analyzer in Elasticsearch using the synonym_graph filter.

    Important: Because the sudachi_split filter produces a token graph, you cannot use it with a synonym filter. Instead, use the sudachi_tokenizer as the base tokenizer.

    This example assumes the converted synonym.txt is located at $ES_PATH_CONF/sudachi/synonym.txt.

    {
    "settings": {
        "analysis": {
            "filter": {
                "sudachi_synonym": {
                    "type": "synonym_graph",
                    "synonyms_path": "sudachi/synonym.txt"
                }
            },
            "tokenizer": {
                "sudachi_tokenizer": {
                    "type": "sudachi_tokenizer"
                }
            },
            "analyzer": {
                "sudachi_synonym_analyzer": {
                    "type": "custom",
                    "tokenizer": "sudachi_tokenizer",
                    "filter": [
                        "sudachi_synonym"
                    ]
                }
            }
        }
    }
}

  6. Release and Tagging procedure

    develop

    Releases are performed by tagging commits on series branches and the default branch.

    Tagging Format:

    • Series tags: [release version]-[branch] (e.g., v3.5.0-es-8.10-plus, v3.5.0-os-3).
    • Default branch tag: [release version] (e.g., v3.5.0). Tagging the default branch triggers the release-orchestrator workflow to build assets for all supported versions.

    Release Steps:

    1. Apply the [release version]-[branch] tags to the appropriate release positions on each series branch.
    2. Apply the [release version] tag to the develop branch.
    3. If a series branch tag is missed or a build fails, manually re-run the workflow after fixing.
    4. Review the draft release and publish.

  7. Configure user dictionaries in Elasticsearch or OpenSearch

    develop

    Once the binary dictionary is compiled, follow these steps to integrate it into your cluster:

    1. Place the dictionary in the config directory: Move the .dic file to <es>/config/sudachi/. The sudachi-analysis plugin resolves paths relative to this directory.

      cp user_test.dic `<es>/config/sudachi/user_test.dic`

    2. Update Index Settings: Use the additional_settings key within the sudachi_tokenizer configuration to point to your dictionary. The value should be a JSON string containing the userDict array.

    Example: Using C split mode (Long tokens)

    curl -X PUT "localhost:9200/test_sudachi_user" -H 'Content-Type: application/json' -d @- <<EOF
{
  "settings": {
    "analysis": {
      "tokenizer" : {
        "sudachi_tokenizer": {
            "type": "sudachi_tokenizer",
            "additional_settings": "{\"userDict\": [\"user_test.dic\"]}"
        }
      }
    }
  }
}
EOF

    Example: Using A split mode (Short tokens)

    curl -X PUT "localhost:9200/test_sudachi_user_a" -H 'Content-Type: application/json' -d @- <<EOF
{
  "settings": {
    "analysis": {
      "tokenizer" : {
        "sudachi_tokenizer": {
            "type": "sudachi_tokenizer",
            "additional_settings": "{\"userDict\": [\"user_test.dic\"]}",
            "split_mode": "A"
        }
      }
    }
  }
}
EOF

  8. Install analysis-sudachi plugin

    develop

    To install the analysis-sudachi plugin into your Elasticsearch or OpenSearch instance, follow these steps:

    1. Navigate to your $ES_HOME directory.
    2. Install the plugin using the elasticsearch-plugin tool. You can use a release package URL or a local file path in URI format.
    3. Download the Sudachi dictionary archive from the SudachiDict repository.
    4. Extract the dictionary file and place it at config/sudachi/system_core.dic. Note: This step is mandatory for Elasticsearch 7.6 or later.
    5. Restart your Elasticsearch/OpenSearch instance.
    # Using a release package
bin/elasticsearch-plugin install https://github.com/WorksApplications/elasticsearch-sudachi/releases/download/v3.5.0/analysis-sudachi-8.13.4-3.5.0.zip

# Using a self-build package (absolute path in URI format)
bin/elasticsearch-plugin install file:///path/to/analysis-sudachi-8.13.4-3.5.0.zip

  10. Select the correct base branch for Pull Requests

    develop

    When creating a Pull Request (PR), choose the base branch based on the scope of your changes:

    Change TypeTarget Base Branch
    Shared logic (filters, tests, docs, build config)develop
    Series-specific (bug fixes, dependency updates)The specific series branch (e.g., es-9)
    Engine-specific (ES-only bug fixes)The latest applicable branch for that engine, then backport
    Build/CI/Gradle updatesdevelop
    Search/Filter logic updatesThe applicable series branch

    Note: Always state the target branch and the requirement for backporting in the PR description.

  11. Compile a binary user dictionary

    develop

    To use a custom dictionary in Sudachi, you must first compile your CSV user dictionary into a binary format using the UserDictionaryBuilder class.

    IMPORTANT: You must compile the user dictionary against the specific system dictionary you intend to use. Using a binary user dictionary compiled with one system dictionary against a different system dictionary can cause analysis errors.

    Requirements:

    • Sudachi Java jar
    • The binary system dictionary (e.g., system_core.dic)
    • Your user dictionary in CSV format
    java -Dfile.encoding=UTF-8 -cp sudachi-*.jar \
    com.worksap.nlp.sudachi.dictionary.UserDictionaryBuilder \
    -o user_test.dic -s system_core.dic \
    -d 'my first user dictionary' \
    input.csv

  12. Configure Sudachi dictionaries

    develop

    After installing the plugin, you must manually provide a Sudachi dictionary.

    1. Download a dictionary (e.g., sudachi-dictionary-latest-core.zip) from SudachiDict.
    2. Unzip the package.
    3. Create a config/sudachi directory within your Elasticsearch configuration path.
    4. Copy the dictionary file (e.g., system_core.dic) into that directory.
    $ wget https://d2ej7fkh96fzlu.cloudfront.net/sudachidict/sudachi-dictionary-latest-core.zip
$ unzip sudachi-dictionary-latest-core.zip
$ mkdir config/sudachi
$ cp sudachi-dictionary-*/system_core.dic config/sudachi/system_core.dic