Kagome Japanese Morphological Analyzer

repository·v2·Indexed 21 days ago

A pure Go Japanese morphological analyzer that tokenizes text into words and analyzes parts of speech. It features embedded dictionaries, multiple segmentation modes, and supports various deployment targets including CLI, REST API, WebAssembly, and a C FFI wrapper for integration with languages such as Python, PHP, and Rust.

Tokens: 10.3K
Snippets: 43
Records: 57
Agent score: 75%

What's inside Kagome

Explore Kagome usage examples
v2
The _examples directory provides several implementation patterns for using Kagome's Japanese morphological analysis capabilities. Depending on your use case, you can explore the following examples:
- Japanese Text Segmentation (Wakati): Learn how to segment Japanese text into individual words.
- Custom Dictionaries: See how to incorporate a user-defined dictionary into the analysis process.
- Tokenization and POS Tagging: Understand how to analyze sentences to retrieve both tokens and their corresponding Part-of-Speech (POS) tags.
- Full-text Search with SQLite3: Learn how to integrate Kagome with SQLite3 for efficient Japanese text searching.
- WebAssembly (Wasm): See how to run Kagome in browser-based or Wasm environments.
- C Library (FFI) Integration: Learn how to use Kagome as a C library to interface with other languages like Python and PHP 8 via the C ABI.
https://github.com/ikawaha/kagome/blob/v2/_examples/README.md

Choose a segmentation mode for tokenization

Kagome supports three segmentation modes that determine how input text is split into tokens. Choosing the right mode depends on your specific use case (e.g., general linguistic analysis vs. search engine indexing).

Normal: Standard segmentation.
Search: Uses heuristics to perform additional segmentation specifically optimized for search purposes.
Extended: Similar to Search mode, but also applies uni-grams to unknown words to improve coverage.

Input	Untokenized	Normal	Search	Extended
関西国際空港	関西国際空港	関西国際空港	関西　国際　空港	関西　国際　空港
日本経済新聞	日本経済新聞	日本　経済　新聞	日本　経済　新聞
デジカメを買った	デジカメを買った	デジカメ　を　買っ　た	デジカメ　を　買っ　た	デ　ジ　カ　メ　を　買っ　た

Memory Management and Thread Safety in the Go FFI Bridge
v2
To ensure stability when calling Kagome from other languages, the FFI bridge follows these strict rules:
Memory Management
- Allocation: All memory returned to the caller (token arrays, string fields within tokens, and the TokenArray container) is allocated via C.malloc. This prevents Go's garbage collector from moving or deleting data while C is still using it.
- Responsibility: The caller is responsible for freeing any memory allocated by the bridge. For example, every call to KagomeTokenizeStruct must eventually be paired with a call to KagomeFreeTokenArray.
- Strings: C.CString is used for inputs, which requires manual freeing. However, C.GoString is used to copy data back to Go, which does not require manual freeing.
Thread Safety
- The bridge uses a sync.Mutex to protect the internal map of tokenizer instances.
- Optimization: The mutex is only held during map access (to find or create a handle). It is not held during the actual tokenization process, allowing multiple threads to perform tokenization concurrently using different handles.
https://github.com/ikawaha/kagome/blob/v2/_examples/clib/go/README.md
Use Kagome via C FFI (Multi-Language)
v2
Kagome can be used from other languages (like Python or PHP) by building it as a C shared library (.so, .dll, or .dylib). This is achieved through a C wrapper that exposes a stable ABI.
C API Functions
The wrapper provides four primary functions for interacting with the morphological analyzer:
- kagome_init(): Initializes the Kagome engine.
- kagome_destroy(): Cleans up and destroys the engine.
- kagome_tokenize(): Performs tokenization on Japanese text.
- kagome_free_token_array(): Frees the memory allocated for the token array.
Supported Language Examples
- Python: Uses ctypes to load the shared library and call the C functions.
- PHP: Uses the FFI extension to interface with the library.
```
/* C API Reference */
void kagome_init();
void kagome_destroy();
void* kagome_tokenize(const char* text);
void kagome_free_token_array(void* token_array);
```
https://github.com/ikawaha/kagome/blob/v2/_examples/clib/README.md
Use Kagome and SQLite3 for Japanese text search applications
v2
The pattern of tokenizing Japanese text with Kagome and indexing it via SQLite3 FTS4 is suitable for several high-scale use cases:
- Search Engines: Building an index to search through large volumes of Japanese text content.
- Document Management Systems: Enabling full-text search capabilities for Japanese-language documents.
- Content Recommendation Systems: Implementing query-based recommendations from a large collection of Japanese content.
- Chatbots and NLP: Assisting in text analysis and searching within a chatbot's knowledge base.
https://github.com/ikawaha/kagome/blob/v2/_examples/db_search/README.md
Understand the Go FFI Bridge Architecture
v2
The Go FFI Bridge layer is a specialized component designed to expose Kagome's tokenizer to non-Go languages (such as Python or PHP) via a C interface. It acts as a mediator between the Kagome library and a C wrapper.
The data flow follows this hierarchy:
1. Other Languages (Python, PHP, etc.)
2. Shared Library (../bin/kagome.(so|dll|dylib))
3. C Wrapper (../c_wrapper/)
4. Go Bridge (this directory)
5. Kagome Library (github.com/ikawaha/kagome/v2)
This layer is necessary because it handles the complexities of Go's cgo system, specifically managing memory across the FFI boundary, ensuring thread-safe tokenizer instance management, and converting data structures between Go and C.
https://github.com/ikawaha/kagome/blob/v2/_examples/clib/go/README.md
Run the Kagome WebAssembly example
v2
Once you have built kagome.wasm and copied wasm_exec.js into your directory, you can run the application by serving the project via a web server and opening the HTML file (e.g., kagome.html) in a browser. The HTML file is configured to load both the wasm_exec.js glue code and the kagome.wasm binary.
An online demo of this implementation is available at: https://ikawaha.github.io/kagome/
https://github.com/ikawaha/kagome/blob/v2/_examples/wasm/README.md
Use Kagome in PHP via FFI
v2
You can use Kagome (a Japanese Morphological Analyzer) in PHP by calling the kagome_* C wrapper functions through the PHP FFI extension. This allows PHP scripts to leverage the high-performance Go-based tokenizer for Japanese text processing.
```
// The example uses the kagome_* C wrapper functions via PHP FFI
// to tokenize Japanese text and access token properties like:
// surface, pos, base_form, reading, pronunciation, start, and end.
```
https://github.com/ikawaha/kagome/blob/v2/_examples/clib/php8/README.md
Build Kagome for WebAssembly
v2
To use Kagome in a web browser, you must compile the Go source code into a WebAssembly (.wasm) binary using the js/wasm target.
After building the binary, you must also copy the wasm_exec.js file from your local Go installation's GOROOT to your project directory. This JavaScript file is required to bridge the Go WebAssembly runtime with the browser environment.
```
# Build the wasm binary
GOOS=js GOARCH=wasm go build -o kagome.wasm main.go

# Copy wasm_exec.js which matches the compiled binary
cp "$(go env GOROOT)/misc/wasm/wasm_exec.js" .
```
https://github.com/ikawaha/kagome/blob/v2/_examples/wasm/README.md

Use Kagome as a Go library

Integrate Kagome into your Go applications by importing the v2 module. You can perform simple word segmentation (wakati) or full morphological analysis (tokenize).

First, install the module: go get github.com/ikawaha/kagome/v2

Then, use the tokenizer package to create a new tokenizer instance with a dictionary (e.g., ipa.Dict()).

package main

import (
  "fmt"
  "strings"

  "github.com/ikawaha/kagome-dict/ipa"
  "github.com/ikawaha/kagome/v2/tokenizer"
)

func main() {
  t, err := tokenizer.New(ipa.Dict(), tokenizer.OmitBosEos())
  if err != nil {
    panic(err)
  }
  // wakati (simple word splitting/segmentation)
  fmt.Println("---wakati---")
  seg := t.Wakati("すもももももももものうち")
  fmt.Println(seg)

  // tokenize w/ morphological analysis
  fmt.Println("---tokenize---")
  tokens := t.Tokenize("すもももももももものうち")
  for _, token := range tokens {
    features := strings.Join(token.Features(), ",")
    fmt.Printf("%s\t%v\n", token.Surface, features)
  }
}

Use the Kagome C Wrapper for FFI integration
v2
The Kagome C wrapper provides a stable, public API designed for use with Foreign Function Interface (FFI) in languages such as Python, PHP, Rust, and others. Instead of calling Go functions directly, which can be unstable and expose internal runtime symbols, you should only call the functions prefixed with kagome_* provided by this layer.
This wrapper ensures stability across Go version updates and prevents symbol name conflicts by hiding internal Go symbols.
```
// Example of the available function patterns (conceptual)
// kagome_init()
// kagome_tokenize()
// kagome_destroy()
// kagome_free_token_array()
```
https://github.com/ikawaha/kagome/blob/v2/_examples/clib/c_wrapper/README.md
Access the Kagome server
v2
Once the environment is bootstrapped using docker compose up -d, the Kagome server is accessible at:
http://localhost:6060/
https://github.com/ikawaha/kagome/blob/v2/_examples/server_docker_graphviz/README.md

Kagome Japanese Morphological Analyzer

What's inside Kagome

Memory Management

Thread Safety

C API Functions

Supported Language Examples