DOCSEARCHbyAlgolia
Submit Docs

Ichiran Japanese Language Processing Tools

repository·master·Indexed 19 days ago

https://github.com/tshatrov/ichiran

A collection of tools for Japanese language processing featuring experimental segmentation and romanization algorithms. Ichiran utilizes the JMDictDB dictionary database to provide detailed word meanings and part-of-speech information. It includes a Common Lisp API via the romanize function and a command-line interface (ichiran-cli) for performing romanization and segmentation.

Tokens
2K
Snippets
11
Records
12
Agent score
64%

What's inside Ichiran

  1. Install Ichiran manually

    master

    To install Ichiran as a regular ASDF system, follow these steps:

    1. Download Dictionary Data: Obtain JMDict data. For a full installation, download JMDict and optionally kanjidic2.xml for kanji functionality.
    2. Configure Settings: Create a settings.lisp file by copying settings.lisp.template. Update it with correct file paths and database connection parameters.
    3. Install Dependencies: Use quicklisp to install all required dependencies.
    4. Initialize Database:
      • Easy mode: Download a database dump from the releases page. Ensure settings.lisp has the correct connection parameters. Run (ichiran/maintenance:add-errata) to update the database.
      • Hard mode: Run (ichiran/maintenance:full-init) for a complete initialization. Alternatively, run (ichiran/maintenance:load-jmdict) followed by (ichiran/maintenance:load-best-readings) to initialize only ichiran/dict (skipping ichiran/kanji). Note that this can take several hours.
    5. Verify Installation: Run (ichiran/test:run-all-tests) to ensure the installation is correct.
    6. Optimize Segmentation: Before using word segmentation, run (ichiran/dict:init-suffixes t) to create a suffix cache for improved quality.

  2. Install Ichiran using Docker

    master

    You can run Ichiran using Docker Compose. This method handles database importation and initialization automatically.

    1. Build the images: Run docker compose build from the repository root.
    2. Start containers: Run docker compose up. The first run will take several minutes as the database is imported and initialized. You can monitor the database growth using du -h -d0 docker/pgdata (it grows to ~4.7 GB). The process is complete when the ichiran container logs "All set, awaiting commands.".
    3. Resetting the database: If you encounter errors during import or wish to re-import a new database, delete the existing postgres data to trigger the initdb scripts: sudo rm -rf docker/pgdata, then run docker compose up again.
    docker compose build
docker compose up

  3. Run Ichiran using Docker Compose

    master

    You can deploy Ichiran using the provided docker-compose.yml file. The setup consists of two services:

    1. pg: A PostgreSQL database service configured with a pre-loaded database dump.
    2. main: The primary Ichiran application service.

    To start the services, run:

    docker-compose up

  4. Use the ichiran-cli command line interface

    master

    The ichiran-cli tool is used to romanize Japanese text. By default, it performs romanization on the provided input string. It supports several flags to change the output format, such as printing dictionary information or returning full segmentation data as JSON.

    Basic Usage: ichiran-cli "your input text"

    Note: The tool loads database connections from environment variables via load-connection-from-env during initialization.

    ichiran-cli "一覧は最高だぞ"

  6. Use the romanize API in Common Lisp

    master

    The core functionality of Ichiran is provided via the romanize function. In an SBCL interpreter (accessible via docker exec -it ichiran-main-1 ichiran-sbcl), you can call this function to get romanized text and detailed word information.

    Use the :with-info t keyword to include dictionary meanings and part-of-speech information in the output.

    * (ichiran:romanize "一覧は最高だぞ" :with-info t)
"ichiran wa saikō da zo"
(("ichiran" . "一覧 【いちらん】\n1. [n,vs] look; glance; sight; inspection\n2. [n] summary; list; table; catalog; catalogue")
 ("wa" . "は\n1. [prt] 《pronounced わ in modern Japanese》 indicates sentence topic\n2. [prt] indicates contrast with another option (or stated or unstated)\n3. [prt] adds emphasis")
 ...)

  7. Configure the PostgreSQL service in Docker Compose

    master

    The pg service in docker-compose.yml manages the database layer. Key configuration details include:

    • Database Dump: The service builds using a specific Dockerfile and pulls a database dump via the ICHIRAN_DB_URL build argument.
    • Shared Memory: shm_size is set to 1gb to support database operations.
    • Environment Variables:
      • POSTGRES_PASSWORD: Sets the database password (defaults to password).
      • PGDATA: Defines the internal data directory path (/var/lib/postgresql/data/pgdata).
    • Persistence: Data is persisted to your local filesystem via the volume mapping ${PWD}/docker/pgdata:/var/lib/postgresql/data.
    services:
  pg:
    build:
      dockerfile: ./docker/postgres-dockerfile
      context: .
      args:
        ICHIRAN_DB_URL: "https://github.com/tshatrov/ichiran/releases/download/ichiran-260118/ichiran-260118.pgdump"
    shm_size: "1gb"
    environment:
      POSTGRES_PASSWORD: "password"
      PGDATA: "/var/lib/postgresql/data/pgdata"
    volumes:
      - ${PWD}/docker/pgdata:/var/lib/postgresql/data

  8. Get full segmentation info as JSON with --full

    master

    To get the complete segmentation data in JSON format, use the --full (or -f) flag. You can combine this with the --limit (or -l) flag to restrict the number of segmentations returned.

    Example command:

    ichiran-cli -f -l 5 "一覧は最高だぞ"
    ichiran-cli -f -l 5 "一覧は最高だぞ"

  9. Evaluate arbitrary expressions with --eval

    master

    The --eval (or -e) flag allows you to pass an arbitrary Lisp expression as a string. The CLI will evaluate the expression and print the result. This is useful for testing internal logic or specific functions within the ichiran environment.

    ichiran-cli -e "(some-ichiran-function)"

  10. Print dictionary info with --with-info

    master

    To see the dictionary information associated with the romanized text, use the --with-info (or -i) flag. This will print the romanized result followed by the word and its corresponding gloss/definition.

    ichiran-cli -i "input text"

  11. Use the ichiran-cli in Docker

    master

    The ichiran-cli allows you to perform romanization and segmentation directly from the command line within the Docker container using the -i flag to provide input text.

    docker exec -it ichiran-main-1 ichiran-cli -i "一覧は最高だぞ"

  12. Reference ichiran-cli command line flags

    master

    The following flags are available for the ichiran-cli tool:

    FlagShortLongDescription
    --help-h--helpPrint this help text
    --eval-e--evalEvaluate an arbitrary expression and print the result
    --with-info-i--with-infoPrint dictionary info
    --full-f--fullFull split info (as JSON)
    --limit-l--limitLimit segmentations to the specified number (useful only with -f or --full)

    Note on --limit: This flag requires an integer argument. The default value is 1.

    ichiran-cli -f -l 5 "一覧は最高だぞ"