DOCSEARCHbyAlgolia
Submit Docs

Rakuten MA

repository·master·Indexed 19 days ago

https://github.com/rakuten-nlp/rakutenma

A pure JavaScript morphological analyzer for Chinese and Japanese, providing word segmentation and Part-of-Speech (PoS) tagging. Compatible with Node.js and modern web browsers, rakutenma version 1.0.0 supports incremental model updates via online learning, domain adaptation, and model minification for client-side distribution.

Tokens
5.8K
Snippets
18
Records
30
Agent score
67%

What's inside rakutenma

  1. Convert models for browser use

    master

    While the internal data structure of models is the same for Node.js and browsers, browser environments require the model to be a JavaScript assignment (e.g., var model = [...]) rather than a raw JSON file.

    To convert a JSON model file (like model_zh.json) into a format usable in a browser (like model_zh.js), use the provided conversion script:

    scripts/convert_for_browser.js

  2. Use Rakuten MA in the browser

    master

    To use Rakuten MA in a web browser, include the required scripts in your HTML <head>. You will typically need rakutenma.js, the model file (e.g., model_ja.js), and hanzenkaku.js for character conversion.

    Note that RakutenMA.tokens2string(tokens) can be used to convert the tokenized output back into a readable string.

    <script type="text/javascript" src="rakutenma.js" charset="UTF-8"></script>
<script type="text/javascript" src="model_ja.js" charset="UTF-8"></script>
<script type="text/javascript" src="hanzenkaku.js" charset="UTF-8"></script>

<script type="text/javascript" charset="UTF-8">
  function Segment() {
    // Initialize with the loaded model
    rma = new RakutenMA(model);
    rma.featset = RakutenMA.default_featset_ja;
    rma.hash_func = RakutenMA.create_hash_func(15);

    var textarea = document.getElementById("input");
    var result = document.getElementById("output");
    
    // Process input (including character conversion via HanZenKaku)
    var tokens = rma.tokenize(HanZenKaku.hs2fs(HanZenKaku.hw2fw(HanZenKaku.h2z(textarea.value))));

    result.style.display = 'block';
    result.innerHTML = RakutenMA.tokens2string(tokens);
  }
</script>

<textarea id="input" cols="80" rows="5"></textarea>
<input type="submit" value="Analyze" onclick="Segment()">
<div id="output"></div>

  3. Train a new Rakuten MA model from scratch

    master

    To train an original model, follow these steps:

    1. Prepare a training corpus consisting of an array of sentences, where each sentence is an array of [token, part-of-speech-tag] pairs.
    2. Initialize a new instance: var rma = new RakutenMA();.
    3. Set the featset (e.g., rma.featset = RakutenMA.default_featset_ja;).
    4. Iterate through your corpus and call rma.train_one(sentence) for each sentence.

    Training typically converges after one epoch (one full pass through the corpus), but performing 2-3 additional epochs can improve accuracy. Reference scripts/train_ja.js or scripts/train_zh.js for implementation examples.

    var rma = new RakutenMA();
rma.featset = RakutenMA.default_featset_ja;

// Example training data format
var corpus = [
  [["うらにわ", "N-nc"], ["に", "P-k"], ["は", "P-rj"]],
  // ... more sentences
];

// Train
corpus.forEach(sentence => {
  rma.train_one(sentence);
});

  4. Reduce model size using minify.js

    master

    If the model size is too large for client-side distribution, you can use the minify.js script to apply feature quantization.

    Run the script via Node.js: node scripts/minify.js [input_model_file] [output_model_file]

    Warning: This process removes the SCW "sigma" component. Once a model has been minified, it cannot be retrained. Always perform any necessary retraining before minifying.

    node scripts/minify.js input_model.json output_model.json

  5. Train a Rakuten MA model from scratch

    master

    To train a new model, follow these steps:

    1. Prepare a training corpus where each sentence is an array of [token, PoS tag] pairs.
    2. Initialize a new instance: var rma = new RakutenMA();.
    3. Set the featset (e.g., RakutenMA.default_featset_ja or RakutenMA.default_featset_zh).
    4. Iterate through your corpus and call rma.train_one(sent) for each sentence.
    5. You can repeat passes (epochs) through the corpus to improve performance.

  6. Configure feature hashing for bundled models

    master

    When using the bundled models (model_zh.json or model_ja.json), you must use a 15-bit feature hashing function to ensure correct tokenization and PoS tagging. If the hashing function does not match what was used during training, the sentence may be split into individual characters without PoS tags.

    Use RakutenMA.create_hash_func(15) to initialize the correct function.

    rma.hash_func = RakutenMA.create_hash_func(15);

  7. Use Rakuten MA in Node.js

    master

    To use Rakuten MA in a Node.js environment, require the library and initialize a new RakutenMA instance. You can use an empty model for incremental training or load a pre-trained model using JSON.parse and passing it to the constructor.

    When using pre-trained models (like the bundled model_ja.json), you must:

    1. Set the appropriate featset (e.g., RakutenMA.default_featset_ja).
    2. Set a 15-bit feature hashing function using RakutenMA.create_hash_func(15).

    Use rma.tokenize(text) to perform morphological analysis.

    var RakutenMA = require('./rakutenma');
var fs = require('fs');

// Initialize with a pre-trained model
var model = JSON.parse(fs.readFileSync("model_ja.json"));
var rma = new RakutenMA(model, 1024, 0.007812); // 1024 and 0.007812 are SCW hyperparameters

// Required configuration for bundled models
rma.featset = RakutenMA.default_featset_ja;
rma.hash_func = RakutenMA.create_hash_func(15);

// Analyze text
console.log(rma.tokenize("うらにわにはにわにわとりがいる"));

  8. Install Rakuten MA via npm

    master

    You can install Rakuten MA as an npm package for use in Node.js projects. The model files are located in node_modules/rakutenma/ after installation.

    npm install rakutenma

  9. Re-train an existing model (Domain Adaptation)

    master

    You can perform domain adaptation or fix specific errors by loading an existing model and feeding it new training data.

    1. Load the model: rma = new RakutenMA(existing_model_json);.
    2. Prepare your new training sentences (arrays of [token, PoS tag]).
    3. Call rma.train_one(sent) for each new sentence to update the model incrementally.

  10. Retrain an existing model (Domain Adaptation)

    master

    You can perform domain adaptation or fix errors by retraining an existing model.

    1. Load your existing model: var model = JSON.parse(fs.readFileSync("model_file"));.
    2. Initialize RakutenMA with that model: var rma = new RakutenMA(model);.
    3. Provide new training data (in the same [token, tag] format used for initial training).
    4. Call rma.train_one(sentence) for the new data. Even a small number of sentences can be used to adapt the model to a specific domain.

  11. Reduce model size using minification

    master

    To reduce the model size for client-side distribution, use the scripts/minify.js script which applies feature quantization.

    Command:

    node scripts/minify.js [input_model_file] [output_model_file]

    Warning: Minification deletes the "sigma" part of the model. This means the resulting minified model cannot be re-trained. Always perform any necessary re-training before minifying.

  12. Using bundled models in a browser

    master

    The JSON model files (e.g., model_ja.json) are intended for Node.js. To use them in a browser, you must convert them into a JavaScript file that includes a variable assignment (e.g., var model = [JSON representation];).

    • Use model_ja.js (the browser-ready version) if available.
    • Or use the provided script scripts/convert_for_browser.js to convert the JSON model for browser use.