DOCSEARCHbyAlgolia
Submit Docs

Bling Fire (FIRE) Documentation

repository·master·Indexed 23 days ago

https://github.com/microsoft/blingfire

A high-performance finite state machine and regular expression manipulation library for linguistic operations including tokenization, stemming, lemmatization, and multi-word expression matching. Optimized for low-latency production inference, it provides a uniform interface for algorithms such as WordPiece, SentencePiece (Unigram LM and BPE), and pattern-based tokenization, with compatibility for models like BERT, XLNET, and XLM-RoBERTa. Supports Python via pip and C# via NuGet, with native build instructions for Windows, macOS, and Linux.

Tokens
13.1K
Snippets
31
Records
49
Agent score
82%

What's inside Bling Fire

  1. What is Bling Fire Tokenizer?

    master

    Bling Fire (FIRE: FInite State machine and REgular expression manipulation library) is a high-performance linguistic manipulation library used for tasks like tokenization, multi-word expression matching, stemming, and lemmatization.

    Its tokenizer provides a uniform interface for several state-of-the-art algorithms, making it compatible with models like BERT, XLNET, or custom models. It is designed for low-latency inference and is significantly faster than alternatives like Hugging Face Tokenizers, SentencePiece, or SpaCy.

    Supported algorithms include:

    • Pattern-based tokenization
    • WordPiece
    • SentencePiece Unigram LM
    • SentencePiece BPE
    • Induced/learned syllabification patterns (W2H)

  3. Use the xlm_roberta_base tokenization model

    master

    The xlm_roberta_base model provides a version of the XLM-RoBERTa tokenization model that includes normalization. This model is intended for use with the BlingFire tokenizer to achieve tokenization compatible with XLM-RoBERTa while benefiting from BlingFire's performance.

    Note that this implementation includes specific modifications to pos.dict.utf8 compared to the original HuggingFace Transformers implementation to support BlingFire's architecture.

  4. Understand the uri100kint model naming and characteristics

    master

    The uri100kint model is a specialized tokenizer designed for international URLs. The name breaks down as follows:

    • URI: Refers to a URL.
    • int: Indicates the model is optimized for international characters.
    • 100k: Indicates the model produces 100,000 unique IDs.

    Input Requirements

    To ensure correct tokenization at runtime, you must prepare your input as follows:

    1. Encoding: Input URLs must be UTF-8 and Punycode decoded.
    2. Case Sensitivity: Do not normalize the case. The model expects the original case as seen in a browser address bar.
    3. Format: Feed the input exactly as it appears in a browser line.

  5. Use the XLNet non-normalization tokenization model

    master

    The xlnet_nonorm model is a version of the XLNet tokenization model that does not perform internal normalization.

    Important: When using this model, you must ensure that the entire input text is normalized before calling the TextToIds method. This model assumes the input is already in its final normalized form.

  6. Build the uri250k language model

    master

    To build a custom language model for the uri250k dataset, follow these steps to prepare the vocabulary, generate the dictionary, and compile the model using the BlingFire Makefile.

    1. Prepare the dictionary and tagset: Navigate to the ldbsrc/uri250k directory and process the vocabulary file to create pos.dict.utf8 and tagset.txt.
    2. Generate charmap (Optional): Create a character map for case folding using generate_charmap.py.
    3. Compile: Return to the ldbsrc directory and run the make command specifying the uri250k language.

    Note: You must have the BlingFire environment set up (e.g., via set_env from the BlingFire directory) so that the PATH is correctly configured.

    cd <BlingFire>/ldbsrc/uri250k

# Produce pos.dict.utf8 and tagset.txt
cat uri250k.vocab | awk 'BEGIN {FS="\t"} NF == 2 { if (NR > 1) { print $1 "\tWORD_ID_" NR-1 "\t" ($2 == 0 ? "-0.00001" : $2); }  print "WORD_ID_" NR " " NR > "tagset.txt"; }' > pos.dict.utf8

# Optional: create charmap
python ./generate_charmap.py > charmap.utf8

# Build the model
cd <BlingFire>/ldbsrc
make -f Makefile.gnu lang=uri250k all

  7. Build and publish the Bling Fire NuGet package

    master

    If you have made changes to the NuGet package, follow these steps to rebuild and test it locally before publishing.

    1. Create the NuGet package

    Run this command from the lib folder using the Release configuration:

    dotnet pack --configuration Release

    2. Test locally using a local NuGet repository

    To avoid publishing directly to a public registry, add the package to a local source. You must have nuget.exe installed.

    On Linux:

    nuget.exe add lib/bin/Release/BlingFireNuget.0.1.5.nupkg -Source /home/sergei/BlingFire2/tmplocalnugetrepo/packages/

    On Windows:

    C:\src\BlingFire\nuget\lib>c:\Users\sergeio\Downloads\nuget.exe add bin\Release\BlingFireNuget.0.1.7.nupkg -Source C:/Users/sergeio/mytestnugets

    Configure NuGet.config: Ensure your NuGet.config includes your local directory as a feed:

    <?xml version="1.0" encoding="utf-8"?>
<configuration>
  <packageSources>
    <add key="nuget.org" value="https://api.nuget.org/v3/index.json" protocolVersion="3" />
    <add key="Test Source" value="/home/sergei/BlingFire2/tmplocalnugetrepo/packages" />
  </packageSources>
</configuration>

    3. Consume the new version

    Update your test project (e.g., BlingUtilsTest.csproj) to use the new version and run it in Debug configuration (required because model file paths are hardcoded to the Debug configuration):

    mcedit BlingUtilsTest.csproj
dotnet run --configuration Debug BlingUtilsTest.csproj

    4. Publish to GitHub

    To publish to GitHub, use the dotnet nuget push command with your API token:

    dotnet nuget push "bin/Release/BlingFire.x.y.z.nupkg" --source "github" --api-key a123687fef23b8bfake0234820394802ffebc233
    dotnet pack --configuration Release

  8. Use the uri250kint model for international URLs

    master

    The uri250kint model is a specialized version of the URI tokenizer designed for internationalized URLs. It produces up to 250,000 unique IDs.

    Data Preparation Requirements

    To ensure compatibility with this model at runtime, you must perform the following preprocessing on your input URLs:

    1. UTF-8 Decoding: Ensure the URL is UTF-8 decoded.
    2. Punycode Decoding: Ensure the URL is Punycode decoded.
    3. Case Sensitivity: Do not normalize the case. The model expects the original case as seen in a browser address bar.

    Input Format

    Feed the URL exactly as it appears in a browser address bar, using UTF-8 encoding.

  9. Build the laser100k LDB model

    master

    To build the laser100k LDB (Language Data Base) model, follow these steps to prepare the corpus, train a SentencePiece model, generate the dictionary, and compile the tools:

    1. Prepare Corpus: Download the WikiMatrix corpus, extract unique sentences, and sample 30M-50M sentences.
    2. Normalize: Use the normalize_corpus.py script to normalize spaces (using identity normalization).
      cat all_plain_text.30m.txt | python ../../scripts/normalize_corpus.py > normalized_corpus.txt
    3. Train SPM Model: Train a unigram language model using spm_train.
      spm_train --input=normalized_corpus.txt --model_prefix=laser_100k --vocab_size=100000 --character_coverage=0.9999 --model_type=unigram --normalization_rule_name=identity --num_threads=10 --input_sentence_size=50000000 --shuffle_input_sentence=true
    4. Create Dictionary: Generate pos.dict.utf8.zip from the vocabulary.
      cat laser_100k.vocab | awk 'BEGIN {FS="\t"} NF == 2 { if (NR > 1) { print $1 "\tWORD_ID_" NR-1 "\t" ($2 == 0 ? "-0.00001" : $2); } print "WORD_ID_" NR " " NR > "tagset.txt"; }' > pos.dict.utf8
zip pos.dict.utf8.zip pos.dict.utf8
    5. Configure: Add options.small and ldb.conf.small (using xlnet_nonorm as the example since normalization is not used).
    6. Compile: Build the tools from the ldbsrc directory.
      cd <BlingFire>/ldbsrc
make -f Makefile.gnu lang=laser100k all

  11. Build the XLNet component of BlingFire

    master

    To build the XLNet component, you must first prepare the model files and then compile the source code using the provided Makefile.

    1. Prepare Model Files

    Navigate to the XLNet source directory and export the vocabulary from the spiece.model file:

    cd <BlingFire>/ldbsrc/xlnet

# Export the model vocabulary
spm_export_vocab --model spiece.model --output spiece.model.exportvocab.txt --output_format txt

# Produce the pos.dict.utf8 and tagset.txt files
cat spiece.model.exportvocab.txt | awk 'BEGIN {FS="\t"} NF == 2 { if (NR > 1) { print $1 "\tWORD_ID_" NR-1 "\t" ($2 == 0 ? "-0.00001" : $2); }  print "WORD_ID_" NR " " NR > "tagset.txt"; }' > pos.dict.utf8

# Zip the dictionary
zip pos.dict.utf8.zip pos.dict.utf8

# Optional: Generate a charmap for normalization (e.g., NFC to NFKC)
python ./generate_charmap.py > charmap.utf8

    2. Compile

    Navigate back to the ldbsrc directory and run the make command specifying the xlnet language:

    cd <BlingFire>/ldbsrc
make -f Makefile.gnu lang=xlnet all

    Upon successful compilation, the binary xlnet.bin will be located in the ldb/ directory.

    cd <BlingFire>/ldbsrc/xlnet

# export the model:
spm_export_vocab --model spiece.model --output spiece.model.exportvocab.txt --output_format txt

# produce pos.dict.utf8 file and tagset.txt:
cat spiece.model.exportvocab.txt | awk 'BEGIN {FS="\t"} NF == 2 { if (NR > 1) { print $1 "\tWORD_ID_" NR-1 "\t" ($2 == 0 ? "-0.00001" : $2); }  print "WORD_ID_" NR " " NR > "tagset.txt"; }' > pos.dict.utf8

# zip it:
zip pos.dict.utf8.zip pos.dict.utf8

# optional step: create a charmap for NFC --> NFKC normalization or anything else
python ./generate_charmap.py > charmap.utf8

# build all as usual
cd <BlingFire>/ldbsrc
make -f Makefile.gnu lang=xlnet all

  12. Verify tokenization parity and performance

    master

    After building the laser100k model, you can verify its accuracy against the SentencePiece model and measure performance.

    Parity Verification

    Use test_bling_with_offsets.py to check if the BlingFire output matches the SentencePiece model output. A high parity (e.g., 99.999%) is expected.

    cat laser100k/normalized_corpus.txt | python ../scripts/test_bling_with_offsets.py -m ldb/laser100k.bin -p laser100k/laser_100k.model > output.txt

# Check for errors
cat output.txt | awk '/ERROR:/' | wc -l

    Performance Measurement

    Compare the execution time of the SentencePiece model (test_sp.py) against the BlingFire LDB model (test_bling.py).

    # Measure SentencePiece performance
time -p cat test.norm1m.txt | python ../scripts/test_sp.py -m laser100k/laser_100k.model -s 1

# Measure BlingFire performance
time -p cat test.norm1m.txt | python ../scripts/test_bling.py -m ldb/laser100k.bin -s 1