DOCSEARCHbyAlgolia
Submit Docs

sacremoses

repository·master·Indexed 19 days ago

https://github.com/hplt-project/sacremoses

A multilingual NLP library providing tools for text preprocessing, including tokenization, detokenization, truecasing, and punctuation normalization. It features a Python API with classes like MosesTokenizer, MosesDetokenizer, MosesTruecaser, and MosesPunctNormalizer, as well as a CLI that supports processing pipelines for tasks such as tokenize, detokenize, truecase, detruecase, and normalize.

Tokens
3.4K
Snippets
16
Records
17
Agent score
17%

What's inside sacremoses

  2. Use the Sacremoses CLI for text processing pipelines

    master

    The Sacremoses CLI allows you to perform tokenization, detokenization, normalization, and truecasing via command-line pipes. The CLI accepts input from stdin and outputs the processed text to stdout.

    Global options apply to all commands:

    • --language, -l: Specify the language (default: en).
    • --processes, -j: Number of processes for parallel execution (default: 1).
    • --encoding, -e: Specify file encoding (default: utf8).
    • --quiet, -q: Disable the progress bar.
    # Example: Tokenize an English file using 4 processes
cat input.txt | sacremoses -l en -j 4 tokenize > output.txt

  3. Use MosesPunctNormalizer in Python

    master

    MosesPunctNormalizer can be used to normalize punctuation within a string.

    from sacremoses import MosesPunctNormalizer
mpn = MosesPunctNormalizer()
normalized = mpn.normalize('TEXT TO NORMALIZE')
    from sacremoses import MosesPunctNormalizer
mpn = MosesPunctNormalizer()
mpn.normalize('THIS EBOOK IS OTHERWISE PROVIDED TO YOU "AS-IS."')

  4. Use MosesTokenizer and MosesDetokenizer in Python

    master

    Use MosesTokenizer to split text into tokens and MosesDetokenizer to reconstruct text from tokens. You can specify the language using the lang parameter.

    MosesTokenizer.tokenize(text, return_str=True) returns the tokenized text as a string. Without return_str=True, it returns a list of tokens.

    MosesDetokenizer.detokenize(tokens) takes a list of tokens and returns the reconstructed string.

    from sacremoses import MosesTokenizer, MosesDetokenizer

# Tokenization
mt = MosesTokenizer(lang='en')
text = 'This, is a sentence with weird\xbb symbols\u2026 appearing everywhere\xbf'
tokenized_text = mt.tokenize(text, return_str=True)

# Detokenization
md = MosesDetokenizer(lang='en')
tokens = ['This', 'is', 'a', 'sentence']
detokenized_text = md.detokenize(tokens)

  5. Use MosesTruecaser for training and truecasing in Python

    master

    MosesTruecaser allows you to train a model to correctly capitalize text (truecasing) and then apply that model to strings.

    Training

    You can train a model by passing tokenized documents to .train(). You can save the model during training using the save_to argument, or save it after training using .save_model(path).

    Applying a Model

    • Initialize with a model file: MosesTruecaser('path/to/model').
    • .truecase(text): Returns a list of truecased tokens.
    • .truecase(text, return_str=True): Returns the truecased text as a string.
    • .truecase(text, use_known=True): Uses known words to assist truecasing.
    from sacremoses import MosesTruecaser, MosesTokenizer

# Training and saving
mtr = MosesTruecaser()
mtok = MosesTokenizer(lang='en')
tokenized_docs = [mtok.tokenize(line) for line in open('big.txt')]
mtr.train(tokenized_docs, save_to='big.truecasemodel')

# Using a trained model
mtr = MosesTruecaser('big.truecasemodel')
result = mtr.truecase("THE ADVENTURES OF SHERLOCK HOLMES", return_str=True)
# Output: 'the adventures of Sherlock Holmes'

  6. Reference: sacremoses truecase CLI options

    master

    Options for the truecase command:

    • -m, --modelfile TEXT: Filename to save/load the modelfile. [required]
    • -a, --is-asr: A flag to indicate that model is for ASR.
    • -p, --possibly-use-first-token: Use the first token as part of truecase training.
    • -h, --help: Show this message and exit.
    $ sacremoses truecase --help
Usage: sacremoses truecase [OPTIONS]

Options:
  -m, --modelfile TEXT            Filename to save/load the modelfile.
                                  [required]
  -a, --is-asr                    A flag to indicate that model is for ASR.
  -p, --possibly-use-first-token  Use the first token as part of truecase
                                 training.
  -h, --help                      Show this message and exit.

  7. Reference: sacremoses tokenize CLI options

    master

    Options for the tokenize command:

    • -a, --aggressive-dash-splits: Triggers dash split rules.
    • -x, --xml-escape: Escape special characters for XML.
    • -p, --protected-patterns TEXT: Specify file with patterns to be protected in tokenisation.
    • -c, --custom-nb-prefixes TEXT: Specify a custom non-breaking prefixes file.
    • -h, --help: Show this message and exit.
    $ sacremoses tokenize --help
Usage: sacremoses tokenize [OPTIONS]

Options:
  -a, --aggressive-dash-splits   Triggers dash split rules.
  -x, --xml-escape               Escape special characters for XML.
  -p, --protected-patterns TEXT  Specify file with patters to be protected in
                                 tokenisation.
  -c, --custom-nb-prefixes TEXT  Specify a custom non-breaking prefixes file, 
                                 add prefixes to the default ones from the
                                 specified language.
  -h, --help                     Show this message and exit.

  8. Reference: sacremoses detokenize CLI options

    master

    Options for the detokenize command:

    • -x, --xml-unescape: Unescape special characters for XML.
    • -h, --help: Show this message and exit.
    $ sacremoses detokenize --help
Usage: sacremoses detokenize [OPTIONS]

Options:
  -x, --xml-unescape  Unescape special characters for XML.
  -h, --help          Show this message and exit.

  9. Reference: sacremoses detruecase CLI options

    master

    Options for the detruecase command:

    • -j, --processes INTEGER: Number of processes.
    • -a, --is-headline: Whether the file are headlines.
    • -e, --encoding TEXT: Specify encoding of file.
    • -h, --help: Show this message and exit.
    $ sacremoses detruecase --help
Usage: sacremoses detruecase [OPTIONS]

Options:
  -j, --processes INTEGER  No. of processes.
  -a, --is-headline        Whether the file are headlines.
  -e, --encoding TEXT      Specify encoding of file.
  -h, --help              Show this message and exit.

  10. Reference: sacremoses normalize CLI options

    master

    Options for the normalize command:

    • -q, --normalize-quote-commas: Normalize quotations and commas.
    • -d, --normalize-numbers: Normalize number.
    • -p, --replace-unicode-puncts: Replace unicode punctuations BEFORE normalization.
    • -c, --remove-control-chars: Remove control characters AFTER normalization.
    • -h, --help: Show this message and exit.
    $ sacremoses normalize --help
Usage: sacremoses normalize [OPTIONS]

Options:
  -q, --normalize-quote-commas  Normalize quotations and commas.
  -d, --normalize-numbers       Normalize number.
  -p, --replace-unicode-puncts  Replace unicode punctuations BEFORE
                                 normalization.
  -c, --remove-control-chars    Remove control characters AFTER normalization.
  -h, --help                    Show this message and exit.

  11. Use the sacremoses CLI

    master

    The sacremoses CLI allows you to run NLP tasks via the command line. Since version 0.0.42, the CLI supports a pipeline feature where global options are set before the command.

    Global Options

    • -l, --language TEXT: Use language specific rules.
    • -j, --processes INTEGER: Number of processes.
    • -e, --encoding TEXT: Specify file encoding.
    • -q, --quiet: Disable progress bar.

    Commands

    • tokenize: Tokenize text.
    • detokenize: Detokenize tokens.
    • truecase: Apply truecasing.
    • detruecase: Reverse truecasing.
    • normalize: Normalize text.
    • train-truecase: Train a truecase model.

    Pipeline Example

    You can chain commands together in a single execution:

    cat big.txt | sacremoses -l en -j 4 \
    normalize -c tokenize -a truecase -a -m big.truemodel \
    > big.txt.norm.tok.true