LASER (Language-Agnostic SEntence Representations)

You can pre-download encoders and tokenizers for specific languages using the langs build argument. This argument accepts a space-separated list of language codes (e.g., eng_Latn, fra_Latn).

docker build --build-arg langs="eng_Latn fra_Latn" -t laser -f docker/Dockerfile .

To demonstrate the use of multilingual sentence embeddings for cross-lingual Natural Language Inference (NLI), you can run the provided XNLI application. This process trains an NLI classifier on the English MultiNLI corpus (optimizing meta-parameters on the English XNLI development corpus) and then applies it to the test sets of 14 transfer languages.

Running the script performs the following tasks:

1. Installs the XNLI and MultiNLI corpora.
2. Calculates the multilingual sentence embeddings.
3. Trains the classifier.
4. Displays the results.

bash ./xnli.sh

To download encoders for all 24 supported languages for the WMT '22 shared task, run the download_models.sh script located in the current directory. The models will be stored in $LASER/models/wmt22.

Language Model Formats:

• Most focus languages use the laser3-xxx format.
• Afrikaans (afr), English (eng), and French (fra) are supported by the laser2 model.

Supported Languages: amh, fuv, hau, ibo, kam, kin, lin, lug, luo, nso, nya, orm, sna, som, ssw, swh, tsn, tso, umb, wol, xho, yor, and zul.

bash ./download_models.sh

To quickly download a sample dataset (flores200), a sample encoder (laser2), and calculate the sentence embeddings and the xSIM error rate for a set of languages, run the provided evaluation script via bash.

bash ./eval.sh

To demonstrate LASER's application to multilingual similarity search, you can run a provided script that downloads the newstest2012 (WMT 2012) dataset, calculates sentence embeddings for an N-way parallel corpus, and computes the similarity search error rate for each language pair.

Note: The parallel corpus used for this task must not contain duplicate sentences, as the error rate calculation relies on matching sentence indices between the source and target languages in the joint embedding space.

bash ./wmt.sh

You can explore generating multilingual embeddings and performing clustering on the MASSIVE dataset using the provided Google Colab notebook. This tutorial demonstrates how LASER encodes sentences from multiple languages into a shared embedding space for cross-lingual clustering.

https://colab.research.google.com/github/Paulooh007/LASER/blob/laser-clustering/tasks/clustering/LaserClusteringExample.ipynb

Use the extract.py tool to process WikiMatrix TSV files and extract parallel texts based on a margin score threshold.

Usage

Run the script with the following arguments:

• --tsv: Path to the .tsv.gz file.
• --bitext: Path to the output .txt bitext file.
• --src-lang: ISO code for the source language.
• --trg-lang: ISO code for the target language.
• --threshold: The margin score threshold.

Threshold Optimization

• Higher threshold: Increases the likelihood that sentences are mutual translations, but results in less data.
• Recommended value: A threshold of 1.04 is a good starting point for most language pairs.

Note: The data is distributed under the Creative Commons Attribution-ShareAlike license.

python3 extract.py \
  --tsv WikiMatrix.en-fr.tsv.gz \
  --bitext WikiMatrix.en-fr.txt \
  --src-lang en --trg-lang fr \
  --threshold 1.04

To use LASER for cross-lingual document classification using the MLDoc corpus, follow these steps:

1. Download the MLDoc corpus: Obtain the corpus from the MLDoc repository.
2. Install the corpus: Place the downloaded files in a directory named MLDoc.
3. Run the training and embedding pipeline: Execute the provided shell script to calculate multilingual sentence embeddings for all languages and train the classifier.

The system allows training a document classifier on one language (e.g., English) and applying it to other languages (e.g., German, Spanish, French, Italian, Japanese, Russian, and Chinese) without requiring language-specific resources for the target languages.

bash ./mldoc.sh

To run the standard LASER container, use the following command:

docker run -it laser

You can install the core sentence embedding package using pip:

pip install laser_encoders