fastText

FastText provides vectors in two formats:

1. Binary format (.bin): Optimized for fast loading and allows obtaining vectors for out-of-vocabulary (OOV) words. You can extract OOV vectors via the CLI: ./fasttext print-word-vectors <model.bin> < oov_words.txt

2. Text format (.vec): Each line contains a word followed by its space-separated vector values, sorted by frequency. These can be loaded into Python using a custom parser (e.g., using io.open).

import io

def load_vectors(fname):
    fin = io.open(fname, 'r', encoding='utf-8', newline='\n', errors='ignore')
    n, d = map(int, fin.readline().split())
    data = {}
    for line in fin:
        tokens = line.rstrip().split(' ')
        data[tokens[0]] = map(float, tokens[1:])
    return data

Hyperparameter optimization (autotune) is activated by providing a validation file via the -autotune-validation argument. This allows fastText to automatically find the best parameters based on a specific metric.

Autotune arguments:

• -autotune-validation: Validation file to be used for evaluation.
• -autotune-metric: Metric objective {f1, f1:labelname} [default: f1].
• -autotune-predictions: Number of predictions used for evaluation [default: 1].
• -autotune-duration: Maximum duration in seconds [default: 300].
• -autotune-modelsize: Constraint model file size (leave empty to do not quantize).

fastText offers two unsupervised learning architectures:

1. skipgram: Learns to predict a target word using a nearby context word. It is generally more effective when using subword information.
2. cbow (continuous bag-of-words): Predicts a target word based on the sum of vectors of words in a fixed-size surrounding window.

To train a cbow model specifically:

CLI:

./fasttext cbow -input data/fil9 -output result/fil9

Python:

import fasttext
model = fasttext.train_unsupervised('data/fil9', 'cbow')

Choosing the right loss function depends on your dataset balance and performance requirements:

• Hierarchical Softmax: An approximation of full softmax that allows efficient training on a large number of classes. It may result in a few percent loss in accuracy and is optimized for unbalanced datasets.
• Negative Sampling: If your dataset has a balanced number of examples per class, try negative sampling using the -loss ns and -neg <count> options (e.g., -loss ns -neg 100).

Note: Negative sampling is faster during training but will still be slow at test time because the full softmax must be computed.

The aligned word vectors use the default fastText text format. A .vec file consists of:

1. Header Line: The first line contains two integers: the total number of vectors and the dimensionality of the vectors.
2. Vector Lines: Every subsequent line contains a single word followed by its corresponding vector values, all separated by spaces.

Example structure:

10000 300
apple 0.123 -0.456 ...
banana 0.789 0.012 ...

When training on large datasets with many labels, use the hierarchical softmax loss function to speed up training. This approximates the regular softmax using a Huffman tree, making lookup times optimal for frequent outputs.

CLI flag: -loss hs
Python parameter: loss='hs'

./fasttext supervised -input cooking.train -output model_cooking -lr 1.0 -epoch 25 -wordNgrams 2 -bucket 200000 -dim 50 -loss hs

FastText supervised learning requires a specific input format. Each line in the training file must contain a list of labels followed by the document text.

Crucial Requirement: All labels must start with the __label__ prefix (e.g., __label__cooking). This prefix allows fastText to distinguish between labels and regular words in the document.

Example line format: __label__tag1 __label__tag2 document text goes here...

To represent phrases (like "New York") or entire sentences, use one of the following approaches:

• Bag of Words: Take a bag of words composed of individual word vectors.
• Token Preprocessing: Preprocess the data so that phrases are joined into a single token (e.g., transform "New York" into "New_York").

Pre-trained fastText text models follow a specific structure:

1. Header Line: The first line contains two integers: the number of words in the vocabulary and the size (dimension) of the vectors.
2. Data Lines: Each subsequent line contains a single word followed by its vector values, all space-separated.
3. Ordering: Words are ordered by descending frequency.

Example structure:

[vocab_size] [vector_size]
word1 [v1] [v2] ... [vd]
word2 [v1] [v2] ... [vd]

To measure how well your model performs, use the test command (CLI) or .test() method (Python) on a validation dataset.

Metrics:

• Precision (P@k): The proportion of predicted labels that are correct among the top k predictions.
• Recall (R@k): The proportion of actual labels that were successfully predicted among the top k predictions.

In both CLI and Python, you can specify k to calculate precision and recall for the top $N$ labels.

# CLI: Test top 5 predictions
./fasttext test model_cooking.bin cooking.valid 5

# Python: Test top 5 predictions
# Returns (number of samples, precision, recall)
results = model.test("cooking.valid", k=5)