OCTIS Documentation

The EarlyStopping class in pytorchtool.py is a regularization tool designed to prevent overfitting in PyTorch models. It monitors validation loss during training and terminates the training loop if the loss fails to decrease for a specified number of epochs.

Key features:

• Validation Monitoring: Tracks validation loss across epochs.
• Checkpointing: Automatically saves a model checkpoint whenever the validation loss decreases.
• Patience Control: The patience argument determines how many consecutive epochs to wait for an improvement in validation loss before stopping the training.

To use it, instantiate the class and call its methods within your training loop to evaluate the loss and decide whether to continue or stop.

# Conceptual usage pattern
# (Refer to MNIST_Early_Stopping_example.ipynb for a full implementation)
from octis.models.early_stopping.pytorchtool import EarlyStopping

# patience: number of epochs to wait after last improvement
esp = EarlyStopping(patience=20)

# Inside training loop:
# esp.check(validation_loss, model)

To add a new model to OCTIS, your class must inherit from AbstractModel (defined in octis/models/model.py).

Implementation Steps

1. Define Hyperparameters: Create a dictionary of default hyperparameter values. This allows users to override only specific parameters.
2. Override train_model: Implement the train_model(self, dataset, hyperparameters={}, top_words=10) method.

Required Return Format

The train_model method must return a dictionary containing at least:

• topics: A list of lists of strings (the most significant words for each topic).
• topic-word-matrix: An $N imes V$ matrix of weights ($N$ = topics, $V$ = vocabulary length).
• topic-document-matrix: An $N imes D$ matrix of weights ($N$ = topics, $D$ = number of documents).

If your model supports training/test partitioning, you should also return:

• test-topic-document-matrix: The document-topic matrix for the test set.

# Example hyperparameter definition
hyperparameters = {
    'corpus': None, 
    'num_topics': 100, 
    'id2word': None, 
    'alpha': 'symmetric', 
    'eta': None, 
    'callbacks': None
}

# Example method signature to override
def train_model(self, dataset, hyperparameters={}, top_words=10):
    # ... implementation ...
    return {
        'topics': list_of_lists, 
        'topic-word-matrix': matrix_nv, 
        'topic-document-matrix': matrix_nd
    }

To optimize a topic model, you must provide a dataset, an evaluation metric, and a search space defined using scikit-optimize types. The Optimizer class handles the optimization process.

Optimization Workflow

1. Define Search Space: Use skopt.space.space types (like Real) to define the range for hyperparameters. Refer to the specific topic model's initialization signature to identify which hyperparameters are available.
2. Initialize Optimizer: Create an instance of octis.optimization.optimizer.Optimizer.
3. Run Optimization: Call .optimize() with your model, dataset, metric, and search space.
4. Save Results: Use .save_to_csv() on the resulting object to persist the optimization history.

Visualization

To visualize the optimization process, set the plot attribute of the Bayesian_optimization object to True.

from octis.optimization.optimizer import Optimizer
from skopt.space.space import Real

# Define the search space using scikit-optimize types
search_space = {"alpha": Real(low=0.001, high=5.0), "eta": Real(low=0.001, high=5.0)}

# Initialize an optimizer object and start the optimization.
optimizer = Optimizer()
optResult = optimizer.optimize(
    model, 
    dataset, 
    eval_metric, 
    search_space, 
    save_path="../results", # path to store the results
    number_of_call=30,       # number of optimization iterations
    model_runs=5             # number of runs of the topic model
)

# Save the results of the optimization in a csv file
optResult.save_to_csv("results.csv")