DOCSEARCHbyAlgolia
Submit Docs

OpenLTM Documentation

repository·main·Indexed 19 days ago

https://github.com/thuml/openltm

An open codebase providing a pipeline to develop, pre-train, and evaluate Large Time-Series Models (LTMs) or Time Series Foundation Models (TSFMs). It includes support for supervised training, large-scale pre-training on datasets like UTSD and ERA5-Family, and model adaptation. The repository provides tools for implementing custom models and instructions for loading and using Timer and Timer-XL checkpoints for zero-shot forecasting and fine-tuning.

Tokens
1.5K
Snippets
4
Records
7
Agent score
19%

What's inside OpenLTM

  1. Develop a custom large time-series model

    main

    To add a new model to the OpenLTM pipeline, follow these steps:

    1. Define the model: Add your model implementation file to the ./models folder (refer to ./models/timer_xl.py for a template).
    2. Register the model: Include your new model in the Exp_Basic.model_dict located in ./exp/exp_basic.py.
    3. Create execution scripts: Add corresponding training/evaluation scripts under the ./scripts folder.

  2. Prepare datasets for OpenLTM

    main

    Place all downloaded datasets in the ./dataset directory. Depending on your task, you will need different data sources:

    • Univariate Pre-training: Use UTSD (1 billion time points) or ERA5-Family for domain-specific models.
    • Supervised Training or Model Adaptation: Use datasets from TSLib.

  3. How to load and use Timer checkpoints

    main
    For the 260B pre-trained Timer in PyTorch, you can use the provided notebook to learn how to load and use the checkpoint for fine-tuning. This version is designed to be more user-friendly for fine-tuning than the standard HuggingFace model.

  5. Run training and adaptation scripts

    main

    OpenLTM provides pre-configured bash scripts in the ./scripts/ directory for various workflows. You can run these scripts to perform supervised training, large-scale pre-training, or model adaptation.

    Supervised Training

    • One-for-one forecasting: bash ./scripts/supervised/forecast/moirai_ecl.sh
    • One-for-all (rolling) forecasting: bash ./scripts/supervised/rolling_forecast/timer_xl_ecl.sh

    Large-scale Pre-training

    • Pre-training on UTSD: bash ./scripts/pretrain/timer_xl_utsd.sh
    • Pre-training on ERA5: bash ./scripts/pretrain/timer_xl_era5.sh

    Model Adaptation

    • Full-shot fine-tune: bash ./scripts/adaptation/full_shot/timer_xl_etth1.sh
    • Few-shot fine-tune: bash ./scripts/adaptation/few_shot/timer_xl_etth1.sh
    # Example: Supervised training (one-for-one forecasting)
bash ./scripts/supervised/forecast/moirai_ecl.sh

# Example: Model adaptation (full-shot fine-tune)
bash ./scripts/adaptation/full_shot/timer_xl_etth1.sh

  6. Initialize and load a pre-trained Timer-XL model

    main

    To use a pre-trained Timer-XL model, you must initialize an argparse.Namespace object with the specific architecture hyperparameters used during training, then instantiate the timer_xl.Model and load the checkpoint.

    Note: You must download the checkpoint from this URL before loading it.

    Required architecture arguments:

    • input_token_len: Length of input tokens (e.g., 96).
    • output_token_len: Length of output tokens (e.g., 96).
    • d_model: Model dimension (e.g., 1024).
    • n_heads: Number of attention heads (e.g., 8).
    • e_layers: Number of encoder layers (e.g., 8).
    • d_ff: Feed-forward dimension (e.g., 2048).
    • dropout: Dropout rate (e.g., 0.1).
    • activation: Activation function (e.g., 'relu').
    • use_norm: Boolean for normalization.
    • flash_attention: Boolean for flash attention.
    • covariate: Boolean for covariate usage.
    • output_attention: Boolean for output attention.
    import torch
import argparse
from models import timer_xl

args = argparse.Namespace()
args.input_token_len = 96
args.output_token_len = 96
args.d_model = 1024
args.n_heads = 8
args.e_layers = 8
args.d_ff = 2048
args.dropout = 0.1
args.activation = 'relu'
args.use_norm = True
args.flash_attention = False
args.covariate = False
args.output_attention = False

model = timer_xl.Model(args)
model.load_state_dict(torch.load('checkpoint.pth'))

  7. Perform zero-shot forecasting with Timer-XL

    main

    Timer-XL can perform zero-shot forecasting by passing a lookback window of time-series data into the model.

    Important Implementation Detail: The model output represents a sequence of next-token predictions. To obtain the final forecast for a specific prediction length (e.g., 96), you must select the last token_len tokens from the output sequence.

    1. Prepare input: Convert your time-series data into a torch tensor of shape (1, lookback_length, 1).
    2. Forward pass: Call model(input.unsqueeze(-1), None, None).
    3. Extract prediction: Slice the output to get the last 96 tokens: output[:, -96:, 0].
    # Assuming 'model' is initialized and 'df' is a pandas DataFrame with an 'OT' column
lookback_length = 1440
prediction_length = 96

# Prepare input tensor (1, L, 1)
input_tensor = torch.tensor(df["OT"][:lookback_length]).unsqueeze(0).float()

# Generate forecast
# Note: input is unsqueezed to (1, L, 1) for the model
output = model(input_tensor.unsqueeze(-1), None, None)

# Extract the last 96 tokens as the final prediction
pred = output[:, -96:, 0].squeeze().detach().numpy()