Lag-Llama: Foundation Models for Probabilistic Time Series Forecasting

Lag-Llama is an open-source foundation model designed for probabilistic time series forecasting. It is capable of zero-shot forecasting on datasets of any frequency and any prediction length. It also supports fine-tuning on specific datasets to improve performance.

Key capabilities:

• Zero-shot forecasting: Predict future values without specific training on your target dataset.
• Fine-tuning: Adapt the model to your specific data for maximum performance.
• Probabilistic output: The model outputs a probability distribution for each predicted timestep rather than a single point estimate.

To achieve maximum performance or conduct accurate benchmarking during fine-tuning, consider the following:

• Hyperparameter Tuning: Specifically tune the context length and the learning rate for each dataset.
  • Suggested Context Lengths: 32, 64, 128, 256, 512, 1024.
  • Suggested Learning Rates: 1e-2, 5e-3, 1e-3, 5e-4, 1e-4.
• Early Stopping: Use a validation split of your dataset to implement early stopping. A recommended patience is 50 epochs to prevent overfitting.

When using Lag-Llama for zero-shot forecasting, follow these recommendations to optimize performance:

• Tune Context Length: Do not rely solely on the training context length of 32. Test different context lengths to find what works best for your specific data. Performance generally improves as context length increases until a data-specific threshold is reached.
• Enable RoPE Scaling: If you use context lengths larger than what the model was originally trained on, it is highly recommended to enable Rotary Positional Embedding (RoPE) scaling to maintain accuracy.

To replicate the experiments presented in the Lag-Llama paper, use the provided shell scripts:

1. Pretraining: Use scripts/pretrain.sh to replicate the pretraining setup.
2. Finetuning: Once a model is pretrained, use scripts/finetune.sh to replicate the finetuning setup used in the paper.