DOCSEARCHbyAlgolia
Submit Docs

HierarchicalForecast

repository·main·Indexed 20 days ago

https://github.com/nixtla/hierarchicalforecast

A library for probabilistic hierarchical time series forecasting. It provides a suite of deterministic and probabilistic reconciliation methods—including BottomUp, TopDown, MiddleOut, MinTrace, ERM, Normality, Bootstrap, PERMBU, and Conformal—to ensure forecasts across different levels of aggregation (cross-sectional or temporal) are coherent and consistent.

Tokens
37.6K
Snippets
100
Records
125
Agent score
71%

What's inside hierarchicalforecast

  1. Understand Cross-sectional vs Temporal Hierarchies

    main

    Hierarchical forecasting ensures coherence between different levels of time series. There are two main types of hierarchies:

    1. Cross-sectional hierarchies: Reconcile aggregations across different groups (e.g., product demand $\rightarrow$ product group $\rightarrow$ department $\rightarrow$ store). This uses a summation matrix $S$ and a contribution matrix $P$ to find optimal reconciled forecasts $\tilde{\textbf{Y}}$.
    2. Temporal hierarchies: Reconcile aggregations across different time granularities (e.g., daily $\rightarrow$ weekly $\rightarrow$ monthly). This uses a temporal summation matrix $S_{te}$ and a temporal contribution matrix $P_{te}$.

    Cross-temporal reconciliation is achieved by performing cross-sectional reconciliation followed by temporal reconciliation in a two-step procedure.

  2. How hierarchical forecasting reconciliation works

    main

    Hierarchical forecasting aims to achieve coherency, meaning that forecasts at aggregate levels (e.g., Total, State) precisely sum up to the forecasts of their disaggregate components (e.g., Regions).

    The process follows a two-stage workflow:

    1. Base Forecast: Obtain initial forecasts $\hat{\mathbf{y}}$ for all series (both bottom and aggregate levels) using standard forecasting models (e.g., Naive, ARIMA).
    2. Reconciliation: Apply a reconciliation method to transform these base forecasts into coherent forecasts $\tilde{\mathbf{y}}$ that satisfy the hierarchical constraints.

    Available reconciliation methods include:

    • Deterministic: BottomUp, TopDown, MiddleOut.
    • Optimal/Statistical: MinTrace, ERM.
    • Probabilistic: Normality, Bootstrap, PERMBU.

  3. How to create an aggregation constraints matrix with aggregate()

    main

    To perform hierarchical reconciliation, you must first transform your flat time series data into a hierarchical structure. The hierarchicalforecast.utils.aggregate function creates the necessary hierarchical DataFrames and the aggregation constraints matrix (S_df).

    Workflow:

    1. Define hierarchy_levels: A list of lists where each inner list represents a level of the hierarchy (e.g., [['Country'], ['Country', 'State']]).
    2. Call aggregate(df=df, spec=hierarchy_levels).
    3. The function returns Y_df (the aggregated hierarchical series), S_df (the aggregation constraints matrix), and tags (metadata describing the hierarchy structure).
    from hierarchicalforecast.utils import aggregate

hierarchy_levels = [
    ['Country'],
    ['Country', 'State'],
    ['Country', 'Purpose'],
    ['Country', 'State', 'Region'],
    ['Country', 'State', 'Purpose'],
    ['Country', 'State', 'Region', 'Purpose']
]

Y_df, S_df, tags = aggregate(df=df, spec=hierarchy_levels)

  4. Generate hierarchically coherent probabilistic distributions

    main

    The hierarchicalforecast.probabilistic_methods module provides methods to generate samples of multivariate time series that satisfy hierarchical linear constraints. These methods extend the core HierarchicalForecast capabilities to ensure that probabilistic forecasts are coherent across all levels of a hierarchy.

    Available methods include:

    • Normality
    • Bootstrap
    • PERMBU

    Each of these methods implements a get_samples function to produce the required multivariate samples.

  5. Reconciliation methods in HierarchicalForecast

    main

    HierarchicalForecast provides several categories of reconciliation methods:

    Classic Methods

    • BottomUp: Aggregates forecasts from the bottom level up to the top.
    • TopDown: Distributes forecasts from the top level down through the hierarchy.

    Alternative Methods

    • MiddleOut: Anchors predictions at a middle level; uses BottomUp for levels above and TopDown for levels below.
    • MinTrace: Minimizes total forecast variance using the Minimum Trace approach.
    • ERM: Optimizes the reconciliation matrix by minimizing an L1 regularized objective.

    Probabilistic Coherent Methods

    • Normality: Uses MinTrace variance-covariance under a normality assumption.
    • Bootstrap: Uses Gamakumara's bootstrap approach to generate a distribution of reconciled predictions.
    • PERMBU: Reconciles independent sample predictions by reinjecting multivariate dependence via rank permutation copulas and BottomUp aggregation.
    • Conformal: Provides distribution-free prediction intervals using conformal prediction.

    Temporal Reconciliation

    Most methods (except in-sample methods) are also compatible with temporal hierarchies.

  6. Use external forecast adapters for ML compatibility

    main
    To ensure compatibility with other machine-learning libraries, HierarchicalForecast provides external forecast adapters. These adapters transform output base forecasts from external libraries into the specific dataframe format required by HierarchicalForecast reconciliation methods.

  8. Run HierarchicalForecast baseline experiments

    main

    You can run experiments for specific datasets and probabilistic reconciliation methods using the src/run_baselines.py script.

    Available Parameters

    • --intervals_method: Choose from ['bootstrap', 'normality', 'permbu'].
    • --dataset: Choose from ['Labour', 'Traffic', 'OldTraffic', 'TourismSmall', 'TourismLarge', 'OldTourismLarge', 'Wikitwo'].

    Example Command

    To run the bootstrap method on the OldTourismLarge dataset:

    python src/run_baselines.py --intervals_method 'bootstrap' --dataset 'OldTourismLarge'

  9. How to run HierE2E baseline predictions

    main

    The HierE2E baseline uses the DeepVARHierarchicalEstimator from GluonTS to learn coherent probabilistic forecasts. The process involves:

    1. Loading and processing hierarchical data using HierarchicalDataset.load_process_data.
    2. Converting data into HierarchicalTimeSeries objects for training and testing.
    3. Initializing the DeepVARHierarchicalEstimator with a specific configuration (epochs, learning rate, etc.).
    4. Training the estimator and generating probabilistic samples.
    5. Calculating point forecasts (mean) and quantile forecasts from the samples.
    # Example workflow
DATASET = 'OldTourismLarge'
config = configs[DATASET]
data = HierarchicalDataset.load_process_data(dataset=DATASET)

# run_hiere2e returns (Yq_hat, Y_hat, Y_test, Y_train)
Yq_hat, Y_hat, Y_test, Y_train = run_hiere2e(config, data)

  10. Fit and predict using DeepVARHierarchicalEstimator

    main

    The run_hiere2e function demonstrates how to wrap the GluonTS DeepVARHierarchicalEstimator to perform end-to-end hierarchical forecasting.

    Steps involved:

    1. Initialize Estimator: Configure DeepVARHierarchicalEstimator with parameters like freq, prediction_length, S (structure matrix), and various training hyperparameters (e.g., learning_rate, num_layers, coherent_train_samples).
    2. Train: Call .train(training_data=...) on the estimator.
    3. Predict: Generate probabilistic forecasts using .predict(dataset=...).
    4. Post-process Samples: Convert the forecast iterator into a structured NumPy array of shape [n_items, n_hier, horizon, n_samples] to compute mean forecasts (Y_hat) and quantiles (Yq_hat).
    # Example configuration for DeepVARHierarchicalEstimator
config = dict(
    epochs=25,
    num_batches_per_epoch=50,
    scaling=True,
    pick_incomplete=False,
    batch_size=32,
    num_parallel_samples=200,
    hybridize=False,
    learning_rate=0.0005,
    context_length=24,
    rank=4,
    assert_reconciliation=False,
    num_deep_models=1,
    num_layers=2,
    num_cells=40,
    coherent_train_samples=True,
    coherent_pred_samples=True,
    likelihood_weight=0.0,
    CRPS_weight=1.0,
    num_samples_for_loss=50,
    sample_LH=True,
    seq_axis=[1],
    warmstart_epoch_frac=0.1
)

# Running the process
Yq_hat, Y_hat, Y_test, Y_train = run_hiere2e(config=config, data=data)

  11. Reconcile forecasts using Bottom-Up, Top-Down, and Middle-Out methods

    main

    The HierarchicalForecast package implements classic reconciliation algorithms. These methods can be used to reconcile base forecasts using numpy arrays.

    Available Methods

    • Bottom-Up: Reconciles by aggregating bottom-level forecasts upwards.
    • Top-Down: Reconciles by disaggregating top-level forecasts downwards. Note: TopDown requires strictly hierarchical structures. If the structure is not strictly hierarchical, it will raise an error.
    • Middle-Out: Reconciles by aggregating from a middle level upwards and disaggregating downwards.

    Each method provides fit, predict, fit_predict, and sample interfaces. Sparse versions (e.g., BottomUpSparse, TopDownSparse, MiddleOutSparse) are available for efficiency with large hierarchies.

    # Example of Top-Down reconciliation usage pattern
# Note: This is a conceptual snippet based on the documentation's method signatures
cls_top_down(S=S, y_hat=S @ y_hat_bottom, y_insample=S @ y_bottom, tags=tags)["mean"]

  12. Reconcile forecasts using Min-Trace, Optimal Combination, and ERM

    main

    For more advanced reconciliation that optimizes the contribution of different levels, use these methods:

    • Min-Trace (MinT): Minimizes the trace of the error covariance matrix to find optimal reconciliation weights.
    • Optimal Combination: Combines forecasts from different levels using optimal weights.
    • Empirical Risk Minimization (ERM): Uses empirical risk to determine reconciliation parameters.

    Like the basic methods, these support both standard and sparse implementations (e.g., MinTraceSparse, OptimalCombination, ERM).