LightGBMLSS

LightGBMLSS is an extension of LightGBM designed for probabilistic modelling. It models the full conditional distribution of a univariate target as a function of covariates, allowing for probabilistic forecasts, prediction intervals, and quantiles.

Key capabilities include:

• Distributional Estimation: Estimates all parameters of chosen distributions.
• Complex Distributions: Supports Normalizing Flows (for multi-modal distributions), Mixture-Densities, and Zero-Adjusted/Zero-Inflated distributions (for excess zeros).
• Automatic Differentiation: Uses PyTorch to automatically derive Gradients and Hessians of all distributional parameters.
• Optimization & Explainability: Integrates with Optuna for automated hyper-parameter search and SHAP (SHapley Additive exPlanations) for model interpretability.
• Compatibility: Maintains full compatibility with standard LightGBM features.

LightGBMLSS (LightGBM for Location, Scale and Shape) uses a multi-parameter optimization approach. Instead of predicting a single value, the framework grows a separate tree for each parameter of a specified distribution.

Key characteristics of the modeling process:

• Simultaneous Estimation: Gradients, Hessians, and loss function evaluations are performed simultaneously for all parameters.
• Automatic Differentiation: Gradients and Hessians (the first and second order derivatives of the log-likelihood) are derived using PyTorch's automatic differentiation. This allows for the use of custom parametric distributions or Normalizing Flows without manual derivation.
• Stability via Offsets: To improve convergence and stability, the model uses unconditional Maximum Likelihood estimates of the parameters as offset values.
• Interpretability: The framework provides attribute importance and partial dependence plots using the Shapley-Value (SHAP) approach to help understand the data generating process.

Mixture distributions (or mixture densities) extend univariate modeling by allowing the data to arise from multiple underlying processes. This is useful when a single parametric distribution cannot capture the data, such as when the distribution is multi-modal.

A mixture distribution is a weighted combination of $M$ component distributions:

$$f\bigl(y_{i} | \boldsymbol{\theta}{i}(x{i})\bigr) = \sum_{m=1}^{M} w_{i,m}(x_{i}) \cdot f_{m}\bigl(y_{i} | \boldsymbol{\theta}{i,m}(x{i})\bigr)$$

Key components:

• Component Densities ($f_m$): Each component has its own parameters $\boldsymbol{\theta}_{i,m}$ that depend on covariates.
• Mixing Coefficients ($w_{i,m}$): Weights that indicate the importance of each component, also modeled as functions of covariates.

LightGBMLSS supports combinations like Gaussian-Mixture or StudentT-Mixture, where the same distribution type is used with different parameterizations.

LightGBMLSS supports a wide variety of univariate distributions for probabilistic modeling and uncertainty estimation, built upon PyTorch and Pyro. These distributions are categorized by their type (Continuous, Discrete Count, or Discrete-Continuous) and their support (the range of possible values for $y$).

When selecting a distribution, consider the support of your target variable $y$ and the number of parameters required for the model.

| Distribution | Usage | Type | Support | Number of Parameters |
|:---|:---|:---|:---|:---|
| Beta | `Beta()` | Continuous | $y \in (0, 1)$ | 2 |
| Cauchy | `Cauchy()` | Continuous | $y \in (-\infty,\infty)$ | 2 |
| Expectile | `Expectile()` | Continuous | $y \in (-\infty,\infty)$ | Number of expectiles |
| Gamma | `Gamma()` | Continuous | $y \in (0, \infty)$ | 2 |
| Gaussian | `Gaussian()` | Continuous | $y \in (-\infty,\infty)$ | 2 |
| Gumbel | `Gumbel()` | Continuous | $y \in (-\infty,\infty)$ | 2 |
| Laplace | `Laplace()` | Continuous | $y \in (-\infty,\infty)$ | 2 |
| Logistic | `Logistic()` | Continuous | $y \in (-\infty,\infty)$ | 2 |
| LogNormal | `LogNormal()` | Continuous | $y \in (0,\infty)$ | 2 |
| Mixture | `Mixture(CompDist(), M)` | Continuous & Discrete Count | $y \in (-\infty,\infty)$, $[0, \infty)$, $(0, 1)$, or $[0, 1, 2, \dots)$ | CompDist + M |
| Negative Binomial | `NegativeBinomial()` | Discrete Count | $y \in [0, 1, 2, 3, \dots)$ | 2 |
| Poisson | `Poisson()` | Discrete Count | $y \in [0, 1, 2, 3, \dots)$ | 1 |
| Spline Flow | `SplineFlow()` | Continuous & Discrete Count | $y \in (-\infty,\infty)$, $[0, \infty)$, $(0, 1)$, or $[0, 1, 2, \dots)$ | 2xcount_bins + (count_bins-1) (quadratic) or 3xcount_bins + (count_bins-1) (linear) |
| Student-T | `StudentT()` | Continuous | $y \in (-\infty,\infty)$ | 3 |
| Weibull | `Weibull()` | Continuous | $y \in [0, \infty)$ | 2 |
| Zero-Adjusted Beta | `ZABeta()` | Discrete-Continuous | $y \in [0, 1)$ | 3 |
| Zero-Adjusted Gamma | `ZAGamma()` | Discrete-Continuous | $y \in [0, \infty)$ | 3 |
| Zero-Adjusted LogNormal | `ZALN()` | Discrete-Continuous | $y \in [0, \infty)$ | 3 |
| Zero-Inflated Negative Binomial | `ZINB()` | Discrete-Count | $y \in [0, 1, 2, 3, \dots)$ | 3 |
| Zero-Inflated Poisson | `ZIPoisson()` | Discrete-Count | $y \in [0, 1, 2, 3, \dots)$ | 2 |

To find the best-fitting distribution for a univariate target variable, use the dist_select method from DistributionClass.

1. Define a list of candidate distribution classes (e.g., Gaussian, StudentT, Gamma).
2. Call dist_select passing your training target data and the list of candidates.
3. The method returns the negative log-likelihood (NLL) for each distribution. The distribution with the lowest NLL is considered the best fit.
4. If plot=True is passed, the method will visualize the density of the target variable against the fitted density of the best-selected distribution.

Note on selection: Choose candidate distributions based on the nature of your data. For count data, include Poisson or NegativeBinomial. For positive real-scale data, select appropriate continuous distributions.

from lightgbmlss.distributions import *
from lightgbmlss.distributions.distribution_utils import DistributionClass

# Initialize the distribution utility class
lgblss_dist_class = DistributionClass()

# Define candidate distributions
candidate_distributions = [Gaussian, StudentT, Gamma, Cauchy, LogNormal, Weibull, Gumbel, Laplace]

# Select the best distribution based on negative log-likelihood
dist_nll = lgblss_dist_class.dist_select(
    target=y_train, 
    candidate_distributions=candidate_distributions, 
    max_iter=50, 
    plot=True, 
    figure_size=(10, 5)
)

print(dist_nll)