XGBoostLSS

XGBoostLSS uses a one vs. all estimation strategy. This means a separate tree is grown for every distributional parameter being estimated.

Runtime Calculation: Total trees trained = [number of iterations] * [number of distributional parameters]

Implications:

• Univariate distributions: Runtime is higher than standard XGBoost (which only trains [number of iterations] trees).
• Multivariate targets: Estimation can become computationally expensive as the number of parameters increases with the number of targets and their dependencies.

Generalized Additive Models for Location, Scale, and Shape (GAMLSS) is a framework for distributional modeling where a univariate response $y$ follows a distribution $\mathcal{D}(\boldsymbol{\theta}(x))$ that depends on covariates.

In this framework, the distributional parameters $\boldsymbol{\theta}(x)$ are modeled as functions of explanatory variables. A typical parameter set includes:

• Location ($\mu$): The center of the distribution.
• Scale ($\sigma^2$): The spread or variance.
• Shape ($\nu, \tau$): Parameters controlling skewness and kurtosis.

Unlike Generalized Linear Models (GLM), GAMLSS relaxes the assumption of the exponential family, allowing for highly skewed, kurtotic, discrete, or zero-inflated distributions. XGBoostLSS extends this by using tree-based models to estimate these parameter functions $\eta_{k} = f_{k}(\mathbf{x})$.

When parametric distributions (like Gaussian or Poisson) are not flexible enough to approximate your dataset—for example, when dealing with multi-modal distributions—you can use Normalizing Flows in XGBoostLSS.

Normalizing Flows approximate the data non-parametrically by transforming a simple base distribution (e.g., $N(0,1)$) into a complex target distribution through a series of bijective transformations. This approach allows for fitting highly complex distributions with relatively few parameters.

In XGBoostLSS, the implementation is based on element-wise rational splines (linear or quadratic order), providing a balance of functional flexibility and numerical stability.

Traditional regression models often focus only on estimating the conditional mean $\mathbb{E}(Y|\mathbf{X} = \mathbf{x})$, which implicitly assumes a Normal distribution with constant variance. Distributional Gradient Boosting Machines (DGBM) extend this by modeling the entire conditional distribution $F_{Y}(y|\mathbf{x})$.

This approach is useful when data exhibits:

• Heteroskedasticity: Varying levels of variance.
• Skewness: Asymmetric distributions.
• Kurtosis: Varying degrees of 'tailedness'.
• Multimodality: Data arising from multiple underlying processes.

By modeling all distributional parameters (location, scale, shape, etc.) as functions of covariates, DGBM provides a more complete picture of uncertainty and the data generating process compared to standard point-forecast models.

XGBoostLSS supports multivariate distributional regression to model dependencies in $D$-dimensional responses $\mathbf{y} = (y_{i1}, \ldots, y_{iD})^{T}$.

Instead of modeling each dimension independently, the framework relates all parameters $\theta_{k}$ of a multivariate density to the covariates $\mathbf{x}$. Common modeling approaches include:

• Multivariate Gaussian: Modeling the conditional mean vector $\mu_{\mathbf{x}}$ and a positive definite symmetric covariance matrix $\Sigma_{\mathbf{x}}$.
• Bivariate Modeling: For $D=2$, the covariance matrix $\Sigma_{\mathbf{x}}$ is parameterized using variances $\sigma^2$ and correlations $\rho$.
• Covariance Approximations: Using Cholesky Decomposition or Low-Rank Covariance approximations to manage the covariance matrix structure.

For practical guidance and code examples on how to implement the framework, refer to the official example documentation. A common starting point is the Gaussian Regression example.

Detailed usage guides can be found in the example section.

XGBoostLSS optimizes Gradients and Hessians for distributional parameters. If these values have vastly different magnitudes, convergence may be unstable or slow.

To improve convergence:

1. Built-in Stabilization: XGBoostLSS includes internal stabilization for Gradients and Hessians.
2. Response Standardization: For datasets where the response range differs significantly from the Gradient/Hessian ranges, manually standardize the continuous response variable (e.g., by dividing it by a constant like y/100).

Evaluate both techniques based on your specific dataset to ensure stable training.

You can install XGBoostLSS via PyPI for the stable version, or directly from the GitHub repository for the development version.

Install stable version from PyPI

pip install xgboostlss

Install development version from GitHub

pip install git+https://github.com/StatMixedML/XGBoostLSS.git

You can tune any XGBoost hyperparameter using the xgblss.hyper_opt method. The param_dict must follow a specific structure based on the parameter type:

1. Float/Int: {"param_name": ["sample_type", low, high, log]}
   • sample_type: `