StatsBase.jl Documentation

StatsBase.jl is a Julia package providing fundamental statistical support. It is designed to handle a wide range of common statistical operations on data, including:

• Scalar statistics: Basic descriptive statistics.
• High-order moments: Computation of moments beyond the mean and variance.
• Counting and Ranking: Frequency counts and order statistics.
• Covariances: Measuring relationships between variables.
• Sampling: Methods for drawing samples from data.
• Empirical density estimation: Estimating the underlying probability density function of a dataset.

StatsBase.jl provides fundamental statistical support in Julia. Its core capabilities include:

• Scalar statistics: Basic descriptive statistics.
• High-order moments: Computation of moments beyond the mean and variance.
• Counting and Ranking: Frequency counts and order statistics.
• Covariances: Measuring relationships between variables.
• Sampling: Various methods for drawing samples from distributions or datasets.
• Empirical density estimation: Estimating probability density functions.
• Other features: Robust statistics, deviation measures, signal correlation, statistical models, and data transformations.

The RegressionModel abstract type is a subtype of StatisticalModel. It extends the base statistical interface with methods specifically required for regression analysis, such as handling model matrices, responses, and predictions.

Additional methods implemented by RegressionModel include:

• Data Access: modelmatrix, crossmodelmatrix, response, responsename.
• Predictions & Fitted Values: predict, predict!, fitted, meanresponse.
• Residuals & Diagnostics: residuals, leverage, cooksdistance, dof_residual.

When working with low-level sampling functions in StatsBase, the following notation is used:

• a: The source array representing the population.
• x: The destination array (must be pre-allocated).
• wv: The weight vector (of type AbstractWeights) used for weighted sampling.
• n: The length of the source array a.
• k: The length of the destination array x. For sampling without replacement, k must not exceed n.
• rng: An optional random number generator. It defaults to Random.default_rng() on Julia >= 1.3 and Random.GLOBAL_RNG on older versions.

The CovarianceEstimator is an abstraction used to define how covariance, variance, and standard deviation are calculated, particularly when weights are involved.

Commonly used types include:

• CovarianceEstimator: The abstract type for covariance estimation strategies.
• SimpleCovariance: A specific implementation of a covariance estimator.

You can use these estimators with the following functions:

• cov(estimator, x)
• cov(estimator, x, w)
• cov(estimator, X)
• var(estimator, x)
• std(estimator, x)

In StatsBase.jl, statistical models are represented by the StatisticalModel abstract type (defined in StatsAPI.jl). Any object that implements the StatisticalModel interface can be used with standard statistical diagnostic and estimation methods.

Key methods implemented by StatisticalModel include:

• Estimation & Fitting: fit, fit!, weights.
• Coefficients: coef, coefnames, coeftable, stderror, confint, vcov.
• Goodness of Fit & Information Criteria: aic, aicc, bic, r2, adjr2, loglikelihood, deviance, nulldeviance, nullloglikelihood.
• Degrees of Freedom & Observations: dof, nobs.
• Residuals & Sum of Squares: rss, mss.
• Other Diagnostics: informationmatrix, isfitted, islinear, score.

The sample and sample! methods act as a poly-algorithm. Instead of forcing a single method, they automatically select the most efficient underlying algorithm based on your input parameters (such as population size, sample size, and whether replacement is allowed).

This selection is based on extensive benchmarking. While sample is optimized for most general use cases, if you have specific knowledge that a particular algorithm is better suited for your data distribution or constraints, you can call the specific internal algorithm functions directly to avoid the selection overhead.

StatsBase.jl uses the AbstractWeights type to represent weight vectors. This abstraction serves two primary purposes:

1. Type Distinction: It distinguishes weight vectors from other data vectors in function arguments.
2. Efficiency: The weight vector maintains its own sum of weights, which is computed during construction. This prevents statistical functions from having to re-calculate the sum repeatedly.

Key Implementation Details:

• Lightweight: Weight vectors are lightweight wrappers around the input vector; the input vector is not copied during construction.
• Pre-computed Sums: If you have already calculated the sum of weights, you can pass it as the second argument to the constructor to avoid redundant computation.

StatsBase.jl provides functions to compute the scatter matrix and various forms of covariance matrices, including support for weighted covariance using the CovarianceEstimator abstraction.

Key functions include:

• scattermat: Computes the scatter matrix.
• cov: Computes the covariance matrix. It supports several method signatures including estimation via a CovarianceEstimator for vectors or matrices.
• var and std: Compute variance and standard deviation using a CovarianceEstimator.
• cor: Computes the correlation matrix.
• mean_and_cov: Computes both the mean and the covariance matrix in a single pass.
• cov2cor / cor2cov: Convert between covariance and correlation matrices.

To use weighted estimation, you can pass a CovarianceEstimator (such as SimpleCovariance) to the covariance, variance, or standard deviation functions.

You can install StatsBase.jl using the Julia package manager (Pkg).

Via Julia REPL (Package Mode):

1. Enter package mode by typing ].
2. Run add StatsBase.

Via Julia Code: Use the Pkg.add function within a Julia script or REPL session.

using Pkg
Pkg.add("StatsBase")

StatsBase.jl provides several functions for calculating robust statistics, which are less sensitive to outliers than standard measures. These functions allow you to 'trim' or 'winsorize' data to mitigate the influence of extreme values.

Available functions:

• trim(x) / trim!(x): Returns a version of the data with a specified proportion of the smallest and largest values removed.
• winsor(x) / winsor!(x): Replaces extreme values with the nearest non-extreme values (winsorization) rather than removing them.
• trimvar(x): Calculates a robust version of the variance.

# Example usage (conceptual based on available symbols)
trim(data)
winsor(data)
trimvar(data)

StatsBase.jl provides a suite of functions to compute various statistical deviations and distances between two arrays. These functions are implemented to be memory-efficient, avoiding the creation of temporary intermediate arrays during computation.

Available deviation functions include:

• counteq: Counts the number of elements that are equal.
• countne: Counts the number of elements that are not equal.
• sqL2dist: Squared L2 distance.
• L2dist: L2 distance (Euclidean distance).
• L1dist: L1 distance (Manhattan distance).
• Linfdist: L-infinity distance (Chebyshev distance).
• gkldiv: Generalized Kullback-Leibler divergence.
• meanad: Mean absolute deviation.
• maxad: Maximum absolute deviation.
• msd: Mean squared deviation.
• rmsd: Root mean squared deviation.
• psnr: Peak Signal-to-Noise Ratio.