OnlineStats.jl

Many OnlineStats support a weight function $w(t)$ that determines the influence of the $t$-th observation. This allows for different types of moving averages or statistical models:

• Analytical Mean: Use $w(t) = t^{-1}$ (e.g., using the inv function).
• Exponentially Weighted Mean: Use $w(t) = \lambda$ where $0 < \lambda < 1$.

The update formula used is: $\mu^{(t)} = [1 - w(t)] \mu^{(t-1)} + w(t) y_t$.

Because OnlineStat objects can be merged, you can perform embarrassingly parallel computations by splitting data across multiple processes, fitting local statistics on each, and then merging the results.

Important Considerations:

• fit! is generally a cheaper operation than merge!.
• Not all OnlineStat types support merging. If a type does not support merging, OnlineStats may use an approximation or issue a warning that no merging occurred.

To use distributed parallelism, use the @distributed macro with merge to combine results from different processes.

using Distributed
addprocs(3)
@everywhere using OnlineStats

# Use @distributed merge to combine results from multiple processes
s = @distributed merge for i in 1:3
    o = Series(Mean(), Variance(), KHist(20))
    fit!(o, randn(10_000))
end

When using StatLearn, you can customize the estimation process using several components:

1. Fitting Algorithms: You can plug in different algorithms. Some use stochastic gradients (e.g., SGD()), while others use the Majorization-Minimization (MM) principle (e.g., MSPI()) for better numerical stability.
2. Loss and Penalty Functions: You can specify interchangeable loss and penalty functions (e.g., OnlineStats.l2regloss) for both regression and classification.
3. Learning Rate: Controlled via the rate keyword argument.

Note on Tuning: Different algorithms respond differently to the rate parameter. It is recommended to test different combinations of algorithms and learning rates on a sample of your data to find an optimal pairing. You can monitor stability by plotting coefficients over time.

A Series is a collection of OnlineStats that are all updated simultaneously using the same input data. Use Series when you want to monitor several different metrics (e.g., Mean and Variance) for a single sequence of values.

using OnlineStats

y = rand(1000)

# Tracks both Mean and Variance for the same stream 'y'
s = Series(Mean(), Variance())

fit!(s, y)

The StatLearn OnlineStat uses stochastic approximation to estimate models that minimize a loss function plus a penalty function (regularization). It is designed for models of the form:

$$\hat\beta = \argmin_\beta \frac{1}{n} \sum_i f(y_i, x_i'\beta) + \sum_j \lambda_j g(\beta_j)$$

This allows for online estimation of complex models like LASSO regression or logistic regression that would otherwise require $O(n)$ memory in an offline setting.

Key Trade-off: StatLearn provides noisy estimates rather than exact coefficients. This makes online estimation possible for models where sufficient statistics scale with the number of observations, but you sacrifice the precision of offline counterparts.

using OnlineStats

# Example conceptual structure
# StatLearn(algorithm; loss, penalty, rate)
# o = StatLearn(SGD(), OnlineStats.l2regloss; rate=0.8)

A Group is a collection of OnlineStats where each statistic is applied to a different part of the input. This is typically used with an iterator of pairs (e.g., (key, value)), where the key determines which internal statistic to update with the value.

using OnlineStats

# Tracks Mean and a CountMap for different streams identified by a Bool key
g = Group(Mean(), CountMap(Bool))

# Example input: a stream of (value, key) pairs
itr = zip(randn(100), rand(Bool, 100))

fit!(g, itr)

OnlineStats.jl is designed for streaming and big data applications. The core mental model relies on three principles:

• Single-pass: Algorithms process data in one pass, making them suitable for data that is too large to fit in memory or arrives as a stream.
• Constant Memory: Algorithms use $O(1)$ memory, meaning the memory footprint does not grow with the number of observations processed.
• Incremental Updates: Statistics are updated one observation at a time (or in batches), allowing for real-time monitoring of data distributions and models.

The Partition type summarizes sections of a data stream using an OnlineStat. This allows you to plot summaries of massive datasets (e.g., millions of points) without plotting every individual observation.

• Continuous Data: Use Partition with statistics like KHist or Series to visualize distributions or running statistics over time.
• Categorical Data: Use Partition with CountMap to visualize the distribution of categories over time.

# Continuous example
y = cumsum(randn(10^6)) + 100randn(10^6)
o = Partition(KHist(10))
fit!(o, y)
plot(o)

# Categorical example
y = rand(["a", "a", "b", "c"], 10^6)
o = Partition(CountMap(String), 75)
fit!(o, y)
plot(o)

OnlineStats.jl provides high-performance, single-pass algorithms that use $O(1)$ memory. You can create a collection of statistics using Series, update them with single observations or large arrays using fit!, and retrieve the current state using value().

Key workflow:

1. Initialize: Use Series(...) to group multiple statistics together.
2. Update: Use fit!(object, data) to incorporate new data points. This works for both scalars and arrays.
3. Retrieve: Use value(object) to get the current computed values.

using OnlineStats

# Create several statistics
o = Series(Mean(), Variance(), Extrema())

# Update with single data point
fit!(o, 1.0)

# Iterate through and update with lots of data
fit!(o, randn(10^6))

# Get the values of the statistics
value(o)  # Returns a tuple: (value(mean), value(variance), value(extrema))

To use OnlineStats.jl in your Julia project, add it via the Pkg package manager.

import Pkg
Pkg.add("OnlineStats")