Criterium Clojure Benchmarking Library

Criterium's streaming statistics implementation is designed to estimate and represent statistical distributions from streaming data samples with minimal storage. The design prioritizes handling uni-modal or bi-modal distributions (approximately log-normal) and supports both quantile queries and density estimation.

While several approaches were evaluated (including Reservoir Sampling, P-square, and Streaming Moments), the project's design direction favors T-digest for its balance of efficiency and accuracy.

Key characteristics of the chosen approach (T-digest):

• High storage efficiency for simple modal patterns.
• Fast quantile queries.
• Robustness against gradual changes in distribution shape.
• Low sensitivity to parameter tuning.

When extraction fails, Criterium uses structured errors to identify exactly where the process broke down. Errors are categorized by a :stage key.

Common extraction stages include:

• :lock-dir-create
• :open-lock
• :acquire-lock
• :create-temp
• :copy
• :verify-hash (triggered if SHA256 integrity check fails)
• :atomic-move
• :set-executable
• :verify-permissions
• :resolve-binary

If an unsupported platform is detected, the system will log a clear error to stderr including the system's os.name and os.arch via platform/describe.

Criterium uses a content-addressed extraction mechanism for its native agent. When an agent is extracted, it is placed in a temporary directory (typically java.io.tmpdir) with a filename derived from its SHA256 hash.

Persistence Model:

• Default Behavior: Extraction is persistent. The extracted binary is NOT deleted when the producing JVM exits. This ensures that the path returned by (jvm-opts) remains valid for subsequent JVM processes.
• Ephemeral Behavior (Opt-in): If you require the agent to be cleaned up automatically, you must explicitly pass the :cleanup-on-exit? true option to the extraction function.
• Deduplication: Because files are named by their SHA256 hash, multiple processes or different versions of the JAR will reuse the same file if the binary is identical, preventing unbounded growth in the temp directory.

Criterium is designed with graceful degradation. If the agent is unavailable (due to an unsupported platform or a missing binary), the library continues to function with reduced capabilities:

• Timing measurements: Basic timing continues to work normally.
• Allocation tracking: Returns empty results.
• API behavior: (agent/loaded?) returns false and (agent/jvm-opts) returns [].
• Logging: A warning is logged on the first attempt to use agent features.

This ensures that your benchmarking code remains portable across all platforms, even if allocation data is unavailable.

Bench plans define how Criterium analyzes and presents data.

• default: The standard plan. Includes JIT warmup, bootstrap confidence intervals, outlier detection, and KDE density estimation.
• histogram: Uses criterium.bench-plans/histogram. Provides non-parametric distribution analysis with Knuth optimal binning, KDE density estimation, and mode detection.
• distribution-analysis: Uses criterium.bench-plans/distribution-analysis. Performs parametric distribution fitting (gamma, log-normal, Weibull), shape statistics (skewness, kurtosis), goodness-of-fit tests, and Q-Q plots.

(require '[criterium.bench-plans :as plans])

;; Histogram analysis
(bench/bench (my-function) :bench-plan plans/histogram)

;; Distribution analysis
(bench/bench (my-function) :bench-plan plans/distribution-analysis)

Criterium uses semantic versioning for tags and conventional commits for automatic changelog generation via git-cliff.

Version Format

Tags follow the pattern vMAJOR.MINOR.PATCH (e.g., v0.5.0, v0.5.1).

Changelog Commit Types

The changelog is automatically categorized based on commit prefixes:

• feat: - Features
• fix: - Bug Fixes
• docs: - Documentation
• perf: - Performance
• refactor: - Refactor
• test: - Testing
• Other commits are categorized as Miscellaneous.

Criterium estimates the time taken by its own measurement overhead and stores it in the criterium.core/estimated-overhead-cache var.

If you notice small negative times for quick functions (often due to high system load during estimation), you can force a recalculation by calling criterium.core/estimated-overhead!. For consistency across different JVM processes, you may choose to set this value to a constant manually.

To ensure the automated release workflow functions correctly, the following prerequisites must be met:

1. GitHub Secrets: The Release environment in the GitHub repository settings must contain the following secrets:
   • CLOJARS_USERNAME
   • CLOJARS_PASSWORD
2. Version Configuration: The version number must be correctly set in build/src/build/version.clj before initiating the release.

To use Criterium in a Leiningen project, add the following dependency to your :dependencies vector in project.clj:

[criterium "0.4.6"]

To perform a basic benchmark in Criterium 0.5.x, require criterium.bench and use the bench macro. This macro handles JVM warmup and provides statistical analysis automatically.

Note: The 0.4.x API criterium.core/bench is deprecated. Always use criterium.bench/bench for new code.

(require '[criterium.bench :as bench])

(bench/bench (+ 1 1))