Criterion.rs Documentation

criterion-plot is the graphing sub-crate of the [Criterion.rs] benchmarking framework.

Warning: This is an unstable implementation detail of Criterion.rs. The public API and implementation may change at any time without warning.

When using Criterion::benchmark_group to compare implementations or inputs, Criterion automatically generates visual reports:

• Violin Plot: Displays the median times and the Probability Density Function (PDF) for each implementation.
• Line Chart: Shows how different functions perform as the input size increases.

Criterion.rs follows a four-phase process for every benchmark to ensure statistical reliability:

1. Warmup: Executes the routine repeatedly to populate CPU/OS caches and allow JIT compilation to stabilize. This phase is controlled by the warm_up_time parameter in the Criterion struct.
2. Measurement: Collects performance data by executing the routine in multiple samples. Each sample contains an increasing number of iterations to meet the measurement_time requirement. Data is stored locally for future comparisons.
3. Analysis: Distills collected samples into statistics using outlier classification (Tukey's Method) and linear regression via bootstrap sampling to provide confidence intervals.
4. Comparison: Compares current statistics against data from the previous run using a T-test to detect performance regressions or optimizations.

To benchmark asynchronous functions, you must convert the standard bencher to async mode using the .to_async() method. This method requires a futures executor to run the benchmark. The timing loops function identically to synchronous benchmarks.

use criterion::BenchmarkId;
use criterion::Criterion;
use criterion::{criterion_group, criterion_main};

// Import the desired executor
use criterion::async_executor::FuturesExecutor;

async fn do_something(size: usize) {
    // Async logic here
}

fn from_elem(c: &mut Criterion) {
    let size: usize = 1024;

    c.bench_with_input(BenchmarkId::new("input_example", size), &size, |b, &s| {
        // Use `to_async` with an executor to benchmark the async function
        b.to_async(FuturesExecutor).iter(|| do_something(s));
    });
}

criterion_group!(benches, from_elem);
criterion_main!(benches);

To use a custom measurement, you must:

1. Provide the measurement to the Criterion instance using .with_measurement(YourMeasurement).
2. Update your benchmark function signatures to use Criterion<YourMeasurement> instead of the default.
3. Configure your benchmark group using the custom Criterion instance.

Note: As of version 0.3.0, only a single measurement can be used per benchmark.

fn my_benchmark(criterion: &mut Criterion<HalfSeconds>) {
    // Use the criterion struct as normal here.
}

fn alternate_measurement() -> Criterion<HalfSeconds> {
    Criterion::default().with_measurement(HalfSeconds)
}

criterion_group! {
    name = benches;
    config = alternate_measurement();
    targets = my_benchmark
}

When using parameterized benchmarks where input sizes scale exponentially, you can switch from the default linear axis to a logarithmic axis in generated plots. This is done by configuring a PlotConfiguration with AxisScale::Logarithmic and applying it to a BenchmarkGroup via .plot_config().

use criterion::*;

fn do_a_thing(x: u64) {}

fn bench(c: &mut Criterion) {
    let plot_config = PlotConfiguration::default()
        .summary_scale(AxisScale::Logarithmic);

    let mut group = c.benchmark_group("log_scale_example");
    group.plot_config(plot_config);
    
    for i in [1u64, 10u64, 100u64, 1000u64].iter() {
        group.bench_function(BenchmarkId::from_parameter(i), i, |b, i| b.iter(|| do_a_thing(i)));
    }
    group.finish();
}

criterion_group!(benches, bench);
criterion_main!(benches);

Criterion.rs cannot accurately measure only a portion of a function due to the granularity of the system clock.

Recommended Approach: Extract the specific logic into a new function, name it, and benchmark that function. You can use #[inline(always)] to ensure the compiler inlines it back into the original callsite in the final executable.

Advanced/Less Accurate Approach: If you accept reduced accuracy, you can use Bencher::iter_custom to implement your own timing loop. This is useful for complex cases like multi-threaded code, but you are responsible for the measurement accuracy.