DOCSEARCHbyAlgolia
Submit Docs

Criterion.rs Documentation

repository·master·Indexed 18 days ago

https://github.com/criterion-rs/criterion.rs

A statistics-driven micro-benchmarking library for Rust that helps developers detect performance improvements or regressions with high accuracy. It features statistical analysis using linear regression and bootstrap sampling, HTML reports, and visual analysis through charts via gnuplot or plotters. The ecosystem includes the criterion-macro for procedural macros, criterion-bencher-compat for bencher crate compatibility, and the experimental cargo-criterion Cargo extension.

Tokens
43K
Snippets
132
Records
195
Agent score
61%

What's inside Criterion.rs

  1. Use the Criterion.rs User Guide

    master

    The User Guide covers a wide range of benchmarking tasks and advanced configurations. Key topics include:

    • Migration: Moving from libtest to Criterion.rs.
    • Output Formats: Understanding Command-Line output, HTML reports, Plots & Graphs, and CSV output.
    • Benchmarking Techniques: Using inputs in benchmarks, comparing functions, benchmarking async functions, and handling timing loops.
    • Advanced Features: Custom measurements, profiling, custom test frameworks, and WebAssembly/WASI support.
    • Configuration: Using Quick Mode and advanced configuration settings.

  2. Convert bencher benchmarks to Criterion.rs using criterion-bencher-compat

    master
    The criterion-bencher-compat crate acts as a shim to facilitate the conversion of most bencher benchmarks into [Criterion.rs] benchmarks. This allows you to leverage Criterion's advanced statistical analysis and reporting features while maintaining compatibility with existing bencher-style benchmark code.

  3. Core Features of Criterion.rs

    master

    Criterion.rs provides the following capabilities for microbenchmarking:

    • Statistical Analysis: Automatically detects performance changes (improvements or regressions) by comparing current results against previous runs using statistical methods.
    • Detailed Charts: Integrates with gnuplot to generate visual graphs of benchmark results.
    • Stable Rust Compatibility: Allows you to run sophisticated benchmarks on the stable Rust toolchain without requiring nightly features.
    • HTML Reports: When the html_reports feature is enabled, it generates detailed web-based reports of your benchmark data.

  4. Interpret Performance Change and Noise

    master

    Criterion.rs compares current runs against data stored in target/criterion. The change output shows a confidence interval for the difference and the p-value indicating the probability the difference occurred by chance.

    Possible outcomes:

    • Performance has improved: Strong statistical evidence of improvement.
    • Performance has regressed: Strong statistical evidence of regression.
    • No change in performance detected: The change was not statistically significant (usually $p > 0.05$).
    • Change within noise threshold: The change is within the configurable noise_threshold (defaults to +-2%).

    Example of a regression:

    change: [+40.772% +43.934% +47.801%] (p = 0.00 < 0.05)
Performance has regressed.
    alloc                   time:   [1.2421 ms 1.2540 ms 1.2667 ms]
                        change: [+40.772% +43.934% +47.801%] (p = 0.00 < 0.05)
                        Performance has regressed.

  5. Understand comparison visualizations: Violin Plots and Line Charts

    master

    When comparing multiple implementations or inputs within a benchmark_group, Criterion generates specific visualizations:

    • Violin Plot: Displays the median execution times and the Probability Density Function (PDF) for each implementation, showing the distribution of performance.
    • Line Chart: Shows how the performance of different functions changes as the input size or input value increases. This is automatically generated when using Criterion::benchmark_group with varying inputs.

  7. How Criterion.rs compares benchmark runs

    master

    In the Comparison phase, Criterion.rs determines if performance has changed compared to the last run using a T-test on bootstrap samples.

    • Statistical Significance: It calculates the probability that the observed difference is due to chance. If the probability is very low, the change is reported as a real optimization or regression.
    • Noise Threshold: To prevent reporting tiny, insignificant fluctuations (like those caused by background OS processes) as meaningful changes, Criterion.rs uses a configurable noise threshold. For example, changes within $\pm 1%$ are often treated as noise and ignored.

    Best Practice: To minimize noise and false positives in comparisons, run benchmarks on a quiet computer where possible.

  8. How Criterion.rs handles outliers

    master

    Criterion.rs uses a modified version of Tukey's Method to classify outliers during the Analysis phase. This helps identify if a benchmark is unreliable due to environmental interference.

    • Outliers: Data points falling below (25th percentile - 1.5 * IQR) or above (75th percentile + 1.5 * IQR), where IQR is the Interquartile Range.
    • Severe Outliers: Data points falling below (25th percentile - 3 * IQR) or above (75th percentile + 3 * IQR).

    Note: Outlier samples are not dropped from the dataset; they are included in all subsequent statistical calculations (like linear regression) to maintain data integrity. If outliers are detected, Criterion.rs will print a warning and generate a plot showing the outlier fences.

  9. Understand the Criterion.rs analysis process

    master

    Criterion.rs follows a four-phase lifecycle for every benchmark to ensure statistical reliability:

    1. Warmup: Executes the routine repeatedly to populate CPU/OS caches and allow JIT compilation to stabilize. This prevents 'cold' starts from skewing results.
    2. Measurement: Collects performance data by executing the routine in multiple samples. Each sample contains an increasing number of iterations to meet the target measurement time.
    3. Analysis: Distills raw samples into meaningful statistics (mean, median, etc.) using linear regression and bootstrap sampling. It also performs outlier classification.
    4. Comparison: Compares the current run's statistics against data from the previous run using a T-test to determine if performance changes are statistically significant or merely noise.

  11. Choosing between Criterion.rs and Iai

    master

    Criterion.rs and Iai are complementary tools designed for different benchmarking needs. For most projects, the recommended approach is to use both.

    When to use Iai

    • CI/CD Environments: Use Iai for benchmarks running in noisy CI environments (e.g., GitHub Actions, Travis-CI) where Criterion.rs may struggle with reliability.
    • Precision: Use Iai when you need to detect very small performance changes.
    • Profiling: Use Iai if you want to generate profile output from the benchmark automatically.
    • Large Benchmarks: Iai scales better to larger benchmarks.
    • Platform Support: Note that Iai requires Valgrind support and cannot be used on Windows.

    When to use Criterion.rs

    • Windows/Non-Valgrind Platforms: Use Criterion.rs on Windows or any platform not supported by Valgrind.
    • Setup Exclusion: Use Criterion.rs if you need to exclude setup/initialization code from your measurements.
    • Statistical Analysis: Use Criterion.rs to measure performance variation caused by OS thread scheduling, hash-table randomization, or other non-deterministic factors.
    • Direct Timing: Use Criterion.rs when you need direct measurements of wall-clock time rather than correlations.

    Summary Comparison Table

  12. Understand the Criterion.rs report file structure

    master

    Benchmark results, raw data, and generated plots are stored in target/criterion/$BENCHMARK_NAME/.

    Key directories include:

    • $BENCHMARK_NAME/base/: Statistics for the last run on the base baseline.
    • $BENCHMARK_NAME/new/: Statistics for the most recent benchmarking run.
    • $BENCHMARK_NAME/report/: Contains the visual output.
      • report/both/: Plots comparing both the base and new runs on a single chart.
      • report/change/: Plots showing the differences (deltas) between the last two runs.
      • report/index.html: The entry point for viewing the HTML report.

    Note: Criterion.rs only maintains historical data for the last run.

    $BENCHMARK_NAME/
├── base/
│  ├── raw.csv
│  ├── estimates.json
│  ├── sample.json
│  └── tukey.json
├── change/
│  └── estimates.json
├── new/
│  ├── raw.csv
│  ├── estimates.json
│  ├── sample.json
│  └── tukey.json
└── report/
   ├── both/
   │  ├── pdf.svg
   │  ├── regression.svg
   │  └── iteration_times.svg
   ├── change/
   │  ├── mean.svg
   │  ├── median.svg
   │  └── t-test.svg
   ├── index.html
   ├── MAD.svg
   ├── mean.svg
   ├── median.svg
   ├── pdf.svg
   ├── pdf_small.svg
   ├── regression.svg (optional)
   ├── regression_small.svg (optional)
   ├── iteration_times.svg (optional)
   ├── iteration_times_small.svg (optional)
   ├── relative_pdf_small.svg
   ├── relative_regression_small.svg (optional)
   ├── relative_iteration_times_small.svg (optional)
   ├── SD.svg
   └── slope.svg