critcmp

The primary use case for critcmp is comparing different Criterion baselines. A baseline is a collection of benchmark data produced by a single Criterion run.

To compare two baselines, first save them using Criterion's --save-baseline flag, then pass their names to critcmp.

You can also compare more than two baselines at once.

# 1. Save baselines using Criterion
$ cargo bench -- --save-baseline before
$ cargo bench -- --save-baseline change

# 2. Compare them with critcmp
$ critcmp before change

# Comparing multiple baselines
$ critcmp before change1 change2

If you want to compare benchmarks that have different names but represent the same logic (e.g., optimized/input1 vs naive/input1), you can use the -g flag with a regex containing capturing groups.

critcmp determines the matching criteria based on the values matched by all capturing groups in the regex. Benchmarks with equivalent capturing groups will be grouped into a single comparison.

# Compare benchmarks in the 'benches' baseline by matching their suffixes
# This matches 'optimized/input1' and 'naive/input1' via the 'input1' group
$ critcmp benches -g '\w+/(input1)'

Since critcmp is designed for use with the Criterion benchmark harness in Rust, install it using cargo install.

Note on Compatibility:

• Minimum supported Rust version: The current stable release.
• Criterion compatibility: Known to work with Criterion 0.3.3.
• WARNING: This tool reads undocumented internal data emitted by Criterion. It may break if Criterion's internal data format changes.

$ cargo install critcmp

To hide comparisons where the difference between benchmarks is very small, use the -t or --threshold flag. The value is a percentage.

Example: To hide any comparisons with differences of 5% or less:

critcmp before change -t 5

To see a list of all available baselines known to critcmp within the Criterion directory, use the --baselines flag:

critcmp --baselines

If the comparison results are too large to fit in the default column-oriented display, use the --list flag to flatten the results into lists.

critcmp before change1 change2 --list

The primary use case for critcmp is comparing baselines produced by the Criterion benchmark harness. You can compare two or more baselines by providing their names (as saved by Criterion) or paths to exported JSON files as positional arguments.

Basic Usage

To compare two baselines named before and change:

critcmp before change

To compare multiple baselines:

critcmp before change1 change2

Using Exported JSON Files

You can export a baseline to a JSON file for permanent storage and then use that file path as an argument:

critcmp --export before > before.json
critcmp before.json change

Use the --export flag to print the benchmark data for a specific baseline as JSON to stdout. This is useful for saving data outside of the Criterion target directory.

critcmp --export before > before.json

When comparing baselines, you can use regex flags to refine which benchmarks are included in the output.

Filter benchmarks by name

Use the -f or --filter flag with a regex to only show benchmarks whose names match the pattern:

critcmp before change -f 'foo.*bar'

Group benchmarks by regex capturing groups

Use the -g or --group flag to compare benchmarks that have different names but share common characteristics. This requires a regex with at least one capturing group. Benchmarks are grouped based on the values matched by the capturing groups.

For example, to compare optimized/input1 and naive/input1 from a baseline named benches:

critcmp benches -g '\w+/(input1)'

Use critcmp to manage and persist your benchmark data:

• --baselines: Prints a list of all baselines known to critcmp.
• --export <BASELINE_NAME>: Exports a specific baseline to a JSON file. This allows you to store benchmark data outside of Criterion's target directory and use it later by referencing the file path.

# List available baselines
$ critcmp --baselines

# Export a baseline to JSON
$ critcmp --export before > before.json
$ critcmp --export change > change.json

# Use exported JSON files for comparison
$ critcmp before.json change.json

You can refine the output of your comparisons using the following flags:

• -f, --filter <REGEX>: Limit comparisons to benchmarks that match a specific regular expression.
• -t <THRESHOLD>: Filter out comparisons where the difference is below a certain percentage. For example, -t 5 hides differences of 5% or less.
• --list: If the default column-oriented display is too wide, use this flag to flatten the results into lists.

# Filter by regex
$ critcmp before change -f 'foo.*bar'

# Filter by threshold (e.g., 5%)
$ critcmp before change -t 5

# Flatten output to lists
$ critcmp before change1 change2 --list