DOCSEARCHbyAlgolia
Submit Docs

hyperfine

repository·master·Indexed 12 days ago

https://github.com/sharkdp/hyperfine

A command-line benchmarking tool version 1.20.0 that provides statistical analysis across multiple runs, support for arbitrary shell commands, and outlier detection. It features parameterized benchmarks, a structured lifecycle (setup, prepare, conclude, cleanup), and the ability to export results to JSON, CSV, Markdown, AsciiDoc, and org-mode.

Tokens
7.9K
Snippets
39
Records
44
Agent score
97%

What's inside hyperfine

  1. Install hyperfine

    master

    Hyperfine can be installed on various platforms using package managers or from source.

    Linux

    • Ubuntu/Debian: apt install hyperfine or download the .deb from the Release page.
    • Fedora: dnf install hyperfine
    • Arch Linux: pacman -S hyperfine
    • Alpine Linux: apk add hyperfine
    • NixOS: nix-env -i hyperfine
    • openSUSE: zypper install hyperfine
    • Void Linux: xbps-install -S hyperfine

    macOS

    • Homebrew: brew install hyperfine
    • MacPorts: sudo port install hyperfine

    Windows

    • Chocolatey: choco install hyperfine
    • Scoop: scoop install hyperfine
    • Winget: winget install hyperfine

    Other

    • Cargo (Rust): cargo install --locked hyperfine (Requires Rust 1.76+)
    • Conda: conda install -c conda-forge hyperfine
    # Example: Installing via apt on Ubuntu
apt install hyperfine

# Example: Installing via Homebrew on macOS
brew install hyperfine

# Example: Installing via Cargo
cargo install --locked hyperfine

  2. Install dependencies for hyperfine visualization scripts

    master

    The visualization scripts (like plot_whisker.py) require numpy, matplotlib, and scipy to function.

    Using PEP-723 compliant runners (Recommended)

    If you use a tool like uv or pipx that supports inline script requirements, you can run the scripts directly without manual dependency management:

    uv run plot_whisker.py sleep.json

    Using pip

    Alternatively, install the required packages manually using pip or your system package manager:

    pip install numpy matplotlib scipy

  3. How the benchmark lifecycle works

    master

    A Benchmark in hyperfine follows a structured lifecycle to ensure accurate measurements. The process includes:

    1. Setup: Runs the command specified by --setup. This is run once before any other phase.
    2. Warmup (Optional): If --warmup is configured, hyperfine runs the command multiple times to prime caches or stabilize the environment. Each warmup run is wrapped by preparation and conclusion commands.
    3. Initial Measurement: A single run is performed to estimate the execution time. This estimate is used to calculate how many subsequent runs are needed to satisfy the --min-run-time requirement.
    4. Main Benchmarking Loop: Hyperfine executes the command for the calculated number of iterations. Each iteration is preceded by a Preparation command (--prepare) and followed by a Conclusion command (--conclude).
    5. Cleanup: Finally, the command specified by --cleanup is run once after all benchmarks and measurements are complete.

    Note on Errors: If a setup, preparation, or conclusion command fails (returns a non-zero exit code), hyperfine will raise an error and stop. If you want to ignore failures in these intermediate commands, append || true to the command string.

    // The lifecycle follows this sequence:
// 1. Setup
// 2. Warmup (if configured)
// 3. Initial measurement
// 4. Loop: Preparation -> Benchmark -> Conclusion
// 5. Cleanup

  4. Represent a command to be benchmarked

    master

    In hyperfine, a Command represents a single shell command that will be executed during a benchmark. A command consists of a name (optional), an expression (the actual shell command string), and zero or more parameters used for substitution.

    Key behaviors:

    • Parameter Substitution: Parameters are defined using curly braces (e.g., {param_name}). When the command is resolved, these placeholders are replaced by their corresponding values.
    • Command Resolution: The get_command_line() method performs the substitution to produce the final string that will be executed.
    • Execution: The get_command() method parses the resolved command line using shell-word splitting and returns a std::process::Command ready for execution.
    // Conceptual usage of the Command structure
// A command with parameters: 'echo {foo} {bar}'
// Parameters: foo='baz', bar='quux'
// Resolved command line: 'echo baz quux'

  5. Configure the shell used for benchmarks

    master

    By default, hyperfine uses sh on non-Windows systems and cmd.exe on Windows. You can specify a custom shell using the --shell flag. The shell string is parsed using shell-word splitting, allowing you to pass arguments to the shell itself (e.g., bash -c).

    If you use --no-shell, hyperfine will execute commands directly (Raw mode) without a shell wrapper.

    # Example: Using a specific shell with arguments
# hyperfine --shell "bash -c" "your-command"

  6. Manage lifecycle with setup, prepare, conclude, and cleanup

    master

    Hyperfine allows you to execute commands at different stages of the benchmarking lifecycle to manage state or clear caches.

    • --setup CMD: Execute CMD once before the entire set of timing runs for a command (e.g., compiling software).
    • --prepare CMD: Execute CMD before each individual timing run (e.g., clearing disk caches).
    • --conclude CMD: Execute CMD after each individual timing run.
    • --cleanup CMD: Execute CMD once after all timing runs for a specific command are completed (e.g., removing artifacts).
    hyperfine -L n 1,2 -r 2 --show-output \
	--setup 'echo setup n={n}' \
	--prepare 'echo prepare={n}' \
	--conclude 'echo conclude={n}' \
	--cleanup 'echo cleanup n={n}' \
	'echo command n={n}'

  7. Perform parameterized benchmarks

    master

    Vary command arguments dynamically using placeholders in the format {VAR}.

    Parameter Scan

    Use --parameter-scan VAR MIN MAX to run benchmarks for every value in the range MIN..MAX.

    • Use --parameter-step-size DELTA to control the increment.
    • You can use shell arithmetics within the command to create non-linear patterns (e.g., power of 2).

    Parameter List

    Use --parameter-list VAR VALUES to run benchmarks for a comma-separated list of specific values.

    • If specified multiple times, hyperfine runs all possible combinations of the provided parameters.
    # Parameter scan (linear)
hyperfine -P threads 1 8 'make -j {threads}'

# Parameter scan with step size
hyperfine -P delay 0.3 0.7 -D 0.2 'sleep {delay}'

# Parameter scan with shell arithmetic (power of 2)
hyperfine -P size 0 3 'sleep $((2**{size}))'

# Parameter list
hyperfine -L compiler gcc,clang '{compiler} -O2 main.cpp'

  8. Run hyperfine benchmarks via CLI

    master

    hyperfine is a command-line benchmarking tool. It executes commands multiple times to provide statistical analysis of their performance. The tool handles warmup runs, parameterization, and exporting results to various formats.

    When running hyperfine, the execution flow follows these steps:

    1. Argument Parsing: CLI arguments are parsed into Options and Commands.
    2. Validation: The provided commands are validated against the options.
    3. Scheduling: A Scheduler is initialized with the commands, options, and an ExportManager.
    4. Execution: The scheduler runs the benchmarks, performs relative speed comparisons, and performs the final export of results.
    # Example of how hyperfine is typically invoked (based on CLI entrypoint logic)
hyperfine 'command1' 'command2'

  9. Configure benchmark runs and warmup

    master

    Control the number of times a command is executed to ensure statistical significance or to prepare the environment.

    • --warmup NUM: Perform NUM warmup runs before the actual benchmark (useful for filling disk caches).
    • --min-runs NUM: Perform at least NUM runs (default: 10).
    • --max-runs NUM: Perform at most NUM runs.
    • --runs NUM: Perform exactly NUM runs.
    hyperfine --warmup 3 'grep -R TODO *'

# Specify exact number of runs
hyperfine --min-runs 5 'sleep 0.2' 'sleep 3.2'

  10. Use Parameter Scanning to benchmark ranges

    master

    Parameter scanning allows you to benchmark a command across a numeric range. This is useful for testing how performance scales with input size or specific numeric configurations.

    When using --parameter-scan <name> <min> <max> [--parameter-step-size <step>]:

    • The <name> is the placeholder used in the command (e.g., {val}).
    • The range can be defined using integers (e.g., 1 10 1) or decimals (e.g., 0.0 1.0 0.1).
    • If a step size is not provided, it defaults to 1 for integers.
    • For decimal scans, a step size is required.
    # Example of a parameter scan via CLI
hyperfine 'echo {val}' --parameter-scan val 1 5 --parameter-step-size 1

# This generates benchmarks for:
echo 1
echo 2
echo 3
echo 4
echo 5

  11. Run basic benchmarks with hyperfine

    master

    To benchmark a single command or compare multiple commands, pass the commands as arguments to hyperfine. By default, it performs statistical analysis across multiple runs and provides constant feedback on progress.

    hyperfine 'find . -name todo.txt'

# Compare multiple commands
hyperfine 'sleep 0.2' 'sleep 3.2'

  12. Visualize benchmark results with plot_whisker.py

    master

    You can visualize benchmark data by exporting hyperfine results to a JSON file and passing that file to the plot_whisker.py script. This script generates whisker plots from the exported data.

    Workflow:

    1. Run hyperfine with the --export-json flag to create a JSON report.
    2. Execute plot_whisker.py pointing to that JSON file.
    hyperfine 'sleep 0.020' 'sleep 0.021' 'sleep 0.022' --export-json sleep.json
./plot_whisker.py sleep.json