DOCSEARCHbyAlgolia
Submit Docs

BenchmarkTools.jl

repository·main·Indexed 20 days ago

https://github.com/juliaci/benchmarktools.jl

A framework for writing, running, and comparing groups of benchmarks in Julia. It provides tools for detailed performance analysis via the `@benchmark` macro, quick sanity checks with `@btime`, and profiling with `@bprofile`. The library includes features for handling setup costs, interpolating variables to avoid global overhead, and automatic tuning of evaluations and samples to ensure robust timing results.

Tokens
7.7K
Snippets
33
Records
38
Agent score
70%

What's inside BenchmarkTools.jl

  1. Organize benchmarks using the `BenchmarkGroup` type

    main

    A BenchmarkGroup acts as an organizational unit for benchmark suites. It can store and structure benchmark definitions (@benchmarkable), raw Trial data, estimation results, and nested BenchmarkGroup instances. It supports a subset of the Julia AbstractDict interface, allowing you to map IDs to values and assign descriptive "tags" for filtering.

    Defining a suite

    You can create a hierarchy by indexing into a BenchmarkGroup. Keys are automatically created upon access, even if they do not exist. To remove unused keys created by accidental access, use clear_empty!(suite).

    # Define a parent group
suite = BenchmarkGroup()

# Add child groups with tags for filtering
suite["utf8"] = BenchmarkGroup(["string", "unicode"])
suite["trig"] = BenchmarkGroup(["math", "triangles"])

# Add benchmarks to a specific group
teststr = join(rand('a':'d', 10^4))
suite["utf8"]["replace"] = @benchmarkable replace($teststr, "a" => "b")

# Nested creation via indexing
suite2 = BenchmarkGroup()
suite2["my"]["nested"]["benchmark"] = @benchmarkable sum(randn(32))

  2. Interpolate variables into benchmarks using $

    main

    To avoid the performance pitfalls of benchmarking with global variables, use the $ operator to interpolate external variables or expressions into your benchmark. Interpolated values are pre-computed before the benchmarking process begins, ensuring the benchmark measures only the target expression.

    Warning: If a benchmark reports a time of less than one nanosecond, the compiler may have 'cheated' by hoisting the calculation out of the benchmark code. To prevent this in very simple expressions, reference and dereference the interpolated variables using Ref(var)[].

    using BenchmarkTools

# Interpolating a global variable
A = rand(3,3);
@btime inv($A)

# Interpolating an expression (pre-computed before benchmarking)
@btime inv($(rand(3,3)))

# INCORRECT: This includes the rand(3,3) call in the benchmark time
@btime inv(rand(3,3))

# Preventing compiler hoisting for simple expressions
a = 1; b = 2
@btime $(Ref(a))[] + $(Ref(b))[]

  3. Core terminology in BenchmarkTools.jl

    main

    To use BenchmarkTools effectively, understand the following hierarchy of measurement units:

    • evaluation: A single execution of the benchmark expression being measured.
    • sample: A single time or memory measurement. Because a single evaluation might be faster than the system's timing resolution, a sample is often obtained by running multiple evaluations and calculating the average time per evaluation.
    • trial: An experiment consisting of gathering multiple samples.
    • benchmark parameters: The configuration settings that control how a benchmark trial is performed.

  4. Prevent compiler optimizations from eliding code

    main

    The Julia compiler may optimize away code if it determines the result is unused or if all values are known at compile time. This can lead to unrealistically fast benchmarks.

    1. Unused results: If a function call like view(a, 1:2, 1:2) is not returned by the benchmark expression, the compiler might elide it. Ensure the expression returns the value you want to measure.
    2. Constant folding: If you benchmark $a + $b where a and b are known constants, the compiler might replace the expression with the pre-calculated sum.

    Workaround: To stop the optimizer from eliding simple operations, reference and dereference the interpolated variables using Ref():

    a = 1; b = 2
@btime $(Ref(a))[] + $(Ref(b))[]
    a = 1; b = 2
# Prevents constant folding
@btime $(Ref(a))[] + $(Ref(b))[]

  5. Choose the right estimator for benchmark timing

    main

    Because benchmark time distributions are typically right-skewed (due to positive machine noise), different estimators serve different purposes:

    • minimum: A robust estimator for the location parameter of the time distribution; it should not be considered an outlier.
    • median: A robust measure of central tendency that is relatively unaffected by outliers.
    • mean: A non-robust measure of central tendency that is usually positively skewed by outliers.
    • maximum: Primarily noise-driven; it can change drastically between trials and should be treated as an outlier.

  6. Handle benchmark results with Trial and TrialEstimate

    main

    Running a benchmark (e.g., using @benchmark) produces a Trial object. A Trial contains all collected samples and the parameters used for the benchmark.

    You can extract statistical summaries from a Trial using estimation functions, which return a TrialEstimate. These estimates provide a snapshot of time, GC time, memory usage, and allocation counts.

    Common estimation functions include:

    • minimum(t)
    • maximum(t)
    • median(t)
    • mean(t)
    • std(t)

    Note: median, mean, and std are re-exported from the Statistics package.

    t = @benchmark eigen(rand(10, 10))

# Get various estimates
min_est = minimum(t)
med_est = median(t)
mean_est = mean(t)

  7. Best practices for consistent benchmarking

    main

    To minimize noise and ensure reproducibility, follow these guidelines:

    • Seed RNGs: If your benchmark uses rand or other random number generators, seed the RNG (or provide a seeded RNG) so values are consistent across trials, samples, and evaluations.
    • Manage BLAS threads: On some systems, BLAS worker threads may exceed available cores, causing scheduling issues. Use BLAS.set_num_threads(i::Int) to match or stay below the number of available cores.
    • Reduce Machine Noise: Use CPU/memory shielding tools (like cset) to dedicate resources to the Julia process. While BenchmarkTools handles noise between samples, it cannot mitigate noise occurring between trials.
    • Scope Awareness: Note that the @benchmark macro is evaluated in the global scope, even if called from within a local scope.

  8. Understand how samples are calculated

    main

    BenchmarkTools does not always measure a single evaluation. If an evaluation is extremely fast, the timing resolution might be insufficient. To compensate, BenchmarkTools runs n evaluations to produce one sample, where the sample time is estimated as total_time / n.

    Instead of manually guessing the number of evaluations needed per sample, use the tune! method to automatically determine the optimal configuration for your specific benchmark.

  9. Define and execute benchmarks with @benchmark

    main

    To quickly benchmark a Julia expression, use the @benchmark macro. It automatically handles defining the benchmark, tuning configuration parameters, and running the trial.

    For more control, you can split this into three explicit steps:

    1. Define: Use @benchmarkable to create a benchmark object with default parameters.
    2. Tune: Use tune!(b) to automatically find the optimal number of evals (evaluations per sample) and samples.
    3. Run: Use run(b) to execute the benchmark and see the results.
    # Quick way
@benchmark sin(1)

# Explicit way
b = @benchmarkable sin(1)
tune!(b)
run(b)

  11. Visualize benchmark results with histograms

    main

    You can customize the histogram visualization of a Trial by using an IOContext. This is useful for comparing multiple benchmarks on the same scale.

    Key options for IOContext:

    • :histmin: Sets the minimum value for the histogram range.
    • :histmax: Sets the maximum value for the histogram range.
    • :logbins: Set to true or false to control vertical scaling (log frequency vs. linear frequency).
    # Set a specific range and use log scaling for bins
io = IOContext(stdout, :histmin=>0.5, :histmax=>8, :logbins=>true)

b = @benchmark x^3 setup=(x = rand())
show(io, MIME("text/plain"), b)

  12. Cache and reuse benchmark parameters

    main

    To ensure consistency and reduce turnaround time when comparing different versions of a package, you can pre-tune a BenchmarkGroup and save its parameters to a file. This avoids the expensive tune! process in every new session and guarantees that the same 'evaluations per sample' are used across experiments.

    1. Tune and Save: Use tune!(suite) to configure the parameters, then use BenchmarkTools.save with params(suite) to serialize them.
    2. Load and Apply: Use loadparams! to apply the saved parameters back to a suite. Note that BenchmarkTools.load returns an array, so you typically access the first element [1] to get the parameters.
    # 1. Tune the suite
tune!(suite);

# 2. Save the parameters to a JSON file
BenchmarkTools.save("params.json", params(suite));

# 3. In a new session, load and apply the parameters
# Syntax: loadparams!(group, paramsgroup, fields...)
loadparams!(suite, BenchmarkTools.load("params.json")[1], :evals, :samples);