criterion

Benchmark results include several key metrics. For high-quality results, look for an R² goodness-of-fit value between 0.99 and 1.0. Values below 0.9 are considered unreliable.

Criterion HTML reports provide two primary visual representations for each benchmark:

1. Kernel Density Estimate (KDE) Chart: A graph of the probability of any given time measurement occurring. Spikes indicate frequent measurements at that specific time.
2. Raw Measurements Chart: Shows the number of loop iterations on the x-axis and measured execution time on the y-axis. A linear regression line is drawn through the data points to help identify trends and noise.

To measure and regress against GHC garbage collector (GC) statistics, you must explicitly enable statistics collection at runtime using the GHC runtime system flag +RTS -T.

Without this flag, metrics like allocated, numGcs, and other GC-related data will not be available.

# Example of running a benchmark with GC statistics enabled
./my-benchmark +RTS -T

A Criterion benchmark suite is composed of Benchmark values, typically organized into groups. You use defaultMain to run the suite and bgroup to cluster related benchmarks under a common name.

To define an individual benchmark, use the bench function, which takes a descriptive name and a Benchmarkable value (the code to be measured).

Example structure:

main = defaultMain [
  bgroup "fib" [
    bench "1"  $ whnf fib 1,
    bench "5"  $ whnf fib 5
  ]
]

main = defaultMain [
  bgroup "fib" [
    bench "1"  $ whnf fib 1
               , bench "5"  $ whnf fib 5
               , bench "9"  $ whnf fib 9
               , bench "11" $ whnf fib 11
               ]
  ]

To post-process benchmark data into a visual HTML report, use the criterion-report executable.

1. Run your benchmark with the --json flag to save the results to a file.
2. Use criterion-report to convert that JSON file into an HTML report.
3. You can optionally use the --template flag (supported by Criterion) to customize the report.

# 1. Run benchmark and save to JSON
criterion --json data.json

# 2. Generate HTML report
criterion-report data.json report.html

To create a benchmark suite, import Criterion.Main and use the defaultMain function. defaultMain accepts a list of Benchmark values. You can group related benchmarks using bgroup and define individual benchmarks using bench.

To ensure you are measuring the actual computation and not just the creation of a thunk, use whnf (Weak Head Normal Form) to evaluate the function to its head normal form during the benchmark.

{- cabal:
build-depends: base, criterion
-}

import Criterion.Main

-- The function we're benchmarking.
fib :: Int -> Int
fib m | m < 0     = error "negative!"
      | otherwise = go m
  where
    go 0 = 0
    go 1 = 1
    go n = go (n - 1) + go (n - 2)

-- Our benchmark harness.
main = defaultMain [
  bgroup "fib" [
    bench "1"  $ whnf fib 1
  , bench "5"  $ whnf fib 5
  , bench "9"  $ whnf fib 9
  , bench "11" $ whnf fib 11
  ]
  ]

If external factors are making measurements noisy, look for these 'yellow flags':

• Low R² value: An R² goodness-of-fit measure dropping below 0.9 suggests the regression does not fit the data well.
• Severe outliers: In HTML reports (using --output), look for data points sitting far from the linear regression line.
• Loose confidence bounds: If the lower and upper bounds on an estimate are not 'tight' (far from the estimate), noise is likely affecting the result.
• Outlier variance warning: Criterion may explicitly print a warning if the standard deviation is heavily inflated by outliers, e.g., variance introduced by outliers: 91% (severely inflated).

Criterion can export benchmark results into several formats:

• HTML: Use --output <file> (the most user-friendly format).
• JSON: Use --json <file> or --template json --output <file>.
• CSV: Use --csv <file>.
• JUnit XML: Use --junit <file> (compatible with JUnit-style XML readers).

$ ./Fibber --json mydata.json
$ ./Fibber --csv mydata.csv
$ ./Fibber --junit mydata.xml

When running benchmarks in a terminal, the output follows this format:

benchmarking <name>
time                 <estimate>   (<lower bound> .. <upper bound>)
                     <R²>         (<lower bound> .. <upper bound>)
mean                 <estimate>   (<lower bound> .. <upper bound>)
std dev              <estimate>

• time is the OLS regression estimate.
• R² is the goodness-of-fit metric.
• mean and std dev are the average execution time and standard deviation respectively.

benchmarking ByteString/HashMap/random
time                 4.046 ms   (4.020 ms .. 4.072 ms)
                     1.000 R²   (1.000 R² .. 1.000 R²)
mean                 4.017 ms   (4.010 ms .. 4.027 ms)
std dev              27.12 μs   (20.45 μs .. 38.17 μs)