benchmark-ips

When benchmarking extremely small workloads, the overhead of the benchmark loop itself can introduce errors. There are two ways to mitigate this:

1. Manual Iteration Loop: Pass a block that accepts a times argument. You must run your code exactly times times inside a loop. This is the recommended approach for reducing overhead.
2. Code Grafting: Pass the code as a string as the second argument to x.report. This grafts the code directly into the internal loop. This is typically not needed if you use the |times| form.

# Method 1: Manual loop (Recommended)
x.report("addition2") do |times|
  i = 0
  while i < times
    i += 1
    1 + 2
  end
end

# Method 2: Code grafting
x.report("addition3", "1 + 2")

To use benchmark-ips in your Ruby project, install the gem using the following command:

$ gem install benchmark-ips

You can share your results on benchmark.fyi by running your script with the SHARE=1 environment variable:

SHARE=1 ruby my_benchmark.rb

If you are using a custom instance of benchmark.fyi, use the SHARE_URL variable:

SHARE_URL=https://ips.example.com ruby my_benchmark.rb

The Benchmark::IPS::Report::Entry#to_s method (and its display method for printing to $stdout) produces a formatted string containing the label, iterations per second, error percentage, and iteration count.

The format changes based on the global Benchmark::IPS.options[:format] setting:

1. :human format: Uses scaled values for readability (e.g., 1.2k i/s) and includes the error percentage (e.g., ±4.1%).
2. Default format: Provides a fixed-width numeric representation.

If show_total_time! has been called on the entry, the total runtime in seconds will be appended to the output.

By default, benchmark-ips shows a margin of error based on one standard deviation. For more mathematically sound results, you can use a bootstrap confidence interval. This requires the kalibera gem to be installed (gem install kalibera).

When using :bootstrap, the report uses the median of the interval rather than the mean.

# Requires 'gem install kalibera'
Benchmark.ips do |x|
  x.config(:stats => :bootstrap, :confidence => 95)
  # or
  x.stats = :bootstrap
  x.confidence = 95
end

You can customize the execution behavior using x.config or direct attribute assignment within the Benchmark.ips block:

• warmup: Seconds used during the warmup phase (default 2).
• time: Seconds used during the calculation phase (default 5).
• iterations: Number of times to run the warmup and calculation stages. Useful for Ruby implementations that optimize using tracing/on-stack-replacement. Default is 1.
• suite: A custom suite object to control behavior like Garbage Collection.

You can pass a custom suite object to x.config(:suite => suite) to manage environment states, such as enabling/disabling Garbage Collection between runs.

class GCSuite
  def warming(*); run_gc; end
  def running(*); run_gc; end
  def warmup_stats(*); end
  def add_report(*); end

  private
  def run_gc
    GC.enable
    GC.start
    GC.disable
  end
end

suite = GCSuite.new

Benchmark.ips do |x|
  x.config(:suite => suite)
  x.report("job1") { ... }
end

If you want to ensure measurements are independent (e.g., to avoid cross-contamination between different Ruby invocations), use these commands:

• x.hold! 'filename': Runs only one benchmark each time you run the command, storing results in the specified file. The file is deleted once all results are gathered and the report is shown.
• x.save!: An alternative command for saving results (see examples/save.rb in the repo for details).

Benchmark.ips do |x|
  x.hold! 'filename'
end

You can output results in JSON format for use with other tools.

• To save to a file: x.json! 'filename.json'
• To output to STDOUT: Set x.quiet = true and then call x.json! STDOUT.

Benchmark.ips do |x|
  x.report("some report") {  }
  x.json! 'filename.json'
end

# To STDOUT
Benchmark.ips do |x|
  x.report("some report") {  }
  x.quiet = true
  x.json! STDOUT
end