DOCSEARCHbyAlgolia
Submit Docs

Benchee Documentation

repository·main·Indexed 23 days ago

https://github.com/bencheeorg/benchee

A library for high-precision micro-benchmarking in Elixir. Benchee allows developers to compare code performance using metrics such as execution time, memory consumption, and BEAM reductions. It features a pipeline-based architecture for extensibility, support for warmup phases, outlier removal, and the ability to save and load results for comparative analysis. It provides detailed statistical analysis including average, ips, deviation, median, and 99th percentile.

Tokens
7.9K
Snippets
24
Records
40
Agent score
80%

What's inside Benchee

  1. Overview of Benchee features

    main

    Benchee is a versatile benchmarking library for Elixir with the following capabilities:

    • Warmup phase: Runs functions for a set time before recording to simulate a running system.
    • Multi-metric support: Measures time, memory consumption, and reductions.
    • Extensible architecture: Supports plugins for different output formats like HTML, Markdown, and JSON.
    • Precision: Provides up to nanosecond precision (OS dependent).
    • Overhead compensation: Optionally measures and subtracts the overhead of function calls.
    • Parallel execution: Runs benchmark jobs in parallel to gather more results or simulate load.
    • Outlier removal: Optionally removes outliers from the statistical calculation.

  2. Understand Benchee statistics

    main

    Benchee provides several statistical metrics to help you evaluate performance:

    Core Statistics:

    • average: The average execution time or memory usage (lower is better).
    • ips: Iterations per second; how many times the function can execute in one second (higher is better).
    • deviation: The standard deviation expressed as a percentage of the average, indicating how much results vary.
    • median: The middle value of all measured values; often more stable than the average.
    • 99th %: The 99th percentile; indicates worst-case performance.

    Optional Extended Statistics:

    • minimum: The smallest (fastest) value measured.
    • maximum: The largest (slowest) value measured.
    • sample size: The total number of measurements taken.
    • mode: The most frequently occurring value.

  3. Understand the lifecycle of Benchee hooks

    main

    Benchee supports several hooks to manage setup and teardown logic at different granularities: Global (applying to the entire suite) and Local (applying to a specific scenario).

    Hook Types

    • before_scenario: Runs once before a scenario starts.
    • after_scenario: Runs once after a scenario finishes.
    • before_each: Runs before every individual execution within a scenario.
    • after_each: Runs after every individual execution within a scenario.

    Execution Order

    When running a suite, Benchee follows a nested execution pattern. For a scenario with local hooks, the order is:

    1. suite_set_up (Internal)
    2. global_before_scenario
    3. local_before_scenario
    4. Loop (Repeated for each execution):
      • global_before_each
      • local_before_each
      • The benchmarked function
      • local_after_each
      • global_after_each
    5. local_after_scenario
    6. global_after_scenario
    7. suite_tear_down (Internal)

  4. Understanding memory consumption measurements

    main

    When measuring memory in Benchee, note the following constraints:

    1. Process Scoped: Measurement is limited to the specific process Benchee uses to execute your code. It does not account for other processes or the entire BEAM.
    2. BEAM Managed Only: Only memory reported by the BEAM is counted. Memory allocated by NIFs is excluded.
    3. Total Allocation: The value represents the total amount of memory allocated during the scenario, including memory that was subsequently garbage collected.

  5. How Benchee's architecture works

    main

    Benchee's architecture follows a "pipes" model designed for extension and customization. The core structure is Benchee.Suite, which is progressively enhanced through a series of discrete, exchangeable steps.

    Each step in the pipeline is a public API function, allowing you to replace or reorder them to customize the benchmarking process. The standard execution flow follows this sequence:

    1. Initialization: Benchee.init/1 creates the initial suite.
    2. System Check: Benchee.system/1 prepares the environment.
    3. Job Addition: Benchmarking jobs are added to the suite.
    4. Collection: Benchee.collect/1 runs the benchmarks.
    5. Statistics: Benchee.statistics/1 calculates raw statistics.
    6. Loading: Benchee.load/1 handles data loading.
    7. Relative Statistics: Benchee.relative_statistics/1 calculates comparative metrics.
    8. Output: Formatter.output/1 renders the results.
    9. Profiling: Benchee.profile/1 performs additional profiling if requested.
    def run(jobs, config \ []) when is_list(config) do
  config
  |> Benchee.init()
  |> Benchee.system()
  |> add_benchmarking_jobs(jobs)
  |> Benchee.collect()
  |> Benchee.statistics()
  |> Benchee.load()
  |> Benchee.relative_statistics()
  |> Formatter.output()
  |> Benchee.profile()
end

  6. How to measure time, memory, and reductions

    main

    Benchee can measure execution time, memory consumption, and reductions.

    • Time: The default metric. Measures how fast a system processes something.
    • Memory: Measures total memory allocated during the execution of a scenario (including garbage collected memory). It is limited to the process Benchee executes your code in and only counts BEAM-managed memory.
    • Reductions: Measures reductions during execution.

    To enable these, you must set non-zero values for time, memory_time, and reduction_time respectively.

  7. Extend Benchee functionality with plugins

    main

    Benchee uses a plugin system to provide additional features like data export and visualization without bloating the core library. Common plugins include:

    • benchee_html: Generates HTML reports including data tables and various graphs (e.g., boxplots, bar charts with standard deviation). Supports exporting individual graphs as PNG.
    • benchee_csv: Generates CSV files from benchmark results for use in spreadsheet tools.
    • benchee_json: Exports suite results as JSON for integration with other tools or JavaScript applications.
    • benchee_markdown: Writes benchmark results directly into Markdown files.

  8. How Benchee hooks work

    main

    Benchee provides three levels of hooks to perform setup and teardown tasks without including those tasks in the actual performance measurements.

    Note: Hooks are generally not included in measurements, except for before_each and after_each when measuring functions that execute faster than the native resolution (primarily on Windows).

    Hook Types

    1. Suite hooks: Manual setup/teardown performed before and after calling Benchee.run/2 (e.g., seeding a database).
    2. Scenario hooks: Executed for every combination of a benchmarking function and an input.
    3. Benchmarking function hooks: Executed before and after every single invocation of the benchmarking function.

    Hook Configuration: Global vs. Local

    • Global hooks: Defined in the configuration map passed to Benchee.run/2. They run for every scenario in the suite.
    • Local hooks: Defined within the benchmark map by passing a tuple {function, hooks_keyword_list}. They only run for that specific benchmarking function.

  9. Save and load Benchee results for comparison

    main

    Benchee allows you to persist benchmark results to a file and reload them later to compare performance across different runs (e.g., comparing a feature branch against main or different Elixir/Erlang versions).

    Saving Results

    Use the :save configuration option. You can specify a :path and an optional :tag to annotate the results (e.g., with a branch name). If :path is omitted, it defaults to "benchmark.benchee". The default :tag is a timestamp.

    Loading Results

    Use the :load option to specify a file path or a list of paths (including glob expressions) to load previous results.

    To simply view loaded results without running new benchmarks, use Benchee.report/1 with a configuration containing the :load key.

    Benchee.run(
  %{
    "something_great" => fn -> cool_stuff end
  },
  save: [path: "save.benchee", tag: "first-try"]
)

Benchee.report(load: "save.benchee")

  10. Install Benchee via Mix

    main

    To use Benchee in your Elixir project, add it to your mix.exs dependencies. It is recommended to restrict it to the :dev environment to avoid including benchmarking tools in production builds.

    defp deps do
  [
    {:benchee, "~> 1.0", only: :dev}
  ]
end

  11. Configure Benchee via Benchee.run/2

    main

    Configuration options are passed as a keyword list in the second argument of Benchee.run/2. Most options are optional as Benchee provides sensible defaults.

    Common configuration patterns:

    # Example: Disabling benchmarking output
Benchee.run(%{"some function" => fn -> magic end}, print: [benchmarking: false])

# Example: Measuring time, memory, and reductions
Benchee.run(
  %{
    "something_great" => fn -> cool_stuff end
  },
  warmup: 1,
  time: 5,
  memory_time: 2,
  reduction_time: 2
)
    Benchee.run(%{"some function" => fn -> magic end}, print: [benchmarking: false])

  12. Run Benchee benchmarks from Erlang

    main

    You can use Benchee in Erlang projects, though it is recommended to set up a small Elixir project as a dependency to leverage Mix/Rebar3 interoperability.

    To use Benchee in an Erlang project with rebar3, use the rebar3_elixir_compile plugin.

    Rebar3 Configuration

    Add the following to your rebar.config:

    deps, [{enchee, {elixir, "benchee", "0.9.0"}}].

{plugins, [
    {rebar3_elixir_compile, ".*", {git, "https://github.com/barrel-db/rebar3_elixir_compile.git", {branch, "main"}}}
]}.

{provider_hooks, [
  {pre, [{compile, {ex, compile}}]}],
  {pre, [{release, {ex, compile}}]}
]}.

{elixir_opts, [{env, dev}]}.

    Erlang Usage

    Benchee provides a :benchee interface for Erlang compatibility. You can call benchee:run/2 directly from the Erlang shell or your code.