tinybench Documentation

repository·main·Indexed 25 days ago

https://github.com/tinylibs/tinybench

A lightweight (10KB) benchmarking library for measuring code performance across multiple JavaScript runtimes. tinybench provides high-precision timing, statistical analysis (mean, median, p99, etc.), and support for various concurrency modes. It includes features for timer overhead correction, custom timestamp providers, and the ability to abort benchmarks via AbortSignal.

Tokens
7.5K
Snippets
15
Records
40
Agent score
79%

What's inside tinybench

  1. Compare Tinybench results with other tools

    main

    When comparing Tinybench results to other benchmarking tools, expect differences due to:

    • Timing APIs: Differences in resolution (e.g., Date.now() vs process.hrtime.bigint()).
    • Methodology: Tinybench collects many samples over a configurable time window and performs statistical analysis, whereas some tools may only measure a single iteration.
    • Statistics: Tinybench reports mean, median (p50), p75, p99, standard deviation, and margin of error. Other tools may use different aggregation methods (e.g., geometric mean).
    • Warmup: Tinybench performs a separate warmup phase by default (warmup: true) that is excluded from statistics. Disabling it (warmup: false) includes JIT warmup costs in the results.

    Best Practice: Focus on relative performance (is A faster than B?) rather than absolute numbers, and ensure both tests run on the same machine and environment.

  2. How events work in Tinybench

    main

    Both Bench and Task classes extend EventTarget. You can attach listeners using addEventListener to monitor benchmark progress or warnings.

    Bench Events:

    • cycle: Runs on each benchmark task's cycle.
    • warning: Runs when timer saturation is detected (reasons: 'zero-dominated', 'low-distinct', or 'zero-mad').

    Task Events:

    • cycle: Runs only on that specific task's cycle.
    // runs on each benchmark task's cycle
    bench.addEventListener('cycle', (evt) => {
      const task = evt.task!;
    });
    
    // runs when timer saturation is detected for a task's measured samples
    bench.addEventListener('warning', (evt) => {
      const task = evt.task!;
      const reason = evt.reason; // 'zero-dominated' | 'low-distinct' | 'zero-mad'
    });
    
    // runs only on this benchmark task's cycle
    task.addEventListener('cycle', (evt) => {
      const task = evt.task!;
    });
  3. Abort benchmarks at the Bench level

    main

    To abort all tasks within a benchmark simultaneously, pass an AbortSignal to the Bench constructor. When the signal is aborted, all tasks will be stopped.

    const controller = new AbortController()
    
    const bench = new Bench({ signal: controller.signal })
    
    bench
      .add('task1', () => {
        // This will be aborted
      })
      .add('task2', () => {
        // This will also be aborted
      })
    
    // Abort all tasks
    controller.abort()
    
    await bench.run()
  4. Abort individual tasks

    main

    You can abort specific tasks without affecting others by passing an AbortSignal in the options object when calling bench.add(). When the signal is aborted, only the associated task will stop.

    const controller = new AbortController()
    
    const bench = new Bench()
    
    bench
      .add(
        'abortable task',
        () => {
          // This task can be aborted independently
        },
        { signal: controller.signal }
      )
      .add('normal task', () => {
        // This task will continue normally
      })
    
    // Abort only the first task
    controller.abort()
    
    await bench.run()
  5. Handle measurement precision issues

    main

    If a task runs faster than the timer's resolution, samples may measure as zero, skewing results. Tinybench detects this and provides diagnostics:

    1. Detection: A task exposes detectedResolution (or undefined if no positive sample was measured). If samples are dominated by timer resolution, Tinybench dispatches a 'warning' event.
    2. Resolution: You can improve precision by selecting a different timestampProvider via the benchmark options.
    3. Statistical Confidence: If precision is sufficient but results are unstable, increase the benchmark time to collect more samples.

    Note: Do not rely solely on the relative margin of error (rme) to detect saturation; use the 'warning' event and detectedResolution as the primary signals.

  6. Mitigate JS JIT de-optimization and latency spikes

    main

    JIT de-optimization (deopt) occurs when a JavaScript engine discards an optimized code path, causing sudden latency spikes. This is often caused by type instability or dynamic property access.

    To handle this during benchmarking:

    1. Use Statistical Indicators: Monitor standard deviation, variance, and percentiles reported by Tinybench to identify outliers.
    2. Enable Warmup: Ensure warmup is enabled (it is true by default) to allow the engine to reach an optimized state before measurement begins. You can tune this via warmupIterations or warmupTime.
    3. Environment Consistency: In browsers, close DevTools and run in a production-like configuration to avoid overhead that can interfere with JIT optimizations.
  7. Basic usage of Tinybench

    main

    To benchmark code, instantiate the Bench class, add tasks using the .add() method, and then run the benchmark with .run(). Each task is identified by a unique name. The .add() method returns the Bench instance, allowing for method chaining.

    import { Bench } from 'tinybench'
    
    const bench = new Bench({ name: 'simple benchmark', time: 100 })
    
    bench
      .add('faster task', () => {
        console.log('I am faster')
      })
      .add('slower task', async () => {
        await new Promise(resolve => setTimeout(resolve, 1)) // we wait 1ms :)
        console.log('I am slower')
      })
    
    await bench.run()
    
    console.log(bench.name)
    console.table(bench.table())
  8. Override task duration manually

    main

    If you need to provide a specific duration for a task execution (for example, to bypass timer resolution limits or simulate specific latencies), your task function can return an object containing an overriddenDuration property.

    When this is returned, Tinybench will use this value as the sample instead of measuring the actual elapsed time.

  9. Configure timestamp providers

    main

    To control the precision of measurements, use the timestampProvider option in the Bench constructor.

    Available shorthand values:

    • auto: Lets Tinybench pick the most precise available provider for the current runtime.
    • hrtimeNow: High-resolution time.
    • performanceNow: Uses the Performance API.
    • bunNanoseconds: Specific to the Bun runtime.

    Alternatively, you can provide a plain millisecond clock using the now option. Note that now and timestampProvider cannot be used together; if now is provided, it is converted to a provider internally.

  10. Understand Task timer resolution detection

    main

    The Task class attempts to detect the effective timer resolution of the environment. This is the smallest strictly positive latency sample that repeats among the measured samples.

    If you use overriddenDuration, the detectedResolution is calculated using only the measured-only subset (excluding your manual values) to prevent constant user-supplied values from skewing the diagnostic.

  11. Override task duration with FnReturnedObject

    main
    If you need to measure a specific part of a task's execution rather than the whole function, your task function can return an object containing an overriddenDuration field. Tinybench will use this value instead of its own measured duration for statistics.