repository·master·Indexed 11 days ago
https://github.com/dotnet/benchmarkdotnetA library for transforming methods into high-precision benchmarks, tracking performance, and generating reproducible measurement experiments across multiple runtimes. It supports .NET 5+, .NET Framework 4.6.1+, .NET Core 2.0+, Mono, and NativeAOT across Windows, Linux, and macOS. Features include memory and disassembly diagnostics, baseline comparisons, and customizable configurations via Jobs, Diagnosers, and Analysers.
BenchmarkDotNet supports a wide range of environments and languages:
EventPipeProfiler is a cross-platform profiler available since version 0.12.1. It allows you to profile .NET code on Windows, Linux, and macOS.
When a benchmark runs with this profiler, the collected data is exported to trace files in .speedscope.json and .nettrace formats. These files can be analyzed using the following tools:
A job describes how to run your benchmark. It is a collection of characteristics (environment, runtime, execution strategy, etc.) that define the execution context. You can specify one or several jobs for your benchmarks to test how code performs across different platforms, runtimes, or configurations.
Key categories of job characteristics include:
IOrderer interface.Throughput strategy is the default RunStrategy. It is designed for microbenchmarking and automatically manages the number of operations in main iterations based on pilot iterations. It also automatically adjusts the total number of iterations based on the accuracy settings of the job. This strategy is ideal for benchmark methods that reach a steady state.You can mark a specific benchmark method as a baseline to provide a reference point for all other benchmarks in your suite. When a baseline is defined, BenchmarkDotNet calculates the ratio of each benchmark's performance relative to that baseline. This is useful for scaling results and understanding the relative cost of different implementations.
[Benchmark(Baseline = true)]
public void BaselineMethod()
{
// This method serves as the reference
}
[Benchmark]
public void OtherMethod()
{
// Results for this will be shown as a ratio relative to BaselineMethod
}A Config in BenchmarkDotNet is a collection of components that define how a benchmark is executed and reported. You can customize your benchmark by specifying sets of:
jobs: Define the runtime environments and execution settings.columns: Define the data columns in the output reports.exporters: Define how results are written to files (e.g., CSV, JSON, Markdown).loggers: Define how results are printed to the console or other logs.diagnosers: Provide additional measurements (e.g., memory allocation, hardware counters).analysers: Perform statistical analysis on the results.validators: Ensure the benchmark run meets specific criteria.filters: Control which benchmarks are executed.logical group rules: Group benchmarks together in reports.When benchmarking across different runtime configurations (e.g., comparing .NET Core vs. .NET Framework vs. Mono), you can designate one specific job as the reference point by setting baseline = true in its configuration.
When a baseline is defined, BenchmarkDotNet adds Ratio and RatioSD columns to the summary table. The Ratio column shows the performance of each job relative to the baseline (where the baseline itself has a ratio of 1.00). A ratio less than 1.00 indicates the job is faster than the baseline, while a ratio greater than 1.00 indicates it is slower.
// Example of marking a job as the baseline in a Benchmark class
[Config(typeof(Config))]
public class IntroJobBaseline
{
private class Config : ManualConfig
{
public Config()
{
// Define multiple jobs, marking one as the baseline
AddJob(Job.Default.WithId("Clr").WithBaseline(true));
AddJob(Job.Default.WithId("Core"));
AddJob(Job.Default.WithId("Mono"));
}
}
[Benchmark]
public void SplitJoin()
{
// ... implementation ...
}
}BenchmarkDotNet can detect when a benchmark's results do not follow a normal (unimodal) distribution. When the statistical analysis identifies multiple peaks in the data, it issues a MultimodalDistribution warning.
These warnings include an mValue (the number of modes detected). A high mValue suggests that the benchmark results are inconsistent, potentially due to background processes, thermal throttling, or non-deterministic behavior in the code being measured. When you see these warnings, you should investigate the stability of your benchmarking environment or the code under test.
// * Warnings *
MultimodalDistribution
IntroMultimodal.Bimodal: MainJob -> It seems that the distribution is bimodal (mValue = 3.57)
IntroMultimodal.Trimodal: MainJob -> It seems that the distribution is multimodal (mValue = 4.65)The SummaryStyle class in BenchmarkDotNet allows you to customize how benchmark result summaries are formatted and displayed. You can fine-tune the output by controlling unit printing, column widths, measurement units (size and time), and culture-specific formatting.
// Example of how SummaryStyle properties might be used to configure a report
// Note: Actual implementation depends on the specific BenchmarkDotNet configuration API used
var style = new SummaryStyle
{
PrintUnitsInHeader = true,
PrintUnitsInContent = true,
TimeUnit = TimeUnit.Nanoseconds,
SizeUnit = SizeUnit.Bytes
};[Params]), the cost of allocating and creating those arrays is not included in the benchmarked method's execution time. The arrays are allocated before the benchmark loop begins. This ensures that the measured Mean time reflects only the logic inside your benchmarked method, not the overhead of setting up the test data.When using setup and cleanup attributes, BenchmarkDotNet follows a specific lifecycle. [GlobalSetup] and [GlobalCleanup] wrap the entire benchmark process, while [IterationSetup] and [IterationCleanup] wrap every single invocation (including Warmup and Target phases).
The execution sequence is:
[GlobalSetup][IterationSetup] $\rightarrow$ [Benchmark] $\rightarrow$ [IterationCleanup] (repeated for Warmup iterations)[IterationSetup] $\rightarrow$ [Benchmark] $\rightarrow$ [IterationCleanup] (repeated for Target iterations)[GlobalCleanup]// GlobalSetup
// IterationSetup (1) // IterationSetup Jitting
// IterationCleanup (1) // IterationCleanup Jitting
// IterationSetup (2) // MainWarmup1
// Benchmark // MainWarmup1
// IterationCleanup (2) // MainWarmup1
// IterationSetup (3) // MainWarmup2
// Benchmark // MainWarmup2
// IterationCleanup (3) // MainWarmup2
// IterationSetup (4) // MainTarget1
// Benchmark // MainTarget1
// IterationCleanup (4) // MainTarget1
// IterationSetup (5) // MainTarget2
// Benchmark // MainTarget2
// IterationCleanup (5) // MainTarget2
// IterationSetup (6) // MainTarget3
// Benchmark // MainTarget3
// IterationCleanup (6) // MainTarget3
// GlobalCleanup