repository·main·Indexed 18 days ago
https://github.com/apple/swift-collections-benchmarkA tool for collecting and visualizing performance data for Swift data structures and collection algorithms. It provides a Benchmark class with addSimple and add methods to define tests, a CLI for running benchmarks and rendering results as charts, and the ability to compare result files or manage benchmarks via Benchmark Libraries.
A Benchmark Library is a JSON file that organizes multiple benchmarks into hierarchical, thematic charts. This is useful for managing large sets of benchmarks (like those in the Swift Collections package).
Library Structure:
Libraries consist of group kinds (for organization) and chart kinds (which list specific tasks).
Commands:
library run: Collects data for all tasks defined in the library.library render: Renders the library into an interactive Markdown file or a self-contained HTML file.Key Options for library render:
--theme-file <FILE>: Specify a JSON theme file.--output <FILE>: Set the output destination (e.g., .html for a self-contained file).# Run a library
$ swift-collections-benchmark library run results.json --library Library.json --max-size 16M --cycles 20
# Render a library to HTML
$ swift-collections-benchmark library render results.json --library Library.json --output results.htmlTo use this package in a SwiftPM project, add it to your Package.swift dependencies. It is recommended to set up a standalone executable target dedicated to benchmarking to keep your production code separate from benchmark logic.
// swift-tools-version:6.1
import PackageDescription
let package = Package(
name: "MyPackage",
products: [
.executable(name: "my-benchmark", targets: ["MyBenchmark"]),
],
dependencies: [
.package(url: "https://github.com/apple/swift-collections-benchmark", from: "0.0.4"),
// ... other dependencies ...
],
targets: [
// ... other targets ...
.executableTarget(
name: "MyBenchmark",
dependencies: [
.product(name: "CollectionsBenchmark", package: "swift-collections-benchmark"),
]),
]
)Use the Benchmark class to define and run performance tests.
addSimple: Use this for benchmarks where the performance depends on a single input type. The closure receives the input and should perform the work.add: Use this for more complex benchmarks where the performance depends on multiple inputs (e.g., a collection and a set of lookup values). The closure receives the inputs and returns a closure that accepts a timer to measure repeated operations.Note: Use blackHole() to prevent the compiler from optimizing away the code you are trying to measure.
import CollectionsBenchmark
var benchmark = Benchmark(title: "Demo Benchmark")
// Simple benchmark
benchmark.addSimple(
title: "Array<Int> sorted",
input: [Int].self
) { input in
blackHole(input.sorted())
}
// Complex benchmark returning a measurement closure
benchmark.add(
title: "Set<Int> contains",
input: ([Int], [Int]).self
) { input, lookups in
let set = Set(input)
return { timer in
for value in lookups {
precondition(set.contains(value))
}
}
}
benchmark.main()To create a benchmark, instantiate a Benchmark object and use the addSimple method. You must provide a title, the input type (e.g., [Int].self), and a closure containing the code to be measured.
Important: Use the blackHole() function inside your closure to consume the result of your computation. This prevents the Swift compiler from optimizing away the code you are trying to measure.
import CollectionsBenchmark
// Create a new benchmark instance.
var benchmark = Benchmark(title: "Kalimba")
// Define a simple benchmark
benchmark.addSimple(
title: "kalimbaOrdered",
input: [Int].self
) { input in
blackHole(input.kalimbaOrdered())
}
// Execute the benchmark tool
benchmark.main()To measure the impact of an optimization, use the results compare command to compare two different result files. This provides a differential analysis, including a score and counts of improvements and regressions.
Key Options:
--output <FILE>: Generates a detailed HTML report (e.g., diff.html) containing graphical renderings of the differences.# Compare two result files
$ swift run -c release <your-benchmark-target> results compare results results-deque
# Generate an HTML differential report
$ swift run -c release <your-benchmark-target> results compare results results-deque --output diff.htmlAfter implementing your benchmark in a Swift executable, use the following commands to collect data and visualize it.
swift run -c release benchmark run results to execute the tasks. Use the --cycles flag to specify the number of measurement cycles.swift run -c release benchmark render results <filename> to generate a visual chart (e.g., a .png) from the collected results.# Run benchmarks with 5 cycles
$ swift run -c release benchmark run results --cycles 5
# Render the results to a PNG chart
$ swift run -c release benchmark render results chart.png
# Open the chart
$ open chart.pngThe render command converts the JSON results file into visual charts. On Linux, use the .svg extension instead of .png.
Key Options:
--linear-size / --linear-time: Switch from log-log scales to linear scales.--amortized <bool>: If false, shows full execution time instead of per-element value.--percentile <N>: Ignore outliers above a certain percentile (e.g., --percentile 90).--min-time <TIME> / --max-size <SIZE>: Control the ranges shown in the chart.$ swift run -c release <your-benchmark-target> render results chart.pngUse the run command to execute your defined benchmarks and collect data into a JSON file. By default, the tool measures execution time for sizes between 1 and 1,000,000. Results are appended to the output file if it already exists.
Key Options:
--cycles <N>: Specifies the number of times to cycle through the input sizes.--mode <MODE>: Controls whether to append to or overwrite existing data.--amortized-cutoff <TIME>: Sets the per-element execution time threshold (default is 10µs) at which the tool stops running larger sizes to prevent extremely long runtimes. Use --disable-cutoff to turn this off.$ swift run -c release <your-benchmark-target> run --cycles 3 results