repository·main·Indexed 21 days ago
https://github.com/airspeed-velocity/asvA Python history benchmarking tool that tracks performance over time. It allows developers to run benchmark suites, discover benchmarks in repositories, and view results through an interactive web interface. The tool includes statistical utilities for comparing benchmark samples using Mann-Whitney U tests and confidence intervals, as well as a ProfilerGui system for visual profiling.
asv (airspeed velocity) is a tool designed for benchmarking Python packages over their lifetime. It is optimized for benchmarking a single project using a specific suite of benchmarks. Results are visualized in an interactive web frontend that can be hosted on any basic static webserver.airspeed velocity (asv) is a benchmarking tool designed to track Python packages over their lifetime. It allows developers to monitor runtime performance, memory consumption, and custom-computed metrics.
Key features include:
For inspiration, you can view deployed examples from major projects like astropy, numpy, and scipy, or explore the asv_sample repository for user-specific examples.
To perform initialization or cleanup that should not be included in the benchmark timing, use setup and teardown.
setup(self) or teardown(self) method within your benchmark class.setup function to the benchmark function object.setup function runs for every benchmark in that module before specific function/class setups.Note: If setup raises NotImplementedError, the benchmark is marked as skipped. For version 0.6.0+, you can also raise asv_runner.benchmark.mark.SkipNotImplemented to skip a benchmark.
# Example using a class with setup
class Suite:
def setup(self):
with open("/usr/share/words.txt", "r") as fd:
self.words = fd.readlines()
def time_upper(self):
for word in self.words:
word.upper()
# Example using function attributes
words = []
def my_setup():
global words
with open("/usr/share/words.txt", "r") as fd:
words = fd.readlines()
def time_upper():
for word in words:
word.upper()
time_upper.setup = my_setupASV uses a version number to track changes in benchmark code. If the source code of a benchmark (including setup and setup_cache) changes, the version number changes, and ASV will ignore previous results for that benchmark.
Automatic Versioning: By default, ASV computes the version by hashing the benchmark source code text.
Manual Versioning: You can manually control the version by setting the .version attribute on the benchmark function to any Python string. ASV only checks if the version recorded with a measurement matches the current .version attribute.
def time_my_func():
pass
time_my_func.version = "v1.2.3"The asv benchmarking process follows these conceptual steps:
asv_runner).ASV detects regression changes by identifying stepwise changes in data graphs. The algorithm assumes that curves are piecewise constant plus random noise.
Key aspects of the detection process:
If your setup is computationally expensive, use setup_cache. Unlike setup, which runs for every benchmark and every repeat, setup_cache runs only once and caches the result to disk.
There are two ways to use setup_cache:
setup method.Attributes:
setup_cache.timeout: You can specify a timeout for the cache calculation by setting the .timeout attribute on the function. If not set, it defaults to the maximum timeout of the benchmarks using it.default_benchmark_timeout configuration option.# Example 1: Returning a pickled data structure
class Suite:
def setup_cache(self):
fib = [1, 1]
for i in range(100):
fib.append(fib[-2] + fib[-1])
return fib
def track_fib(self, fib):
return fib[-1]
# Example 2: Explicitly saving files
class Suite:
def setup_cache(self):
with open("test.dat", "wb") as fd:
for i in range(100):
fd.write(f'{i}\n')
def setup(self):
with open("test.dat", "rb") as fd:
self.data = [int(x) for x in fd.readlines()]
def track_numbers(self):
return len(self.data)ASV recognizes several benchmark types based on the function name prefix. Use these prefixes to determine what metric is being measured:
def time_*(): Measures the time taken by the function (Timing benchmarks).def timeraw_*(): Measures time taken by the function after interpreter start (Raw timing benchmarks).def mem_*(): Measures the memory size of the object returned (Memory benchmarks).def peakmem_*(): Measures the peak memory size of the process when calling the function.def track_*(): Uses the returned numerical value directly as the benchmark result (Tracking benchmarks).Starting from version 0.6.0, asv functionality is split into two distinct parts to ensure a clean separation of concerns:
asv_runner: This is the component responsible for the execution phase. It loads benchmark types, discovers benchmarks, and runs them within a specific environment. It is designed to have minimal dependencies—ideally only the minimum Python version required and the dependencies of the project being benchmarked.asv: This component handles the orchestration and post-processing. It manages setting up environments, loading plugins, and collecting/visualizing results after the benchmarks have been run.This architecture ensures that the subprocess running your benchmarks remains lightweight and isolated from the orchestration logic.
To manage benchmarking environments, asv requires one of the following tools to be installed on your system:
py-rattler: Used for the rattler backend. This is the fastest option when non-pythonic dependencies are required.virtualenv: Required if you are not using venv, as venv is not compatible with managing multiple different Python versions.anaconda or miniconda: Requires the conda command to be available on your PATH. This is preferred for projects with many compiled C/C++ extensions, as conda can fetch precompiled binaries.uv: Used for the uv backend.When you specify pythons in asv.conf.json, ASV handles them in two ways depending on the input:
Version Strings (e.g., "3.9"):
conda is installed, ASV uses it to create a temporary environment for that version via an environment.yml file.virtualenv is installed, ASV searches for that Python version on your PATH and creates a new virtual environment.Executable Paths or Names:
PATH, ASV assumes the environment is already fully loaded and read-only.