DOCSEARCHbyAlgolia
Submit Docs

pyperf Documentation

repository·main·Indexed 21 days ago

https://github.com/psf/pyperf

A Python toolkit for writing, running, and analyzing reliable benchmarks. pyperf provides tools for calibration, statistical analysis, and performance comparison across environments. It includes a Runner class for executing benchmarks (including microbenchmarks via timeit and bench_time_func), a BenchmarkSuite for managing collections of results, and a set of analysis commands such as show, check, stats, and compare_to to identify outliers and determine statistical significance using t-tests.

Tokens
19.3K
Snippets
75
Records
95
Agent score
75%

What's inside pyperf

  1. Overview of the pyperf toolkit

    main

    The pyperf module is a toolkit designed for writing, running, and analyzing benchmarks in Python. It provides tools for reliable benchmarking, including automatic calibration for a time budget, support for multiple worker processes, and statistical analysis of results (mean, standard deviation, percentiles, etc.).

    Key capabilities include:

    • Reliable Execution: Automatic calibration and system tuning to ensure stable results.
    • Statistical Analysis: Commands like pyperf stats for distribution analysis and pyperf compare_to for significance testing.
    • Microbenchmarking: The pyperf timeit CLI tool for quick Python microbenchmarks.
    • Resource Tracking: Options to track memory usage via --track-memory and --tracemalloc.
    • Metadata Collection: Automatic collection of system and benchmark metadata.
    • Data Formats: Support for JSON storage and multiple units (seconds, bytes, and integers).

  2. CPU pinning and isolation in pyperf

    main

    To improve benchmark stability on Linux, you should isolate CPU cores and pin worker processes to them.

    • Automatic Pinning: The pyperf.Runner class automatically pins worker processes to isolated CPUs if they are detected.
    • Manual Pinning: You can manually specify CPU affinity using the --affinity command line option to make benchmarks more stable even if no CPUs are isolated.
    • Verification: CPU pinning status can be verified in benchmark metadata via the cpu_affinity key.
    • Implementation: pyperf uses os.sched_setaffinity for pinning.

    It is also recommended to check CPU topology for HyperThreading and NUMA configurations to ensure optimal performance.

    # Example of using the --affinity flag (conceptual)
$ pyperf run --affinity 2,3,6,7 my_benchmark.py

  3. How to benchmark microbenchmarks efficiently

    main

    When benchmarking very fast functions (those taking less than 1 millisecond), bench_func introduces non-negligible overhead because it calls the Python function in a loop.

    To achieve higher precision for microbenchmarks, use one of these two methods:

    1. timeit: Use this to run a specific Python statement. It allows you to define setup and teardown code that runs around the benchmark statement.
    2. bench_time_func: Use this if you have a function that manually manages its own timing loop. The function should use time.perf_counter to measure the total elapsed time of all loops and return that raw value. The Runner will then normalize the result by dividing it by the number of loops.

  4. NUMA and CPU Pinning

    main

    On high-performance Intel and AMD systems, multiple NUMA (Non-uniform memory access) nodes may exist, where each node is assigned a specific memory region. Accessing memory from a different node increases latency.

    To optimize performance on NUMA systems:

    1. Use lscpu -a -e to list CPUs and their associated NUMA nodes.
    2. CPU Pinning is critical: Ensure your benchmark processes are pinned to the CPUs belonging to the same NUMA node as the memory they are accessing. Use the numactl command for advanced NUMA control.
    $ lscpu -a -e

  5. Understand the pyperf JSON data format

    main

    pyperf stores benchmark results as JSON files. By default, these files are optimized for small size (minified).

    Key Features

    • Compression: pyperf supports gzip compression; use filenames ending in .gz to enable it.
    • Readability: To generate human-readable (indented) JSON, use the pyperf convert command with the --indent flag.
    • Structure: The JSON contains a top-level metadata object (describing the overall benchmark like name, loops, cpu_count, etc.) and a benchmarks array. Each benchmark contains a runs array, where each run includes its own metadata, values (the measured results), and warmups data.

    For processing these files, the jq tool is recommended.

    {
    "benchmarks": [
        {
            "runs": [
                {
                    "metadata": {
                        "date": "2016-10-21 03:14:19.670631",
                        "duration": 0.33765527700597886,
                    },
                    "warmups": [
                        [1, 0.023075559991411865],
                        [2, 0.022522017497976776]
                    ]
                },
                {
                    "metadata": {
                        "date": "2016-10-21 03:14:20.496710",
                        "duration": 0.7234010050015058,
                    },
                    "values": [
                        0.022752201875846367,
                        0.022529058374857414,
                        0.022569017250134493
                    ],
                    "warmups": [
                        [8, 0.02249833550013136]
                    ]
                }
            ]
        }
    ],
    "metadata": {
        "cpu_count": 4,
        "cpu_model_name": "Intel(R) Core(TM) i7-3520M CPU @ 2.90GHz",
        "description": "Telco decimal benchmark",
        "hostname": "selma",
        "loops": 8,
        "name": "telco",
        "perf_version": "0.8.2",
        "tags": ["numeric"]
    },
    "version": "1.0"
}

  6. Understand pyperf benchmark and system metadata

    main

    The Run class automatically collects extensive metadata for every benchmark run. This metadata is categorized into several groups to help you understand the environment and context of your performance measurements:

    Benchmark Metadata

    • date (str): ISO 8601 formatted start date.
    • duration (int or float >= 0): Total run duration in seconds.
    • name (non-empty str): The name of the benchmark.
    • loops (int >= 1): Number of outer-loops per value.
    • inner_loops (int >= 1): Number of inner-loops.
    • timer: The timer implementation used (e.g., time.perf_counter()).
    • tags (list of str, optional): Tags used to aggregate results.

    Python Metadata

    Includes python_compiler, python_cflags, python_executable, python_hash_seed, python_implementation (e.g., cpython, pypy), and python_version (including architecture).

    Memory Metadata

    • command_max_rss (int): Max resident set size in bytes measured by Runner.bench_command.
    • mem_max_rss (int): Max resident set size in bytes (requires Linux kernel 2.6.32+).
    • mem_peak_pagefile_usage (int): Peak Commit Charge (Windows only).

    CPU Metadata

    Includes cpu_affinity, cpu_config (e.g., scaling governor), cpu_count, cpu_freq, cpu_machine, cpu_model_name, and cpu_temp.

    System Metadata

    Includes aslr, boot_time, hostname, platform, load_avg_1min, runnable_threads, and uptime.

    Other

    Includes perf_version, unit (byte, integer, or second), and calibration details like calibrate_loops, recalibrate_loops, calibrate_warmups, and recalibrate_warmups.

  7. Plot benchmark values with matplotlib

    main

    The plot.py script uses matplotlib to visualize benchmark data from a JSON file.

    Usage:

    • To plot all benchmarks in a file: python3 plot.py telco.json
    • To plot a specific benchmark: python3 plot.py -b telco suite.json (where -b is the benchmark name).
    python3 plot.py telco.json

  8. Export benchmark averages to CSV

    main

    The export_csv.py script extracts the average values from a pyperf JSON results file and saves them to a CSV file.

    Usage:

    • Export all: python3 export_csv.py telco.json telco.csv
    • Export specific benchmark: python3 export_csv.py result.json -b telco telco.csv
    python3 export_csv.py telco.json telco.csv

  9. Compare two benchmark result files

    main

    Use the compare_to command to compare a new benchmark result against a baseline JSON file. pyperf uses a Student's two-sample, two-tailed t-test (with alpha = 0.95) to determine if the difference between samples is statistically significant.

    To view the comparison in a clean table format, use the --table option.

    # Basic comparison
$ python3 -m pyperf compare_to py36.json py38.json

# Comparison as a table
$ python3 -m pyperf compare_to mult_list_py36.json mult_list_py38.json --table

  10. Use the pyperf CLI

    main

    The pyperf module provides a command-line interface. You can use the pyperf command directly, or if that is unavailable, use python3 -m pyperf .... The -m pyperf syntax is preferred for the timeit command because it uses the running Python program.

    Note: If a filename is provided as -, pyperf will read the JSON content from stdin.

    python3 -m pyperf <command> [options] [filenames]

  11. Isolate CPUs on Linux using isolcpus

    main

    Isolating at least one core on a multicore Linux system significantly improves benchmark stability.

    1. Identify physical cores

    Use lscpu --extended to identify physical CPU cores and their relationship to sockets and NUMA nodes. For example:

    $ lscpu --extended
CPU NODE SOCKET CORE L1d:L1i:L2:L3 ONLINE MAXMHZ    MINMHZ
0   0    0      0      0:0:0:0       oui    5900,0000 1600,0000
1   0    0      1      1:1:1:0       oui    5900,0000 1600,0000
2   0    0      2      2:2:2:0       oui    5900,0000 1600,0000
3   0    0      3      3:3:3:0       oui    5900,0000 1600,0000

    2. Enable isolation

    To isolate specific cores (e.g., cores 2, 3, 6, and 7), modify your Linux kernel command line in GRUB to include the isolcpus parameter:

    isolcpus=2,3,6,7

    3. Verify isolation

    After rebooting, verify that the CPUs are isolated by checking the sysfs entries:

    $ cat /sys/devices/system/cpu/isolated
2-3,6-7
$ cat /sys/devices/system/cpu/nohz_full
2-3,6-7
    $ lscpu --extended
# ... output ...
# Modify GRUB with:
isolcpus=2,3,6,7

  12. Identify and handle outliers in benchmarks

    main

    Outliers (values significantly slower than the average) often occur if the system is not tuned. pyperf will issue a WARNING: the benchmark result may be unstable if the maximum value is significantly greater than the mean.

    To address outliers:

    1. Rerun the benchmark with more runs, more values, or more loops.
    2. Tune the system by running python3 -m pyperf system tune to reduce system jitter.
    3. Analyze the distribution using pyperf stats, pyperf dump, and pyperf hist to understand the spread.
    4. Use robust statistics: If you cannot achieve stability, rely on the Median and Median Absolute Deviation (MAD) provided by pyperf stats, as these ignore outliers unlike the mean and standard deviation.