BenchExec: Framework for Reliable Benchmarking and Resource Measurement

BenchExec is a framework for reliable benchmarking and resource measurement consisting of three primary programs:

• benchexec: The main utility used for managing and executing large sets of benchmark runs.
• table-generator: A utility for generating result tables based on the output produced by benchexec.
• runexec: A tool for benchmarking a single tool execution. It serves as a more accurate and feature-rich replacement for the standard time command and can be integrated into other benchmarking frameworks or scripts.

Before installing, ensure your system meets these core requirements:

• Python: 3.10 or newer.
• OS: Linux.
• Architecture: x86 or ARM.
• Cgroups: Access to cgroups is required for resource measurement and limiting.
• Kernel:
  • Ideal: Linux 5.19 or newer with cgroups v2.
  • Minimum recommended: Linux 4.14 or newer.
  • Note: BenchExec will remove support for cgroups v1 in April 2027.
• NUMA: Support is expected; running on kernels without NUMA support is not guaranteed to work.

BenchExec strictly follows SI standard units for all measurements. Note that binary prefixes (IEC standards) are not supported.

• Time: Measured in seconds.
• Memory: Measured in bytes. For prefixes like Kilo, Mega, etc., BenchExec uses the SI factor of 1000 (e.g., 1 KB = 1000 bytes).

Warning: Kibibytes (KiB), Mebibytes (MiB), etc., which use a factor of 1024, are not supported.

P4 benchmark files follow the standard Benchexec XML structure but must include three specific option tags to define the P4 environment:

1. <option name="ptf_test_folder">path/to/ptf_test_folder</option>: Path to the PTF test folder.
2. <option name="switch_folder">path/to/switch_folder</option>: Path to the switch folder.
3. <option name="network_config">path/to/network_config_file</option>: Path to the network configuration file.

Switch Folder Layout

The switch_folder (e.g., switch_bmv2) must contain the following subdirectories:

• log/: Used by the program.
• P4/: Where the switch expects to find the setup file.
• tables/: Where the switch looks for defined table entries.

BenchExec distinguishes between CPU time and Wall time.

CPU Time:

• Refers to the time the tool was actually using CPU cores (excluding sleep time).
• For multi-core tools, the CPU time is the sum of usage across all cores.
• Note: Enforcement is not perfectly exact; a tool might run ~1 second longer than the limit. If it terminates within this window, BenchExec still reports it as a timeout in the status value.

Wall Time:

• BenchExec automatically sets a wall-time limit slightly higher than the CPU-time limit to prevent deadlocks from hanging indefinitely.
• For runexec, you can explicitly specify a wall-time limit, but you cannot disable it entirely if a CPU-time limit is active.

BenchExec defines a "core" as the smallest hardware unit capable of executing a thread in parallel (often called a "virtual core" or "processor").

When assigning multiple cores to a run, BenchExec attempts to select cores that are physically close in the CPU architecture to optimize performance. For example, on a system with hyper-threading, assigning 8 cores will allocate 4 physical cores (each with 2 hyper-threading threads).

Key behaviors:

• Hyper-threading: By default, BenchExec uses virtual cores. Use --no-hyperthreading to ensure only one virtual core per physical core is used, leaving the others idle.
• Core Selection: You can restrict which cores are used by providing a list of allowed cores via --allowedCores.

The SLURM extension automatically calculates and passes several parameters to srun based on the benchmark's requirements:

• -t <hh:mm:ss>: CPU time limit.
• -c <cpus>: Number of CPUs.
• --threads-per-core=1: Forces one thread per core.
• --mem-per-cpu <mem/cpus>: Memory allocation in MB per CPU.
• --ntasks=1: Number of tasks per node.

Resource Data Collection

After a job completes, the script runs seff to collect the following metrics:

• Exit code
• Status
• CPU time [s]
• Wall time [s]
• Memory [MB]

BenchExec reports result values for every executed run through two primary interfaces:

1. XML Result Files (benchexec): Results are stored as <column title="<key>" value="<value>"/> tags inside <run> tags. The start time is stored in ISO 8601 format with a time zone.
2. Python API (runexec): The RunExecutor.execute_run() method returns results as a dictionary. In this dictionary:
   • Integer values are int.
   • Decimal numbers are arithmetic Python types.
   • Start time is an "aware" datetime.datetime instance in local time.
   • Other values are strings.
   • Complex values are dict objects.
   • Note: Instead of returnvalue and exitsignal, the API uses a field named exitcode which contains an instance of benchexec.util.ProcessExitCode.

The contrib/plots directory provides helper files for two different plotting engines:

Gnuplot

Uses *.gp files for plot definitions. Available plot types include:

• quantile.gp: Standard quantile plot.
• quantile-score.gp: Score-based quantile plot.
• quantile-split.gp: Quantile plot with a linear scale for y-range [0,1] and a logarithmic scale beyond.
• scatter.gp: Standard scatter plot.
• scatter-counted.gp: Scatter plot where point color indicates the number of occurrences (ideal for discrete data).

LaTeX (TikZ/PGFPlots)

Uses *.tex files for plot definitions. These are designed to be used with the pgfplots and standalone packages. These files can be compiled into standalone figures or their tikzpicture environments can be copied into existing LaTeX documents.