vtebench

Benchmarks are defined as directories within the ./benchmarks folder. Each benchmark directory must contain:

• A benchmark executable: The core logic. Its stdout is used as the benchmark payload.
• A setup executable (optional): Used for one-time initialization tasks.

Note on execution: The stdout of the benchmark executable is automatically repeated to ensure a sufficient sample size. To prevent repeated overhead, move any logic that should only run once into the setup executable.

./benchmarks/
└── my-new-benchmark/
    ├── benchmark
    └── setup (optional)

vtebench is a specialized tool that only measures the speed at which a terminal emulator reads from the PTY. It is not a general-purpose performance benchmark. It does not account for critical terminal performance factors such as:

• Frame rate
• Latency

Results should not be used to draw broad conclusions about the overall performance of a terminal emulator.

To visualize benchmark performance, you must first export the results to a .dat file using the --dat flag. Once the data file is generated, use the provided shell scripts in the ./gnuplot directory to create SVG plots. You can plot a single data file or combine multiple .dat files to compare results.

# 1. Generate the .dat file
cargo run --release -- --dat results.dat

# 2. Generate a summary SVG plot
./gnuplot/summary.sh results.dat output.svg

# Or combine multiple results
./gnuplot/summary.sh *.dat output.svg

# Or generate detailed plots in a directory
./gnuplot/detailed.sh *.dat output/

To execute the collection of default benchmarks provided in the repository, ensure you have a Rust toolchain installed and run the project using cargo in release mode. This will execute all benchmarks located in the ./benchmarks directory.

cargo run --release

The benchmark process follows a three-stage lifecycle:

1. Discovery: find_benchmarks scans directories for setup and benchmark files. Each directory becomes a BenchmarkLoader.
2. Loading: BenchmarkLoader::load executes the setup and benchmark scripts. The output is captured into memory. If the output is smaller than min_bytes, it is tiled (repeated) to reach the target size.
3. Execution: Benchmark::run performs warmup runs, then enters a loop that writes the setup and benchmark data to STDOUT, measuring the time the write operation blocks, until max_secs or max_samples is reached.

To execute a loaded Benchmark, use the run method. This method performs warmup runs to fill the PTY buffer and then collects samples for a specified duration or count.

Each sample involves:

1. Resetting the terminal (writing \x1bc).
2. Executing the setup data (if provided).
3. Writing the benchmark data to STDOUT and measuring the time taken for the write operation to block.

// warmup_runs: number of initial runs to fill buffer
// max_secs: maximum duration to collect samples
// max_samples: optional limit on the number of samples
let results = benchmark.run(5, 30, Some(100));

The Format trait defines the interface for displaying benchmark results. To create a custom output format, implement this trait and provide a format method that accepts a slice of Results.

use crate::format::Format;
use crate::bench::Results;

struct MyCustomFormat;

impl Format for MyCustomFormat {
    fn format(&self, results: &[Results]) {
        // Implement custom logic to display results
    }
}

The Results object contains the statistical analysis of the benchmark execution. You can extract various metrics in milliseconds:

• min(): Fastest sample.
• max(): Slowest sample.
• mean(): Arithmetic mean.
• median(): Middle value.
• stddev(): Standard deviation.
• percentile(n): The value below which n percent of samples fall.
• samples(): Returns the raw chronological samples.
• bench_size(): The size of the benchmark data in bytes.

println!("Mean: {}ms", results.mean());
println!("90th Percentile: {}ms", results.percentile(90));
println!("Samples collected: {}", results.sample_count());

To find available benchmarks, use find_benchmarks by providing a list of directory paths. The function recursively searches for directories containing two specific files: setup and benchmark.

Each directory containing these files is treated as a single benchmark named after the directory. A valid benchmark must have a benchmark file; the setup file is optional.

let paths = vec![PathBuf::from("./benchmarks")];
let loaders = find_benchmarks(&paths);

The DatFormat struct implements the Format trait to export benchmark Results into a .dat file format compatible with Gnuplot.

When formatted, the output file follows a columnar structure:

1. The first line contains the names of the benchmark runs as column headers.
2. Subsequent lines contain the sample values for each run.
3. If a specific benchmark run has fewer samples than the maximum sample count in the set, the missing values are represented by an underscore (_) placeholder.

Example output structure:

benchmark_one benchmark_two 
1 3 
2 2 
3 _ 

// To use DatFormat, initialize it with a target file path
let formatter = DatFormat::new(PathBuf::from("results.dat"));

// Call the format method (from the Format trait) to write the results
formatter.format(&results);

A BenchmarkLoader holds the metadata needed to initialize a benchmark without having loaded the data into memory. To create a Benchmark instance, call load on a loader and specify min_bytes.

If the data produced by the benchmark script is smaller than min_bytes, the data will be repeated in full until the min_bytes threshold is met or exceeded. If the benchmark script produces no output, an Error::Empty is returned.

// Assuming 'loader' is a BenchmarkLoader
let benchmark = loader.load(1024 * 1024)?; // Load at least 1MB of data

The StdoutFormat implementation provides a human-readable text representation of benchmark results printed directly to the standard output. For each benchmark result, it displays:

• The benchmark name
• The number of samples taken
• The memory size in MiB
• The average execution time (mean)
• The 90th percentile latency
• The standard deviation (±) of the execution time

// Example of the output structure produced by StdoutFormat:
Results:

  benchmark_name (100 samples @ 1.50 MiB):
    10.50ms avg (90% < 12.00ms) +-0.50ms