rlt Load Testing Library

rlt allows you to track performance changes over time by saving and comparing results.

Saving a Baseline

Save current results with a name:

mybench --url http://localhost:8080 -c 10 -d 30s --save-baseline v1.0

Baselines are stored in target/rlt/baselines/ by default. You can change this using --baseline-dir or the RLT_BASELINE_DIR environment variable.

Comparing Against a Baseline

Compare a new run against a saved baseline. The output shows color-coded deltas:

• Green: Performance improved
• Red: Performance regressed
• Yellow: Within noise threshold (unchanged)

mybench --url http://localhost:8080 -c 10 -d 30s --baseline v1.0

CI/CD Integration

To automatically fail a pipeline if performance regresses, use --fail-on-regression. You can also specify which metrics to monitor using --regression-metrics (e.g., latency-p99, success-ratio).

# Fail on regression
mybench --baseline main --fail-on-regression

# Fail on specific metrics
mybench --baseline main --fail-on-regression --regression-metrics latency-p99,success-ratio

Compare and Save

You can compare against an old version and save the new result as a new baseline in one command:

mybench --baseline v1 --save-baseline v2

You can use rlt in CI/CD pipelines to automatically detect and fail on performance regressions.

Fail on Regression

Use the --fail-on-regression flag. When this flag is set, the CLI will exit with code 1 if a regression is detected. If the verdict is improved, unchanged, or mixed (where mixed is treated as a regression), the exit code will be 1.

Customizing Regression Metrics

By default, rlt uses a specific set of metrics to calculate the performance verdict. You can override this using --regression-metrics <METRICS>, where <METRICS> is a comma-separated list of metric names.

Supported metrics include:

• Throughput: iters-rate, items-rate, bytes-rate
• Latency: latency-mean, latency-median, latency-p90, latency-p99, latency-max
• Reliability: success-ratio

Noise Threshold

Use --noise-threshold <PERCENT> to define the threshold below which changes are considered "unchanged". The default is 1.0 (1.0%).

# CI mode: fail on regression
mybench --url http://localhost:8080 --baseline main --fail-on-regression

# Custom regression metrics
mybench --url http://localhost:8080 --baseline main \
    --regression-metrics iters-rate,latency-mean,latency-p99

rlt allows you to save benchmark results as named baselines and compare future runs against them to detect performance regressions or improvements.

Baseline Management Commands

• Save a baseline: Use --save-baseline <NAME> to store the current run. Note that this overwrites existing baselines of the same name by default.
• Compare against a baseline: Use --baseline <NAME> to compare the current run against a previously saved named baseline.
• Compare against a file: Use --baseline-file <PATH> to compare against a specific JSON baseline file (this is mutually exclusive with --baseline).

Baseline Directory Resolution

If you do not specify a directory using --baseline-dir <PATH>, rlt searches for baselines in this order:

1. The directory provided via --baseline-dir.
2. The RLT_BASELINE_DIR environment variable.
3. ${CARGO_TARGET_DIR}/rlt/baselines.
4. target/rlt/baselines (default).

Baseline Name Rules

Names must match the pattern [a-zA-Z0-9_.-]+.

# Save current run as baseline "v1.0"
mybench --url http://localhost:8080 -c 10 -d 30s --save-baseline v1.0

# Compare against baseline "v1.0"
mybench --url http://localhost:8080 -c 10 -d 30s --baseline v1.0

# Compare against main and save as feature-branch
mybench --url http://localhost:8080 -c 10 -d 30s --baseline main --save-baseline feature-branch

For benchmarks requiring per-worker state (such as HTTP clients or database connections), implement the BenchSuite trait.

1. Use the bench_cli! macro to define your CLI options (which uses clap).
2. Implement BenchSuite, defining your WorkerState type.
3. Implement setup to initialize the WorkerState for each worker.
4. Implement bench to perform the actual workload using the provided WorkerState.
5. Use the bench_cli_run! macro in your main function to run the test.

bench_cli!(HttpBench, {
    /// Target URL.
    #[clap(long)]
    pub url: Url,
});

#[async_trait]
impl BenchSuite for HttpBench {
    type WorkerState = Client;

    async fn setup(&mut self, _worker_id: u32) -> BenchResult<Self::WorkerState> {
        Ok(Client::new())
    }

    async fn bench(&mut self, client: &mut Self::WorkerState, _: &IterInfo) -> BenchResult<IterReport> {
        let t = Instant::now();
        let resp = client.get(self.url.clone()).send().await?;
        let status = resp.status().into();
        let bytes = resp.bytes().await?.len() as u64;
        let duration = t.elapsed();
        Ok(IterReport { duration, status, bytes, items: 1 })
    }
}

#[tokio::main]
async fn main() -> rlt::Result<()> {
    bench_cli_run!(HttpBench).await
}