cargo-nextest

Nextest is a next-generation test runner for Rust. The repository consists of the following primary components:

• cargo-nextest: A faster Cargo test runner.
• nextest-runner: The core logic used by cargo-nextest.
• nextest-metadata: A library designed for calling cargo-nextest via the command line.
• nextest-filtering: A parser and evaluator for filtersets.

The cargo-nextest binary follows semantic versioning. Its public API is strictly defined as:

1. Command-line arguments, options, and flags.
2. Machine-readable output.
3. The configuration file format.

Note: Experimental features are not part of the public API and may change or be removed in patch releases. Human-readable output (standard output/error) is not considered part of the public API and is subject to change.

When recording is enabled, nextest persists a full event stream, captured test outputs, and workspace metadata for every test run. This data enables the replay and analysis of test runs and serves as the foundation for iterative reruns.

Key characteristics of the recording subsystem:

• Fault Tolerance: Recording failures (e.g., disk full, I/O errors) will not cause a test run to fail or produce incorrect results. Errors are reported as warnings, and the test run proceeds normally.
• Isolation: Each run is an independent fault domain; corruption in one run's data does not affect others.
• Encapsulation: The complexity of replays and reruns is encapsulated in a dedicated subsystem, keeping the runner loop and user interface clean.

The Nextest runner loop is responsible for executing units of work (typically test attempts, but also setup/teardown scripts) and coordinating scheduling and responses. It uses an actor model where two main components, the dispatcher and the executor, communicate via message passing rather than shared state.

Key components:

• Dispatcher: Interacts with the outside world (signals, input, reporter errors) and manages the high-level state.
• Executor: Responsible for scheduling units of work and running them to completion.
• Units: Async Rust state machines that represent a single unit of work. Each unit has two dedicated channels to the dispatcher: one for sending responses (progress, completion, errors) and one for receiving requests (state queries, job control, cancellation).

nextest uses Tokio's native signal support. A SignalHandler multiplexer receives signals and generates a stream of events. The runner loop's dispatcher selects over this stream and, upon receiving a signal, broadcasts a message to all active units.

Signal Flow:

1. Tokio installs signal handlers.
2. Signal Muxer streams signals to the dispatcher.
3. Dispatcher receives the event.
4. Dispatcher broadcasts the signal to all units.
5. Units respond to the signal.

Nextest integrates with various tools for testing, benchmarking, and analysis:

• Test coverage: Use llvm-cov for coverage reporting.
• Interpreter: Support for the Miri interpreter.
• Debugging: Integration with debuggers and system call tracers.
• Benchmarking: Integration with Criterion benchmarks.
• Mutation testing: Support for cargo-mutants.
• System tracing: Support for USDT probes using bpftrace and DTrace.

nextest records test runs using a structured format designed for high fidelity and efficient storage. The recording consists of three main components:

1. Global Index: A Zstandard-compressed JSON file (runs.json.zst) that tracks all runs. It is stored at projects/<escaped path to workspace>/records/runs/runs.json.zst and contains run IDs, versions, timing, status, and metadata.
2. Run Event Log: An event stream of TestEvent instances serialized as Zstandard-compressed JSON Lines. This allows for full-fidelity replays and ensures that if a run crashes, existing events are preserved.
3. Test Output Store: A zip file containing compressed test outputs (stdout/stderr). It uses content-addressed hashing to deduplicate outputs and employs Zstandard with pre-trained dictionaries to optimize the compression of small text blobs.

Nextest manages interactive input through three main mechanisms:

1. Terminal settings: It modifies terminal settings to disable line buffering and echoing. This allows nextest to read individual characters immediately as they are typed without displaying them on the screen.
2. Event stream: It utilizes crossterm's EventStream to convert keyboard events into a stream compatible with the Tokio runtime.
3. Dispatcher control: The runner loop's dispatcher asynchronously polls the input handler and is responsible for restoring terminal settings (e.g., during SIGTSTP).

To ensure robustness, nextest attempts to restore original terminal settings using both a Drop implementation and a panic hook to handle abnormal exits.

Nextest's execution model generally provides faster test runs than cargo test. You will see the greatest performance improvements in the following scenarios:

• Large workspaces: Workspaces with many crates and test binaries benefit more from Nextest's ability to manage multiple binaries.
• Long-pole test bottlenecks: Nextest can run tests from multiple binaries in parallel, whereas cargo test runs them serially. This helps mitigate bottlenecks caused by individual long-running tests in different binaries.
• Optimized build environments: When using build caching tools like sccache or the Rust Cache GitHub Action, the time saved on compilation makes the faster test execution of Nextest a more significant part of the total end-to-end execution time.

Unlike cargo test, which runs entire test binaries serially and relies on exit codes, cargo-nextest uses a two-phase execution model to provide structured test results and better parallelism:

1. The list phase: cargo-nextest builds all test binaries using cargo test --no-run and then queries them to generate a complete list of all individual tests.
2. The run phase: cargo-nextest executes each individual test in its own separate process in parallel. It collects, displays, and aggregates results for every single test.

Important Note: Because cargo-nextest uses a much deeper interface with test binaries than cargo test, custom test harnesses may need to be adapted to work correctly with this model.