trybuild Documentation

repository·master·Indexed 21 days ago

https://github.com/dtolnay/trybuild

A test harness for verifying Rust compiler diagnostics. trybuild allows developers to write UI tests to assert whether code fails to compile with specific expected error messages or compiles successfully, which is particularly useful for testing procedural macros. It includes features for normalizing compiler output to ensure consistent tests across environments and a TestCases API for managing pass and compile_fail assertions.

Tokens
1.7K
Snippets
8
Records
9
Agent score
76%

What's inside trybuild

  1. How trybuild normalizes compiler error messages

    master

    When comparing compiler output (stderr) against saved snapshots, trybuild uses a normalization process to ensure tests don't break due to non-semantic changes (like absolute paths, line numbers, or minor compiler version differences).

    Instead of a single comparison, trybuild generates a set of Variations. A test passes if the actual compiler output matches any of these variations. This allows the library to introduce new normalization steps without breaking existing snapshots.

    Key concepts in normalization include:

    • Path Normalization: Replacing absolute paths with placeholders like $DIR/, $WORKSPACE/, $RUST/, or $CARGO/.
    • Stripping: Removing noise like error: Could not compile... or For more information... messages.
    • Unindenting: Adjusting the indentation of error blocks to make them more resilient to formatting changes.
    • Placeholder Substitution: Replacing specific crate names or dependency versions with generic tokens like $CRATE or $VERSION.
  2. Use trybuild for pass tests

    master

    Use trybuild::TestCases::pass to assert that a Rust file compiles successfully and that its main function executes without panicking. This is useful for testing successful usage patterns or for workshop-style environments where you want to control the execution of multiple test cases in a single harness.

    #[test]
    fn ui() {
        let t = trybuild::TestCases::new();
        t.pass("tests/01-parse-header.rs");
        t.compile_fail("tests/03-expand-four-errors.rs");
    }
  3. Use trybuild for compile-fail tests

    master

    Use trybuild::TestCases::compile_fail to assert that specific Rust files fail to compile with the expected error messages.

    For each test file (e.g., tests/my_test.rs), trybuild expects an adjacent file with the same name but a .stderr extension (e.g., tests/my_test.stderr) containing the exact expected compiler output. If the compiler output does not match the .stderr file, the test fails and displays a diff.

    Note: A compile_fail test will also fail if the code actually compiles successfully.

    #[test]
    fn ui() {
        let t = trybuild::TestCases::new();
        t.compile_fail("tests/ui/*.rs");
    }
  4. Update expected .stderr files

    master

    When your compiler output changes, you need to update your .stderr files. There are two recommended workflows:

    1. The wip directory method: If a compile_fail test is run and no .stderr file exists, trybuild saves the actual output into a wip/ directory in your project root. You can then move these files from wip/ into your test directory.
    2. The TRYBUILD=overwrite method: Run your tests with the TRYBUILD=overwrite environment variable. This tells trybuild to skip the wip/ directory and write the actual compiler output directly into your existing .stderr files. Always check git diff after using this method to verify the changes.
    TRYBUILD=overwrite cargo test
  5. Ensure consistent diagnostics with rust-src

    master

    The Rust compiler's diagnostic output (including source snippets) can change depending on whether the rust-src component is installed via rustup. To ensure consistent test results across different environments (like local dev vs. CI), add a rust-toolchain.toml file to your project:

    [toolchain]
    components = ["rust-src"]
  6. TestCases API reference

    master

    The TestCases struct is the primary entry point for the trybuild harness. It is typically instantiated within a standard #[test] function.

    • new() -> TestCases: Creates a new test runner instance.
    • pass<P: AsRef<Path>>(&self, path: P): Registers a file that is expected to compile successfully and run without panicking.
    • compile_fail<P: AsRef<Path>>(&self, path: P): Registers a file that is expected to fail compilation. It expects a corresponding *.stderr file to exist for comparison.
    let t = trybuild::TestCases::new();
    t.pass("tests/success.rs");
    t.compile_fail("tests/failure.rs");
  7. Understand the Variations API

    master

    The Variations struct represents the collection of possible normalized versions of a compiler output.

    • preferred(): Returns the 'last' variation. This is the version used for display when a test fails or when the stderr file is missing.
    • any<F>(mut f: F): A utility to check if any of the generated variations satisfy a predicate f. This is how trybuild determines if a test passes.
    • concat(&mut self, other: &Self): Merges another set of variations into the current one.
    // Example of how Variations are used conceptually
    if variations.any(|stderr| stderr == expected_output) {
        // Test passes
    }
    
    let preferred_output = variations.preferred();
  8. Reference: Normalization steps

    master

    The following Normalization enum variants define the different levels of transformation applied to compiler output. These are applied cumulatively based on their order. New steps are added to the end of the list to maintain backward compatibility with older snapshots.

    // Available Normalization levels:
    Basic,
    StripCouldNotCompile,
    StripCouldNotCompile2,
    StripForMoreInformation,
    StripForMoreInformation2,
    TrimEnd,
    RustLib,
    TypeDirBackslash,
    WorkspaceLines,
    PathDependencies,
    CargoRegistry,
    ArrowOtherCrate,
    RelativeToDir,
    LinesOutsideInputFile,
    Unindent,
    AndOthers,
    StripLongTypeNameFiles,
    UnindentAfterHelp,
    AndOthersVerbose,
    UnindentMultilineNote,
    DependencyVersion,
    HeadingNote,
    UnindentSuggestion,
    CustomRegistry,
    MorePathDependencies,
  9. Filter trybuild tests via CLI

    master

    You can run a subset of your trybuild tests by passing a filter string to cargo test. The filter string must be prefixed with trybuild= and passed as an argument after the -- separator. Only test cases whose filenames contain the filter string will be executed.

    Example usage:

    $ cargo test -- ui trybuild=tuple_structs.rs