Catch2 C++ Unit Testing Framework

repository·devel·Indexed 12 days ago

https://github.com/catchorg/catch2

A lightweight, natural C++ unit testing framework supporting micro-benchmarking and BDD-style testing. It allows assertions to look like standard C++ expressions using macros like REQUIRE and CHECK. Catch2 v3 is a compiled library with multiple headers, migrating from the single-header v2.x version. Key features include expression decomposition, Matchers, exception testing, and CMake integration via catch_discover_tests.

Tokens
47.4K
Snippets
165
Records
223
Agent score
92%

What's inside Catch2

  1. Key features and benefits of Catch2

    devel

    Catch2 is a C++ testing framework designed for ease of use and minimal setup. Key advantages include:

    • Minimal Setup: Can be integrated by adding files directly to your project.
    • No External Dependencies: Requires only a C++14 compatible compiler and the C++ standard library.
    • Self-Registering Tests: Test cases are written as functions or methods that register themselves.
    • Section-Based Isolation: Use SECTION blocks to divide test cases. Each section runs in isolation, which eliminates the need for traditional test fixtures.
    • BDD Support: Supports Behavior-Driven Development (BDD) style using GIVEN, WHEN, and THEN sections.
    • Natural Assertion Syntax: Uses standard C/C++ operators (e.g., ==, !=) within a single core assertion macro. The framework decomposes the expression to log both the left-hand side (lhs) and right-hand side (rhs) values upon failure.
    • Free-form Naming: Test cases are identified by free-form strings rather than being restricted to valid C++ identifiers.
  2. Understand the differences between Old and New filtering behaviour

    devel

    Catch2 provides two distinct ways to filter test paths, which behave differently regarding generators.

    Old Behaviour (Deprecated)

    Trigger: Using only -c or --section.

    • Generators: Completely ignored. They are not filtered and do not affect filter depth. If a generator is a sibling to a filtered section, the generator will still run to completion, potentially causing the code outside sections to execute multiple times.
    • Example: In a test where a section A is filtered but a sibling generator exists, the generator will still exhaust all its elements.

    New Behaviour

    Trigger: Using -g or -p.

    • Generators: Can be explicitly filtered by index (-g 0) or wildcard (-g *).
    • Execution Flow: Unlike sections, a generator must be active. If a generator fails a filter (e.g., a section filter is applied at a depth where a generator exists), the generator cannot proceed, and the test case is skipped (equivalent to SKIP()).
    • Precision: Allows for much tighter control over nested generators and dynamic sections.
  3. Running tests in parallel

    devel
    Catch2 does not support native parallel test execution. Parallelism is intended to be handled by external test runners that can manage separate processes, execution timeouts, and other orchestration tasks. Catch2 provides tools to assist external runners, which can be found in the best practices documentation.
  4. Share setup and teardown using SECTIONS

    devel

    Instead of using class-based fixtures, idiomatic Catch2 uses SECTION macros to share setup and teardown code.

    When a TEST_CASE contains multiple SECTIONs, the TEST_CASE is re-executed from the start for every single section. This ensures that each section (and each leaf section in a nested tree) starts with a fresh state of any local variables declared at the top of the TEST_CASE.

    Sections can be nested to create a tree of execution paths. Each run through a TEST_CASE will follow exactly one path from the root to a single leaf section.

    TEST_CASE( "vectors can be sized and resized", "[vector]" ) {
        // This setup runs once for every SECTION
        std::vector<int> v( 5 );
    
        REQUIRE( v.size() == 5 );
    
        SECTION( "resizing bigger changes size and capacity" ) {
            v.resize( 10 );
            REQUIRE( v.size() == 10 );
        }
        SECTION( "resizing smaller changes size but not capacity" ) {
            v.resize( 0 );
            REQUIRE( v.size() == 0 );
        }
    }
  5. Understand test running event pairs

    devel

    Test running events follow a strict lifecycle where every fooStarting event is paired with a fooEnded event. This allows you to track the lifecycle of the entire test run, individual test cases, sections, and assertions.

    Key pairs include:

    • testRunStarting / testRunEnded: Bookend the entire test execution.
    • testCaseStarting / testCaseEnded: Bookend one full run of a specific test case.
    • testCasePartialStarting / testCasePartialEnded: Bookend a single partial run of a test case (e.g., a single leaf section or a single GENERATE value).
    • sectionStarting / sectionEnded: Emitted only for active SECTIONs that are actually entered.
    • assertionStarting / assertionEnded: Emitted around the evaluation of an assertion (e.g., REQUIRE).
    void testRunStarting( TestRunInfo const& testRunInfo );
    void testRunEnded( TestRunStats const& testRunStats );
    
    void testCaseStarting( TestCaseInfo const& testInfo );
    void testCaseEnded( TestCaseStats const& testCaseStats );
    
    void testCasePartialStarting( TestCaseInfo const& testInfo, uint64_t partNumber );
    void testCasePartialEnded(TestCaseStats const& testCaseStats, uint64_t partNumber );
    
    void sectionStarting( SectionInfo const& sectionInfo );
    void sectionEnded( SectionStats const& sectionStats );
    
    void assertionStarting( AssertionInfo const& assertionInfo );
    void assertionEnded( AssertionStats const& assertionStats );
  6. Perform data and type driven tests

    devel

    Catch2 supports advanced testing patterns where test cases are driven by:

    • Types: Using type-parameterized test cases to run the same logic against different data types.
    • Data: Using Generators to run the same test logic against a variety of input values.
  7. Details on Random Number Generators

    devel

    Catch2's random(a, b) generators produce values uniformly distributed in the closed interval [a, b]. This differs from std::uniform_real_distribution (which is typically [a, b)) to ensure that random(a, a) is a valid operation.

    Reproducibility

    • Integers: Integral generators are fully reproducible across different platforms if the same seed is used.
    • Floating Point: Reproducibility is only guaranteed on platforms obeying the IEEE-754 standard and between binaries using the same floating-point math implementation (e.g., both using SSE2, not one using x87). Using compiler flags like -ffast-math may break reproducibility.
    • long double: No reproducibility guarantees are provided for long double due to platform variations.
  8. How Data Generators work in Catch2

    devel

    Data generators (parametrized test cases) allow you to reuse the same set of assertions across different input values. When you use a generator, the TEST_CASE or SECTION containing it is re-entered for every value produced by the generator.

    Key behaviors:

    • Nesting: Generators respect the ordering and nesting of TEST_CASE and SECTION macros. Nested sections are run once per each value in a generator.
    • Cartesian Product: If multiple GENERATE macros are used at the same scope, the test case will run a number of times equal to the Cartesian product of all elements (e.g., two generators with 2 and 3 elements result in 6 runs).
    • Implicit Sections: GENERATE acts as an implicit SECTION that extends from the point of use to the end of the current scope.

    To use generators, you must include:

    #include <catch2/generators/catch_generators.hpp>
    TEST_CASE("Generators") {
        auto i = GENERATE(1, 3, 5);
        REQUIRE(is_odd(i));
    }
  9. Limitations of Catch2 assertions

    devel

    When using Catch2 assertions, be aware of the following constraints:

    • Logical Operators: You cannot use && or || inside a REQUIRE or CHECK macro because they cannot be safely overloaded for expression decomposition. Split these into multiple assertions or use Matchers.
    • Thread Safety: Assertions in Catch2 are not thread safe.
    • Comma Parsing: As noted in the 'Expressions with commas' section, commas within expressions passed to multi-argument macros can confuse the preprocessor.
  10. Catch2 repeatability guarantees

    devel

    Catch2 provides repeatability guarantees in two specific areas:

    1. Test Case Shuffling: Repeatable across different platforms since v2.12.0. While generally stable across versions, shuffling logic may change to improve randomness.
    2. Random Generators: Since Catch2 3.5.0, random generators use custom distributions designed to be repeatable across different platforms.

    Warning: Prior to v3.5.0, random generators relied on platform-specific <random> distributions, which are not guaranteed to be repeatable across different platforms.

  11. Use tags to group and filter test cases

    devel

    Tags are strings enclosed in square brackets [] associated with a TEST_CASE. They allow you to group related tests and select them via the command line using expressions.

    Tag Selection Logic:

    • "[tag]": Selects all tests with [tag].
    • "[tag1][tag2]": Selects tests that have both [tag1] and [tag2].
    • "[tag1],[tag2]": Selects tests that have either [tag1] or [tag2].

    Rules:

    • Tag names are case-insensitive.
    • Tags can contain any ASCII characters (including spaces), but escapes are not supported (e.g., [\ ] is invalid).
    • The combination of test name and tags must be unique.
    TEST_CASE( "A", "[widget]" ) { /* ... */ }
    TEST_CASE( "B", "[widget]" ) { /* ... */ }
    TEST_CASE( "C", "[gadget]" ) { /* ... */ }
    TEST_CASE( "D", "[widget][gadget]" ) { /* ... */ }
  12. C++ Standard and Polyfilling in Catch2

    devel

    Catch2 targets C++14 as the minimum supported version. While features from higher standards can be used, they should be used sparingly to avoid maintenance overhead.

    When using newer features that are not available in C++14, use a polyfilling pattern. For example, if std::conjunction is available, use it; otherwise, provide a custom implementation. This allows the library to benefit from compiler built-ins when available while maintaining compatibility.