DOCSEARCHbyAlgolia
Submit Docs

Apache Maven Surefire Documentation

repository·master·Indexed 19 days ago

https://github.com/apache/maven-surefire

A test execution framework for Maven providing the maven-surefire-plugin for unit tests, maven-failsafe-plugin for integration tests, and maven-surefire-report-plugin for HTML reports. It supports JUnit 4, JUnit 5/6, and TestNG. The documentation covers its forked JVM architecture, classloader isolation, binary event stream communication protocol, and the unified provider model introduced in version 3.6.0.

Tokens
6.8K
Snippets
18
Records
31
Agent score
66%

What's inside Apache Maven Surefire

  1. What is Apache Maven Surefire?

    master

    Apache Maven Surefire is a test execution framework for Maven. It provides three distinct plugins for different testing stages and reporting needs:

    • maven-surefire-plugin: Used to run unit tests during the Maven test phase.
    • maven-failsafe-plugin: Used to run integration tests during the integration-test and verify phases.
    • maven-surefire-report-plugin: Used to generate HTML test reports from XML results.

    Surefire supports JUnit 4 (4.12+), JUnit 5/6 (Jupiter), and TestNG (6.14.3+). Since version 3.6.0, it uses a single unified provider for all supported testing frameworks.

  2. How the Unified Provider Architecture works in Surefire 3.6.0

    master

    Starting with version 3.6.0, Apache Maven Surefire uses a single unified provider (surefire-junit-platform) based on the JUnit Platform to execute all test frameworks.

    Instead of having separate providers for different frameworks, Surefire auto-detects the available engines on your classpath and delegates execution accordingly. This provides consistent behavior for filtering, parallel execution, and reporting across all frameworks.

    Test FrameworkExecution EngineMinimum Version
    JUnit 5 (Jupiter)Jupiter Engine (native)5.x/6.x
    JUnit 4Vintage Engine4.12
    JUnit 3 testsVintage Engine (via JUnit 4 compatibility)Requires JUnit 4.12+
    TestNGTestNG JUnit Platform Engine6.14.3

  3. How ClassLoader Isolation works in Surefire

    master

    To prevent classpath conflicts between the test framework and Surefire itself, Surefire uses a layered classloader strategy with child-first delegation.

    The Layered Hierarchy

    1. Bootstrap ClassLoader: Contains JDK classes.
    2. System ClassLoader: Contains ForkedBooter, surefire-booter, and surefire-api.
    3. IsolatedClassLoader: A URLClassLoader that uses child-first delegation. It contains the Provider, test classes, and test dependencies. This ensures test classes take precedence over Surefire internals.

    Managed Classpaths

    • surefireClasspath: Contains surefire-booter, surefire-api, and extensions (System classloader).
    • testClasspath: Contains test classes, test dependencies, and the provider (IsolatedClassLoader).
    • inprocClasspath: Contains classes required by both loaders to facilitate bridging (e.g., the SurefireProvider interface).

  4. Understand the 1-line error summary format

    master

    Starting from version 2.13, Maven Surefire provides a compact one-line error summary designed to help you quickly locate test failures. This summary is intended for a high-level overview; for full details, refer to the main test reports or the files generated on disk.

    Formatting Rules

    • Assertion Failures: Only the assertion message is displayed.
    • Exceptions/Errors: The exception name is stripped to save space.
    • Message Trimming: Exception messages are trimmed to approximately 80 characters.
    • The » Symbol: Indicates that the exception occurred in library code called by the test method, rather than in the test method itself.
    • Superclass Methods: Methods in superclasses are typically displayed as SuperClassName.methodName.
    • Stacktrace Origin: If the first method in the stacktrace belongs to a superclass, it is displayed using the format TestClass>Superclass.method.
    Failures:
      Test1.assertion1:59 Bending maths expected:<[123]> but was:<[312]>
      Test1.assertion2:64 True is false

    Errors:
      Test1.nullPointerInLibrary:38 » NullPointer
      Test1.failInMethod:43->innerFailure:68 NullPointer Fail here
      Test1.failInLibInMethod:48 » NullPointer
      Test1.failInNestedLibInMethod:54->nestedLibFailure:72 » NullPointer
      Test2.test6281:33 Runtime FailHere

  5. Understand Fork Configuration variants

    master

    Surefire uses different ForkConfiguration implementations to build the command line for the forked JVM, depending on the environment and classpath requirements:

    • ClasspathForkConfiguration: The default strategy; uses the -cp <full classpath> argument.
    • JarManifestForkConfiguration: Used when the classpath exceeds operating system limits. It creates a temporary JAR file with a Class-Path manifest entry to manage dependencies.
    • ModularClasspathForkConfiguration: Used for Java 9+ module paths; utilizes --module-path and --add-modules.

  6. Handle JDK deprecated modules in Java 9+

    master

    Since Java 9, several modules previously bundled with the JDK are disabled by default.

    • Plugin version 2.20.1: Automatically added the --add-modules java.se.ee option to forked JVM command lines to ease the transition.
    • Plugin version 2.21 and later: Does not add this option.

    Recommended approach: Instead of relying on plugin flags, add explicit dependencies for the maintained versions of these libraries to your project.

    Commonly used libraries that were bundled with Java 8 include:

    • Commons Annotations: javax.annotation:javax.annotation-api:1.3.1
    • JavaBeans Activation Framework: javax.activation:javax.activation-api:1.2.0
    • Java Transaction API: javax.transaction:javax.transaction-api:1.2
    • JAXB: javax.xml.bind:jaxb-api:2.3.0 and org.glassfish.jaxb:jaxb-runtime:2.3.0 (implementation)
    • JAX-WS: javax.xml.ws:jaxws-api:2.3.0 and com.sun.xml.ws:jaxws-rt:2.3.0 (implementation)

  7. How the Forked JVM Architecture works

    master

    A core architectural principle of Surefire is that tests never run in the Maven JVM. Instead, they execute in a separate, forked process. This isolation prevents test execution from interfering with the Maven build process.

    Execution Flow

    1. Detection & Resolution: The AbstractSurefireMojo auto-detects the appropriate provider and resolves its classpath via SurefireDependencyResolver.
    2. Forking: The ForkStarter serializes configuration to a .properties file and builds the JVM command line.
    3. Bootstrapping: The forked JVM launches ForkedBooter.main(), which reads the properties, sets system properties, and establishes a communication channel.
    4. Test Execution: The provider (e.g., JUnit) is loaded via an IsolatedClassLoader and executes the tests.
    5. Communication: Test results are sent from the Forked JVM back to the Maven JVM via a binary event stream protocol (using EventChannelEncoder and EventDecoder) over a pipe or TCP connection. This allows the Maven JVM to update the console and write XML reports in real-time.

  8. Understand the Forked JVM Communication Protocol

    master

    The forked JVM communicates with the Maven process via a binary event stream. Events flow one-way from the fork to Maven using a specific wire format.

    Wire Format

    Events are encoded as: :<magic>:<opcode>:<data>:. This is handled by EventChannelEncoder on the fork side and EventDecoder on the Maven side.

    Transport Channels

    • Pipe (default): Uses pipe:// via stdout/stderr of the forked process.
    • TCP: Uses tcp://host:port via a socket connection (configured via Extensions SPI).

    Event Categories

    Events are categorized by ForkedProcessEventType:

    • Test lifecycle: testset-starting, testset-completed, test-starting, test-succeeded, test-failed, test-skipped, test-error, test-assumption-failure.
    • Console output: std-out-stream, std-out-stream-new-line, std-err-stream, std-err-stream-new-line.
    • Logging: console-info-log, console-debug-log, console-warning-log, console-error-log.
    • Control: bye, stop-on-next-test, next-test.
    • System: sys-prop, jvm-exit-error.

  9. Handle multi-line exception messages in Surefire reports

    master

    Since version 2.19, Maven Surefire includes special handling for multi-line exception messages to improve vertical alignment and readability in test reports. This feature automatically reformats exceptions that use vertical bars (|) for line continuation, ensuring the message content is properly indented and aligned.

    This feature also supports Groovy assertion output formatting.

    java.lang.IllegalArgumentException: My Couch
   |
   May not contain whitespace

becomes:

java.lang.IllegalArgumentException:
The Couch
   |
   May not contain whitespace

  10. Handle breaking changes when upgrading to Surefire 3.6.0+

    master

    If upgrading to version 3.6.0 or later causes issues, use one of the following mitigation strategies:

    Option 1: Pin to Surefire 3.5.x

    Stay on the previous version to avoid the new provider model:

    <plugin>
    <groupId>org.apache.maven.plugins</groupId>
    <artifactId>maven-surefire-plugin</artifactId>
    <version>3.5.4</version>
</plugin>

    Option 2: Use a legacy provider as a plugin dependency

    If you need 3.6.0 features but require legacy provider behavior, add the specific provider as a dependency to the plugin:

    <plugin>
    <groupId>org.apache.maven.plugins</groupId>
    <artifactId>maven-surefire-plugin</artifactId>
    <version>3.6.0</version>
    <dependencies>
        <dependency>
            <groupId>org.apache.maven.surefire</groupId>
            <artifactId>surefire-junit3</artifactId>
            <version>3.5.4</version>
        </dependency>
    </dependencies>
</plugin>

  11. Build and run Maven Surefire tests

    master

    Use the following Maven commands to build the project or run specific tests.

    Requirements:

    • Maven 3.6.3+
    • JDK 8+

    Build Commands

    • Full build with unit tests: mvn clean install
    • Full build with integration tests: mvn clean install -P run-its
    • Build a single module: mvn clean install -pl surefire-booter

    Test Commands

    • Run a single test: mvn test -pl surefire-booter -Dtest=ForkedBooterTest
    • Run a single integration test: mvn verify -pl surefire-its -Prun-its -Dit.test=JUnit47RedirectOutputIT

    IDE Setup

    Before importing the project into an IDE, run:

    # Install shared utils
mvn install -P ide-development -f surefire-shared-utils/pom.xml

# Compile the grouper
mvn compile -f surefire-grouper/pom.xml
    # Full build with unit tests
mvn clean install

# Full build with integration tests
mvn clean install -P run-its

# Build a single module
mvn clean install -pl surefire-booter

# Run a single test
mvn test -pl surefire-booter -Dtest=ForkedBooterTest

# Run a single integration test
mvn verify -pl surefire-its -Prun-its -Dit.test=JUnit47RedirectOutputIT

# IDE setup (required before importing)
mvn install -P ide-development -f surefire-shared-utils/pom.xml
mvn compile -f surefire-grouper/pom.xml

  12. Migrate TestNG tests to Surefire 3.6.0

    master

    TestNG tests are now executed via the TestNG JUnit Platform Engine.

    Requirements & Changes:

    1. Ensure your TestNG version is 6.14.3 or later.
    2. Breaking Change: The suiteXmlFiles configuration is no longer supported. You must now use groups or JUnit suite support for TestNG configuration.
    3. TestNG configuration (groups, listeners, suites, etc.) is mapped to the JUnit Platform infrastructure through Surefire's plugin configuration.
    <dependency>
    <groupId>org.testng</groupId>
    <artifactId>testng</artifactId>
    <version>7.10.2</version>
    <scope>test</scope>
</dependency>