Gremlins Mutation Testing for Go

Gremlins is a mutation testing tool designed for Go, specifically optimized for small-to-medium sized modules like microservices. It validates the effectiveness of your test suite by injecting mutations (small code changes) into the code exercised by your tests and checking if your tests fail as a result.

Key Use Cases:

• Validating test suite quality beyond simple code coverage.
• Aiding the Test-Driven Development (TDD) process.
• Acting as a quality gate in Continuous Integration (CI) pipelines.

Limitations:

• Not recommended for very large Go modules, as execution times can extend to several hours.

Gremlins provides several capabilities for mutation testing in Go:

• Mutant Discovery: Automatically discovers mutant candidates and executes relevant tests.
• Efficiency: Only tests mutants that are actually covered by existing tests.
• Incremental Testing: Supports testing mutants only within PR (Pull Request) changes.
• Mutant Types: Supports five distinct types of mutants.
• Configuration: Uses YAML-based configuration files.
• CI Integration: Designed to run as a quality gate in CI environments.

Gremlins follows a specific order of precedence for configuration settings. If a setting is defined in multiple places, the higher priority source will override the lower ones:

1. Command flags (Highest priority)
2. Environment variables
3. Configuration files (Lowest priority)

The Conditionals negation mutation type inverts the direction of conditional operators. This is used during mutation testing to ensure that your test suite correctly catches logic errors where comparison directions are flipped.

Supported operator transformations:

• == becomes !=
• != becomes ==
• > becomes <=
• <= becomes >
• < becomes >=
• >= becomes <

The Invert loop control mutation type targets loop control operations by swapping their behavior. This is used to test if your test suite can detect when the flow of a loop is incorrectly altered. Specifically, it transforms continue statements into break statements and vice versa.

// Original
for i := 0; i < 3; i++ {
    continue
}

// Mutated
for i := 0; i < 3; i++ {
    break
}

When running in Integration mode, Gremlins is significantly more CPU-intensive. To compensate and prevent system exhaustion, Gremlins automatically halves the effective values for both workers and test CPU.

For example, if you explicitly configure --workers 4, Gremlins will effectively run with only 2 workers when in integration mode. The same halving logic applies to the --test-cpu setting.

The Invert logical mutation type targets logical operators in Go code, swapping them for their opposites to test the robustness of test suites against logical errors.

Specifically, it performs the following transformations:

• Replaces && (logical AND) with || (logical OR).
• Replaces || (logical OR) with && (logical AND).

// Original
a := true && false

// Mutated
a := true || false

The Remove self-assignments mutation tests the robustness of your code by replacing compound assignment operators with simple assignment operators. Instead of performing an operation between the current value and a new value (e.g., a += 2), the mutation performs a trivial assignment (e.g., a = 2). This helps identify if your tests actually verify that the arithmetic or bitwise operation occurred, rather than just checking that the variable was updated.

// Original
a := 1
a += 2

// Mutated
a := 1
a = 2

The Invert bitwise mutation performs inversions on basic bitwise operations. This mutation tests the robustness of your code by swapping bitwise operators with different ones to see if your tests can detect the logic change.

Mutation Mapping

The Arithmetic base mutation set performs inversions on basic arithmetic operations to test the robustness of your code's logic. When this mutation is applied, the following transformations occur:

This is useful for ensuring that your unit tests are sensitive enough to detect incorrect mathematical logic.

// Original code
a := 1 + 2

// Mutated code (Arithmetic base)
a := 1 - 2

Gremlins is currently in 0.x.x release status. Users should be aware of the following SemVer implications:

• No backward compatibility guarantee: Configuration flags and configuration file schemas may change between minor releases.
• Maintenance policy: Only the current minor release is maintained. For example, if the current version is v0.2.0, patch releases will not be provided for the v0.1.0 line.
• Future stability: Backward compatibility guarantees will begin once the project reaches 1.x.x release.