DOCSEARCHbyAlgolia
Submit Docs

mutmut

repository·main·Indexed 23 days ago

https://github.com/boxed/mutmut

A mutation testing system for Python 3 designed to improve test suite quality by injecting small changes (mutants) into code to verify if existing tests detect them. It features incremental progress, a TUI for browsing mutants, and support for various execution strategies (fork, collect, import, none). Configuration is supported via setup.cfg or pyproject.toml, with options for line-level coverage filtering, type-checker integration (mypy, pyrefly), and custom mutation exclusions using pragmas or regex.

Tokens
6.4K
Snippets
21
Records
44
Agent score
79%

What's inside mutmut

  1. How mutmut run works

    main

    The mutmut run command operates in several distinct phases to ensure mutation testing is performed accurately and efficiently.

    1. Generating mutants: Creates a ./mutants/ directory. It copies source files to this directory, applies mutations to .py files, and copies additional required files (specified by also_copy).
    2. Collecting tests and stats: Identifies and executes tests to map which tests cover which mutants and tracks execution time for performance optimization. Results are stored in ./mutants/mutmut-stats.json.
    3. Collecting mutation results: Loads existing results from .meta files located next to the mutated code (e.g., mutants/foo/bar.py.meta).
    4. Running clean tests: Runs the full test suite with all mutants disabled to verify the baseline test setup is functional.
    5. Running forced fail test: Verifies the mutation mechanism by forcing all mutants to raise an Exception. This ensures the test suite actually detects changes in the code.
    6. Running mutation testing: The core phase where each mutant is executed against the relevant subset of tests. If a test fails, the mutant is considered 'killed'. Results are persisted in .meta files.

  2. Understand how mutmut performs mutations

    main

    mutmut performs subtle changes to your source code to test the effectiveness of your test suite. Examples of mutations include:

    • Integer literals: Changed by adding 1 (e.g., 0 becomes 1, 5 becomes 6).
    • Comparison operators: < is changed to <=.
    • Control flow: break is changed to continue and vice versa.

    The goal is to make mutations as subtle as possible to ensure that only meaningful changes trigger test failures.

  3. Enable a specific mutant using MUTANT_UNDER_TEST

    main
    Mutmut uses the MUTANT_UNDER_TEST environment variable to control which specific mutant is active during a run. When a mutant is enabled via this variable, the mutated code is executed; otherwise, the original code is used. This is useful for debugging specific mutation failures.

  4. Disable mutation using code comments (Pragmas)

    main

    You can use special comments to skip mutation in specific parts of your code:

    • # pragma: no mutate: Disables mutation on a single line.
    • # pragma: no mutate block: Disables mutation for the entire indentation block (function, class, if branch, etc.).
    • # pragma: no mutate start / # pragma: no mutate end: Disables mutation for a specific range of lines, regardless of indentation.

    Restrictions: You cannot nest a new block or start context while another is active; doing so will raise a PragmaParseError.

    # Single line
some_code_here()  # pragma: no mutate

# Entire block (function or class)
def complex_algorithm():  # pragma: no mutate block
    return some_complex_calculation()

# Specific range
a = mutate_this()

# pragma: no mutate start
b = skip_this()
c = skip_this_too()
# pragma: no mutate end

d = mutate_this_too()

  6. Lint and format code with ruff and pre-commit

    main

    The project uses ruff for linting and formatting, managed via pre-commit.

    To run linting and formatting on all files manually: uv run pre-commit run --all-files

    To ensure these checks run automatically during git commit, install the hooks locally: pre-commit install

    uv run pre-commit run --all-files
pre-commit install

  7. Install and run mutmut

    main

    To start mutation testing, install mutmut via pip and execute the run command. By default, mutmut run will attempt to locate your source code and execute pytest on tests found in tests or test directories.

    Key behaviors:

    • Incremental Progress: You can stop a mutation run at any time; mutmut will remember completed work and restart from where it left off.
    • Platform Requirement: mutmut requires a system with fork support. On Windows, you must use WSL (Windows Subsystem for Linux).
    • Dependency Note: If installation fails on libcst due to a missing Rust compiler, you may need to install the rustc and cargo toolchain from the Rust website.
    pip install mutmut
mutmut run

  9. Run mutmut tests

    main

    Run the test suite using uv run pytest.

    If you are performing E2E or integration tests and the output changes intentionally, use the --inline-snapshot=fix flag to update the snapshots managed by inline-snapshot.

    Troubleshooting: If pytest terminates abruptly without reporting failures, it may be due to mutmut calling os._exit(...). Check these calls in the source code to troubleshoot.

    uv run pytest
# To update snapshots if output changes are expected:
uv run pytest --inline-snapshot=fix

  10. Run mutation testing in the Benchmark 1K project

    main

    To execute mutation testing within the synthetic Benchmark 1K project, navigate to the project directory and run the mutmut run command. This project is designed to validate mutmut's process isolation and hot-fork warmup strategy performance using 1000 mutants.

    cd e2e_projects/benchmark_1k
mutmut run

  11. Use the mutmut workflow to improve tests

    main

    To use mutmut to enhance your test suite, follow this iterative workflow:

    1. Run mutations: Execute mutmut run to start the mutation testing process. You can stop the run early if you want to focus on immediate results.
    2. Browse mutants: Use mutmut browse to view the list of generated mutants.
    3. Fix tests: Identify a surviving mutant and write a new test case specifically designed to catch (kill) that mutation.
    4. Verify: Within the mutmut browse interface, press r to rerun the specific mutant and verify if your new test successfully killed it.

    Note on state: mutmut stores its progress and mutant data in a mutants/ directory. To perform a completely fresh run from scratch, delete the mutants/ directory.

  12. Work with mutants using browse and apply

    main

    After running mutations, use the following commands to interact with the results:

    • mutmut browse: Opens an interactive terminal UI to view mutants and retest them after you have updated your tests.
    • mutmut apply <mutant>: Writes a specific mutant to disk.

    Warning: Always ensure your source code is committed to version control before using apply to avoid losing uncommitted changes.

    mutmut browse
mutmut apply <mutant>