DOCSEARCHbyAlgolia
Submit Docs

Cosmic Ray Documentation

repository·master·Indexed 20 days ago

https://github.com/sixty-north/cosmic-ray

A mutation testing tool for Python 3 that evaluates test suite effectiveness by injecting small changes into source code and running tests to see if they fail. It utilizes AST manipulation via parso to create mutants and supports various distributors, such as LocalDistributor and HttpDistributor, to execute tests. The tool includes a system for sessions, TOML-based configuration, and filters like cr-filter-operators, cr-filter-pragma, and cr-filter-git to refine mutation scope.

Tokens
17.9K
Snippets
77
Records
107
Agent score
70%

What's inside Cosmic Ray

  1. What is Cosmic Ray?

    master
    Cosmic Ray is a mutation testing tool designed for Python 3. It works by injecting small, controlled changes (mutations) into your source code and then running your existing test suite against each version. This process helps you evaluate the effectiveness of your tests by determining if they can detect the changes made to the code.

  2. What is Cosmic Ray mutation testing?

    master

    Cosmic Ray is a mutation testing tool for Python 3. It works by making small, automated changes (mutations) to your production source code and then running your existing test suite against each mutated version.

    Key distinction from coverage analysis:

    • Coverage analysis tells you if a line of code was executed during tests.
    • Mutation testing determines if your tests actually validate the behavior of that code.

    If your test suite passes even after the code has been mutated, it indicates a 'survived mutant,' meaning your tests are not effectively checking that specific functionality.

  3. Overview of the cosmic_ray.operators package

    master
    The cosmic_ray.operators package contains the core mutation logic for Cosmic Ray. Operators are responsible for defining how code is transformed to create mutants. The package is organized into specialized submodules, each handling a specific type of code mutation (e.g., replacing boolean values, changing comparison operators, or removing decorators).

  4. What is mutation testing and how does it work?

    master

    Mutation testing is a technique used to verify the effectiveness of your test suite. It works by making controlled, intentional changes (mutations) to your code under test. You then run your existing test suite against this mutated code to see if your tests can detect the changes.

    There are three primary outcomes for a mutant:

    1. Killed: Your test suite fails when run against the mutated code. This is the desired outcome, as it means your tests successfully detected the incorrect behavior.
    2. Survived: Your test suite passes despite the mutation. This indicates that your tests are not adequately checking the specific logic that was changed.
    3. Incompetent: The mutation causes the code to crash (e.g., a syntax error or a fatal runtime error) rather than just changing the logic.

    The ultimate goal is to maximize the number of 'killed' mutants. A surviving mutant suggests you either need to write better/more tests or remove the unnecessary code that the mutation affected.

  5. How Cosmic Ray performs mutation testing

    master

    Cosmic Ray implements mutation testing by manipulating the Abstract Syntax Trees (AST) of the Module Under Test (MUT) and its submodules.

    1. Parsing: It uses parso to parse the source code into ASTs.
    2. Mutation: Mutation operators walk these trees to modify or delete specific nodes.
    3. Code Generation: The modified ASTs are converted back into source code and written to disk.
    4. Execution: Cosmic Ray runs user-supplied test commands against the mutated code on disk.

    Outcomes of a mutation run:

    • Mutant Survived: The tests passed despite the mutation (the mutation was not caught).
    • Mutant Killed: The tests failed as expected due to the mutation.
    • Mutant Incompetent: An exception occurred during the test run (e.g., a syntax error or runtime error caused by the mutation itself).
    for mod in modules_under_test:
    for op in mutation_operators:
        for site in mutation_sites(op, mod):
            mutant_ast = mutate_ast(op, mod, site)
            write_to_disk(mutant_ast)

            try:
                if discover_and_run_tests():
                    print('Oh no! The mutant survived!')
                else:
                    print('The mutant was killed.')
            except Exception:
                print('The mutant was incompetent.')

  6. Best practices for separating test and production code

    master

    Cosmic Ray attempts to mutate all code within a specified module. If test code resides in the same module as production code, Cosmic Ray will mutate the tests, which is usually undesirable.

    Recommendations:

    1. Separate Modules: Keep test code in separate modules from production code. This allows you to target only production code for mutation.
    2. Separate Packages: Ideally, keep tests in a different package entirely. This allows you to mutate an entire package without filtering.
    3. Use excluded-modules: If tests must reside in the same package, use the excluded-modules setting in your config.toml to prevent Cosmic Ray from mutating them.

  7. What are Sessions and how to initialize them

    master

    A session is a database (typically a .sqlite file) that records the work to be done and the results returned by workers. Sessions allow Cosmic Ray to be interrupted and restarted safely, as the database tracks completed work. They also enable post-facto analysis and report generation.

    To initialize a session, use the init command, specifying a configuration file and a session name (which becomes the database filename).

    cosmic-ray init config.toml session.sqlite

  9. What are Operators in Cosmic Ray

    master

    An operator is a class that represents a specific type of mutation. It has two primary roles:

    1. Identification: It finds points in the code where a specific mutation can be applied.
    2. Execution: It performs the mutation (e.g., replacing a break node with a continue node).

    Operators are implemented as subclasses of cosmic_ray.operators.operator.Operator and are exposed via plugins. Users can extend the available set by providing their own operators.

    from cosmic_ray.operators.operator import Operator

class MyCustomOperator(Operator):
    # Implementation details for identifying and applying mutations
    pass

  10. How the HTTP distributor works for distributed mutation testing

    master

    The cosmic_ray.distributors.http.HttpDistributor allows you to parallelize mutation testing by sending requests to one or more workers running locally or remotely.

    Key Concepts:

    • Workers: Each worker is a small HTTP server that handles one mutation request at a time.
    • Isolation: Each worker must have its own independent copy of the code under test. Because Cosmic Ray modifies code on disk to perform mutations, workers cannot share the same directory without interfering with each other.
    • Execution Flow: You must start your workers before running the exec command. The exec command does not start workers automatically; it only communicates with the endpoints provided in your configuration.

  11. What are Distributors in Cosmic Ray

    master

    Distributors determine the execution context for tests. They are implemented as plugins and are responsible for taking a sequence of pending mutations and executing them in the appropriate environment.

    Common distributors include:

    • LocalDistributor: Runs tests on the local machine, modifying code in-place and running tests serially (no concurrency).
    • HttpDistributor: Distributes tests to remote workers via HTTP. This allows for parallel execution, where each worker typically maintains its own copy of the code under test.

    Distributors can require specific infrastructure; for example, the HttpDistributor requires workers to be started prior to execution and requires each worker to have its own copy of the code.