SlipCover

repository·main·Indexed 18 days ago

https://github.com/plasma-umass/slipcover

A high-performance Python code coverage tool designed for near-zero overhead. It utilizes JIT instrumentation and de-instrumentation, or the sys.monitoring API in Python 3.12+, to track code execution with minimal impact on program speed. SlipCover supports branch and line coverage, multiple output formats including JSON, XML, and LCOV, and integrates with test harnesses like pytest and pytest-xdist.

Tokens
5.8K
Snippets
19
Records
28
Agent score
66%

What's inside slipcover

  1. How SlipCover works: JIT instrumentation

    main

    Unlike tools like Coverage.py that use Python's sys.settrace (which adds significant overhead), SlipCover uses just-in-time (JIT) instrumentation and de-instrumentation.

    1. Instrumentation: SlipCover modifies Python bytecodes by inserting instructions to track executed lines.
    2. De-instrumentation: As the program executes, SlipCover removes instrumentation that is no longer needed, allowing those parts to run at full speed.
    3. Python 3.12+: On Python 3.12 and newer, SlipCover utilizes the sys.monitoring API instead of rewriting bytecode to collect coverage information.
  2. How Slipcover handles annotation-only lines

    main

    Slipcover is designed to ignore lines that only contain bytecode for loading type annotations (e.g., LOAD_NAME, LOAD_GLOBAL, LOAD_ATTR, BINARY_SUBSCR). This ensures that coverage percentages are not skewed by metadata that does not represent actual program logic.

    In Python versions < 3.14, where annotations are evaluated eagerly, Slipcover identifies these lines by collecting all opcodes per line and checking if the set of opcodes is a subset of _ANNOTATION_ONLY_OPS.

  3. Configure SlipCover via pyproject.toml

    main

    You can persist SlipCover settings in your project's pyproject.toml file under the [tool.slipcover] section. SlipCover automatically discovers the nearest pyproject.toml by walking up from the current working directory.

    Note: Command-line arguments always take precedence over pyproject.toml settings. The following options are not configurable via TOML and must be passed per-invocation: --merge, -m/module, the script argument, --version, and --help.

    [tool.slipcover]
    branch = true
    source = "src"        # or ["src", "lib"]
    omit = "tests/*"       # or ["tests/*", "*.pyc"]
    fail-under = 80.0
    json = true
    xml = false
    pretty-print = true
    skip-covered = true
    immediate = false
    out = "coverage.json"
    threshold = 75
    missing-width = 120
    xml-package-depth = 3
  4. How slipcover works with pytest-xdist

    main

    Slipcover includes a pytest plugin designed to support parallel testing via pytest-xdist. When running tests in parallel, the plugin coordinates coverage collection across multiple worker processes using a shared temporary directory.

    The Workflow:

    1. Controller (Main Process): Detects xdist usage, creates a shared temporary directory, and sets the SLIPCOVER_COVERAGE_DIR environment variable.
    2. Workers: Inherit the shared directory from the controller. Each worker collects its own coverage and writes it to a unique file named coverage-<worker_id>.json (e.g., coverage-gw0.json) within that directory.
    3. Session End: The controller identifies all worker coverage files, merges them into a single dataset using sc.merge_coverage, and writes the final result to merged.json in the shared directory.

    This plugin is automatically activated when the SLIPCOVER_ENABLED environment variable is set (typically by running slipcover via its __main__.py entry point).

  5. Run coverage with a test harness like pytest

    main

    SlipCover can execute Python modules directly. To run pytest with coverage, use the -m flag to specify the module and pass any pytest arguments after it. No pytest plugin is required.

    python3 -m slipcover -m pytest -x -v
  6. Retrieve collected coverage data

    main

    Call Slipcover.get_coverage() to obtain the final coverage dictionary containing metadata and per-file statistics (executed lines, missing lines, branches, etc.).

    This method automatically handles:

    • Adding file summaries.
    • Calculating global summary statistics (total lines, total branches, total percentage).
    • Simplifying file paths using PathSimplifier.
    from slipcover import Slipcover
    
    sc = Slipcover(source=['./my_module'])
    # ... run tests ...
    coverage_data = sc.get_coverage()
    print(coverage_data['summary']['percent_covered'])
  7. Export coverage data to XML

    main

    Use print_xml to generate an XML report of the collected coverage. This is useful for integration with CI/CD tools or other coverage aggregators.

    Arguments:

    • coverage: The Coverage dictionary object.
    • source_paths: An iterable of strings representing the source file paths.
    • with_branches: Boolean; if True, includes branch coverage in the report.
    • xml_package_depth: Integer; controls the depth of the XML package structure.
    • outfile: The file-like object to write to (defaults to sys.stdout).
    from slipcover import print_xml
    
    # Assuming 'cov' is a Coverage object
    print_xml(cov, source_paths=['/path/to/source'], with_branches=True, outfile=open('report.xml', 'w'))
  8. Instrument dynamically loaded files with spec_from_file_location

    main

    Some tools (like Alembic) load Python files dynamically using importlib.util.spec_from_file_location. To capture coverage for these files, you must wrap this function to ensure the resulting loaders are intercepted by Slipcover.

    Use wrap_spec_from_file_location to patch importlib.util.spec_from_file_location globally within your process.

    import importlib.util
    from slipcover import Slipcover, FileMatcher, wrap_spec_from_file_location
    
    sci = Slipcover()
    matcher = FileMatcher()
    
    # Patch the importlib utility
    wrap_spec_from_file_location(sci, matcher)
    
    # Now dynamic loads will be instrumented if they match the FileMatcher
    importlib.util.spec_from_file_location("dynamic_mod", "/path/to/dynamic_file.py")
  9. Merge two SlipCover coverage results

    main

    Use merge_coverage(a, b) to combine two coverage dictionaries. This is particularly useful when running tests in multiple steps or processes.

    Key behaviors:

    • Canonicalization: Files are matched using their canonical (absolute/resolved) paths to ensure that relative paths and absolute paths for the same file are merged correctly.
    • Display Keys: The resulting merged dictionary uses the shortest original spelling of the file path as the key.
    • Requirements:
      • Both dictionaries must have meta['software'] == 'slipcover'.
      • You cannot merge coverage if one has show_contexts=True.
      • If branch coverage is enabled in dictionary a, it must also be present in dictionary b.
    from slipcover import merge_coverage
    
    # Merges coverage 'b' into 'a'
    merged_cov = merge_coverage(coverage_step_1, coverage_step_2)
  10. Generate LCOV reports with LcovReporter

    main

    The LcovReporter class is used to generate coverage reports in the LCOV format, which is compatible with many standard coverage visualization tools. You can instantiate the reporter with a Coverage object and specify whether to include branch coverage data.

    To generate the report, call the .report() method. If no outfile is provided, the report is written to sys.stdout.

    from slipcover.lcovreport import LcovReporter
    
    # Assuming 'coverage_data' is a valid Coverage object
    reporter = LcovReporter(
        coverage=coverage_data,
        with_branches=True,
        test_name="my_test_suite",
        comments=["Generated by SlipCover"]
    )
    
    # Write to a file
    with open("coverage.lcov", "w") as f:
        reporter.report(outfile=f)