CrossHair Documentation

repository·main·Indexed 23 days ago

https://github.com/pschanely/crosshair

CrossHair is a Python analysis tool that uses symbolic execution and SMT solvers to verify code contracts and find counterexamples. It supports contract verification via asserts, PEP316, icontract, and deal, as well as unit test generation, behavioral comparison between functions, and integration as a backend for Hypothesis. The tool provides a CLI with commands like check, search, cover, and diffbehavior, and offers IDE integrations for VS Code and PyCharm.

Tokens
10.2K
Snippets
26
Records
66
Agent score
79%

What's inside CrossHair

  1. What is CrossHair

    main

    CrossHair is an analysis tool for Python that blurs the line between testing and type systems. It works by repeatedly calling functions with symbolic inputs, using an SMT solver (a theorem prover) to explore execution paths and find counterexamples to your code's logic.

    Key capabilities include:

    • Contract Verification: Automatically finding counterexamples for functions that have type annotations and defined contracts.
    • Unit Test Generation: Using the cover feature to generate unit tests.
    • Behavioral Comparison: Using diffbehavior to find differences in behavior between two functions.
    • Symbolic Reasoning: Supports built-in types, user-defined classes, and much of the Python standard library.
    • Hypothesis Integration: Can be used as an optional backend for the Hypothesis property-based testing tool.
  2. Reimplement classes and functions for plugins

    main

    A common plugin pattern involves re-implementing the classes and functions of a native package in pure Python, then telling CrossHair to use these implementations instead of the originals.

    Key API components:

    • register_type(original_class, replacement_class): Maps an original class to your pure-Python replacement.
    • register_patch(original_function, replacement_function): Maps an original function to your pure-Python replacement.
    • SymbolicFactory: Passed to your replacement class's __init__ method. You can use it to create symbolic values for other types (e.g., factory(int)).
    from crosshair import register_patch, register_type, SymbolicFactory
    from bunnies import Bunny, introduce_bunnies
    
    class _Bunny:
        happiness: int
        def __init__(self, factory: SymbolicFactory):
            self.happiness = factory(int)
    
        def pet(self: _Bunny) -> None:
            self.happiness += 1
    
    def _introduce_bunnies(bunny1: AnyBunny, bunny2: AnyBunny) -> None:
        bunny1.happiness += 1
        bunny2.happiness += 1
    
    register_type(bunnies.Bunny, _Bunny)
    register_patch(bunnies.introduce_bunnies, _introduce_bunnies)
  3. Understand CrossHair's analysis limitations

    main

    When using CrossHair for property-based testing, be aware of the following constraints to avoid false confidence or unexpected results:

    • Absence of counterexamples: If CrossHair does not find a counterexample, it does not guarantee that the property holds. It only means no counterexample was found within the current search space/constraints.
    • Symbolic value identity: Symbolic values are implemented as Python proxy values. While CrossHair attempts to maintain the illusion of real values, identity checks (e.g., x is y) may not be correctly analyzed.
    • Scope of analysis: CrossHair only analyzes function and class definitions located at the top level. It does not analyze nested functions or classes.
    • Determinism requirement: CrossHair can only analyze deterministic behavior. If the code produces different results for the same input, CrossHair may raise a NotDeterministic error.
    • Iterator and Generator consumption: Consuming values from an iterator or a generator within a pre-condition or post-condition will result in unexpected behavior.
  4. Requirements and caveats for using CrossHair

    main

    To ensure CrossHair works correctly, observe the following requirements and limitations:

    Requirements:

    • Python Version: Supported on Python 3.8+ and only on CPython.
    • Type Annotations: Your arguments must have proper type annotations.
    • Object Properties: Arguments must be deep-copyable and equality-comparable. Return values must also be equality-comparable and have reprs that faithfully reconstruct object state.
    • Determinism: Only deterministic behavior can be analyzed.

    Caveats:

    • Complexity: CrossHair may not be able to fully explore highly complex code.
    • Execution Risk: CrossHair actually runs your code and applies generated arguments to it. Use caution when running against sensitive logic.
    • User-defined Classes: Classes used as arguments must meet specific expectations (see documentation for hints).
  5. Supported Contract Syntaxes in CrossHair

    main

    CrossHair supports several ways to define contracts (pre-conditions and post-conditions) in your Python code. You can choose the syntax that best fits your project:

    • asserts: Uses standard Python assert statements.
    • PEP316: Uses docstring-based contracts.
    • icontract: Uses the icontract 3rd party library (decorator-based).
    • deal: Uses the deal 3rd party library (decorator-based).
  6. Configure analysis using code directives

    main

    You can customize CrossHair's behavior directly within your Python source code using special comments called "directives". These allow you to enable/disable checking or change the contract kind for specific scopes. Lower-level directives (e.g., inside a function) take precedence over higher-level ones (e.g., at the module level).

    Common directives:

    • # crosshair: off - Disable contract checking.
    • # crosshair: on - Re-enable contract checking.
    • # crosshair: analysis_kind=<KIND> - Set the specific contract syntax to use (e.g., asserts, PEP316, icontract, or deal).

    Directives can be placed in the body of a function, at the top level of a module, or in a package's __init__.py file.

    # crosshair: off
    
    def grow(age: int):
        # crosshair: on
        # crosshair: analysis_kind=asserts
        assert age >= 0
        ...
  7. Important: Reachability and Entry Points

    main

    CrossHair only evaluates code that is reachable by running some function with a contract.

    Even if you target a specific function, CrossHair will not analyze it unless that function has at least one pre- or post-condition. To force CrossHair to treat a function as a valid entry point for analysis, you can add a trivial post-condition such as assert True.

  8. How CrossHair performs symbolic execution

    main

    CrossHair uses a concolic-style approach to explore code paths without performing AST or bytecode analysis. Instead, it repeatedly calls your target function using special proxy objects that mimic standard Python types but hold underlying Z3 SMT solver expressions.

    Type Mapping

    CrossHair maps Python types to specific symbolic types and Z3 sorts:

    Python TypeCrossHair TypeZ3 Sort
    intSymbolicIntIntSort()
    boolSymbolicBoolBoolSort()
    strAnySymbolicStrStringSort()
    dictSymbolicDictArraySort(K, V) and IntSort() for length

    Path Exploration Mechanism

    When the Python interpreter encounters a conditional (e.g., if symbolic_x > 0:), it calls the __bool__ method on the resulting symbolic boolean object. CrossHair then:

    1. Consults Z3 to see if the expression is forced to be True or False by current constraints.
    2. If the outcome is ambiguous, it makes a decision (randomly) and adds that decision as a new constraint to the current execution path.
    3. This allows CrossHair to explore different branches in subsequent executions.

    When a target state is reached (like an unhandled exception or a failed postcondition), CrossHair requests a model from Z3 to provide a concrete counterexample.

  9. Use CrossHair with Hypothesis

    main
    CrossHair can be used as an optional backend for the Hypothesis property-based testing library. This allows you to leverage CrossHair's symbolic execution capabilities within your existing Hypothesis test suites.
  10. Understand how CrossHair executes code

    main

    CrossHair performs symbolic execution by actually executing your contracted functions, but it uses special symbolic arguments and intercepts standard Python behaviors.

    Important considerations for developers:

    • Side Effects: Never target code that causes side effects (e.g., disk or network access). While CrossHair uses sys.addaudithook to attempt to block these, it is not foolproof, especially with C-based modules.
    • Printing: Avoid using print() on symbolic values. Printing forces symbolic values to take on concrete values, which can significantly hamper CrossHair's ability to perform effective analysis.
    • Execution Flow: CrossHair may or may not execute preconditions/postconditions, and it may execute subroutines out-of-order.
    • Bypassing Protections: If running in a containerized or isolated environment, you can use the --unblock command-line option to bypass CrossHair's internal protections.
  11. How to use CrossHair to verify consumer tests with contracts

    main
    If a library uses the specs_complete=True directive, CrossHair can help consumers write more robust tests. Consumers can annotate their own unit tests with a trivial contract """post: True""" to enable CrossHair analysis. CrossHair will then attempt to find inputs that satisfy the library's contracts but cause the consumer's test to fail, revealing if the test is too brittle or relies on undocumented implementation details.
  12. Security considerations when running CrossHair

    main
    CrossHair executes your code with symbolic arguments. While it uses sys.addaudithook to attempt to prevent disk and network access, this protection is not absolute. Specifically, it cannot prevent actions taken by C-based modules. Always run CrossHair in a controlled or sandboxed environment if your codebase contains sensitive operations.