Hypothesis

Hypothesis is a property-based testing library for Python. Instead of writing individual test cases with specific inputs, you define the properties your code should satisfy and the range of inputs it should accept. Hypothesis then automatically generates a wide variety of inputs, including edge cases, to attempt to falsify your properties.

To use Hypothesis, you typically use the @given decorator combined with strategies (often imported as st) to describe the data types and structures your test function expects.

from hypothesis import given, strategies as st

@given(st.lists(st.integers() | st.floats()))
def test_sort_correctness_using_properties(lst):
    result = my_sort(lst)
    assert set(lst) == set(result)
    assert all(a <= b for a, b in zip(result, result[1:]))

HypoFuzz is a service designed to extend Hypothesis testing workflows by providing coverage-guided fuzzing. While standard Hypothesis is used for interactive development and catching regressions (often using @example() decorators or the phases= setting), HypoFuzz is optimized for finding deep bugs by dedicating server resources to searching for interesting inputs and covering under-tested code paths.

Key features include:

• Coverage-guided fuzzing: Uses code coverage to generate new inputs or variations of existing ones to trigger rare behaviors.
• Dynamic search optimization: Automatically allocates more search time to tests that haven't yet 'saturated' their coverage.
• Observability: Provides a dashboard to inspect input distributions and individual test cases.
• Scalability: Can be run on laptops, dedicated servers, or cloud instance pools.

The Hypothesis technical API reference is organized into four distinct sections depending on your use case:

• API Reference: Contains non-strategy Hypothesis objects, classes, and functions (e.g., the @given decorator).
• Strategies Reference: Documentation for Hypothesis strategies, including specialized extras.
• Integrations Reference: Details on features that provide a defined interface without a direct code API.
• Hypothesis internals: Documentation for internal APIs intended for developers building tools, libraries, or conducting research on top of Hypothesis.

The /hypothesis command follows a structured four-step process to ensure high-quality property-based tests:

1. Explore: The model examines the provided code to identify candidate properties.
2. Contextualize: The model explores how the codebase calls that code in practice (e.g., checking usage patterns or type hints).
3. Write: The model writes Hypothesis tests grounded in the gathered context.
4. Verify: The model runs the new tests and reflects on failures to determine if they are genuine bugs or unsound tests, refactoring the test if necessary.

The core engine of Hypothesis is called Conjecture. It represents randomized test cases as a choice sequence—a sequence of typed primitives (integer, float, boolean, string, or bytestring) drawn via draw_* methods on ConjectureData.

Key concepts:

• Choice Sequence: The sequence of choices made during test execution. This acts as the single source of truth for test inputs.
• Shortlex Minimality: The goal of shrinking is to find a choice sequence that is the shortest possible length, and among those of minimal length, the smallest when comparing choices one at a time from the left (using sort_key in shrinker.py).
• Shrinking: Conjecture approximates the shortlex minimal sequence by applying various transformations to the choice sequence to reduce the input while satisfying test predicates.

Hypothesis can be extended by various third-party libraries that provide specialized strategies or integrations. You can find these on PyPI by searching for the hypothesis keyword or the Framework :: Hypothesis classifier.

Common types of extensions include:

• Direct Strategies: Packages like hypothesis-fspaths (filesystem paths), hypothesis-geojson (GeoJSON), hypothesis-sqlalchemy (SQLAlchemy objects), and hypothesis-torch (PyTorch structures).
• Schema Inference: Packages that generate strategies from existing schemas, such as hypothesis-jsonschema (JSON Schema), hypothesis-graphql (GraphQL), and hypothesis-pb (Protocol Buffers).
• Framework Integrations: Deep integrations like Pydantic (automatic registration of constrained types), deal (design-by-contract), and Trio (async support via hypothesis-trio).

Hypothesis provides several mechanisms for generating complex data depending on the required level of control:

• st.from_type: Automatically infers strategies from type hints. Best when most random inputs are valid and you only need to handle rare edge cases.
• assume and filter: Simple tools to rule out invalid inputs when using inferred types.
• builds: Used to generate data that must satisfy complex invariants.
• composite: Provides high control by allowing you to relate multiple draws to each other, ideal for interdependent data.
• data: Allows interactive value picking directly within the test body for conditional generation.
• register_type_strategy: Used to associate custom strategies with specific types.

Hypothesis uses several terms to describe test inputs:

• Test case: A single Hypothesis-generated input passed to a test function.
• Failing test case: A test case that causes the test to fail (e.g., by raising an exception).
• Minimal failing test case: The specific failing test case that has been fully shrunk (minimized) by Hypothesis. Hypothesis reports only this minimal case to the user at the end of the test run.

Strategies describe the values that @given will generate for testing. You can pass a strategy to @given, nest strategies within each other, combine them using combinators, or modify them using .filter(), .map(), or .flatmap().

Example: st.lists(st.integers(), min_size=1) tells Hypothesis to generate lists of integers with at least one element.