pytest-check

The maxfail setting in pytest counts tests, not individual checks.

• Using -x or --maxfail=1 will cause the plugin to abort testing after the very first failed check within a test.
• Using -maxfail=2 or greater will cause pytest to behave normally (it will only stop after the specified number of failed test functions).

By default, a standard assert will stop a test immediately upon failure. To allow multiple checks to run even if one fails, use the check object as a context manager with with check: blocks. This allows you to see the full picture of all failures within a single test function.

import httpx
from pytest_check import check

def test_httpx_get():
    r = httpx.get('https://www.example.org/')
    # This standard assert will still stop the test if it fails
    assert r.status_code == 200
    
    # These checks will continue even if one fails
    with check:
        assert r.is_redirect is False
    with check:
        assert r.encoding == 'utf-8'

You can install pytest-check using pip or conda.

$ pip install pytest-check

# Or via conda
$ conda install -c conda-forge pytest-check

Instead of importing check from pytest_check, you can use it directly as a pytest fixture in your test functions.

def test_httpx_get(check):
    r = httpx.get('https://www.example.org/')
    with check:
        assert r.is_redirect == False

Most validation functions in pytest-check (like equal, is_, is_true, etc.) are 'soft checks'. They return a bool (True on success, False on failure) and log the failure without raising an exception. This allows the test to continue running and report multiple failures in a single test run.

from pytest_check import equal, is_true, is_none

def test_multiple_checks():
    # These will log failures but the test continues
    equal(1, 2, msg="Not equal")
    is_true(False, msg="Should be true")
    is_none(None, msg="Is none")

The pytest-check plugin provides a way to perform multiple assertions within a single test without stopping at the first failure. This is useful for verifying multiple independent properties of an object or state. You can use the check fixture or import the check object directly to access validation functions like equal, is_true, etc. Additionally, you can use with check: as a context manager to wrap standard Python assert statements so that they record failures instead of raising exceptions immediately.

from pytest_check import check

def test_example():
    # Using validation functions directly
    check.equal(1, 1)
    
    # Using the context manager to wrap standard asserts
    with check:
        assert 1 == 2

If you expect certain checks to fail, you can use the standard pytest @pytest.mark.xfail decorator. pytest-check supports matching specific check failures to an xfail mark using the raises keyword argument. This allows a test to pass (as skipped) if the failures match the expected exception types.

If the xfail mark includes a raises parameter, the test will only be considered an expected failure if the failure message contains the name of the specified exception(s).

To log every check failure to a file, register a logging function using check.call_on_fail within a pytest fixture (typically in conftest.py).

Example implementation using logging:

import logging
import pytest
from pytest_check import check

@pytest.fixture(scope='session', autouse=True)
def setup_logging():
    # logging config
    log = logging.getLogger(__name__)
    log.setLevel(logging.DEBUG)
    fh = logging.FileHandler('session.log')
    fh.setLevel(logging.DEBUG)
    fh.setFormatter(logging.Formatter('--- %(asctime)s.%(msecs)03d ---
%(message)s', 
                                      datefmt='%Y-%m-%d %H:%M:%S'))
    log.addHandler(fh)
    # log start of tests
    log.info("---------\nStarting test run\n---------")
    
    # register the failure logger
    def log_failure(message):
        log.error(message)
    check.call_on_fail(log_failure)

You can override global CLI settings locally within a specific test function using the following check methods:

• check.set_max_tb(n): Sets the maximum number of full pseudo-tracebacks for the current test.
• check.set_max_report(n): Sets the maximum number of failures to report for the current test.
• check.set_max_fail(n): Sets the number of failures allowed before the test bails.

def test_max_tb():
    check.set_max_tb(2)
    for i in range(1, 11):
        check.equal(i, 100)

def test_max_report():
    check.set_max_report(5)
    for i in range(1, 11):
        check.equal(i, 100)

def test_max_fail():
    check.set_max_fail(5)
    for i in range(1, 11):
        check.equal(i, 100)

You can wrap any helper function containing assert statements to make it a non-blocking check using the @check.check_func decorator. This is the easiest way to create reusable, non-stopping validation logic.

from pytest_check import check

@check.check_func
def is_four(a):
    assert a == 4

def test_all_four():
    is_four(1)  # This will fail but the test continues
    is_four(4)  # This will pass

Use check.any_failures() to determine if any checks within the current test have already failed. This is useful for making subsequent blocks of checks conditional, preventing unnecessary or cascading failures if a prerequisite check fails.

from pytest_check import check

def test_with_groups_of_checks():
    # always check these
    check.equal(1, 1)
    check.equal(2, 3)
    if not check.any_failures():
        # only check these if the above passed
        check.equal(1, 2)
        check.equal(2, 2)

The check.raises context manager works similarly to pytest.raises, but a failure to catch the expected exception will not stop the test execution. It also supports an xfail reason.

from pytest_check import check

def test_raises():
    # If this doesn't raise AssertionError, the test continues
    with check.raises(AssertionError):
        x = 3
        assert 1 < x < 4

# Using with xfail
def test_raises_and_xfail():
    with check.raises(ValueError, xfail="known issue #123"):
        x = 1 / 0