pytest-xdist

pytest-xdist operates using a controller and one or more workers.

• The controller manages the test session, spawns workers, and distributes tests.
• Each worker is a mini pytest runner responsible for performing its own full test collection and executing tests as instructed by the controller.

Communication between the controller and workers is handled via execnet gateways, allowing workers to run in local or remote interpreters.

Execution Flow

1. Spawning: The controller spawns workers.
2. Collection: Each worker performs a full test collection and sends the resulting test-ids back to the controller. The controller verifies that all workers collected the same tests in the same order.
3. Indexing: To save bandwidth, the controller converts test-ids into simple indexes. Instead of sending a full test-id, the controller tells a worker to execute test index N.
4. Distribution: Depending on the dist-mode, the controller sends tests to workers (see Distribution Modes).
5. Execution: Workers wait for instructions. When they receive a test, they execute the pytest_runtest_protocol.
6. Reporting: Workers send results back to the controller, which forwards them to standard pytest hooks like pytest_runtest_logstart and pytest_runtest_logreport. This ensures compatibility with plugins like junitxml.
7. Shutdown: Once all tests are pending, the controller sends a "shutdown" signal. Workers finish their remaining tests and exit.

Because pytest-xdist runs each worker as a separate process, session scoped fixtures will execute once per worker. To ensure a fixture executes exactly once for the entire test session (e.g., for expensive data production or database initialization), use a file lock for inter-process communication.

Recommended approach:

1. Use tmp_path_factory.getbasetemp().parent to find a shared temporary directory.
2. Use filelock.FileLock to coordinate access to a shared file.
3. The first worker to acquire the lock produces the data and writes it to a file; subsequent workers read from that file.

import json

import pytest
from filelock import FileLock


@pytest.fixture(scope="session")
def session_data(tmp_path_factory, worker_id):
    if worker_id == "master":
        # not executing in with multiple workers, just produce the data and let
        # pytest's fixture caching do its job
        return produce_expensive_data()

    # get the temp directory shared by all workers
    root_tmp_dir = tmp_path_factory.getbasetemp().parent

    fn = root_tmp_dir / "data.json"
    with FileLock(str(fn) + ".lock"):
        if fn.is_file():
            data = json.loads(fn.read_text())
        else:
            data = produce_expensive_data()
            fn.write_text(json.dumps(data))
    return data

You can distribute tests to a remote SSH-reachable machine by synchronizing your package directory to the remote host.

Note: The rsync feature is deprecated and scheduled for removal in release 4.0. For long-term stability, consider using SSH or socket servers.

Requirements for successful execution:

• Ensure all code and test directories are included via --rsyncdir.
• Crucial: Every test (sub) directory must contain an __init__.py file. pytest-xdist references tests as fully qualified Python module paths; missing __init__.py files will cause setup errors on the remote side.

Options:

• --rsyncdir <dir>: Specify one or more directories to synchronize to the remote side.
• --rsyncignore <pattern>: Specify one or more glob patterns to ignore during synchronization. Note that internal patterns .*, *.pyc, *.pyo, *~ cannot be overridden.

pytest -d --rsyncdir mypkg --tx ssh=myhostpopen mypkg/tests/unit/test_something.py