pyinstrument

Pyinstrument is a statistical profiler, not a tracing profiler. Instead of tracking every function call, it interrupts the program at a configured interval (defaulting to 1ms) and records the entire call stack.

Key advantages:

• Low Overhead: Significantly lower overhead compared to tracing profilers like cProfile or profile, which can distort results by slowing down code that makes frequent function calls.
• Accuracy: Even with fewer samples, accuracy is maintained because long-running function calls are recorded at the end of the call, effectively 'bunching' samples.
• Full-stack Recording: Unlike cProfile which provides a flat list of functions, pyinstrument records the entire stack, making it easier to understand the context of why expensive calls are being made. It also hides library frames by default to focus on your application code.

To use pyinstrument natively in an IPython notebook:

1. Load the extension at the top of your notebook: %load_ext pyinstrument.
2. Use the magic command in the cell you want to profile: %%pyinstrument.

Run Pyinstrument directly from the command line to profile a Python script. Instead of using python script.py, use pyinstrument script.py. The tool will output a colored summary of time consumption upon completion or when interrupted with ^C.

Use the -r html flag to generate an interactive HTML profile report for deeper exploration.

To generate individual HTML profiling reports for each test, create an autouse fixture in your conftest.py file. This approach automatically starts the Profiler before each test and writes the results to a .profiles directory after the test completes.

from pathlib import Path
import pytest
from pyinstrument import Profiler

TESTS_ROOT = Path.cwd()

@pytest.fixture(autouse=True)
def auto_profile(request):
    PROFILE_ROOT = (TESTS_ROOT / ".profiles")
    # Turn profiling on
    profiler = Profiler()
    profiler.start()

    yield  # Run test

    profiler.stop()
    PROFILE_ROOT.mkdir(exist_ok=True)
    results_file = PROFILE_ROOT / f"{request.node.name}.html"
    profiler.write_html(results_file)

To profile an installed Python script that uses a console_script entry point, use the --from-path flag. Replace cli-script with the actual command name.

pyinstrument --from-path cli-script

Use IPython magics to profile lines or entire cells in Jupyter notebooks or IPython shells.

1. Load the extension: %load_ext pyinstrument

2. Use the %%pyinstrument magic at the top of a cell to profile it.

%load_ext pyinstrument

%%pyinstrument
import time
# ... code to profile ...

You can profile an entire pytest suite by invoking pyinstrument as a module runner. This provides a consolidated profiling report for all tests executed in the command.

pyinstrument -m pytest [pytest-args...]

Install pyinstrument using pip. It supports Python 3.8+.

pip install pyinstrument

When using the Django middleware, you can use the following environment variables and options:

• PYINSTRUMENT_INTERVAL: Set the sampling interval for the middleware.
• PYINSTRUMENT_PROFILE_DIR: Log profiles of all requests to a specified folder.
• PYINSTRUMENT_USE_SIGNAL: Use signal mode if the default mode presents problems.
• PYINSTRUMENT_SHOW_CALLBACK: Add a condition to decide whether to show the profile (useful for live servers).
• PYINSTRUMENT_PROFILE_DIR_RENDERER: Configure the renderer for the Django middleware file output.

Note: Recent versions also allow customizing the filename of saved profile runs via a callback in the Django configuration.

Docker

Profiling code inside a Docker container can cause strange results because the gettimeofday syscall used by pyinstrument can be slow in those environments.

Pickle and main

When running pyinstrument script.py where script.py contains a class serialized with pickle, you might encounter errors because the serialization machinery doesn't know where __main__ is.

Use an async middleware to profile FastAPI requests.

Note: This approach only profiles async def path operation functions. Routes defined with standard def are executed in a separate thread and will not be captured by this middleware.

To invoke, add the GET parameter profile=1 to your request.

from fastapi import Request
from fastapi.responses import HTMLResponse
from pyinstrument import Profiler

@app.middleware("http")
async def profile_request(request: Request, call_next):
    profiling = request.query_params.get("profile", False)
    if profiling:
        profiler = Profiler()
        profiler.start()
        await call_next(request)
        profiler.stop()
        return HTMLResponse(profiler.output_html())
    else:
        return await call_next(request)