line_profiler

The kernprof tool allows you to profile Python scripts without modifying the source code. It encapsulates profiling concerns and ensures that script execution remains robust by correctly setting variables like __name__, __file__, and sys.path.

By default, running kernprof script_to_profile.py writes the results to script_to_profile.py.prof.

Profiling results generated by kernprof are typically marshalled files that can be read using the pstats.Stats() module. You can view them interactively using the following command:

$ python -m pstats script_to_profile.py.prof

For versions older than 4.1.0, use the kernprof script:

1. Decorate the functions you want to profile with @profile. The decorator is automatically made available in the __builtins__ namespace by kernprof.
2. Run the script using kernprof -lv <script_name>.py.

The -l flag enables line-by-line profiling, and -v tells kernprof to immediately view the formatted results in the terminal.

By using the prof-mod key in your pyproject.toml configuration, you can instruct kernprof to automatically profile specific modules or even specific functions using their dotted names. This eliminates the need to manually add the @profile decorator to your source code.

• To profile an entire module: prof-mod = ['module_name']
• To profile a specific function: prof-mod = ['package.module.function_name']

# Profile an entire package
[tool.line_profiler.kernprof]
prof-mod = ['demo_pkg']

# Profile only a specific function
[tool.line_profiler.kernprof]
prof-mod = ['demo_pkg.core.fib']

You can use the %lprun magic command in IPython to profile specific functions and statements.

To enable the extension, you can either:

1. Explicitly load it in a session: %load_ext line_profiler.
2. Add 'line_profiler' to the extensions list in your IPython configuration file (~/.ipython/profile_default/ipython_config.py).

Example usage:

%load_ext line_profiler
%lprun -f my_function my_function(args)

When analyzing profile data with line_profiler, you can use the --unit command line argument to change the scale of the time reported in the output. This is useful for adjusting the precision and readability of the Time and Per Hit columns.

To use this, first generate a profile file (e.g., profile_output.lprof) by running your script with the LINE_PROFILE=1 environment variable. Then, pass the desired multiplier to the --unit flag when running the line_profiler module.

# 1. Generate profile data
LINE_PROFILE=1 python script.py

# 2. View results with different time units
python -m line_profiler -rtmz --unit 1 profile_output.lprof
python -m line_profiler -rtmz --unit 1e-3 profile_output.lprof
python -m line_profiler -rtmz --unit 1e-6 profile_output.lprof
python -m line_profiler -rtmz --unit 1e-9 profile_output.lprof

Use the -m flag to run and profile installed Python modules or packages, similar to how python -m works. The -m flag terminates kernprof argument parsing; all subsequent arguments are passed directly to the module being run.

# Module execution
# Syntax: kernprof [kernprof_options] -m module_name [module_arguments]

PYTHONPATH="${PYTHONPATH}:${PWD}" \
    kernprof --prof-mod fib --line-by-line --view -m \
    fib --verbose 10 20 30

Use the -c flag to profile Python code provided as a string on the command line, similar to python -c.

Important: Because the code is executed in a temporary file that is deleted after the process ends, profiling results for functions defined within that snippet cannot be viewed later using python -m line_profiler. You must use the --view flag to see the results immediately.

# Literal-code execution
# Syntax: kernprof [kernprof_options] -c "code_string" [arguments]

PYTHONPATH="${PYTHONPATH}:${PWD}" \
    kernprof --prof-mod fib._run_fib --line-by-line --view -c "
    import sys
    from fib import _run_fib, fib_no_cache as fib
    for n in sys.argv[1:]:
        print(f'fib({n})', '=', fib(int(n)))
    " 10 20

Install the core package using pip:

pip install line_profiler

To ensure compatibility with IPython, install with the [ipython] extra:

pip install line_profiler[ipython]

Note: Source releases require a C compiler. Git checkouts require Cython.

If kernprof saved results to a binary file (e.g., script_to_profile.py.lprof), you can view the formatted results later using the line_profiler module:

python -m line_profiler script_to_profile.py.lprof