PyStack

repository·main·Indexed 22 days ago

A high-performance tool for inspecting Python stack traces from running processes and core dump files. PyStack allows for non-intrusive analysis of production issues, crashes, and performance bottlenecks, providing insights into GIL status, garbage collection cycles, and local variables. It supports merged native (C/C++/Rust) and Python frames, and is compatible with processes using multiple Python interpreters (Python 3.11+).

Tokens: 8.5K
Snippets: 20
Records: 54
Agent score: 78%

What's inside pystack

Overview of PyStack capabilities
main
PyStack is a diagnostic tool designed to inspect the stack frames of running Python processes or Python core dump files. It allows developers to understand process behavior (or crash causes) without needing to manually interpret CPython internals.
Key Capabilities:
- Dual Mode: Works with both live running processes and core dump files.
- GIL Awareness: Identifies if threads hold the Python GIL, are waiting for it, or are dropping it.
- GC Visibility: Shows if a thread is currently executing a garbage collection cycle.
- Multi-Interpreter Support: Reports on multiple interpreters residing within the same process.
- Hybrid Stack Traces: Optionally merges native function calls (C/C++/Rust) with Python callables, replacing internal interpreter C-code with the actual Python code being executed.
- Symbol Handling: Automatically demangles symbols in native stacks and includes inlined functions when debug info is available.
- Variable Inspection: Optionally displays local variables and function arguments within Python stack frames.
- Low Impact/High Safety:
  Does not modify memory or execute code in the target process.
  Supports a non-pausing mode for Python stack analysis to minimize impact (though this carries a risk of data races).
- Performance: Optimized for speed, capable of analyzing core files significantly faster than general-purpose tools like GDB.
- Robustness: Works with optimized binaries, binaries lacking symbols (for Python stacks), and is resilient to memory corruption.
https://github.com/bloomberg/pystack/blob/main/docs/overview.rst
Overview of PyStack
main
PyStack is a tool for inspecting the stack frames of a running Python process or a Python core dump. It allows you to see what a process is doing (or what it was doing during a crash) without needing to interpret CPython internals.
Key capabilities include:
- Analyzing both running processes and core dump files.
- Showing GIL (Global Interpreter Lock) status per thread.
- Reporting on garbage collection cycles.
- Showing native (C/C++/Rust) function calls alongside Python frames.
- Displaying local variables and function arguments.
- Safe, non-intrusive inspection of running processes.
- High-speed analysis (up to 10x faster than GDB for core files).
https://github.com/bloomberg/pystack/blob/main/README.md
Analyzing processes with multiple interpreters
main
PyStack supports processes using multiple Python interpreters (e.g., via concurrent.interpreters or concurrent.futures.InterpreterPoolExecutor). When analyzing such processes, PyStack identifies which interpreter each stack is associated with and visualizes the transition points where a thread switches from one interpreter to another.
Requirements:
- The process must be running Python 3.11 or newer.
Key Features in Output:
- Interpreter Association: The traceback explicitly labels which interpreter a thread is running in (e.g., In the main interpreter [], In interpreter 1 []).
- Transition Tracking: Shows the stack trace path as a thread moves from the main interpreter into a subinterpreter.
- Independent Status: GIL (Global Interpreter Lock) status and GC (Garbage Collector) status are reported separately for each interpreter, as these states can differ across subinterpreters within the same process.
```
import os
import signal
from concurrent.futures import InterpreterPoolExecutor


def interpreter1_body(read_fd):
    print("Hello from interpreter 1!")
    open(read_fd, closefd=False).read()
    print("Goodbye from interpreter 1!")


def interpreter2_body(read_fd):
    print("Greetings from interpreter 2!")
    open(read_fd, closefd=False).read()
    print("Farewell from interpreter 2!")


read_fd, write_fd = os.pipe()

with InterpreterPoolExecutor(max_workers=2) as executor:
    executor.submit(interpreter1_body, read_fd)
    executor.submit(interpreter2_body, read_fd)

    try:
        signal.pause()
    except KeyboardInterrupt:
        print("\nCtrl-C received, signaling workers to stop...")
        os.close(write_fd)
os.close(read_fd)
```
https://github.com/bloomberg/pystack/blob/main/docs/multiple_interpreters.rst
How blocking and non-blocking modes work in `pystack remote`
main
When using pystack remote, you can choose between two operation modes depending on your requirements for correctness versus process availability:
- Blocking mode (default): The process is stopped, memory is read to produce a coherent report, and then the process continues. This guarantees a correct report. The overhead is minimal (usually under 10ms).
- Non-blocking mode (--no-block): The process is analyzed while it is running. This avoids pausing the process, but because the interpreter state might change while memory is being read, the resulting report might be incorrect or fail to produce.
https://github.com/bloomberg/pystack/blob/main/docs/process.rst
Identify the origin of a core file
main
PyStack attempts to report why a core file was generated, which helps determine if the process crashed or was stopped intentionally. This information is crucial for interpreting stack traces.
Common origin messages include:
- On demand: The core file seems to have been generated on demand (the process did not crash)
- Killing signal: The process died due receiving signal SIGBUS sent by pid 23
- Segmentation fault: The process died due a segmentation fault accessing address: 0x75bcd15
https://github.com/bloomberg/pystack/blob/main/docs/corefile.rst
Analyze a running process with `pystack remote`
main
Use the remote command to analyze the status of a running process by providing its Process ID (PID). The analysis is non-intrusive; no code is loaded into the process memory and no memory is modified.
By default, PyStack uses blocking mode, which stops the process momentarily (typically < 10ms) to ensure memory coherence and report correctness. If you cannot afford to pause the process even briefly, use the --no-block flag to enable non-blocking mode, though this may result in incorrect or incomplete reports if the interpreter state changes during memory retrieval.
https://github.com/bloomberg/pystack/blob/main/docs/process.rst
Build PyStack from source
main
To build PyStack from source on Linux, you must first install the following binary dependencies:
- libdw
- libelf
On Debian-based systems, you can typically install these via: apt-get install libdw-dev libelf-dev.
Note: For Alpine Linux or distributions not using glibc, you require elfutils 0.188 or newer. If pkg-config is installed, it will be used to locate these libraries automatically.
Once dependencies are installed, follow these steps to build:
1. Clone the repository.
2. Create and activate a virtual environment.
3. Install PyStack in development mode with extra dependencies.
```
git clone git@github.com:bloomberg/pystack.git pystack
cd pystack
python3 -m venv ../pystack-env/
source ../pystack-env/bin/activate
python3 -m pip install --upgrade pip
python3 -m pip install -e . --group test --group extra
```
https://github.com/bloomberg/pystack/blob/main/README.md
Inspect local variables and arguments with --locals
main
To understand the internal state of a program (such as why a specific code path was taken), you can use the --locals option. This provides a string representation of the local variables in different frames as well as the function arguments.
Notes:
- Most common built-in types are supported, but not all types can be printed.
- Performance: Using --locals can slightly increase report generation time because extra memory must be copied.
- Best Practice: It is advised not to use the --no-block option when using --locals, as the process might change too quickly while PyStack is fetching the variable data.
- Tip: For the most comprehensive view, combine --locals with --native or --native--all.
```
$ pystack remote 117 --locals
```
https://github.com/bloomberg/pystack/blob/main/docs/customizing_the_reports.rst
Generate a test core dump
main
You can verify your core dump configuration by forcing a Python process to abort. Run the following command:
```
python3 -c 'import os; os.abort()'
```
If successful, your shell should report Aborted (core dumped) or similar. You can then locate the file using ls based on your core_pattern (e.g., ls /tmp/core-*).
```
python3 -c 'import os; os.abort()'
```
https://github.com/bloomberg/pystack/blob/main/docs/tutorials/core_tutorial.rst
Enable core dumps in Linux
main
To analyze crashes with PyStack, your Linux environment must be configured to generate core dump files.
1. Check current limit: Run ulimit -c. If it returns 0, core dumps are disabled.
2. Enable core dumps: Set the limit to unlimited or a specific size (in KB) for the current shell session:
  Unlimited: ulimit -c unlimited
  Limited (e.g., 100 MB): ulimit -c 100000
3. Make permanent: Add the command to your ~/.bashrc or ~/.zshrc.
Verify core dump location: Check the kernel pattern to see where files are written:
```
cat /proc/sys/kernel/core_pattern
```
You can customize the naming and location of core files using sysctl. For example, to write to /tmp with the executable name, PID, hostname, and timestamp:
```
sudo sysctl -w kernel.core_pattern="/tmp/core-%e.%p.%h.%t"
```
```
ulimit -c unlimited
```
https://github.com/bloomberg/pystack/blob/main/docs/tutorials/core_tutorial.rst
Get merged native and Python stack traces with --native
main
By default, PyStack shows only Python stack traces. If you need to see what is happening during internal interpreter operations or within C extensions, use the --native option. This provides a "merged" stack trace where native calls used to execute Python code are substituted with the actual Python functions being executed, allowing you to see the full sequence of calls between Python and native code (e.g., seeing how time.sleep() leads into libc calls).
Requirement: The interpreter, C extensions, and shared libraries must have debugging symbols installed. If symbols have been stripped, the resulting stack traces may be incomplete.
```
$ pystack remote 112 --native
```
https://github.com/bloomberg/pystack/blob/main/docs/customizing_the_reports.rst
Debug infinite loops using pystack remote --locals
main
When a Python program enters an infinite loop, it may not produce a standard stack trace, making it difficult to identify the logical error. You can use pystack to inspect the state of the running process to find the cause.
To identify the bug in a running process that is stuck in a loop, use the remote command with the --locals flag to inspect the local variables of the active frames. This allows you to see the actual values being processed (e.g., seeing that a random number generator is only producing floats when integers are expected) without needing the program to crash or exit.
```
pystack remote --locals {PID}
```
https://github.com/bloomberg/pystack/blob/main/docs/tutorials/random_prime_number.rst