psutil: Python System and Process Utilities

repository·master·Indexed 11 days ago

https://github.com/giampaolo/psutil

A cross-platform Python library for retrieving information on running processes and system utilization, including CPU, memory, disks, network, and sensors. It provides a consistent API for system monitoring, profiling, and process management, serving as a portable alternative to the os, resource, and subprocess modules.

Tokens
50.7K
Snippets
181
Records
251
Agent score
93%

What's inside psutil

  1. Notable software using psutil

    master

    psutil is widely adopted across various domains of the Python ecosystem. It is used by major projects for system monitoring, automation, and performance profiling. Notable examples include:

    Infrastructure & Automation

    • Home Assistant: System monitor integration.
    • Ansible: System fact gathering.
    • Apache Airflow: Process supervision and unit testing.
    • Celery: Worker process monitoring and memory leak detection.
    • Salt: Deep system data collection (grains).
    • Dask: Metrics dashboard and profiling.
    • Ajenti: Monitoring plugins.

    AI & Machine Learning

    • TensorFlow: Used in unit tests.
    • PyTorch: Used in benchmark scripts.
    • Ray: Metrics dashboard.
    • MLflow: Deep system monitoring integration.

    Developer Tools

    • Sentry: Telemetry metrics.
    • Locust: Monitoring of the Locust process.
    • Spyder: Process management and UI statistics.
    • psleak: Detecting memory leaks in Python C extensions using heap_info().

    System Monitoring

    Many core system monitoring tools rely on psutil as a primary dependency for collecting metrics, including:

    • Glances, bpytop, s-tui, asitop, and psdash (core metrics).
    • auto-cpufreq (CPU monitoring).
    • GRR (system data collection).
    • Datadog (dd-agent and dd-trace-py) for system metrics collection.
  2. Font configuration and usage

    master

    The project uses self-hosted web fonts (latin subset) for its UI. These fonts are integrated into the project's styling via the ../css/fonts.css file.

    The following font families are used:

    • Inter
    • JetBrains Mono
    • Merriweather

    All fonts are licensed under the SIL Open Font License 1.1.

  3. What is free-threaded Python and how does it affect psutil?

    master

    Free-threaded Python (introduced in Python 3.13) refers to Python builds where the GIL (Global Interpreter Lock) is disabled. This allows multiple threads to execute Python bytecodes in true parallel, which is beneficial for CPU-bound applications on multi-core processors.

    Because psutil contains C extensions, it requires specific wheels to run on these builds. psutil 7.1.2+ provides these wheels, allowing users to run psutil in a no-GIL environment without manual compilation.

  4. Compare psutil to Python standard library

    master
    The primary difference between psutil and the Python standard library (os, sys, resource, etc.) is scope: standard library functions typically only operate on the current process, whereas psutil can operate on any process by its PID. This makes psutil a more powerful tool for system monitoring and process management.
  5. Analyze context switches for performance bottlenecks

    master

    Context switches occur when the CPU stops executing one process/thread for another. Use Process.num_ctx_switches() or cpu_stats().ctx_switches to monitor them:

    • Voluntary context switches: Occur when a process gives up the CPU (e.g., waiting for I/O or a lock). High rates in I/O-bound workloads are normal.
    • Involuntary context switches: Occur when the OS forcibly takes the CPU from a process. High rates indicate too many active threads/processes competing for too few CPU cores.
  6. Use psutil.Process instances in sets and dicts

    master

    psutil.Process instances are hashable, meaning they can be compared for equality and used in sets or as dictionary keys. Equality is determined by both the PID and the process creation time, which prevents confusion if a PID is reused by the kernel. This is useful for diffing process snapshots.

    >>> before = set(psutil.process_iter())
    >>> # ... some time passes ...
    >>> after = set(psutilutil.process_iter())
    >>> new_procs = after - before  # processes spawned in between
  7. Understand memory metrics: RSS, VMS, PSS, and USS

    master

    When monitoring process memory, choose the metric that fits your needs:

    • RSS (Resident Set Size): Total physical RAM currently used by a process, including shared memory. It can be misleading because shared memory is counted in full for every process using it.
    • VMS (Virtual Memory Size): Total virtual address space reserved, including mapped files and swap. Usually much larger than RSS.
    • PSS (Proportional Set Size): A fairer estimate than RSS; it divides shared memory pages proportionally among all processes using them (Linux only).
    • USS (Unique Set Size): The most accurate single-process metric. It represents the private memory that would be freed if the process exited, excluding all shared memory. Available via Process.memory_footprint() on Linux, macOS, and Windows.
  8. Choose between Process.memory_footprint() and Process.memory_info()

    master

    To measure a process's memory usage, choose based on your requirements for accuracy versus performance:

    • Process.memory_info(): Returns rss (Resident Set Size). It is fast but includes shared libraries. If multiple processes use the same library, that library's size is counted in the rss of every process.
    • Process.memory_footprint(): Returns uss (Unique Set Size), representing the process's private memory. This is the amount of memory that would be freed if the process were terminated. It is more accurate for determining actual per-process impact but is substantially slower and requires higher privileges.

    Note: On Linux, memory_footprint() also returns pss (Proportional Set Size) and swap.

  9. CPU percent interval changes in psutil 2.0+

    master

    In psutil 2.0.0+, the timeout parameter for CPU percent functions now defaults to 0.0 instead of 0.1. This prevents accidental blocking/sleeping when iterating over many processes.

    Affected functions:

    • Process.cpu_percent
    • psutil.cpu_percent
    • psutil.cpu_times_percent
    # This will be slow if timeout is not specified as 0.0
    for p in psutil.process_iter():
        print(p.cpu_percent())
  10. Monitor swap activity and thrashing

    master

    Swap memory is disk space used as RAM extension. Monitor swap_memory() for:

    • swap-in (sin): Memory moved from disk to RAM.
    • swap-out (sout): Memory moved from RAM to disk. High sout rates indicate memory pressure.

    Thrashing occurs when high, sustained rates of both sin and sout are observed, meaning the system is spending more time moving memory than doing actual work, leading to unresponsiveness.

  11. Compare psutil with Python standard library modules

    master

    When deciding between psutil and Python's built-in modules, consider the following trade-offs:

    • os module: Use for cheap, non-portable wrappers around POSIX syscalls if you only need information about the current process (e.g., os.getpid, os.getuid, os.cpu_count). Use psutil for cross-platform support, access to any process, and structured named tuple returns.
    • resource module (UNIX only): Use specifically to enforce or inspect ulimit-style resource limits (RLIMIT_*) via resource.getrusage. psutil's Process.rlimit provides a similar interface but extends it to all processes, not just the caller.
    • subprocess module: Avoid using subprocess to parse output from tools like ps, top, or netstat. Parsing is fragile due to OS/locale differences and spawning processes is slow. psutil reads kernel data directly without external processes.
    • platform module: Use for identifying the OS name, kernel version, or architecture. It does not provide runtime metrics or process information. It overlaps with psutil's OS constants (e.g., LINUX, WINDOWS, MACOS).
    • /proc filesystem (Linux only): While reading /proc/pid/status directly is fast and dependency-free, it is Linux-specific and requires manual text parsing. psutil handles the parsing and provides a consistent cross-platform API.