DOCSEARCHbyAlgolia
Submit Docs

nv-monitor

repository·main·Indexed 43 days ago

https://github.com/wentbackward/nv-monitor

A lightweight, single-binary monitoring tool for NVIDIA GPU systems. It features a TUI for local observation, a CSV logger for historical data, and a Prometheus exporter for cluster-wide monitoring. Includes a companion utility, demo-load, for generating synthetic CPU and GPU loads to validate monitoring pipelines.

Tokens
2.7K
Snippets
10
Records
16
Agent score
56%

What's inside nv-monitor

  1. Understand nv-monitor resource impact and performance

    main

    nv-monitor is designed for minimal resource impact, making it suitable for edge devices. It provides GPU monitoring, Prometheus export, CSV logging, and RDMA monitoring with significantly lower CPU overhead than standard tools like top or htop.

    Key performance characteristics:

    • Low CPU Usage: Uses significantly less CPU than top (e.g., ~7x to 10x less depending on the hardware) while providing more features.
    • Stable Memory Footprint: Uses a dynamic memory allocation model where all necessary memory is allocated at startup. There are zero allocations during normal operation, preventing memory growth, fragmentation, or malloc/free churn.
    • Shared Library Efficiency: The NVML shared library code is mapped once in physical RAM and shared across all GPU processes (like nvidia-smi or VLLM), minimizing the unique memory cost.

  2. Evaluate Prometheus export overhead

    main

    Adding Prometheus export to nv-monitor has negligible impact on system resources:

    • CPU: The Prometheus server thread uses poll() with a 1-second timeout and only wakes on incoming scrape requests. It consumes zero CPU between scrapes.
    • Memory: Enabling Prometheus adds approximately 128 KB of overhead for the thread stack.
    ModeThreadsAdditional CPUAdditional memory
    Headless only2baselinebaseline
    Headless + Prometheus3+0.00%+128 KB (thread stack)

  3. How nv-monitor manages memory and prevents leaks

    main

    nv-monitor employs a strict memory allocation model to ensure stability for long-running monitoring tasks.

    Core Principles:

    • Startup Allocation: All buffers are sized to the detected hardware (e.g., number of CPUs or GPUs) and allocated at program start or thread start.
    • Zero Runtime Allocation: Functions that execute every frame (e.g., compute_cpu_usage(), draw_screen(), log_csv_row()) or every Prometheus scrape (e.g., format_metrics(), prom_handle()) perform zero allocations, reusing pre-allocated buffers instead.
    • Stability: This design ensures that RSS (Resident Set Size) remains completely stable over time, with no memory growth or fragmentation.

    Allocation Details:

    BufferSized toWhen allocatedWhen freed
    prev_ticksnum_cpus + 1Program startProgram exit
    cur_ticksnum_cpus + 1Program startProgram exit
    cpu_pctnum_cpus + 1Program startProgram exit
    cpu_partnum_cpusProgram startProgram exit
    prom_bodygpu_count + num_cpusPrometheus thread startThread exit
    prom_gpusgpu_countPrometheus thread startThread exit

  4. Run performance and stability benchmarks

    main

    The repository includes scripts to verify resource usage and memory stability on your target hardware.

    • bench.sh: Provides a resource usage snapshot comparing nv-monitor against top and htop. It measures binary sizes, RSS, private/shared memory, and CPU usage.
    • soak-test.sh: Performs a memory stability test by running at maximum collection rates (100ms log interval + 2 scrapes/sec) for a specified duration. Use this to verify that RSS remains flat and no memory leaks occur.

    Usage:

    ./bench.sh              # Resource usage snapshot vs top/htop
./soak-test.sh 10       # 10-minute memory stability soak test
./soak-test.sh 60       # 60-minute production validation soak test
    ./bench.sh              # Resource usage snapshot vs top/htop
./soak-test.sh 10       # 10-minute memory stability soak test

  5. Control the nv-monitor TUI interactively

    main

    When running in TUI mode, use the following keyboard shortcuts:

    KeyAction
    q / EscQuit
    sToggle sort (GPU memory / PID)
    cCycle CPU column count (auto → 1 → 2 → 3 → 4)
    j / k or ↑ / ↓Scroll CPU cores when they exceed the viewport
    + / -Adjust refresh rate (250ms steps)

  6. Expose Prometheus metrics with nv-monitor

    main

    To use nv-monitor as a Prometheus exporter, use the -p flag to specify a port. You can run it in TUI mode or in headless mode (-n) for unattended collection.

    Run as TUI + Exporter:

    ./nv-monitor -p 9101

    Run as Headless Exporter:

    ./nv-monitor -n -p 9101

    Verify endpoint:

    curl -s localhost:9101/metrics
    ./nv-monitor -n -p 9101
curl -s localhost:9101/metrics

  7. Use demo-load to generate synthetic CPU and GPU loads

    main

    demo-load is a lightweight test harness used to validate nv-monitor by generating controllable, sinusoidal CPU and GPU loads. It is useful for end-to-end testing of TUI displays, CSV logging, and Prometheus pipelines.

    Common Commands

    • CPU load only (all cores): ./demo-load
    • CPU + GPU load: ./demo-load --gpu
    • GPU load only: ./demo-load --gpu-only
    • High-intensity GPU load: ./demo-load --gpu --blocks 4096
    ./demo-load --gpu --blocks 4096

  8. Install nv-monitor

    main

    You can install nv-monitor using pre-built binaries or by building from source.

    Arch Linux

    If the AUR is enabled, install via pacman:

    pacman -S nv-monitor

    Build from Source

    Requires gcc and libncurses-dev:

    sudo apt install build-essential libncurses-dev
make

    To install the binary system-wide after building:

    sudo make install
    sudo apt install build-essential libncurses-dev
make
sudo make install

  9. Build the demo-load tool

    main

    You can build the demo-load synthetic load generator using make or by compiling the C source directly with gcc. The tool requires a C compiler and NVIDIA drivers, but does not require the full CUDA toolkit because it uses the CUDA driver API via dlopen.

    Using Make

    make demo-load

    Using GCC directly

    gcc -O2 -o demo-load demo-load.c -lpthread -ldl -lm

  10. Secure the Prometheus metrics endpoint

    main

    The exporter supports Bearer token authentication for the /metrics endpoint. You can provide the token via the -t flag or the NV_MONITOR_TOKEN environment variable. Using the environment variable is preferred as CLI arguments are visible in process lists (ps).

    Using CLI flag:

    ./nv-monitor -p 9101 -t my-secret-token

    Using Environment Variable (Recommended):

    NV_MONITOR_TOKEN=my-secret-token ./nv-monitor -p 9101

    Note: nv-monitor does not implement TLS. For transport security, use a reverse proxy (Nginx/Caddy), an SSH tunnel, or a service mesh.

    NV_MONITOR_TOKEN=my-secret-token ./nv-monitor -p 9101

  11. Generate synthetic load with demo-load

    main

    The demo-load tool is a companion utility that generates sinusoidal CPU and GPU loads for validating monitoring pipelines.

    To build and run it:

    make demo-load
./demo-load --gpu

    This command generates both CPU and GPU sinusoidal loads.

    make demo-load
./demo-load --gpu

  12. Example: Multi-node production test with nv-monitor

    main

    To validate a multi-node cluster setup, run demo-load in the background on each node while running nv-monitor in Prometheus export mode. This allows you to verify that metrics flow from the node through Prometheus to Grafana without needing real production workloads.

    On each node:

    ./demo-load --gpu &
./nv-monitor -n -p 9101
    # On each node:
./demo-load --gpu &
./nv-monitor -n -p 9101