hotpath-rs

hotpath-rs is a Rust performance profiling toolkit designed to identify where code spends time, consumes CPU, and allocates memory. It provides visibility into various layers of a Rust application, including:

• Time, CPU & Memory: Identify expensive functions, allocation hotspots, and memory leaks.
• Async Observability: Monitor futures, channels, and streams.
• I/O Monitoring: Track throughput and latency for sync/async I/O streams (files, TCP, compression).
• SQL Query Profiling: Performance metrics for sqlx and Diesel.
• HTTP Calls Profiling: Per-endpoint latency and error metrics for reqwest.
• Concurrency: Monitor Mutex/RwLock wait times and contention.
• Tokio Runtime: Monitor workers, scheduling, and queues.

Profiling can be performed using a live TUI dashboard for real-time monitoring or by generating static one-off reports for analysis.

The hotpath-macros crate provides procedural macros to instrument your Rust code for CPU and memory profiling. These macros allow you to easily mark functions, entry points, and asynchronous functions for measurement within the hotpath ecosystem.

Available macros include:

• #[measure]: Instrument a function for profiling.
• #[main]: Define the entry point for a profiled application.
• #[future_fn]: Instrument asynchronous functions (futures).

hotpath measures the time taken for reqwest's execute future to resolve. This occurs when the response status line and headers have been fully received.

Included in measurement:

• DNS resolution, TCP connection, and TLS handshake (if no pooled connection is reused).
• Sending the request (including uploading the full request body).
• Redirect hops (each hop's full round trip is counted).
• Server processing time (up to receipt of response headers).

Excluded from measurement:

• Downloading the response body.
• Decompression.
• JSON deserialization.

Key takeaway: The metric behaves like time-to-first-byte plus connection cost. It is a reliable measure of server latency, but not a measure of network throughput. Body-read failures will not appear in the Errors column because the request is considered complete once headers arrive.

To prevent report bloat, hotpath groups requests by METHOD host/path. It normalizes endpoints by dropping query strings, fragments, and credentials, and by collapsing path segments that look like identifiers into {id}.

Collapsed segments include:

• All-digit segments (e.g., /users/123 $\rightarrow$ /users/{id})
• UUIDs (e.g., /jobs/550e8400... $\rightarrow$ /jobs/{id})
• Hex strings of 16+ characters

Example: GET example.com/users/1?verbose=true and GET example.com/users/42 are both reported as GET example.com/users/{id}.

In async contexts, hotpath measures duration from the first poll to Ready. If a future is cancelled mid-Pending (for example, via a tokio::select! timeout), the wrapper cannot observe the cancellation.

When the same direction of operation resumes, the wrapper continues the existing pending span and reports the total time since the abandoned operation began.

The rw_lock! and mutex! macros return an instrumented wrapper rather than the original type. While type inference handles this in let bindings, you must use the hotpath::wrap:: path when explicitly naming the type in struct fields or function signatures.

hotpath::wrap:: mirrors the standard module layout. For example, std::sync::RwLock<T> becomes hotpath::wrap::std::sync::RwLock<T>.

Note: This is zero-overhead when the hotpath feature is disabled; it resolves to a plain re-export of the original type.

// Use hotpath::wrap:: to define fields in a struct
struct App {
    counter: hotpath::wrap::std::sync::RwLock<u32>,
    name: hotpath::wrap::std::sync::Mutex<String>,
}

let app = App {
    counter: hotpath::rw_lock!(std::sync::RwLock::new(0u32)),
    name: hotpath::mutex!(std::sync::Mutex::new(String::new())),
};

The TUI layout adapts based on the type of data being displayed:

Table Splitting

When report tables become too wide for the terminal, they are split into stacked per-kind sub-tables that share a single selection cursor. For example:

• rw_locks reads and writes are split.
• io reads and writes are split.

Log Panels

Different data types support different levels of detail in the logs panel:

• Table-only (No logs): Locks and IO bytes.
• Logs enabled: Channels, streams, futures, functions, SQL, and HTTP.

For SQL and HTTP, the log panels include source attribution provided by caller_stack.rs to help identify the origin of the request.

hotpath provides two ways to manage the profiling lifecycle:

1. #[hotpath::main] macro: The simplest method. It creates a HotpathGuard for the entry-point function (or any function you annotate). The report is generated automatically when the function returns and the guard is dropped. Use this for whole-program profiling with minimal setup.

2. HotpathGuardBuilder API: Provides manual control. You can start profiling later, stop it earlier, or execute custom logic using before_shutdown immediately before the report is generated. Use this to profile specific code segments or control exactly when the report is written.

Critical Constraint: Only one HotpathGuard may be alive at a time. Attempting to create a second guard (e.g., using both the macro and the builder) will cause a panic.

When the hotpath feature is enabled, instrumentation adds a small amount of latency to your operations. The overhead depends on the type of resource being profiled.

General Guidelines:

• Functions: #[hotpath::measure] adds ~40 ns per call. Avoid measuring sub-microsecond functions in tight loops; instead, instrument meaningful units of work.
• Locks: mutex! and rw_lock! add 29-66 ns per lock cycle (uncontended). Each cycle records both wait time and hold time.
• Channels: Default wrap mode adds 47-88 ns per send/recv cycle. Avoid the legacy proxy = true mode unless necessary, as it can increase costs by 4-11x.
• Async: future! adds ~125 ns per poll, while stream! adds ~30 ns per yielded item.

When performing a sync, adhere to these constraints to prevent breaking the meta-profiling infrastructure:

• Scope: Only sync files within src/. Never sync Cargo.toml or other configuration files.
• Methodology: NEVER copy entire files and use bulk find-and-replace. Always apply semantic diffs to existing meta files.
• Preservation: You must preserve all meta-specific naming conventions (feature flags, env vars, crate names).
• Exclusion: Remove any self-instrumentation lines (e.g., #[cfg_attr(feature = "hotpath-meta", ...)]) that appear in the source during the sync process.

By tracking per-worker metrics, hotpath helps identify common runtime bottlenecks:

• Worker utilization: High busy duration relative to wall-clock time indicates a worker is saturated. If one worker is near 100% while others are idle, it signals uneven scheduling or a blocking call.
• Idle vs busy workers: A rising park count means workers are waiting for work. Flat park counts under load mean workers are saturated.
• Work stealing: High steal count and steal operations indicate that workers are frequently pulling tasks from peers, suggesting uneven local-queue distribution.
• Queue depth: Growing local queue depth (per worker) or global queue depth (runtime-wide) indicates tasks are arriving faster than workers can process them, often due to blocking work or insufficient worker threads.