HUATUO Observability Platform

HUATUO is a cloud-native operating system observability project that provides kernel-level insights using eBPF, kprobe, tracepoint, and ftrace. It is designed for general-purpose cloud-native, AI, and bare-metal infrastructure.

Core Capabilities:

• Kernel-Wide Insight: Low-overhead (<1%) observability into MM, CPU scheduling, networking, and block I/O.
• Instant Observability: Event-driven runtime context capture (e.g., on page faults or scheduling delays).
• AutoTracing: Automated snapshot retention to resolve performance jitters (CPU spikes, I/O surges, etc.).
• Continuous Profiling: Ongoing profiling of CPU, Memory, I/O, and Locks.
• Hardware Support: Covers CPUs, memory, PCIe, network, storage, and AI accelerators (GPUs/NPUs).
• Ecosystem Integration: Works with Prometheus, Grafana, Pyroscope, and Elasticsearch, and automatically associates K8s container labels/annotations.

HUATUO monitors Linux kernel hardware error events using a zero-intrusion, low-overhead approach. It leverages eBPF technology to observe kernel subsystems (MCE, EDAC, ACPI GHES, and PCIe AER) via tracepoints.

Key capabilities include:

• Real-time monitoring: Captures hardware errors as they occur.
• Structured storage: Persists error events (including timestamps, device IDs, error types, and raw register values) to local directories or remote storage like ES/OS.
• Prometheus integration: Exposes error counts as Prometheus metrics for alerting and visualization.

Supported Scenarios:

• General Computing: Detecting memory ECC errors (CE) to prevent DIMM failure.
• AI Computing: Monitoring PCIe AER events for GPUs, NVLink, and RDMA NICs to isolate faulty nodes.
• Storage Services: Monitoring PCIe NVMe/HBA errors (e.g., Completion Timeout) to diagnose performance jitter.
• Compliance: Maintaining structured hardware health logs for auditing.

The profiler CLI is a standalone tool for performance profiling that does not require huatuo-apiserver, Elasticsearch, or Grafana. It samples host processes or processes inside containers and outputs call stacks as folded stacks or SVG flame graphs.

Supported Languages & Mechanisms:

• C, C++, Go: Uses an eBPF-based native collector for CPU, virtual memory, physical memory, and residency.
• Java: Uses async-profiler for CPU, object allocation, and live objects.
• Python: Uses py-spy for CPU profiling.

Key Capabilities:

• Locate CPU Hotspots: Identify primary CPU consumers via stack width.
• Native Memory Attribution: Distinguish between virtual address-space, physical page allocation, and resident memory growth.
• JVM Analysis: Identify high allocation rates (GC pressure) or objects that remain referenced (live objects).
• Container Profiling: Use a container_id to profile workloads inside Docker or containerd.

HUATUO is an operating system deep observability project based on eBPF technology. It provides real-time monitoring of core Linux kernel subsystems, including CPU scheduling, memory, network protocols, and hardware errors.

How it works

When the kernel triggers an exception (e.g., softlockup, OOM, or hardware MCE), HUATUO uses eBPF programs to hook into kernel functions (kprobe) or tracepoints. It captures immediate context such as process information, kernel call stacks, and network context. This data is passed via perf event ring buffers to user-space handlers and persisted to Elasticsearch or local disk files.

Key Advantages over dmesg/syslog

• Lower Data Loss Risk: Unlike kernel log buffers which can overflow, eBPF event observation minimizes the risk of losing critical events.
• Captures Transient Exceptions: It can capture short-lived anomalies that are never written to kernel logs (e.g., extended softirq disablement times).
• Cloud-Native Awareness: Provides container-level event correlation, allowing for precise identification of problematic containers in Kubernetes environments.

What is dropwatch?

dropwatch is a kernel network drop observability tool provided by HUATUO. It attaches to the kernel tracepoint tracepoint/skb/kfree_skb to capture network drop events in real time.

Captured Context

When a drop occurs, dropwatch outputs the following context:

• Protocol type
• IP five-tuple
• Process name and PID
• Network device and MAC address
• The complete kernel call stack that triggered the drop

Performance and Filtering

• Kernel-side filtering: Supports tcpdump-style filter expressions. The logic is compiled into eBPF bytecode at load time using the internal/pcapfilter compiler.
• Efficiency: Filtering is performed entirely in kernel mode; only matching packets are reported to user space, minimizing host performance impact.
• Additional Controls: Supports device whitelist/blacklist filtering, global per-second rate limiting, and integration with huatuo-bamai for Elasticsearch storage.