Podtrace is a lightweight, eBPF-driven diagnostic tool designed for Kubernetes applications. It provides full-stack observability by attaching eBPF programs directly to containers to observe real-time behavior without requiring prior configuration or application instrumentation.

Key capabilities include:

• Observability Scope: Correlates kernel events with application-layer behavior (network flows, TCP/UDP performance, file system activity, memory behavior, latency patterns, system calls, and high-level events like HTTP, DNS, and database queries).
• Use Cases: Debugging service slowness, identifying latency origins, monitoring network/IO resource usage, tracing request flows, and performing production incident response.
• Security: Because it runs privileged and loads eBPF into the kernel, release artifacts ship with SLSA v1.0 Build L2 provenance for verification.

Use the PodTraceSchedule CR to automate nightly sweeps or on-call probes. It fires a fresh PodTraceSession on every cron tick, manages child sessions via owner references, and prunes history based on successfulSessionsHistoryLimit.

Manual Trigger: You can manually trigger a scheduled session from the CLI.

apiVersion: podtrace.io/v1alpha1
kind: PodTraceSchedule
metadata: { name: nightly-diagnose, namespace: my-app }
spec:
  schedule: "0 2 * * *"
  concurrencyPolicy: Forbid
  successfulSessionsHistoryLimit: 7
  sessionTemplate:
    spec:
      selector: { matchLabels: { app: api } }
      duration: 5m
      filters: [dns, net]
      exporterRef: { name: prod-otlp }

Manual trigger

kubectl podtrace schedule trigger nightly-diagnose -n my-app

The kubectl podtrace command behaves differently depending on your environment:

• Real Clusters (EKS, GKE, etc.): The CLI spawns a short-lived privileged pod on the target node to run eBPF. This requires a one-time PodSecurity label on the namespace.
• Local/Development (kind, minikube, docker-desktop): Since the workstation is effectively the node host, spawning a pod is unnecessary. Use the --local flag to load eBPF directly onto your workstation's kernel. This requires CAP_BPF or CAP_SYS_ADMIN capabilities on the binary.

# Use --local for kind/minikube/docker-desktop
kubectl podtrace --local -n my-app api-pod

Podtrace uses a dual-path approach to capture DNS activity, ensuring visibility even for workloads that do not use standard system libraries (like statically-linked Go binaries).

1. Packet capture (cgroup_skb): This is the primary method. Podtrace attaches a cgroup_skb program to the pod's cgroup and parses DNS directly from UDP packets on port 53. This method is libc-independent and works for:
   • glibc, musl
   • Statically-linked Go (netgo resolver)
   • Distroless/scratch containers
   • Both IPv4 and IPv6
2. libc getaddrinfo uprobe: A complementary source used to capture the "intent" of DNS lookups for applications using glibc.

Because queries are emitted as soon as they are seen on egress (DNS_QUERY), Podtrace can capture the looked-up name even if a response is never received (e.g., in sparse or idle pods).

Continuous inspections and diagnostic detectors report issues. An issue's id is treated as a contractual address, similar to a metric name.

• Stability: Adding an id is a minor change. Renaming or removing an id is a breaking change (must be documented in CHANGELOG.md under ### Breaking).
• Why: The id is used as a label value in podtrace_issue_active{id=...}, in alerts, and in PodTraceSchedule triggers. Renaming an ID will silently break existing rules.

Non-contractual elements:

• Issue messages (human-readable text).
• Which rule produces a specific ID.
• Thresholds and For durations (tuning parameters).
• Whether a specific rule exists for a failure mode.

Because Podtrace interacts with the Linux kernel via eBPF, it has specific security requirements:

• Privileges: Requires elevated privileges, specifically CAP_SYS_ADMIN or root access.
• Input Validation: The tool validates all inputs including pod names, PIDs, and container IDs.
• Sanitization: Process names are sanitized to prevent injection attacks.
• Metrics Security: The Prometheus metrics endpoint includes rate limiting and security headers on HTTP responses.

Podtrace tracks DNS name resolution by attaching a uprobe to the getaddrinfo function in the workload container's libc.

Limitation: If your application is a statically-linked binary (common in Go) or runs on a distroless or scratch base image, there is no libc available to probe. In these cases, podtrace will emit a warning: DNS tracking disabled: no libc found in the target container...

Impact: This is non-fatal. All other tracing capabilities (network, filesystem, CPU, memory, syscalls) will continue to function normally. To enable DNS tracking, you must use a workload image that ships a dynamic glibc or musl libc.

To help with architectural planning, note the following boundaries of the podtrace project:

• Linux-bound: The tracing core is built on eBPF, meaning it is strictly bound to the Linux kernel.
• Observability only: Podtrace is an observation tool. It is not a policy engine, admission controller, or service mesh. It does not admit, mutate, or route workloads.
• Controlled upgrades: There is no agent self-update mechanism. Agent versions are managed by the operator/cluster owner via standard deployment procedures.