OpenSRE Documentation

Tracer/collect is a core data collection daemon designed for Linux hosts. It provides low-overhead, high-fidelity observability for scientific and cloud-based high-compute workflows by capturing execution signals directly from the operating system kernel.

Key characteristics:

• Framework-agnostic: Works with any workflow engine, scheduler, language, or container runtime.
• Safe and efficient: Uses eBPF kernel tracing without requiring code changes or modifying system binaries.
• Actionable: Produces telemetry describing what ran, where it ran, and how resources were consumed.

Tracer is a multi-layered system designed to provide visibility into execution behavior in compute-intensive environments without requiring application code changes or workload modifications. It operates through three primary components:

• Tracer/collect: An open-source eBPF agent that gathers execution signals directly from the host-layer (Linux kernel).
• Tracer/datalake: A shared execution view that aggregates signals across different pipelines and environments.
• Tracer/tune and Tracer/sweep: Downstream products that consume the execution signal to perform optimization and waste detection.

The system is non-intrusive, meaning it does not require container restarts or application instrumentation.

Performance

Tracer's eBPF-based collection is designed for low overhead:

• Low latency: Probes execute in kernel space.
• Efficient filtering: Event filtering occurs at the source to reduce data volume.
• Scalability: Maintains low overhead even with many short-lived processes.

Security Boundaries

Tracer is an observability tool and does not:

• Modify application behavior.
• Control execution or scheduling.
• Start, stop, or change resources.
• Replace IAM, RBAC, or cloud security controls.

Tracer and Datadog are complementary tools rather than competitors.

• Datadog is a general-purpose observability platform for metrics, logs, traces, dashboards, and alerting across infrastructure and applications. It organizes telemetry around services, hosts, and applications.
• Tracer provides execution-aware insight by observing how tasks, tools, and processes actually execute at runtime. It organizes behavior by pipeline, run, and execution unit.

Use Tracer alongside Datadog when you need to understand pipeline behavior beyond service-level telemetry, diagnose performance issues in short-lived or nested execution, or attribute resource usage and cost to specific workflows or tools.

Tracer is an observation tool and is intentionally scoped to avoid interfering with workloads.

Tracer does NOT:

• Modify application behavior.
• Control execution or scheduling.
• Start, stop, or terminate workloads.
• Replace IAM, RBAC, or cloud security controls.

Tracer operates strictly within the boundaries enforced by the operating system, container runtime, and cloud provider.

The documentation is built using Mintlify and follows this structure:

• docs.json: Configuration file for navigation and theme settings.
• *.mdx: Documentation pages written in MDX (Markdown with JSX).
• images/: Directory for static assets like screenshots and diagrams.
• snippets/: Directory for reusable content blocks.