DOCSEARCHbyAlgolia
Submit Docs

Jaeger Distributed Tracing System

repository·main·Indexed 12 days ago

https://github.com/jaegertracing/jaeger

A CNCF graduated distributed tracing system used to monitor and troubleshoot microservices by collecting, storing, and visualizing traces. Features include the jaeger-es-index-cleaner for Elasticsearch maintenance, a jaeger_storage_exporter with configurable retry behavior, and an AI Gateway that integrates with AI Agent Sidecars using the Agent Client Protocol (ACP) and Model Context Protocol (MCP) for automated trace analysis and skill-based troubleshooting.

Tokens
146.5K
Snippets
327
Records
583
Agent score
97%

What's inside Jaeger

  1. Explore Jaeger Analysis Skills

    main

    Jaeger provides a catalog of analysis skills designed to automate trace inspection. These skills are organized using progressive disclosure; you should read the specific SKILL.md file for a sub-skill before applying it to understand its specific logic and requirements.

    Available built-in skills include:

    • detect-n-plus-one: Identifies N+1 query patterns where a single parent operation triggers multiple near-identical child spans. This is useful for detecting chatty database access or repeated downstream calls.
    • error-root-cause: Traverses a failed trace to locate the first originating error span. Use this when a request fails and you need to identify the source of the error.

  2. Review Jaeger's OpenSSF Best Practices Gold Evidence

    main
    Jaeger maintains evidence for the OpenSSF Best Practices badge. This documentation tracks how the project meets various security and governance criteria, including contribution processes, testing policies, continuous integration, and security architecture (such as TLS configuration and input validation).

  3. Scope of the GenAI Data Layer

    main

    The GenAI observability data layer includes the following capabilities:

    In Scope:

    • Data models for Datasets, Samples, Experiments, Evaluators, and Scores.
    • Schema design for ClickHouse (primary) and guidance for OpenSearch.
    • API surface for storing and querying these entities.
    • Schema evolution strategies without DDL migrations.
    • Integration with external evaluation orchestrators (e.g., Langfuse, DeepEval).
    • Trace correlation via jaeger.eval.trial_id and jaeger.eval.iteration_index.
    • API support for introspective evaluators to retrieve traces by trial/iteration context.

    Out of Scope:

    • Full evaluation execution engines (triggering model calls).
    • LLM prompt management or versioning.
    • Authentication and multi-tenancy (uses existing Jaeger mechanisms).

  4. What is Jaeger Remote Storage?

    main
    The jaeger-remote-storage binary allows you to share single-node storage implementations (such as memory or Badger) over gRPC. It implements the Jaeger Remote Storage gRPC API, which enables other Jaeger components to interact with these storage backends remotely instead of requiring them to be co-located.

  5. What is the storage_cleaner extension?

    main

    The storage_cleaner is a Jaeger extension designed exclusively for testing environments. It provides a mechanism to purge all data from a backend storage by making an HTTP POST request to a specific endpoint.

    Warning: This extension has no authentication and is intended only for use in end-to-end (e2e) tests to clear storage between test runs. It is not included in the standard Jaeger component set to prevent accidental data loss in production.

  6. Understand the Jaeger monitoring metrics

    main

    The Jaeger monitoring dashboards use OpenTelemetry Collector (otelcol_*) metric names and Jaeger-internal metrics. These are compatible with Jaeger v2 and the OTel Collector.

    Key metric categories include:

    • Collector Ingestion/Export: e.g., otelcol_receiver_accepted_spans_total, otelcol_exporter_sent_spans_total.
    • Storage Latency: jaeger_storage_*.
    • Query Rates: http_server_request_duration_seconds_*.

  7. Understand ClickHouse trace storage format in Jaeger

    main

    Jaeger's ClickHouse implementation differs from the otel-collector-contrib implementation primarily in how it handles Attributes. While the OTel-contrib implementation converts all attribute values to strings (causing loss of data types), Jaeger preserves type fidelity by splitting keys and values into separate arrays and using type-specific columns.

    Storage Strategy

    • Basic Types: Values are stored in dedicated type-specific array columns, such as Array(Int64), Array(Bool), Array(Double), and Array(String).
    • Complex Types: Slices and Maps are serialized into JSON format strings and stored in Array(String) columns.
    • Type Preservation: To ensure accuracy for nested telemetry data and avoid JSON's inability to distinguish between integers and doubles, Jaeger converts pdata (from the OTLP specification) into standard OTLP/JSON format before storage.

    This approach allows for more precise querying. For example, you can perform numeric comparisons on attributes that would otherwise be treated as strings in other implementations:

    SELECT * FROM test WHERE resource.attributes['container.restart.count'] > 10

  8. Understand Predicate Anatomy in Structured Query Filters

    main

    In the new Jaeger structured query model, a predicate is a Call consisting of an operator applied to operand expressions.

    There are two types of operands:

    1. References: A value on a span or trace identified by its level, name, and an attr flag.
      • level: The scope (e.g., span, resource, instrumentation, event, link).
      • name: The identifier for the field or attribute.
      • attr: A boolean flag. If true, it refers to an entry in the level's attribute map. If false (default), it refers to a built-in field of that level.
    2. Constants: A scalar value (string, int, double, bool) or a list (used with in/not_in). Constants can optionally include a type to guide backend interpretation.

    Predicates can take several shapes, such as reference op constant (e.g., span.http.status_code = 500) or reference op reference (e.g., span.a > span.b).

    /* Example shapes of predicates */
// reference op constant
{ "ref": { "level": "span", "name": "http.status_code" }, "op": "eq", "args": [{ "scalar": { "value": "500" } }] }

// reference op reference
{ "ref": { "level": "span", "name": "a" }, "op": "gt", "args": [{ "ref": { "level": "span", "name": "b" } }] }

  9. How Jaeger resolves ambiguous attribute filters in ClickHouse

    main

    When a user enters a search term in the Jaeger UI, the system doesn't know if the value is a string, an integer, or a boolean. To avoid scanning every possible typed column at every level (which is computationally expensive), Jaeger uses an attribute_metadata table to resolve types at query time.

    The Resolution Process:

    1. Lookup: For every string-typed key in the search request, Jaeger queries the attribute_metadata table to find all observed (type, level) combinations for that key.
    2. Parsing: The query builder attempts to parse the user's string value into the observed types (e.g., using strconv.ParseInt for int types).
    3. Predicate Generation: For every successful parse, it generates a specific arrayExists(...) predicate against the matching typed column (e.g., int_attributes at the span level).
    4. Fallback: If the key is unknown or parsing fails, the system falls back to a broad "string at every level" search to ensure correctness.

    Note: Programmatic callers using the Jaeger v2 reader API with explicit types (e.g., Bool, Int64) bypass this ambiguity resolution and query the typed columns directly.

  10. Trace correlation for GenAI evaluations

    main

    To enable deterministic trace discovery from evaluators without out-of-band coordination, traces should be tagged with specific correlation attributes. This allows users to drill down from quality metrics directly into the raw spans that generated them.

    Use the following attribute convention for linking traces to evaluation records:

    • jaeger.eval.trial_id
    • jaeger.eval.iteration_index

    Note: The RFC indicates these may be updated to align with official OpenTelemetry GenAI semantic conventions (e.g., gen_ai.eval.*) once standardized.

  11. Understand Jaeger Backend Capabilities and AI Assistant availability

    main

    Jaeger uses a backendCapabilities mechanism to advertise features to the UI. The AI Assistant availability is treated as a dynamic capability based on the liveness of the external AI sidecar.

    • Capability Detection: The backend runs a periodic liveness probe (an ACP initialize handshake) to the agent_url.
    • UI Integration: The capability is exposed via window.getJaegerBackendCapabilities(). The UI uses this to gate the chat surface.
    • State Transitions: If a probe fails, aiAssistant is set to false immediately. If the sidecar becomes reachable again, the capability is updated.
    • Data Format: The capability is delivered as a JSON object:
    {
  "aiAssistant": true
}

  12. How AI Assistant visibility is determined in the UI

    main

    The Jaeger UI does not use a manual toggle to enable the AI assistant. Instead, visibility is driven by a backend-advertised capability called backendCapabilities.aiAssistant.

    The UI will automatically 'light up' (show the chat interface) only when the following conditions are met:

    1. The jaeger_query.ai configuration block is present in the backend config.
    2. A sidecar process is running and reachable.
    3. The backend successfully passes a liveness check to the sidecar.

    This ensures that the UI only presents AI features when the operator has actively opted in via configuration and the necessary sidecar infrastructure is functional.