DOCSEARCHbyAlgolia
Submit Docs

Zipkin Distributed Tracing System

repository·master·Indexed 12 days ago

https://github.com/openzipkin/zipkin

A distributed tracing system designed to collect and visualize timing data to troubleshoot latency and errors in microservice architectures. It supports various storage backends including Cassandra, Elasticsearch, and MySQL, and integrates with messaging systems like Kafka, RabbitMQ, ActiveMQ, and Pulsar. The system includes a web UI for trace visualization and can be deployed via Docker Compose.

Tokens
33.4K
Snippets
116
Records
168
Agent score
97%

What's inside Zipkin

  1. Use the zipkin-eureka Docker image

    master

    The zipkin-eureka testing image provides a Zipkin server integrated with a Eureka Server for service discovery. The Eureka Server listens on port 8761.

    In addition to standard docker-java environment variables, this image supports the following configuration options:

    • EUREKA_USERNAME: Username for authenticating endpoints under /eureka.
    • EUREKA_PASSWORD: Password for authenticating endpoints under /eureka.
    • JAVA_OPTS: Used to modify Eureka settings, such as adjusting the heap size.

  2. Choose the correct Zipkin Docker image

    master

    Zipkin provides two main production images:

    • openzipkin/zipkin: The core server image. It hosts the Zipkin UI, API, and Collector features. Use this if you need storage types other than Elasticsearch (e.g., Cassandra or MySQL).
    • openzipkin/zipkin-slim: A stripped-down server image. It hosts the Zipkin UI and API but only supports in-memory or Elasticsearch storage via HTTP or gRPC span collectors.

    Both images are mirrored at ghcr.io/openzipkin/zipkin and ghcr.io/openzipkin/zipkin-slim respectively.

  3. Encode spans into Kafka messages using JSON or Thrift

    master

    When sending spans to the Kafka topic, the binary data must include a list of spans. The supported encodings are:

    JSON

    The message's binary data is a list of spans in JSON format. The first character must be [ (decimal 91). Use Codec.JSON.writeSpans(spans) for correct encoding.

    Thrift

    The message's binary data includes a list header followed by N spans serialized in TBinaryProtocol. Codec.THRIFT.writeSpans(spans) encodes spans using this structure:

    • write_byte(12) (type of list elements: 12 == struct)
    • write_i32(count) (number of spans)
    • For each span: writeTBinaryProtocol(spans(i))

    Legacy encoding

    Older versions of Zipkin accepted a single span per message instead of a list. This is deprecated but still supported.

    $ kafka-console-producer.sh --broker-list $ADVERTISED_HOST:9092 --topic zipkin
[{"traceId":"1","name":"bang","id":"2","timestamp":1470150004071068,"duration":1,"localEndpoint":{"serviceName":"flintstones"},"tags":{"lc":"bamm-bamm"}}]

  4. Manage trace identifiers and migration

    master

    By default, trace identifiers are unanalyzed keywords requiring exact string matches (16 or 32 lowercase hex characters).

    Migrating from 64 to 128-bit trace IDs: To support mixed lookups during a migration, set ElasticsearchStorage.Builder.strictTraceId to false. This enables tokenization in the index template, allowing a 64-bit lookup to match spans reported with a 128-bit variant of the same ID.

    Testing tokenization: You can verify how a trace ID is tokenized using the _analyze API:

    curl -s 'localhost:9200/zipkin*span-2017-08-22/_analyze' -d '{ "text": "48485a3953bb61246b221d5bc9e6496c", "analyzer": "traceId_analyzer" }' | jq '.tokens|.[]|.token'
    # the output below shows which tokens will match on the trace id supplied.
$ curl -s 'localhost:9200/zipkin*span-2017-08-22/_analyze' -d '{
      "text": "48485a3953bb61246b221d5bc9e6496c",
      "analyzer": "traceId_analyzer"
  }'|jq '.tokens|.[]|.token'
  "48485a3953bb61246b221d5bc9e6496c"
  "6b221d5bc9e6496c"

  5. Handle 64-bit to 128-bit Trace ID Migration

    master

    Zipkin supports both 64-bit and 128-bit trace identifiers. During a migration where some applications use 64-bit and others use 128-bit, a 'mixed state' occurs. 64-bit applications may truncate a 128-bit trace ID to its right-most 16 characters, causing Zipkin to treat the same trace as two different IDs.

    To mitigate this during the transition, set STRICT_TRACE_ID=false.

    When STRICT_TRACE_ID=false, Zipkin only considers the right-most 16 characters of a 32-character trace ID when grouping or retrieving traces.

    Important: Remove this setting or set STRICT_TRACE_ID=true once the migration to 128-bit is complete across all applications.

  6. Use the KafkaCollector to consume spans from Kafka

    master

    The KafkaCollector is a Zipkin collector implemented as a Kafka consumer. It supports Kafka brokers version 0.10.0.0 or later. It polls a specified Kafka topic for messages containing lists of spans encoded in either JSON or TBinaryProtocol (big-endian). These spans are then pushed to a span consumer.

    If using the collector as a library outside of the Zipkin server, use zipkin2.collector.kafka.KafkaCollector.Builder. By default, it operates against a Kafka topic named zipkin.

  7. Understand RabbitMQCollector behavior and caveats

    master

    When using the RabbitMQCollector, be aware of the following operational characteristics:

    • Queue Declaration: The configured queue is automatically declared as a durable queue and is handled idempotently.
    • Concurrency Model: The collector uses a single connection to RabbitMQ. It spawns the number of threads specified by concurrency, where each thread uses its own channel to consume messages.
    • Message Acknowledgement: Consumption is performed with autoAck enabled. Warning: Messages that fail to process successfully are not retried and will be lost.

  8. Understand the Zipkin Core Library design philosophy

    master

    The Zipkin Core Library is designed for use in streaming pipelines and instrumentation libraries (such as Java agents or Android code). Its design is driven by several key constraints:

    • Small API Surface: Only types that are public internally or have significant demand are exposed as public APIs. Most types are package-private to simplify migrations and prevent accidental dependence on utilities.
    • Zero Dependency Policy: To avoid dependency conflicts (especially in modular frameworks like OSGi or Java 9+), Zipkin avoids 3rd-party dependencies, relying only on floor Java version features.
    • Bytecode Optimization: Zipkin avoids using private modifiers on methods and fields to reduce bytecode size and avoid method count limits in environments like Android. Instead, it relies on package-private visibility to share state within a package.
    • Java 8 Requirement: Zipkin requires Java 8 or higher.

  9. Supported Zipkin Storage Components

    master

    Zipkin supports several storage backends depending on your requirements for persistence, scale, and features:

    • In-Memory: Packaged in the core library. Not persistent; intended for testing/local development.
    • Cassandra: Designed for scale using UDTs and SASI/manual indexes. Requires a separate job to aggregate dependency links. Tested against Cassandra 4.1.
    • Elasticsearch: Stores spans as Zipkin v2 JSON. Tested against Elasticsearch 7-8.x and OpenSearch 2.x. Requires a Spark job to aggregate dependency links.
    • MySQL (Legacy v1): Uses Zipkin's V1 Thrift model. Not recommended for high performance due to known issues where queries can become slow as data grows.

  10. Encoding spans into Pulsar messages

    master

    The binary data of a Pulsar message contains a list of spans. The supported encodings match the standard Zipkin POST /spans body formats.

    Supported Encodings

    Json

    The binary data is a JSON list of spans. The first character must be [ (decimal 91). Use Codec.JSON.writeSpans(spans) to ensure correct encoding.

    Thrift

    The binary data includes a list header followed by $N$ spans serialized using TBinaryProtocol. Use Codec.THRIFT.writeSpans(spans) to encode. The structure is:

    1. write_byte(12) (indicates the list element type is a struct)
    2. write_i32(count) (the number of spans)
    3. A loop iterating count times, calling writeTBinaryProtocol(spans(i)) for each span.

    Legacy encoding

    Support is provided for older versions of Zipkin that sent a single span per message instead of a list. This is deprecated but remains functional.

  11. Handle NullPointerExceptions in Zipkin public APIs

    master

    Zipkin's public entry points perform eager null checks. If a required argument is null, the library will throw a NullPointerException with a specific message (e.g., "xxx == null") rather than an IllegalArgumentException or a deferred, uninformative NPE.

    This pattern is used to make debugging easier by explicitly identifying which local variable was null at the point of the call.

  12. Understand Zipkin Docker health check configuration

    master

    To improve startup speed and automation reliability, Zipkin Docker images deviate from Docker's default health check settings:

    • Interval and Timeout: Instead of the Docker default of 30s, Zipkin images typically lower the interval and timeout to 5s. This allows services to be marked as healthy much faster (e.g., in ~10s instead of waiting for a 30s cycle) while still accommodating slow initial requests caused by schema setup in databases like Elasticsearch or Cassandra.
    • Kafka Specifics: For Kafka, the timeout is set to 5s specifically to prevent false negatives. Because kafka-topics.sh uses a large Java classpath, it can be slow during high host contention (e.g., when many containers are starting simultaneously in Docker Compose). A 5s timeout provides enough buffer to avoid breaking orchestration dependencies.