DOCSEARCHbyAlgolia
Submit Docs

VictoriaTraces Documentation

repository·master·Indexed 19 days ago

https://github.com/victoriametrics/victoriatraces

A resource-efficient, high-performance distributed tracing database designed to store and query trace spans using the OpenTelemetry protocol (OTLP). VictoriaTraces is compatible with Jaeger Query Service APIs for integration with Grafana and Jaeger tools. The project includes components such as vtinsert, vtstorage, and vtselect, a Web UI (vmui), and a trace data generator (vtgen) for benchmarking ingestion and query performance.

Tokens
33.9K
Snippets
96
Records
167
Agent score
62%

What's inside VictoriaTraces

  1. Use the http2server package for HTTP/2 support

    master
    The http2server package is an experimental implementation providing HTTP/2 support. It is a port of the github.com/VictoriaMetrics/VictoriaMetrics/lib/httpserver package, modified to enable HTTP/2 capabilities while preserving only the core features. Use this package if your application requires HTTP/2 support and you are working within the VictoriaMetrics ecosystem.

  2. Configure multi-level cluster setups

    master

    You can build hierarchical ingestion and query schemes by nesting components using the -storageNode flag:

    • Multi-level Ingestion: Configure a top-level vtinsert to send spans to multiple lower-level vtinsert nodes using the -storageNode flag. This allows a primary ingestion layer to spread load across multiple sub-clusters.
    • Multi-level Querying: Configure a top-level vtselect to query multiple lower-level vtselect nodes using the -storageNode flag. This allows a single query entry point to aggregate results from multiple sub-clusters.

  3. High Availability (HA) in VictoriaTraces Cluster

    master

    VictoriaTraces handles availability differently for ingestion and querying:

    Ingestion (Write Path)

    vtinsert provides high availability for writes. If some vtstorage nodes are unavailable, vtinsert automatically routes new trace spans to the remaining available nodes to ensure ingestion is not interrupted.

    Querying (Read Path)

    vtselect prioritizes data integrity over partial availability. If any relevant vtstorage node is unavailable, vtselect will return an error instead of returning potentially incomplete or misleading results.

    Achieving Full HA with External Replication

    VictoriaTraces does not perform storage-level replication. To achieve full HA (protecting against cluster failure), use an external trace shipper (like the OpenTelemetry Collector) to replicate spans to two independent VictoriaTraces clusters (e.g., Zone A and Zone B).

    Use a load balancer (like vmauth) in front of the clusters to route query traffic to whichever cluster is healthy.

  4. High Availability (HA) with Single-Node Instances

    master

    To achieve High Availability using VictoriaTraces Single-Node instances, use the following architecture:

    1. Trace Collector: Use a collector like the OpenTelemetry collector that supports multiplexing incoming data to multiple destinations.
    2. VictoriaTraces Instances: Deploy two or more Single-Node instances.
    3. Load Balancer/vmauth: Use vmauth or a standard Load Balancer to distribute read requests across the replicas, ensuring balanced and redundant access.

  5. Understand common components in VictoriaTraces Docker environments

    master

    The Docker Compose environments include several auxiliary components to provide a complete observability stack:

    • vmauth: Acts as a load balancer to spread load across vtselect and vtinsert nodes in a cluster. It routes read queries to VictoriaTraces based on the requested path.
    • vmalert: Evaluates alerting rules.
      • vmalert-metrics: Evaluates rules on VictoriaMetrics based on metrics. Web UI: http://localhost:8880/
      • vmalert-traces: Evaluates rules on VictoriaTraces based on trace spans. Web UI: http://localhost:8881/
    • alertmanager: Receives notifications from vmalert and fires alerts. Web UI: http://localhost:9093/
    • Grafana: Pre-configured with VictoriaMetrics and Jaeger datasources. Web UI: http://localhost:3000 (Credentials: admin/admin)

  6. Optimize LogsQL performance in recording rules

    master

    LogsQL allows you to calculate multiple statistics (like different quantiles) within a single expression. Using this in a recording rule is more efficient than creating separate rules for each statistic.

    When you use a multi-stat expression in a recording rule, vmalert generates a unique metric for each statistic, distinguished by a stats_result label.

    ```yaml
groups:
  - name: requestDuration
    type: vlogs
    interval: 5m
    rules:
      - record: requestDurationQuantile
        expr: '* | stats by (service) quantile(0.5, request_duration_seconds) p50, quantile(0.9, request_duration_seconds) p90, quantile(0.99, request_duration_seconds) p99'

    Resulting Metrics:

    • requestDurationQuantile{stats_result="p50", service="service-1"}
    • requestDurationQuantile{stats_result="p90", service="service-1"}
    • requestDurationQuantile{stats_result="p99", service="service-1"}

  7. How trace span streams work

    master

    VictoriaTraces uses service.name (from resource attributes) and name (from the span) to define trace span streams. This grouping optimizes storage and query performance by allowing better compression and reducing the amount of data scanned during searches.

    Each stream consists of two special fields:

    1. _stream_id: A unique identifier for the stream. You can select all spans in a stream using the _stream_id:... filter.
    2. _stream: Contains stream labels in a format similar to Prometheus, e.g., {resource_attr:service_name="svc name", name="span name"}. This field is used with stream filters.

  8. Use multitenancy in VictoriaTraces

    master

    VictoriaTraces supports multitenancy using an (AccountID, ProjectID) pair, where both are 32-bit unsigned integers.

    • Identification: Set AccountID and ProjectID via request headers during data ingestion and querying.
    • Defaults: If headers are not provided, the default value is 0.
    • Authorization: VictoriaTraces does not perform per-tenant authorization. You must use a tool like vmauth to manage access control for different tenants.

  9. How App Integration Tests work in VictoriaTraces

    master

    Integration tests in the apptest package verify the behavior of VictoriaTraces applications as a whole by starting them in separate processes and interacting with them via HTTP requests. These tests can target a single application or a complex system like a VictoriaTraces cluster to verify component interaction.

    Key components of the testing framework include:

    • app.go: Generic code for starting applications (internal use only).
    • {vtsingle,etc}.go: Specialized helpers for starting specific application types.
    • client.go: Helper functions for issuing HTTP requests to the running applications.

    Tests are located in tests/*_test.go files.

  11. Understand the Trace Data Model

    master

    A trace in VictoriaTraces is composed of multiple Spans. Each span represents a single operation within a trace and contains the following key components:

    • traceID: A unique identifier for the entire trace.
    • spanID: A unique identifier for the specific span.
    • operationName: The name of the operation being performed (e.g., POST /api/checkout or grpc.oteldemo.PaymentService/Charge).
    • startTime: The timestamp when the span began.
    • duration: The length of time the operation took.
    • processID: An identifier for the process where the span was recorded.
    • references: A list of relationships to other spans, typically defining parent-child relationships (e.g., refType: CHILD_OF).
    • tags: A collection of key-value pairs providing metadata about the operation (e.g., http.method, rpc.system, db.system, error).
    • logs: A collection of timestamped events or messages associated with the span.
    • warnings/errors: Indicators of issues encountered during the span's execution.
    {
  "traceID": "9e06226196051d9c3c10dfab343791ad",
  "spans": [
    {
      "spanID": "df89f1712cb212dd6ed",
      "operationName": "grpc.oteldemo.PaymentService/Charge",
      "startTime": 1750044449742000,
      "duration": 3076,
      "tags": [
        { "key": "rpc.method", "type": "string", "value": "Charge" },
        { "key": "rpc.system", "type": "string", "value": "grpc" }
      ],
      "references": [
        { "refType": "CHILD_OF", "spanID": "530667cc212dd6ed", "traceID": "9e06226196051d9c3c10dfab343791ad" }
      ]
    }
  ]
}

  12. VictoriaTraces Cluster Architecture

    master

    VictoriaTraces in cluster mode consists of three primary components designed for horizontal scaling:

    • vtinsert: Handles trace span ingestion via the OpenTelemetry protocol (OTLP). It distributes incoming spans by trace ID across the available vtstorage nodes using the -storageNode command-line flag.
    • vtselect: Receives queries via HTTP endpoints. It fetches data from the configured vtstorage nodes, processes the queries, and returns aggregated results.
    • vtstorage: Acts as the data layer. It stores spans in the directory specified by -storageDataPath and handles local data retrieval and transformation for vtselect queries.

    Communication Model: Components communicate via HTTP over the port defined by -httpListenAddr:

    • vtinsert $\rightarrow$ vtstorage: POST /internal/insert
    • vtselect $\rightarrow$ vtstorage: GET /internal/select/*