DOCSEARCHbyAlgolia
Submit Docs

VictoriaLogs Documentation

repository·master·Indexed 24 days ago

https://github.com/victoriametrics/victorialogs

A high-performance, schema-free log database designed for scalability from small to terabyte-scale deployments, available in single-node and clustered versions. Includes documentation for vlagent (log collection), vlogscli (querying), vlogsgenerator (log generation), and the vmui Web UI.

Tokens
122.9K
Snippets
294
Records
677
Agent score
80%

What's inside VictoriaLogs

  1. Overview of VictoriaLogs

    master
    VictoriaLogs is a high-performance, lightweight, zero-config, and schema-free database designed for log management. It is built to scale both vertically and horizontally, supporting use cases ranging from small individual setups to large-scale deployments processing terabytes of logs per day. Both single-node and cluster versions are available as open-source software.

  3. What is vlagent and when to use it

    master

    vlagent is a log collection agent designed to collect logs from various sources and store them in VictoriaLogs.

    Key Features

    • Kubernetes Log Collection: Automatically discovers and collects logs from all Pods in a Kubernetes cluster with low resource consumption.
    • Protocol Support: Accepts logs via HTTP-based protocols (similar to VictoriaLogs) on TCP port 9429 by default.
    • Replication: Can replicate logs to multiple VictoriaLogs instances for high availability.
    • Buffering: Handles unstable connections by buffering logs to disk if the remote storage is unavailable.
    • Remote Write: Can send logs to any destination supporting newline-delimited JSON.

    When to use vlagent

    • You need to replicate logs to multiple VictoriaLogs instances for high availability.
    • You have unstable connectivity to VictoriaLogs and require on-disk buffering.
    • You run Kubernetes and want automatic Pod log collection with metadata enrichment.

    Note: If you have a single VictoriaLogs instance and a stable network connection, you should send logs directly to VictoriaLogs to reduce operational complexity.

  4. Overview of VictoriaLogs

    master

    VictoriaLogs is an open-source, resource-efficient, and fast database designed for logs. It is built by VictoriaMetrics and is designed to be a single, zero-config executable that is easier to operate than alternatives like Elasticsearch or Grafana Loki.

    Key Capabilities:

    • Resource Efficiency: Uses significantly less RAM and disk space compared to Elasticsearch and Grafana Loki.
    • Scalability: Scales linearly with CPU, RAM, and disk IO. Supports single-node and horizontal cluster modes.
    • Querying: Uses LogsQL, a powerful language for full-text search, analytics, and data transformation. Includes a built-in web UI and an interactive CLI tool (vlogscli).
    • Data Model: Optimized for high-cardinality fields (e.g., trace_id, user_id) and "wide events" (logs with hundreds of fields).
    • Ingestion: Supports popular log collectors, out-of-order ingestion (backfilling), and live tailing.
    • Integrations: Provides a Grafana plugin and integrates with Unix tools like grep, jq, and sort.

  5. Introduction to LogsQL

    master

    LogsQL is the query language used for VictoriaLogs. It is designed for powerful log analysis and supports several core capabilities:

    • Full-text search: Search across any log field. By default, searches are performed on the _msg field.
    • Complex filtering: Combine multiple filters using arbitrary logical operators.
    • Query-time extraction: Extract structured fields from unstructured log data during the query process.
    • Statistical analysis: Use pipes to calculate various statistics over selected log entries.

  6. Overview of VictoriaLogs Dashboard Sections

    master

    The VictoriaLogs dashboards provide comprehensive monitoring organized into several functional areas:

    • Stats: High-level metrics (total log entries, ingestion rates, disk usage, system version).
    • Overview: Real-time visualization (ingestion rates, request patterns, error rates, performance trends).
    • Resource usage: System metrics (CPU, memory, network, garbage collection, system pressure).
    • Troubleshooting: Operational diagnostics (error tracking, configuration validation).
    • Slow Query Troubleshooting: Query performance (optimization metrics, latency diagnostics).
    • Storage: Merge performance and indexing metrics.
    • Ingestion: Pipeline monitoring (flush operations, data flow rates).
    • Querying: Request latencies and timeout tracking.

    Cluster Deployments: For cluster setups, dedicated sections monitor individual components: vlstorage, vlinsert, and vlselect.

  7. Query VictoriaLogs using LogsQL

    master

    VictoriaLogs uses the LogsQL query language to search and analyze logs. You can execute LogsQL queries through several interfaces:

    • vlogscli: A dedicated command-line tool.
    • Command-line interface: Direct CLI interaction.
    • HTTP API: Programmatic access via HTTP requests.
    • Web UI: A web-based graphical interface for manual querying.
    • Grafana plugin: Visualization of logs within Grafana dashboards.

  8. Supported log collectors for VictoriaLogs and vlagent

    master

    VictoriaLogs and vlagent support ingestion from a wide variety of popular log collectors and agents. You can configure these collectors to send logs directly to VictoriaLogs or via vlagent.

    Supported collectors include:

- Syslog, Rsyslog and Syslog-ng
- Filebeat
- Fluentbit
- Fluentd
- Logstash
- Vector
- Promtail (aka Grafana Loki, Grafana Agent or Grafana Alloy)
- Telegraf
- OpenTelemetry Collector
- Journald
- DataDog
- Splunk

  10. Overview of running_stats pipe functions in LogsQL

    master

    The running_stats pipe in LogsQL allows you to perform stateful calculations over a sequence of logs. It provides several functions to track metrics as logs are processed:

    • count(): Calculates the running number of selected logs.
    • last(fieldName): Returns the value of a specific field from the previously seen log entry (based on _time order). Useful for comparing values between consecutive logs.
    • max(field1, ..., fieldN): Returns the running maximum across the specified fields.
    • min(field1, ..., fieldN): Returns the running minimum across the specified fields. Note that missing fields are treated as empty strings.
    • sum(field1, ..., fieldN): Calculates the running sum of numeric values. Non-numeric values are skipped, and if no numeric values are found, NaN is returned.

  11. Understand VictoriaLogs Cluster Architecture

    master

    VictoriaLogs in cluster mode scales horizontally by splitting responsibilities into three main components. All components use the same single-node VictoriaLogs executable, but their behavior is determined by command-line flags.

    Components

    • vlinsert: Accepts logs via supported protocols and shards (distributes) them across vlstorage nodes.
    • vlselect: Accepts queries, executes them in parallel across vlstorage nodes, merges the results, and returns the response to the client.
    • vlstorage: Stores logs at the directory specified by -storageDataPath and executes queries received from vlselect.

    Component Roles and Configuration

    • vlstorage: If the -storageNode flag is not provided, the executable runs as a vlstorage node.
    • vlinsert / vlselect: If a comma-separated list of TCP addresses is passed to the -storageNode flag, the executable converts into an ingestion or query node.
    • Hybrid Node: By default, providing -storageNode makes the node serve both vlinsert and vlselect requests, forwarding them to the specified storage nodes.

    Communication

    Components communicate via HTTP over the port specified by -httpListenAddr (default 9428):

    • vlinsert $\rightarrow$ vlstorage: /internal/insert endpoint.
    • vlselect $\rightarrow$ vlstorage: /internal/select/* endpoints.

  12. Use DNS SRV for remote write URL resolution

    master

    vlagent supports DNS SRV hostname resolution. If the -remoteWrite.url starts with the srv+ prefix, vlagent will resolve the TCP address from the corresponding SRV record. If multiple targets are returned, vlagent randomly chooses a target for every new connection.

    ./vlagent -remoteWrite.url=http://srv+victoria-logs/insert/native