Vector Observability Data Pipeline

Vector is a tool for building flexible and robust data pipelines for logs and metrics. It is designed to act as a complete pipeline rather than just a single component, simplifying observability infrastructure.

Vector's core capabilities include:

• Ingestion: Collecting data from various sources such as tailing files or receiving syslog messages over the network.
• Processing: Manipulating data in flight using techniques like regex parsing, JSON parsing, filtering, and sampling.
• Delivery: Sending raw or processed data to a variety of external storage and querying systems.

Key differentiators include a rich data model (supporting logs, aggregated metrics, and structured events), programmable transforms (using Lua), and high performance/efficiency.

The opentelemetry-proto package defines the protocol used by the OpenTelemetry Collector. It is organized into several specialized packages based on the type of telemetry data being handled:

• common: Contains shared messages used across different services.
• trace: Contains the Trace Service protocol definitions.
• metrics: Contains the Metrics Service protocol definitions.
• logs: Contains the Logs Service protocol definitions.

Vector's architecture is built upon several progressive models that define how data is structured, processed, and managed. To understand how to build and optimize your data pipelines, you should study these models in order:

1. Data Model: Defines the structure and representation of data as it moves through Vector.
2. Pipeline Model: Describes how data flows through sources, transforms, and sinks.
3. Concurrency Model: Explains how Vector handles parallel processing and task execution.
4. Buffering Model: Details how Vector manages data in transit and handles backpressure or crashes.
5. End-to-End Acknowledgements: Covers the mechanisms for ensuring data reliability and delivery guarantees.
6. Runtime Model: Describes the high-level execution environment and lifecycle of the Vector process.

Vector employs a multi-layered testing strategy to ensure reliability and performance, specifically addressing the challenges of its modular architecture and its extensive use of external system interfaces. The strategy is categorized into three main approaches:

1. Example-based testing: Uses specific inputs and asserts expected outputs. This includes Unit tests (isolated logic) and Integration tests (component interactions).
2. Generative testing: Uses automated generation of inputs, including Property-based testing, Model-based testing, and Fuzz testing.
3. Black-box testing: Focuses on observable behavior, including Performance tests, Correctness tests, and Reliability tests.

Vector provides several integrations to export observability data from AWS. As of version 0.11.0, the following AWS-specific components are available:

Sources

• aws_kinesis_firehose: Collect data from AWS Kinesis Data Firehose.
• aws_s3: Collect data from AWS S3 buckets.
• aws_ecs_metrics: Collect metrics from AWS ECS.

Sinks

• aws_sqs: Export data to an AWS SQS queue.

Transforms

• aws_cloudwatch_logs_subscription_parser: Parses AWS CloudWatch logs (Note: This transform was removed in v0.23.0).

To collect AWS CloudWatch logs, a common pattern is to use AWS Firehose to stream logs to Vector, which can then be sent to any supported Vector sink.

Log Namespacing is a feature in Vector that allows different fields of a Log event to be kept under separate namespaces. This prevents field name conflicts where two different data sources might attempt to use the same field name.

Note: Log Namespacing applies only to Log events; it does not apply to Metric or Trace events.

Adaptive Request Concurrency (ARC) is a networking feature in Vector designed to replace static rate limits. Instead of using fixed concurrency settings, ARC automatically optimizes HTTP concurrency limits by monitoring downstream service responses.

It uses a feedback loop mechanism inspired by TCP congestion control algorithms to dynamically adjust how many concurrent requests are sent to a destination. This helps improve performance and reliability across observability infrastructure by preventing downstream services from being overwhelmed while maximizing throughput when they are healthy.

Vector Remap Language (VRL) is an expression-oriented language specifically designed for transforming observability data (logs and metrics). It is intended to replace the trade-off between rigid, configuration-based static transforms and slow, potentially unsafe runtime transforms (like Lua or JavaScript).

Key characteristics include:

• Safety: Designed to be performant and safe for critical observability infrastructure.
• Simplicity: It is not a general-purpose programming language. It intentionally lacks constructs like loops, classes, modules, custom functions, and IO access to ensure sound design and performance.
• Purpose-built: Features a simple syntax and a rich set of built-in functions tailored for observability use cases.

In a Vector deployment, an aggregator is a role used to transform and ship data collected by other agents.

By separating agents from aggregators, you achieve a separation of concerns in your observability pipeline:

• Agents: Act as "thin" pipes that primarily collect from a source and ship data downstream to aggregators.
• Aggregators: Perform "thick" processing tasks such as scrubbing sensitive information, formatting data for downstream consumers, and sampling to reduce volume.

This architecture is particularly useful when integrating Vector into existing infrastructure where you want to centralize processing logic without changing how data is initially collected.

The Log Namespace feature isolates Vector metadata (such as host, timestamp, and source_type) from your actual log data.

In the legacy approach, metadata and log data are stored as top-level fields in the same namespace, which can lead to:

• Field name collisions: Your log data might contain a field named host that conflicts with Vector's metadata.
• Unclear ownership: It is difficult to distinguish between user data and Vector metadata.
• Difficult transformations: Transformations intended for log data may accidentally affect metadata fields.

When log_namespace: true is enabled, the . (the root of the event) represents only your log data, while metadata is moved to a separate, isolated namespace.

Criterion benchmarks are used to establish the baseline performance of Vector's building blocks.

Key Metrics:

• Bytes-per-second throughput: The primary metric for every source, sink, and transform.
• Tolerances: Established throughput values used to calculate the expected cost/resource requirements for specific workloads (e.g., estimating the cost of a syslog -> json_parser -> elasticsearch pipeline in a specific cloud region).

Developers should use these benchmarks to ensure that new PRs do not introduce regressions in the throughput of individual components.