Vector Remap Language

VRL is a stateless, expression-oriented scripting language designed for processing observability data such as logs, metrics, and traces. It is built on two core principles:

• Safety: Programs require explicit handling of all errors from fallible functions, preventing unexpected runtime failures.
• Performance: Programs are compiled at startup and execute with near-native performance, avoiding garbage collection and runtime overhead.

In VRL, every program processes a single event, and every expression returns a value.

By using the new Value::String variant (which uses the bytestring::ByteString type), VRL can avoid expensive UTF-8 validation passes.

When a value is constructed from a guaranteed-UTF-8 source—such as a parser literal, JSON deserialization, or a grok capture—subsequent operations like as_str, display, serde serialization, or string manipulations become $O(1)$ operations instead of requiring repeated calls to from_utf8 or from_utf8_lossy.

VRL is an expression-oriented language that favors returning new copies of data over mutating existing data. While mutation might seem more performant, explicit assignment makes programs easier to reason about and is the standard pattern in VRL.

Correct Pattern (Explicit Assignment):

.message = parse_json(.message)

Avoid (In-place Mutation):

parse_json(.message)

# explicitly assign the parsed JSON to `.message
.message = parse_json(.message)

VRL (Vector Remap Language) is a domain-specific language (DSL) designed for safe, performant, and predictable observability data remapping. It is an expression-oriented language, meaning it is optimized to express data transformations rather than general-purpose programming.

Core Principles:

• Safety and Performance over Ease of Use: VRL is built for mission-critical production systems where reliability and speed are prioritized.
• Clarity of Output: The best VRL programs are those that most clearly express the intended data transformation. Avoid using 'tricks' for performance if they sacrifice readability.
• Predictability: Operations are designed to behave in a consistent, in-memory fashion. Use network-dependent functions with caution, as they fall outside the standard performance envelope and can introduce unpredictable delays.

In VRL, equality (==) is based on the underlying byte content rather than the specific Value variant. This ensures that a string literal or a JSON-parsed string behaves identically to a raw byte array when compared.

Even after the introduction of Value::String, the following behavior is preserved:

• Value::Bytes(b"foo") == Value::String("foo") evaluates to true.
• Hash implementations are custom-built so that byte-equal values produce identical hashes, ensuring they behave correctly in hash-based collections.

// Conceptual behavior:
// Even if the variants differ, equality is structural over byte content
"foo" == b"foo" // true

VRL is specifically designed for operators, DevOps, and SREs (Site Reliability Engineers) who manage infrastructure.

It occupies a middle ground between:

1. Static transforms (like simple field renaming), which are too limited.
2. Turing-complete languages (like Lua), which can be overly complex and introduce performance/readability downsides.

Use VRL when you need a flexible way to remap observability data without the overhead or risks associated with a full programming language.

In later phases of the VRL evolution, Kind::string is introduced as a refinement of Kind::bytes. This means a string kind is a subset of the bytes kind that carries a UTF-8 witness.

Mental Model:

• is_bytes() and contains_bytes() remain the primary way to check for byte-like data and will return true for both Value::Bytes and Value::String.
• is_only_string() returns true only if the kind is exactly the refined Kind::string.
• Refinement Survival in Merges: When merging two values, the string refinement is only preserved if both sides are 'string-y' and both carry the refinement. If one side is unrefined bytes, the result is unrefined bytes.
• String Concatenation Refinement:
  • is_only_string() + is_only_string() $\rightarrow$ Kind::string()
  • Any other combination of string-like values $\rightarrow$ Kind::bytes()

VRL has moved away from manually maintained CUE files to an automated documentation generation system. Documentation is now defined directly within the source code by implementing the Function trait.

The Workflow:

1. Definition: Documentation lives alongside the function implementation in the VRL repository.
2. Validation: All function examples are automatically tested and validated via CI.
3. Generation: A JSON file is generated for each stdlib function and checked into the VRL repository.
4. Consumption: The Vector website dynamically generates documentation using VRL's generation logic when it spins up, consuming these JSON files.

This ensures that the source code remains the single source of truth and that documentation cannot go stale or contain broken examples.

To maintain safety, performance, and simplicity, VRL intentionally excludes several common programming features:

• No Modules: Functions are grouped by naming conventions rather than being split into modules.
• No Classes: Indirection is minimized by avoiding object-oriented class structures.
• No User-Defined Functions: Programs are intended to be read from top to bottom without jumping between function definitions to maintain clarity.
• No Assignable Closures: While closures are supported within function calls, they cannot be assigned to variables or reused elsewhere. This prevents complex indirection and potential infinite loops.

VRL is infallible by default. The compiler will refuse to compile any program where an expression can result in a runtime error unless that error is explicitly handled.

Common causes for compilation failure include:

• Fallible functions: Functions that can fail (e.g., parse_json might fail if the input isn't valid JSON).
• Type uncertainty: When the type of a field cannot be guaranteed at compile-time (e.g., concatenating a field with a string when the field's type is unknown).
• Argument mismatch: Passing an argument to a function that doesn't match the expected type (e.g., passing a non-string to upcase()).

You must use VRL's failure-handling features to manage these cases before the compiler will accept the source.