Numaflow Documentation

Numaflow is a Kubernetes-native, serverless platform designed for running scalable and reliable event-driven applications. It decouples event sources and sinks from processing logic, allowing each component to independently auto-scale based on demand.

Key characteristics include:

• Language Agnostic: Each step in a pipeline can be written in any programming language.
• Serverless: Automatically scales components from zero based on demand.
• Data Integrity: Provides exactly-once semantics for unbounded and near real-time data sources, ensuring no input element is duplicated or lost during pod rescheduling or restarts.
• Auto-scaling: Features auto-scaling with back-pressure support.

Numaflow is language-agnostic, allowing you to develop event-driven applications, stream processing, and serving pipelines using various programming languages. You can write any component of your pipeline—from source to transformer to sink—in a supported language.

For specific implementation details, refer to the SDK Compatibility and SDK Features documentation.

Numaflow UI is a web-based interface for managing and monitoring Numaflow pipelines. Key features include:

• Viewing running pipelines within a specific namespace.
• Inspecting Vertex and Edge information for pipelines.
• Monitoring BackPressure and Pending Messages.
• Accessing container logs for specific vertices.

A User-Defined Function (UDF) is a vertex type in a Numaflow Pipeline that allows you to run custom code to transform data.

Key Characteristics:

• Execution Model: UDFs run as a sidecar container within a Vertex Pod.
• Communication: The Numaflow platform (main container) communicates with your custom code (sidecar container) via gRPC over a Unix Domain Socket.
• Idempotency: Data processing within a UDF should be designed to be idempotent to ensure correctness during retries.
• Processing Types: There are two primary processing models available:
  • Map: For individual record transformations.
  • Reduce: For aggregating data across multiple records.

A Numaflow pipeline is a Directed Acyclic Graph (DAG) of processors connected by inter-step buffers. The pipeline consists of three main types of processors:

1. Source Processors: Ingest data from an external source, write it to the first inter-step buffer, and acknowledge the data in the source.
2. Data Processors: Execute User-Defined Functions (UDFs). They read from one upstream buffer, process the data, write to one or more downstream buffers, and acknowledge the data in the upstream buffer.
3. Sink Processors: Read from an upstream buffer, write the processed data to an external sink, and acknowledge the data in the upstream buffer.

Inter-step buffers are critical components that must support durability, offsets, transactions for exactly-once forwarding, concurrent operations (reader groups), and backpressure (size limits).

Side Inputs allow user-defined functions (UDF, UDSink, Transformer, etc.) to access read-only, slow-updated data or configuration (e.g., from a database or file system) without reloading it for every message. This works for both batch and streaming jobs.

Key Characteristics:

• Data Format: Numaflow treats Side Input data as a []byte array. The pipeline developer is responsible for parsing these bytes into the desired format.
• Size Assumption: Side Input data is expected to be up to 1MB in size and updated at low frequency (e.g., minutes level).
• Mechanism: A Side Inputs Manager retrieves data on a schedule and saves it to a data store. A sidecar container in each vertex pod watches the data store and updates a shared volume, which the user-defined container then accesses via the Numaflow SDK.

Numaflow manages its lifecycle through three primary Custom Resource Definitions (CRDs), each managed by a dedicated controller. These controllers automate the provisioning of buffers, vertices, and pipeline infrastructure.

Managed CRDs:

• interstepbufferservices.numaflow.numaproj.io
• pipelines.numaflow.numaproj.io
• vertices.numaflow.numaproj.io