Apache Flink Stateful Functions Documentation

Stateful Functions provides SDKs for several programming languages. You can find the reference documentation for each below:

• Python API
• JavaScript API
• Java API
• Golang API

The Smoke E2E testing framework is composed of the following modules:

• statefun-smoke-e2e-common: Contains testing utilities, including the SmokeRunner class which organizes the runtime architecture.
• statefun-smoke-e2e-driver: The core logic of the Smoke E2E driver. It is built into a self-contained JAR that runs directly within the Flink StateFun cluster.
• statefun-smoke-e2e-multilang-base: Provides a generic pom.xml with necessary dependencies and the driver JAR required to run Smoke E2E.
• statefun-smoke-e2e-multilang-harness: Provides the MultiLangSmokeHarnessTest which allows running tests as a JUnit process against a local HTTP endpoint (port 8000).

In the Java SDK, FunctionType acts as a logical pointer composed of a namespace and a name. You bind a FunctionType to an implementing class during registration. Once defined, you can use this type to address and message specific function instances via the Context.

1. Define the type:

public static final FunctionType HELLO_TYPE = new FunctionType("apache/flink", "hello");

2. Send a message using the type:

context.send(Identifiers.HELLO_TYPE, "user1", new MyUserMessage());

package org.apache.flink.statefun.docs;

import org.apache.flink.statefun.sdk.FunctionType;

/** A function type that will be bound to {@link FnHelloWorld}. */
public class Identifiers {
    public static final FunctionType HELLO_TYPE = new FunctionType("apache/flink", "hello");
}

Embedded modules are discovered using Java's Service Provider Interface (SPI). To make your module available to the runtime, you must create a service provider configuration file in your JAR.

1. Create a file named org.apache.flink.statefun.sdk.spi.StatefulFunctionModule.
2. Place it in the META-INF/services/ directory of your JAR.
3. The file must contain the fully qualified name of your implementation class.

org.apache.flink.statefun.docs.EmbeddedModule

A Stateful Functions application is composed of four primary primitives:

• Stateful Functions: Small pieces of logic invoked via messages. Each function exists as a virtual instance of a function type. Instances are uniquely addressed by their type and a unique string ID. State is private to the instance and can be accessed via local variables.
  • Mental Model: Think of a function type as a KeyedProcessFunction and the ID as the key.
• Ingresses: The entry points for events into the application (e.g., message queues, logs, or HTTP servers).
• Routers: Attached to ingresses to determine which specific function instance should handle an incoming event.
• Egresses: Standardized ways to send events out of the application. These are optional; functions can also sink events or call other applications directly.

When defining a Kinesis ingress, you can specify where the consumer starts reading from the stream using the startupPosition property:

• latest (default): Starts consuming from the head of the stream shards.
• earliest: Starts consuming from the earliest position possible.
• date: Starts from offsets with an ingestion time greater than or equal to a specified date (format: YYYY-MM-DD HH:mm:ss.SSS Z).

# Latest (default)
startupPosition:
  type: latest

# Earliest
startupPosition:
  type: earliest

# Date
startupPosition:
  type: date
  date: 2020-02-01 04:15:00.00 Z

Stateful Functions provides exactly-once guarantees for both state and messaging by leveraging Apache Flink's snapshotting mechanism.

• State Management: Functions use persisted states, which are locally embedded. During computation, you work with this state via local variables.
• Fault Tolerance: In the event of a failure, the system rolls back the entire 'state of the world'—including both persisted states and messages—to a previous consistent snapshot. This simulates a failure-free execution without requiring an external database.

The urlPathTemplate allows you to use template parameters that are filled dynamically based on the function's type. A common parameter is {function.name}.

For example, if a message is sent to the type com.example/greeter, the runtime will resolve the template https://bar.foo.com/{function.name} to https://bar.foo.com/greeter. This is useful for routing requests through load balancers or service gateways to different physical systems (e.g., Kubernetes, AWS Lambda, or physical servers).

spec:
  functions: com.example/*
  urlPathTemplate: https://bar.foo.com/{function.name}

In Stateful Functions, functions are allocated logically rather than physically. This means a logical function instance does not consume CPU, memory, or threads when it is not actively being invoked.

Because they are virtual, there is no theoretical upper limit on the number of instances you can create. Users are encouraged to model applications with high granularity (many small, specific functions) rather than designing around resource constraints. An instance with no state and no active execution occupies zero computational resources, only consuming storage space for its persisted state.

In Stateful Functions, every function instance is uniquely identified by an Address. An Address consists of two parts:

1. TypeName: Similar to a class in object-oriented programming, this declares the type of function being referenced.
2. Identifier: A primary key that scopes the function call to a specific instance of that TypeName.

All operations performed by a function—including reading from and writing to state—are strictly scoped to the current Address. This ensures that different instances of the same function type do not interfere with each other's data.