Apache Flink

Apache Flink provides several primary APIs and interfaces for stream and batch processing. You can explore the following core surfaces:

• DataStream API: The core API for processing unbounded streams of data.
• DataStream API (V2): The next generation of the DataStream API.
• Table API & SQL: A relational API for processing structured data.
• Stateful Functions: A framework for building stateful, event-driven applications.

For operational details, refer to the Configuration Parameters, REST API, and Command Line Interface (CLI) documentation.

Apache Flink provides several high-level APIs for processing data. The reference documentation covers:

• Flink SQL: For SQL-based stream and batch processing.
• Table API: A declarative relational API for stream and batch processing.
• DataStream API: For low-level, imperative stream processing.
• DataStream API (V2): The next generation of the DataStream API.

For operational and system management, refer to the Configuration guides, the REST API documentation, and the CLI (Command Line Interface) reference.

The OpenAI Model Function enables Flink SQL to perform inference tasks by calling the OpenAI API. It currently supports two primary tasks:

1. Chat Completions: Generates model responses from a conversation of messages.
2. Embeddings: Generates vector representations of input text.

To use these capabilities, you must first define a model using CREATE MODEL and then apply it to your data using the ML_PREDICT function.

The Flink Command-Line Interface (CLI), located at bin/flink, is used to run programs packaged as JAR files and to control their execution. The CLI is available in both local single-node and distributed Flink setups. It communicates with the running JobManager as defined in your Flink configuration files.

# The entrypoint for all Flink CLI operations
bin/flink

Apache Flink provides two relational APIs for unified stream and batch processing: the Table API and SQL.

• Table API: A language-integrated query API for Java, Scala, and Python. It allows you to compose queries using relational operators like selection, filter, and join.
• SQL: A standard relational language for querying data.

Note: This specific README is intended for module contributors. For end-user instructions on how to write and run queries, please refer to the official Flink Table API & SQL documentation.

The flink-architecture-tests module uses ArchUnit to enforce architectural rules. It is divided into three submodules to handle different testing scopes and shared extensions:

• flink-architecture-tests-base: Contains common ArchUnit extensions used by both production and test code tests.
• flink-architecture-tests-production: Contains centralized architectural rules and tests specifically for production code.
• flink-architecture-tests: Contains centralized architectural rules for test code. Note that architectural tests in this category are built individually within each submodule where the test code resides.

PyFlink provides two distinct APIs for building scalable batch and streaming workloads in Python, depending on the required level of abstraction:

1. PyFlink Table API: A high-level relational API similar to SQL or working with tabular data in Python (e.g., Pandas). It is ideal for relational queries and ETL processes.
2. PyFlink DataStream API: A lower-level API that provides direct control over Flink's core building blocks, such as state and time. Use this for complex stream processing use cases that require fine-grained control.

Reactive Mode is a specialized mode for the Adaptive Scheduler designed for single-job clusters (enforced via Application Mode). It automatically scales a job to use all available resources in the cluster: adding a TaskManager scales the job up, and removing one scales it down.

Key Behaviors:

• Automatic Parallelism: Flink manages job parallelism, always setting it to the highest possible value based on available resources.
• Rescaling via Checkpoints: When resources change, Flink restarts the job and restores it from the latest completed checkpoint. This avoids the overhead of manual savepoints.
• Autoscaling Integration: It enables external autoscaling (e.g., Kubernetes replica changes or AWS Auto Scaling Groups) to drive Flink's parallelism by simply adjusting the number of TaskManagers.

The Flink SQL Gateway is a service that enables concurrent execution of SQL statements submitted from multiple clients. It provides a simplified way to submit Flink jobs, query metadata, and perform online data analysis.

Architecture

The SQL Gateway consists of a SqlGatewayService and pluggable endpoints.

• Endpoints: Act as the entry point for users to connect to the service.
• SqlGatewayService: Handles the actual request processing, and multiple endpoints can reuse this service.

Commonly supported endpoints include the REST Endpoint (default) and the HiveServer2 Endpoint.

A DataStream is an immutable collection of data in Flink that can be either finite or unbounded. Because they are immutable, you cannot add or remove elements directly; instead, you derive new streams by applying transformations (API operations) like map or filter.

Key characteristics:

• Immutability: Once created, the stream cannot be modified.
• Transformation-based: You interact with data only through API operations.
• Lazy Evaluation: Flink programs are constructed as a dataflow graph. Operations are not executed immediately when called; they are only triggered when execute() is called on the StreamExecutionEnvironment.