Apache Beam

The Twister2 Runner executes Apache Beam pipelines on a Twister2 cluster. It is designed for large-scale batch jobs that require high performance, particularly in High-Performance Computing (HPC) environments.

Key features include:

• Batch pipeline support.
• Support for proprietary interconnects like Infiniband.
• Distributed massively parallel processing using Bulk Synchronous Parallel (BSP) style execution.
• Native support for Beam side-inputs.

Warning: The Twister2 runner is deprecated and support is planned to be removed in Beam 3.0.

Apache Beam I/O connectors are specialized transforms used to read from and write to external data storage systems. They allow Beam pipelines to interact with popular data storage types in a unified and distributed manner, leveraging native optimized connectivity.

Connectors are categorized by how they are accessed:

• Standard Connectors: Native to the SDK being used.
• X-language Connectors: Available via the Apache Beam multi-language pipelines framework (allowing an SDK to use connectors written in a different language).
• Managed API Connectors: Accessible through simplified Managed I/O APIs.

Apache Beam is an open-source, unified model for defining both batch and streaming data-parallel processing pipelines. It allows you to write a single program that can process finite datasets (batch) or infinite datasets (streaming) using the same classes and transforms.

Once a pipeline is defined using a Beam SDK, it is executed by a Runner on a distributed processing back-end (such as Apache Flink, Apache Spark, or Google Cloud Dataflow). Beam is ideal for embarrassingly parallel tasks, ETL (Extract, Transform, and Load) processes, and data integration.

The Apache Beam Completion Plugin is an IntelliJ IDEA plugin designed to provide Java SDK Beam Transform completions. It assists developers by improving code intelligence specifically for Beam transforms within the IntelliJ IDE.

Note: This project is part of the Google Summer of Code program and is not officially supported by Apache Beam.

Apache Beam provides a variety of built-in I/O connectors to read from (Sources) and write to (Sinks) different data formats and storage systems. These connectors support multiple SDKs including Java, Python, Go, and TypeScript, and can operate in both Batch and Streaming modes.

Key connector categories include:

• File-based: FileIO, TextIO, AvroIO, ParquetIO, XmlIO, TFRecordIO, ThriftIO.
• Cloud Storage: GcsFileSystem (Google Cloud Storage), S3FileSystem (Amazon S3), HadoopFileSystem (HDFS).
• Messaging/Streaming: KafkaIO, KinesisIO, AmqpIO.
• Specialized: TikaIO (for content extraction), In-memory (for Go SDK).

Euphoria is a high-level Java 8 DSL (Domain Specific Language) designed for the Beam Java SDK. It provides an abstraction layer for Beam transformations to make pipelines easier to read and write. Euphoria is designed to be used alongside existing Beam pipelines, as they are convertible back and forth.

Note: The integration of the Euphoria API into the main Beam project is currently in progress (tracked via BEAM-3900).

The Hazelcast Jet Runner allows you to execute Apache Beam pipelines using Hazelcast Jet. It is designed for large-scale continuous jobs, supporting both batch (bounded) and streaming (unbounded) datasets with high throughput, low latency, and natural back-pressure.

Note: The Jet Runner is currently in an EXPERIMENTAL state. It does not currently support Jet's full fault tolerance (if a job fails, it must be restarted) and may not match the raw performance of a native Jet job because Beam pipeline optimization is not yet fully implemented.