DOCSEARCHbyAlgolia
Submit Docs

Irontools

website·Indexed 32 days ago

https://irontools.dev/docs

A suite of Apache Flink extensions designed to improve pipeline efficiency. The suite includes Iron Vector, a native columnar vectorized accelerator for Apache Flink SQL and Table API; Iron Functions, which enables WebAssembly-based function execution; and Iron Serde, providing high-performance Kafka serialization and deserialization libraries.

Tokens
7K
Snippets
31
Records
57
Agent score
58%

What's inside Irontools

  1. Overview of Irontools for Apache Flink

    Irontools is a suite of extensions designed to improve the efficiency, ergonomics, and accessibility of Apache Flink pipelines. It provides tools to optimize data serialization and enable the use of non-JVM languages for stream processing logic.

  2. Overview of Iron Vector for Apache Flink

    Iron Vector is a native, columnar, vectorized, high-performance accelerator designed for Apache Flink SQL and Table API pipelines. It aims to increase compute efficiency (up to 2x) by leveraging the "RAD stack" (Rust, Arrow, and DataFusion) to provide columnar execution without requiring code changes to existing Flink pipelines. It integrates at the Task orchestration level, replacing certain StreamTask subtopologies while continuing to use Flink's Netty-based network exchange and maintaining support for watermarks and metrics.

  3. Overview of Iron Functions for Apache Flink

    Iron Functions is an extension for Apache Flink that enables WebAssembly-based function execution. It allows developers to author User-Defined Functions (UDFs) in languages not natively supported by Flink, specifically TypeScript, Rust, and Go. It leverages the Extism ecosystem to provide a secure, high-performance, and sandboxed environment for executing custom logic within both the DataStream API and Table/SQL API.

  4. Overview of Iron Functions for Apache Flink

    Iron Functions enables WebAssembly (Wasm) based function execution in Apache Flink, allowing developers to write custom functions in TypeScript, Rust, and Go. It uses the Extism ecosystem to provide a secure, sandboxed environment.

    Key Characteristics:

    • Security: Sandboxed execution prevents untrusted code from accessing the host system.
    • Performance: Fast execution via runtime compilation.
    • Integration: Works with both DataStream and Table/SQL APIs.
    • Limitations: Wasm functions cannot read local files, do not support multithreading, and cannot make arbitrary network calls (though a provided HTTP client is available in each language PDK).

  5. Overview of Irontools for Apache Flink

    Irontools is a suite of extensions for Apache Flink designed to improve pipeline efficiency and enable new use cases through high-performance accelerators, multi-language function support, and optimized serialization.

  6. Overview of Iron Serde for Apache Flink

    Iron Serde provides high-performance Kafka serialization and deserialization libraries for Apache Flink. They are designed as drop-in replacements for default Flink libraries and require no additional configuration. The library optimizes performance by generating specialized, compact classes for each observed schema (from a Schema Registry or class inference), which minimizes branching and allows for efficient JIT compilation. Currently, the library focuses primarily on deserialization.

  7. Overview of Iron Vector for Apache Flink

    Iron Vector is a native, columnar, vectorized accelerator for Apache Flink SQL and Table API. Built using Rust, Apache Arrow, and Apache DataFusion, it improves throughput and reduces infrastructure costs by leveraging SIMD operations and cache-efficient processing patterns. It is designed as a drop-in replacement for Flink's default execution engine, requiring no code changes to existing applications.

  8. Implement multi-language UDFs using Iron Functions

    Iron Functions allows developers to author transformation logic or User-Defined Functions (UDFs) in languages such as TypeScript, Rust, and Go. These functions are compiled to WebAssembly (Wasm) to run securely and efficiently within a Flink pipeline. Iron Functions is compatible with both the DataStream and Table/SQL APIs.

  9. Accelerate Flink SQL and Table API with Iron Vector

    Iron Vector is a native, columnar, vectorized high-performance accelerator for Apache Flink SQL and the Table API. It is implemented using Rust, Apache Arrow, and DataFusion to provide optimized execution.

  10. Accelerate Apache Flink SQL and Table API with Iron Vector

    Iron Vector is a native, columnar, and vectorized high-performance accelerator for Apache Flink SQL and Table API. Built using Rust, Apache Arrow, and DataFusion, it serves as a drop-in replacement that requires zero code changes to achieve higher throughput or reduced infrastructure costs.

  11. Understand Iron Vector native format support and data flow

    Iron Vector optimizes Flink pipelines by deserializing input data (e.g., Avro records from Kafka) directly into Apache Arrow batches, bypassing intermediate representations like RowData or GenericRecord. This switch to native format happens automatically when a supported format is detected, though it can be disabled. For unsupported formats, Iron Vector falls back to RowData conversion before converting to Arrow batches, which incurs a performance penalty. For sinks, Iron Vector currently utilizes Flink's ColumnarRowData as a row view into backing columnar vectors.

  12. Execute multi-language code in Flink using Iron Functions

    Iron Functions provides operators and tools to execute code written in languages such as TypeScript, Rust, and Go within Apache Flink by leveraging WebAssembly (WASM). It supports both the DataStream API and Table/SQL API and can be packaged as a standalone User Defined Function (UDF) JAR for fast and secure execution.