Apache Arrow

The Arrow C Device data interface is an experimental extension of the existing C Data Interface. While the standard C Data Interface assumes all data buffers reside in CPU memory, the Device interface allows for the exchange of Arrow-formatted data residing on non-CPU hardware, such as GPUs (e.g., via NVIDIA CUDA) or FPGAs.

By adding a small, stable set of C definitions to the existing interface, it enables zero-copy sharing of device memory between independent runtimes and components within the same process. This avoids expensive host-to-device and device-to-host memory copies when passing data between different libraries or runtimes.

The MATLAB interface to Apache Arrow is an experimental tool designed to bridge MATLAB with Apache Arrow C++ libraries. It enables high-performance data exchange between MATLAB and Arrow-compatible systems.

Core Capabilities:

1. Type Conversion: Converts between a subset of Arrow Array types and MATLAB array types.
2. Tabular Data: Converts between MATLAB tables and arrow.tabular.RecordBatch objects.
3. Schema Management: Allows creating Arrow Fields, Schemas, and Types.
4. File I/O: Supports reading and writing Feather V1 files.

Warning: This interface is under active development and is considered experimental.

Acero is a C++ streaming query engine within the Apache Arrow ecosystem. It is designed to handle complex computations where successive direct invocation of compute functions is inefficient in terms of memory or computation time. Acero allows you to formulate and execute computations by consuming streams of input data, applying transformations, and producing a stream of output data. This makes it suitable for processing arbitrarily large datasets that do not fit in memory.

Warning: Acero is currently experimental, and a stable API is not yet guaranteed.

PyArrow provides the Python API for Apache Arrow, a universal columnar format and multi-language toolbox designed for fast data interchange and in-memory analytics.

Key features include:

• First-class integration: Seamlessly works with NumPy, pandas, and built-in Python objects.
• C++ Core: Built upon the high-performance C++ implementation of Apache Arrow.
• Extensive Ecosystem: Supports reading/writing various formats (Parquet, ORC, CSV, Feather, JSON) and provides capabilities for distributed computing via Flight and dataset management.

Gandiva is a runtime expression compiler that uses LLVM to generate efficient native code for computing on Apache Arrow record batches. It is specifically designed to optimize computations by leveraging the Arrow memory model (separating values from validity bitmaps) and modern hardware features like SIMD instructions.

Key Capabilities:

• Projections: Transforming or selecting specific columns/values.
• Filters: Evaluating boolean expressions to select specific rows.

Note: Gandiva is limited to projections and filters. For other types of data transformations, use the standard Arrow Compute Functions instead.

The Arrow columnar format is a language-agnostic in-memory data structure specification designed for high-performance analytics. It includes metadata serialization (using Google's Flatbuffers) and a protocol for serialization and generic data transport.

Key performance characteristics include:

• Data adjacency: Optimized for sequential access (scans).
• O(1) Random Access: Constant-time access to elements (except for run-end-encoded-layout which is O(log n)).
• SIMD/Vectorization Friendly: Designed to leverage modern CPU instructions.
• Zero-copy: Data is relocatable without "pointer swizzling," allowing for true zero-copy access in shared memory.

Note that the format prioritizes analytical performance and data locality over mutation speed; mutation operations are comparatively more expensive.

The MATLAB Interface for Apache Arrow is designed to enable high-performance columnar analytics within MATLAB by providing access to Arrow memory. The interface aims to support three primary use cases:

1. Memory Management: Creating, accessing, and deleting Arrow memory directly from MATLAB code.
2. Serialization/Deserialization: Reading from and writing to file formats such as Parquet, Feather, JSON, and CSV using Arrow memory.
3. Cross-Language Data Exchange: Moving in-memory tabular data (represented as MATLAB tables) to other languages like Python, R, and Rust with minimal or zero-copy overhead.

Arrow Flight is a high-performance RPC framework designed for efficient data transfer using the Arrow format.

Note: Flight is currently considered unstable. APIs are subject to change, though drastic changes are not expected.