arrow2

Apache Arrow is a language-independent columnar memory format designed for efficient analytic operations on modern hardware (CPUs/GPUs). It provides a standardized specification for how data types (integers, floats, strings, lists, etc.) are stored in memory.

The Arrow2 crate is a Rust implementation of these specifications. It provides the structs and implementations necessary to create Arrow arrays that follow the Apache Arrow standard, enabling high-performance data processing in Rust.

Key Benefits of the Arrow Format:

• Fast in-memory data access: By using a columnar representation instead of a row-based one, similar data types are stored contiguously in memory. This improves cache locality and enables faster columnar querying and SIMD optimizations.
• Zero-copy data sharing: Because the memory format is standardized, different processes (even those written in different languages like Python/Pandas or Scala/Spark) can share the same in-memory data without the overhead of serialization, copying, or conversion.
• Efficient Inter-process Communication (IPC): Data can be sent across networks or between processes as structured packets (Chunks/RecordBatches) that are ready to use immediately upon receipt without decoding.

Arrow2 is a Rust implementation of the Apache Arrow format designed to be 'transmute-free'. It is a feature-complete implementation (second only to the C++ reference implementation) with the following key capabilities:

Data Interoperability

• C Data Interface: Supports reading and writing all Arrow types at zero-copy.
• C Stream Interface: Supports reading and writing all Arrow types.
• Rust Interop: Full interoperability with Rust's Vec.
• MutableArray API: Allows working with bitmaps and arrays in-place.
• Timestamps: Full support for timestamps with timezones, including timezone-aware arithmetic.

Supported Formats (Read/Write)

• CSV
• Apache Arrow IPC (all types)
• Apache Arrow Flight (all types)
• Apache Parquet (except deep nested types)
• Apache Avro (all types)
• NJSON
• ODBC (some types)

Compute Operations

Includes an extensive suite of operations such as:

• Aggregations and arithmetics
• Cast, comparison, and boolean logic (including Kleene logic)
• Sort and merge-sort
• Filter and take
• Hash operations
• If-then-else and nullif
• Temporal operations (day, month, week day, hour, etc.)
• Window functions

arrow2 is a library for efficient in-memory data operations using the Arrow in-memory format. It is designed as a bottom-up rewrite of the official arrow crate, focusing on soundness and type safety.

Key capabilities include:

• Array Operations: Creating and manipulating various array types.
• Computation: High-performance arithmetic and compute operations on arrays.
• Schema Management: Defining data structures using Schema and Field.
• I/O Support: Reading and writing data in formats like Parquet, CSV, JSON, and Arrow IPC.
• Chunking: Managing data in Chunk structures.

use std::sync::Arc;

use arrow2::array::*
use arrow2::datatypes::{Field, DataType, Schema};
use arrow2::compute::arithmetics;
use arrow2::error::Result;
use arrow2::io::parquet::write::*;
use arrow2::chunk::Chunk;

fn main() -> Result<()> {
    // declare arrays
    let a = Int32Array::from(&[Some(1), None, Some(3)]);
    let b = Int32Array::from(&[Some(2), None, Some(6)]);

    // compute
    let c = arithmetics::basic::mul_scalar(&a, &2);
    assert_eq!(c, b);

    // declare a schema with fields
    let schema = Schema::from(vec![
        Field::new("c1", DataType::Int32, true),
        Field::new("c2", DataType::Int32, true),
    ]);

    // declare chunk
    let chunk = Chunk::new(vec![a.arced(), b.arced()]);

    // write to parquet
    let options = WriteOptions {
        write_statistics: true,
        compression: CompressionOptions::Snappy,
        version: Version::V1,
        data_pagesize_limit: None,
    };

    let row_groups = RowGroupIterator::try_new(
        vec![Ok(chunk)].into_iter(),
        &schema,
        options,
        vec![vec![Encoding::Plain], vec![Encoding::Plain]],
    )?;

    let mut file = vec![];
    let mut writer = FileWriter::try_new(file, schema, options)?;

    for group in row_groups {
        writer.write(group?)?;
    }
    let _ = writer.end(None)?;
    Ok(())
}

Arrow2 is a Rust library designed for interoperability with the Arrow format, optimized for CPU and memory-intensive analytics. It supports heterogeneous data structures, null values, and IPC/FFI interfaces across different languages. The library is organized into five primary API categories:

1. Low-level API: For efficient operations on contiguous memory regions.
2. High-level API: For operating directly with Arrow arrays.
3. Metadata API: For declaring and managing logical types and metadata.
4. Compute API: Provides operators to perform computations over arrays.
5. IO API: Provides interfaces for reading from and writing to various formats, including:
   • Arrow: IPC files, IPC streams, and memory-mapped files.
   • CSV: Reading and writing CSV files.
   • Parquet: Reading and writing Parquet files.
   • JSON/NDJSON: Reading and writing JSON and NDJSON.
   • Avro: Reading and writing Avro files.
   • ODBC: Reading and writing via ODBC.

The io module in arrow2 follows a specific architectural pattern to ensure usability and performance. When working with or extending IO functionality, keep these design principles in mind:

1. Dependency Management

• Feature Gating: Any directory depending on external dependencies must be feature-gated using a prefix io_ (e.g., io_csv).
• API Re-exports: To ensure a seamless developer experience, modules must re-export the APIs of external dependencies they use. This allows users to use the crate without adding extra dependencies to their own Cargo.toml.
  • Example: If a module provides write(writer: &mut csv::Writer<W>, ...), it must include pub use csv::Writer;.

2. Module Structure

• Each format directory (e.g., csv, json) should contain two sub-directories: read and write.
• The base module for a format should re-export these sub-modules: pub use read; and pub use write;.

3. Separation of Concerns

• Data vs. Metadata: Reading data should be separated from reading metadata. Schema inference or schema reading should be distinct functions. Functions that read actual data should consume a schema that has been pre-read.
• IO vs. CPU Bounds: To allow consumers to optimize for performance, implementations must separate IO-bound operations from CPU-bound operations:
  1. IO-bound: Functions that consume a Read implementor and output a "raw" struct (e.g., compressed or serialized data).
  2. CPU-bound: Functions that consume a "raw" struct and convert it into Arrow format.

These functions must be offered as independent public APIs so consumers can decide how to balance IO and CPU workloads.