DOCSEARCHbyAlgolia
Submit Docs

Apache Arrow Rust Implementation

repository·main·Indexed 25 days ago

https://github.com/apache/arrow-rs

A native Rust implementation of Apache Arrow (in-memory columnar format) and Apache Parquet (columnar file format). The project includes crates for Avro (arrow-avro), CSV (arrow-csv), IPC (arrow-ipc), JSON (arrow-json), and Flight (arrow-flight), providing high-performance primitives for data processing and analytics.

Tokens
54.9K
Snippets
104
Records
569
Agent score
86%

What's inside arrow-rs

  1. Overview of parquet-geospatial

    main
    The parquet-geospatial crate provides the Rust implementation details for the Geometry and Geography Encoding specification used in Apache Parquet. It includes utilities for calculating geospatial column statistics and supporting tests for geospatial data encoding within the Apache Arrow ecosystem.

  2. Overview of Apache Arrow Rust Compute Kernels

    main
    The compute kernels module provides analytical functions designed to process Arrow columnar data. These kernels are intended for integration into query engines, data frame libraries, and other analytical tools. Kernels are categorized by their behavior and how they handle input/output arrays.

  3. Overview of Apache Arrow Rust Integration Testing binaries

    main

    The arrow-integration-testing crate provides several binaries used by the Archery tool to perform integration testing across different Arrow implementations. These binaries facilitate format conversion and Flight server/client communication.

    | Binary                         | Purpose                                   |
| ------------------------------ | ----------------------------------------- |
| arrow-file-to-stream           | Converts an Arrow file to an Arrow stream |
| arrow-stream-to-file           | Converts an Arrow stream to an Arrow file |
| arrow-json-integration-test    | Converts between Arrow and JSON formats   |
| flight-test-integration-server | Flight integration test: Server           |
| flight-test-integration-client | Flight integration test: Client           |

  4. Overview of Apache Arrow Rust implementation

    main
    The apache/arrow-rs repository provides a native Rust implementation of the Apache Arrow in-memory columnar format and the Apache Parquet columnar file format. It is designed for high-performance analytic computations in Rust.

  5. New features and capabilities in version 53.3.0

    main

    Version 53.3.0 introduced several new capabilities:

    • Casting Support: Added support for casting Utf8View to numeric types in can_cast_types and support for casting StringViewArray to DecimalArray.
    • JSON Support: Added support for Utf8View in JSON serialization and support for Duration in the JSON Reader.
    • Flight API: Exposed known_schema from FlightDataEncoder.
    • Builder Enhancements: Added write_bytes for GenericBinaryBuilder and implemented append_n for BooleanBuilder.
    • Parquet API: Added ParquetObjectReader::with_runtime.
    • Kernel Support: Added support for Utf8View in the bit_length kernel.

  6. Understand Parquet crate versioning and compatibility

    main

    The parquet crate follows SemVer. Note that unlike many Rust crates, this crate uses major versions greater than 0.x (e.g., 19.0.0).

    • Minor releases: Contain only compatible changes.
    • Major releases: May contain breaking API changes.

    Rust Version Compatibility: The crate is tested against the latest stable version of Rust. It is not currently tested against older versions of the Rust compiler.

  7. Understand Arrow Protocol Files

    main
    The format/ directory contains the binary protocol definitions for the Apache Arrow columnar format and other components of the project, such as the Flight RPC framework. These files define the low-level structure of the data formats used across the ecosystem.

  8. Use arrow-ipc for Arrow IPC Format support

    main
    The arrow-ipc crate provides support for reading and writing files and streams using the Arrow IPC (Inter-Process Communication) format. This allows for efficient data exchange between different systems and processes using the Apache Arrow columnar memory layout.