DOCSEARCHbyAlgolia
Submit Docs

nanoarrow

repository·main·Indexed 42 days ago

https://github.com/apache/arrow-nanoarrow

A lightweight C library and Python bindings for producing and consuming Arrow data. It supports Arrow C Data, C Stream, C Device, and serialized Arrow IPC formats, providing a small, highly compatible implementation of the Arrow columnar specification.

Tokens
26.1K
Snippets
108
Records
125
Agent score
52%

What's inside nanoarrow

  1. C API Reference Overview

    main

    The nanoarrow C API provides a lightweight interface for working with Apache Arrow data structures. The API is organized into several functional modules:

    • Schemas: Creating and reading schema definitions.
    • Arrays: Creating and reading Arrow arrays.
    • Buffers: Managing owning and growable buffers.
    • Bitmaps: Utilities for bitmap operations.
    • Metadata: Creating, reading, and modifying schema metadata.
    • Memory Management: Custom allocation and memory control.
    • Error Handling: Mechanisms for detecting and managing errors.
    • Arrow C Data Interface: Support for the standard Arrow C Data Interface for interoperability.

  2. Understand the Arrow C Data interface structures

    main

    nanoarrow is built upon the Arrow C Data interface. The two primary structures are:

    1. ArrowSchema: Represents the data type (the "expression" of the type). It includes fields like format, name, metadata, flags, and n_children for nested types.
    2. ArrowArray: Represents the actual data. It includes fields like length, null_count, offset, n_buffers, and n_children for nested columns.

    Both structures include a release callback for memory management and a private_data pointer for producer-specific information. You must know the ArrowSchema (the type) before accessing the contents of an ArrowArray.

    struct ArrowSchema {
  const char* format;
  const char* name;
  const char* metadata;
  int64_t flags;
  int64_t n_children;
  struct ArrowSchema** children;
  struct ArrowSchema* dictionary;
  void (*release)(struct ArrowSchema*);
  void* private_data;
};

struct ArrowArray {
  int64_t length;
  int64_t null_count;
  int64_t offset;
  int64_t n_buffers;
  int64_t n_children;
  const void** buffers;
  struct ArrowArray** children;
  struct ArrowArray* dictionary;
  void (*release)(struct ArrowArray*);
  void* private_data;
};

  3. How nanoarrow, Schemas, Arrays, and Array Streams work together

    main

    The nanoarrow R package provides bindings to the Arrow C Data and Arrow C Stream interfaces. It wraps three core structures into R objects:

    1. ArrowSchema: Represents the data type of an array.
    2. ArrowArray: Represents the actual values of an array.
    3. ArrowArrayStream: Represents zero or more ArrowArrays that share a common ArrowSchema.

    At the R level, nanoarrow attempts to attach a schema to ArrowArray objects whenever possible to simplify usage.

    # Example of the relationship
# 1. Create a schema
schema <- infer_nanoarrow_schema(1:5)

# 2. Create an array (which attaches a schema)
array <- as_nanoarrow_array(1:5)

# 3. Create a stream from arrays
stream <- basic_array_stream(list(array))

  4. Integrate nanoarrow with the arrow R package

    main

    The nanoarrow package is designed for seamless interoperability with the arrow R package.

    From nanoarrow to arrow: nanoarrow implements the following for most arrow package types:

    • as_nanoarrow_schema()
    • as_nanoarrow_array()
    • as_nanoarrow_array_stream()

    From arrow to nanoarrow: nanoarrow implements the following for nanoarrow objects, allowing them to be passed into many arrow functions:

    • arrow::as_arrow_array()
    • arrow::as_record_batch()
    • arrow::as_arrow_table()
    • arrow::as_record_batch_reader()
    • arrow::infer_type()
    • arrow::as_data_type()
    • arrow::as_schema()

  5. Understand nanoarrow core abstractions: Schema, Array, and ArrayStream

    main

    The nanoarrow package provides high-level Pythonic interfaces for the Arrow C Data and Arrow C Stream interfaces. The core abstractions are:

    • nanoarrow.Schema: Represents the data type of an array (maps to ArrowSchema).
    • nanoarrow.Array: Represents the values of an array (maps to ArrowArray). It can handle multiple chunks, making it compatible with pyarrow.ChunkedArray or polars.Series.
    • nanoarrow.ArrayStream: Represents a sequence of zero or more Array objects sharing a common Schema (maps to ArrowArrayStream). Unlike Array, chunks in a stream may not be fully resolved/materialized until iterated.

    These objects implement the Arrow PyCapsule interface, allowing them to be easily interchanged with pyarrow objects.

    import nanoarrow as na

# Create a schema
schema = na.int32()

# Create an array
array = na.Array([1, 2, 3], schema)

# Create an array stream from chunks
chunked = na.Array.from_chunks([[1, 2, 3], [4, 5, 6]], schema)
stream = na.ArrayStream(chunked)

for chunk in stream:
    print(chunk)

  6. Best practice: Vendoring nanoarrow in C++ libraries

    main

    When building a C++ library that uses nanoarrow, do not include nanoarrow.h in your public header file.

    Because nanoarrow is designed to be vendored (included directly in your source tree) rather than distributed as a system library, exposing it in your public API can cause conflicts if a user of your library also uses a different version of nanoarrow. Instead, keep nanoarrow usage internal to your implementation files and expose only the Arrow C Data interface types in your public headers.

  7. Generate bundled nanoarrow C library

    main

    You can generate bundled versions of the core library and its components (such as device, IPC, and testing support) using the provided Python bundling script. This is the method used to create the versions distributed with the R and Python bindings.

    python ci/scripts/bundle.py \
  --source-output-dir=dist \
  --include-output-dir=dist \
  --header-namespace= \
  --with-device \
  --with-ipc \
  --with-testing \
  --with-flatcc

  9. Read an ArrowArray using ArrowArrayView

    main

    To read data from an ArrowArray without taking ownership, use the ArrowArrayView*() family of functions. This provides a lightweight view into the array's data.

    Workflow:

    1. Create a nanoarrow::UniqueArrayView.
    2. Initialize the view with a type using ArrowArrayViewInitFromType.
    3. Set the view to point to the ArrowArray using ArrowArrayViewSetArray.
    4. Iterate through the array using input->length.
    5. Check for null values with ArrowArrayViewIsNull.
    6. Access data (e.g., strings) using ArrowArrayViewGetStringUnsafe to get an ArrowStringView.
    static int linesplitter_write_internal(ArrowArray* input, std::stringstream& out, ArrowError* error) {
  nanoarrow::UniqueArrayView input_view;
  ArrowArrayViewInitFromType(input_view.get(), NANOARROW_TYPE_STRING);
  NANOARROW_RETURN_NOT_OK(ArrowArrayViewSetArray(input_view.get(), input, error));

  ArrowStringView item;
  for (int64_t i = 0; i < input->length; i++) {
    if (ArrowArrayViewIsNull(input_view.get(), i)) {
      out << "\n";
    } else {
      item = ArrowArrayViewGetStringUnsafe(input_view.get(), i);
      out << std::string(item.data, item.size_bytes) << "\n";
    }
  }

  return NANOARROW_OK;
}

  11. Set up a C/C++ development environment for nanoarrow

    main

    To develop with nanoarrow in C/C++, you need a C/C++ compiler, CMake, and an IDE (like VSCode).

    Prerequisites:

    • Compiler:
      • macOS: xcode-select --install
      • Linux: gcc, g++, and make (e.g., apt-get install build-essential)
      • Windows: Visual Studio
    • CMake: Install via package manager (e.g., brew install cmake, apt-get install cmake, or dnf install cmake).
    • VSCode Extensions: Install CMake Tools and C/C++ extensions.

    Once set up, create a project folder and open it in VSCode to begin.

  12. Set up GoogleTest for nanoarrow projects

    main

    To test your nanoarrow-based C++ code, integrate googletest via CMake's FetchContent.

    1. Declare and make googletest available.
    2. Enable testing with enable_testing().
    3. Create an executable for your tests.
    4. Link against GTest::gtest_main.
    5. Use gtest_discover_tests to automatically find tests.

    To run tests from the terminal, use:

    cd build && ctest .
    FetchContent_Declare(
  googletest
  URL https://github.com/google/googletest/archive/refs/tags/v1.13.0.zip
)
FetchContent_MakeAvailable(googletest)

enable_testing()

add_executable(linesplitter_test linesplitter_test.cc)
target_link_libraries(linesplitter_test linesplitter GTest::gtest_main)

include(GoogleTest)
gtest_discover_tests(linesplitter_test)