DOCSEARCHbyAlgolia
Submit Docs

Apache Arrow ADBC

repository·main·Indexed 20 days ago

https://github.com/apache/arrow-adbc

Arrow Database Connectivity (ADBC) is an API standard for database drivers that leverages Apache Arrow for efficient, bulk columnar data transfer. It provides a unified interface for C, Go, and Java applications. The project includes a Driver Manager for driver isolation and simplified bindings, as well as specific driver implementations for PostgreSQL, SQLite, and Arrow Flight SQL. It also provides a .NET implementation targeting .NET Standard 2.0 and .NET 6 or later, featuring driver discovery via TOML manifests and connection profiles.

Tokens
102.3K
Snippets
274
Records
460
Agent score
67%

What's inside Apache Arrow ADBC

  1. Overview of Arrow Database Connectivity for Rust

    main

    The Rust implementation of ADBC provides several key components for database connectivity using Apache Arrow:

    • Abstract Rust API: A set of traits intended to be implemented by vendor-specific drivers.
    • Driver Manager: A component that implements the abstract API and dynamically loads drivers, forwarding calls via the C API.
    • Driver Exporter: A utility that transforms a Rust implementation of the abstract API into an object file that adheres to the C API.
    • Dummy Driver: A reference implementation used for testing and documentation.

  2. Overview of Thrift-based Apache connectors

    main

    The library provides ADBC drivers built on the Thrift protocol with integrated Arrow support for the following Apache platforms:

    • Hive: Serves as the base class for Spark and Impala implementations.
    • Impala: Currently under development with limited functionality.
    • Spark: Intended for use against native Spark environments.

    Note that implementation states vary across these drivers; some are under active development and may have limited functionality or produce errors.

  3. Overview of ADBC Driver Manager for Python

    main

    The adbc_driver_manager package provides Python bindings for the ADBC Driver Manager. It includes a DBAPI 2.0 (PEP 249) compatible interface, allowing you to load ADBC drivers at runtime.

    Key features:

    • DBAPI 2.0 Compatibility: Works with standard Python database interfaces.
    • Arrow Integration: Includes extensions (similar to turbodbc or duckdb) that allow retrieving Arrow Table objects directly, bypassing the performance limitations of row-oriented DBAPI interfaces.
    • Requirement: Requires PyArrow to use the DBAPI 2.0 interface.

    Note: If a backend-specific package exists (e.g., adbc_driver_postgresql), it is recommended to use that instead, as it provides a more convenient interface.

  5. What is Apache Arrow ADBC?

    main
    Apache Arrow ADBC (Arrow Database Connectivity) is a standardized set of APIs and libraries designed for fast, Arrow-native database access. It allows developers to execute SQL queries, fetch results, ingest bulk data, and explore database catalogs using the Apache Arrow columnar format. By using Arrow natively, ADBC eliminates the need for expensive data copies and conversions, making it highly efficient for analytics and AI applications.

  6. Overview of Apache Arrow ADBC for .NET

    main
    Apache Arrow ADBC (Arrow Database Connectivity) for .NET is an implementation designed to provide high-performance database connectivity using the Apache Arrow memory format. It targets .NET Standard 2.0 and .NET 6 or later, allowing it to be used in a wide range of .NET applications.

  7. Overview of ADBC JavaScript/TypeScript bindings

    main

    ADBC (Arrow Database Connectivity) provides experimental JavaScript and TypeScript bindings to the Rust driver manager. These bindings are built using NAPI-RS and are designed for high-performance database connectivity using Apache Arrow.

    Supported Runtimes:

    • Node.js: version 22+
    • Deno: version 2.0+
    • Bun: version 1.1+

  12. Understand the Apache Arrow ADBC Driver Manager for .NET

    main

    The Apache.Arrow.Adbc.DriverManager namespace provides a .NET implementation of the ADBC Driver Manager, following the C interface defined in adbc_driver_manager.h. It is designed for .NET Standard 2.0 and .NET 6 or later.

    Key capabilities include:

    • Driver discovery: Automatically searching for drivers in configurable directories (environment, user, or system levels).
    • TOML Manifests: Using .toml files to map driver names to platform-specific shared library paths.
    • Connection Profiles: Using .toml files to store reusable driver and connection option configurations.
    • Managed Driver Support: Loading .NET drivers directly using scheme-prefixed entrypoints (dotnet: or netfx:).
    • Extensibility: Implementing IConnectionProfileProvider for custom profile loading.