DOCSEARCHbyAlgolia
Submit Docs

Apache Arrow for Go

repository·main·Indexed 18 days ago

https://github.com/apache/arrow-go

A Go implementation of the Apache Arrow columnar memory format for efficient in-memory analytic operations and zero-copy data streaming. Includes support for Arrow Flight clients, reference counting for memory management, and the parquet_reader CLI tool for exporting Parquet files to TEXT, JSON, or CSV.

Tokens
13.8K
Snippets
49
Records
62
Agent score
62%

What's inside apache-arrow-go

  1. How reference counting works in Apache Arrow for Go

    main

    The library uses reference counting to track memory buffer usage, enabling efficient resource accounting and memory pooling. Objects expose two methods:

    • Retain(): Increases the reference count by 1.
    • Release(): Decreases the reference count by 1.

    When the reference count reaches zero, the associated object is freed. Both methods are safe to call from multiple goroutines.

    When to call Retain and Release

    ScenarioActionReason
    Taking ownershipCall RetainIf you receive an object and need to access it outside the scope of the current function call.
    Creating/ReceivingCall ReleaseYou own any object created via New... or Copy... functions, or objects received over a channel.
    Sending over channelsCall RetainYou must call Retain before sending an object over a channel because the receiver is assumed to take ownership and will eventually call Release.

  2. Use FlightSQL drivers with database/sql

    main

    Go FlightSQL drivers are located in the apache/arrow-adbc repository. To use the standard Go database/sql interface with FlightSQL, import the driver with a blank identifier and provide a Data Source Name (DSN) in the format k=v;k2=v2.

    Example DSN format: uri=grpc://localhost:12345;username=mickeymouse;password=p@55w0RD

    import (
    "database/sql"
    _ "github.com/apache/arrow-adbc/go/adbc/sqldriver/flightsql"
)

func main() {
    dsn := "uri=grpc://localhost:12345;username=mickeymouse;password=p@55w0RD"
    db, err := sql.Open("flightsql", dsn)
    ...
}

  3. Generate .pb.go files from proto definitions

    main

    To generate the Go protocol buffer files (.pb.go) from the message definitions in the arrow/util/messages directory, use the protoc compiler with the Go plugin. This must be run from the go/arrow/util/ directory.

    cd go/arrow/util/
protoc -I ./ --go_out=./messages ./messages/types.proto

  4. Use the FlightSQL driver with database/sql

    main

    The FlightSQL driver implements the database/sql/driver interface. It is registered under the name flightsql. You can use it by calling sql.Open("flightsql", dsn) where dsn is a Data Source Name string.

    import (
    "database/sql"
    _ "github.com/apache/arrow-go/v18/arrow/flight/flightsql"
)

// Open the connection to a FlightSQL backend
db, err := sql.Open("flightsql", "flightsql://localhost:12345?timeout=5s")
if err != nil {
    panic(err)
}
defer db.Close()

// Execute queries using standard database/sql methods
rows, err := db.Query("SELECT * FROM mytable")
if err != nil {
    panic(err)
}
// ...

  6. Install the FlightSQL driver

    main

    To use the FlightSQL driver for Go's database/sql package, ensure you have Go 1.17+ installed and run the following command to add the dependency to your project:

    go get -u github.com/apache/arrow-go/v18/arrow/flight/flightsql

  7. Use the parquet_reader CLI tool

    main

    parquet_reader is a command-line utility designed to read Parquet files and export selected columns into TEXT, JSON, or CSV formats. It supports metadata inspection, column filtering, and output redirection to files.

    # Example: Convert a Parquet file to CSV without printing metadata
./parquet_reader --no-metadata --csv v0.7.1.parquet

  9. Generate Go assembly for arm64 (NEON)

    main

    Generating assembly for arm64 requires manual intervention because c2goasm and asm2plan9s do not fully support arm64.

    1. Generate raw assembly

    Use the Makefile to generate the raw assembly sources using NEON flags:

    make _lib/bit_packing_neon.s
make _lib/unpack_bool_neon.s

    2. Pre-processing for c2goasm

    Before running c2goasm, you must modify the assembly to handle differences in syntax and word sizes:

    • Comments: Convert // (arm64) to # (x86-64/c2goasm expectation) for constants.
    • Word Sizes: Convert .word to .long (since word is 32-bit in arm64 but 16-bit in x86-64) and .xword to .quad.
    • Instructions: MOVQ instructions will be converted to MOVD by the tool.

    3. Post-processing the output

    After running c2goasm, you must perform several manual or automated fixes:

    • Instruction Conversion: Many ARM instructions must be converted to the Go assembly WORD $0x######## format. If an instruction is unrecognized, you can find its byte sequence using objdump -S on a compiled object file.
    • Branching: Convert branching instructions like b.le LBB0_10 to BLE LBB0_10. Convert b instructions to JMP calls.
    • Constants/Labels: Replace adrp/str pairs used for constants with macros and VMOVD or VMOVQ instructions.
    • Return Values: Ensure functions end with a move to the function parameter offset, e.g., MOVD R0, num+32(FP).

    4. Automated Cleanup

    You can use the provided script.sed to automate several of these steps (like converting branches and adrp/ldr pairs):

    sed -f _lib/script.sed -i bit_packing_neon_arm64.s
    make _lib/bit_packing_neon.s
make _lib/unpack_bool_neon.s
sed -f _lib/script.sed -i bit_packing_neon_arm64.s

  10. Configure architecture optimizations via build tags and environment variables

    main

    Apache Arrow for Go uses c2goasm to generate PLAN9 assembly from C/C++ code for high performance. You can control these optimizations in two ways:

    1. Build-time: Use the noasm build tag to compile without assembly optimizations (using pure Go).
    2. Runtime: Use the INTEL_DISABLE_EXT environment variable to dynamically enable or disable specific architecture optimizations (like AVX2 or SSE4).
    # Disable ALL architecture optimizations (use pure Go)
$ INTEL_DISABLE_EXT=ALL go test ./...

# Disable only AVX2 optimizations
$ INTEL_DISABLE_EXT=AVX2 go test ./...

# Enable optimizations (setting to NONE enables AVX2 and SSE4)
$ INTEL_DISABLE_EXT=NONE go test ./...

  11. How FlightSQL and FlightRPC work together

    main

    The flightsql package provides a high-level Server interface that defines SQL-specific operations (like GetFlightInfoStatement, DoGetStatement, BeginTransaction, etc.).

    Because standard Apache Arrow FlightRPC uses generic FlightDescriptor commands, the flightsql.NewFlightServer function acts as a router. It wraps your Server implementation and translates incoming FlightRPC GetFlightInfo and PollFlightInfo calls into the appropriate typed Server methods by unmarshaling the underlying Protobuf commands.

  12. Implement custom Flight client middleware

    main

    You can extend the client's behavior by implementing the CustomClientMiddleware interface and wrapping it with CreateClientMiddleware. This allows you to intercept both Unary and Streaming RPC calls.

    To gain access to specific lifecycle events, implement one or more of these optional interfaces:

    • CustomClientMiddleware: Implement StartCall(ctx context.Context) context.Context to modify the context before a call begins.
    • ClientPostCallMiddleware: Implement CallCompleted(ctx context.Context, err error) to execute logic after a call finishes (success or error).
    • ClientHeadersMiddleware: Implement HeadersReceived(ctx context.Context, md metadata.MD) to inspect server headers/trailers as soon as they are received (especially useful for streaming RPCs like Handshake).
    type myMiddleware struct{}

func (m *myMiddleware) StartCall(ctx context.Context) context.Context {
    return context.WithValue(ctx, "key", "value")
}

func (m *myMiddleware) CallCompleted(ctx context.Context, err error) {
    fmt.Printf("Call finished with error: %v\n", err)
}

// Wrap it
middleware := flight.CreateClientMiddleware(&myMiddleware{})