hyparquet

Hyparquet operates on an AsyncBuffer rather than a standard ArrayBuffer. An AsyncBuffer is an interface that allows the slice method to return a Promise<ArrayBuffer>, enabling efficient asynchronous fetching of remote files via HTTP range requests.

type Awaitable<T> = T | Promise<T>
interface AsyncBuffer {
  byteLength: number
  slice(start: number, end?: number): Awaitable<ArrayBuffer>
}

Common ways to create an AsyncBuffer:

• asyncBufferFromUrl({ url, requestInit, byteLength }): For remote HTTP files. Use requestInit for auth headers and byteLength to avoid an extra HEAD request.
• asyncBufferFromFile(path): For local files in Node.js.
• ArrayBuffer: You can pass a standard ArrayBuffer directly where an AsyncBuffer is expected.

Choosing between these two functions depends on whether you need row-oriented data or high-performance streaming.

parquetReadObjects

This is a convenience wrapper that returns data as Promise<Record<string, any>[]>. It is the simplest way to read a file but performs an expensive transposition from column-oriented to row-oriented format.

parquetRead

This is the base function. It returns a Promise<void> and delivers data via callbacks (onComplete, onChunk, or onPage). Because it avoids the expensive transposition step, it is more memory-efficient and suitable for large datasets or streaming.

// parquetReadObjects returns rows as objects
parquetReadObjects({ file }): Promise<Record<string, any>[]>

// parquetRead uses callbacks for streaming
await parquetRead({
  file,
  onComplete: (data) => { ... }
})

In a Node.js environment, use asyncBufferFromFile to wrap a local file path as an AsyncBuffer before passing it to parquetReadObjects.

Note: hyparquet is published as an ES module, so you may need to use dynamic import() in older Node.js versions.

const { asyncBufferFromFile, parquetReadObjects } = await import('hyparquet')

const file = await asyncBufferFromFile('example.parquet')
const data = await parquetReadObjects({ file })

To read Parquet files in a browser environment, use asyncBufferFromUrl to create an AsyncBuffer from a URL, then use parquetReadObjects to fetch the data. It is highly recommended to provide columns, rowStart, and rowEnd to limit the amount of data fetched via HTTP range requests.

const { asyncBufferFromUrl, parquetReadObjects } = await import('https://cdn.jsdelivr.net/npm/hyparquet/src/hyparquet.min.js')

const url = 'https://hyperparam-public.s3.amazonaws.com/bunnies.parquet'
const file = await asyncBufferFromUrl({ url }) // wrap url for async fetching
const data = await parquetReadObjects({
  file,
  columns: ['Breed Name', 'Lifespan'],
  rowStart: 10,
  rowEnd: 20,
})

By default, hyparquet supports Uncompressed and Snappy codecs. To support other codecs like GZip, Brotli, LZ4, or ZSTD, you must use the hyparquet-compressors package.

import { parquetReadObjects } from 'hyparquet'
import { compressors } from 'hyparquet-compressors'

const data = await parquetReadObjects({ file, compressors })

To use hyparquet, you must provide an AsyncBuffer. This is a file-like object that allows the library to perform asynchronous, partial reads of the Parquet data. It must implement the following interface:

• byteLength: number: The total size of the buffer.
• slice(start: number, end?: number): Awaitable<ArrayBuffer>: A method that returns a slice of the buffer starting at start and ending before end. The return value can be an ArrayBuffer or a Promise<ArrayBuffer>.

export interface AsyncBuffer {
  byteLength: number
  slice(start: number, end?: number): Awaitable<ArrayBuffer>
}

hyparquet supports geospatial data through GeoParquet metadata.

• Enabling GeoParquet: In MetadataOptions or BaseParquetReadOptions, set geoparquet: true (this is the default). This tells the parser to interpret geospatial columns as geometry or geography types based on the metadata.
• Geometry Output: When parsed, geometry data follows the GeoJSON specification (RFC 7946). Supported types include Point, MultiPoint, LineString, MultiLineString, Polygon, MultiPolygon, and GeometryCollection.
• Position: Coordinates are represented as a Position (an array of numbers, typically [longitude, latitude]).

export type Geometry =
  | Point
  | MultiPoint
  | LineString
  | MultiLineString
  | Polygon
  | MultiPolygon
  | GeometryCollection

export interface Point {
  type: 'Point'
  coordinates: Position
}

To improve performance when reading large Parquet files with specific criteria, use the following optimization flags:

1. Filtering: Pass a filter object. Note that using a filter requires rowFormat: 'object' in parquetRead.
2. Bloom Filter Pushdown: Set useBloomFilters: true. This allows the reader to use bloom filters to prune entire row groups that cannot contain matching data.
3. Page Index Pushdown: Set usePageIndex: true. This allows the reader to use column and offset indexes to skip specific pages within a row group that do not match the filter.

When using filters, hyparquet automatically includes the necessary filter columns in the read plan, even if they aren't in your requested columns list, and then performs a projection to remove them before returning the final data.

The parquetQuery function implements different execution paths based on the combination of filter and orderBy options:

1. Filter without orderBy: The engine iterates through row groups and fetches data until the requested rowEnd is reached or all matching rows are found.
2. Filter with orderBy: The engine reads all rows (including the orderBy column if not already requested), sorts the entire result set in memory, and then projects out the orderBy column if it wasn't part of the original columns request.
3. orderBy without Filter: The engine performs a sparse read. It first fetches only the orderBy column to determine the sorted indices, then performs a targeted fetch of the requested columns using those indices.
4. No Filter and No orderBy: The engine falls back to a standard parquetReadObjects call.

When using parquetRead, you can use callbacks to process data as it is loaded. This is useful for chunk streaming.

• onChunk: Returns column-oriented data. It returns top-level columns (including structs) as a single assembled column. This may involve waiting for multiple sub-columns to load.
• onPage: Returns column-oriented page data. It does not assemble struct columns and returns individual sub-column data. It may return data sooner than onChunk.

import { parquetRead } from 'hyparquet'

await parquetRead({
  file,
  onChunk(chunk) {
    // chunk is ColumnData (top-level columns assembled)
    console.log('chunk', chunk)
  },
  onPage(chunk) {
    // chunk is ColumnData (individual sub-columns)
    console.log('page', chunk)
  },
})

Use parquetMetadataAsync to retrieve file metadata (including schema and statistics) without reading the entire file. This is efficient for discovering column names or the total row count.

import { parquetMetadataAsync, parquetSchema } from 'hyparquet'

const file = await asyncBufferFromUrl({ url: '...' })
const metadata = await parquetMetadataAsync(file)

// Get total number of rows (convert bigint to number)
const numRows = Number(metadata.num_rows)

// Get nested table schema
const schema = parquetSchema(metadata)

// Get top-level column header names
const columnNames = schema.children.map(e => e.element.name)

Hyparquet provides options to customize how data is returned and decoded:

Row Format

By default, parquetRead returns an array of values for each row: [value]. To get an object instead ({ columnName: value }), set rowFormat: 'object'. Note that parquetReadObjects uses 'object' by default.

await parquetRead({
  file,
  rowFormat: 'object',
  onComplete: data => console.log(data),
})

Binary and GeoParquet

• Binary Columns: By default, BYTE_ARRAY columns without a LogicalType annotation are decoded as UTF-8 strings. To disable this and get raw binary, set utf8: false.
• GeoParquet: Hyparquet automatically decodes geospatial columns to GeoJSON if it detects a GeoParquet file. To disable this, set geoparquet: false.