parquet2

To optimize IO and avoid reading unnecessary data, Parquet2 supports pushdown filters using:

• Bloom Filters: Column metadata may contain bitsets that allow you to check if an item is not in a column chunk.
• Column and Page Indexes: Metadata that allows you to skip specific column chunks or pages during the IO phase.

The decoding process is split into two distinct phases: an IO-intensive phase and a CPU-intensive phase.

1. IO-Intensive Phase (Handled by Parquet2)

1. Read metadata.
2. Seek to a specific row group and column.
3. Iterate over (compressed) pages within that group/column.

2. CPU-Intensive Phase (Handled by Consumer)

Once a compressed page is in memory, the consumer is responsible for: read -> compressed page -> decompressed page -> decoded bytes -> deserialized

This separation allows you to distribute the second phase across multiple threads to maximize performance.

Understanding the Parquet data structure is essential for efficient reading. The metadata provides the roadmap for accessing data, which is organized as follows:

1. Schema: Defines columns and data types.
2. Row Groups: The file is divided into horizontal slices called row groups.
3. Column Chunks: Each row group contains chunks of data for specific columns.
4. Pages: Each column chunk is further divided into pages.
5. Values: Each page contains the actual multiple values.

Metadata for everything except Pages is available within the FileMetaData object.

Because Parquet2 decouples IO from CPU work, you can maximize throughput by reading compressed pages in the main thread and offloading the decompression and deserialization to worker threads.

When implementing this, note that compressed_page buffers should be moved into the thread rather than cloned to avoid expensive memory operations and ensure memory is released promptly.

let handles = vec![];
for column in columns {
    let column_meta = metadata.row_groups[row_group].column(column);
    let compressed_pages = get_page_iterator(column_meta, &mut file, file)?.collect()?;
    // each compressed_page has a buffer; cloning is expensive(!). We move it so that the memory
    // is released at the end of the processing.
    handles.push(thread::spawn move {
        page_iter_to_array(compressed_pages.into_iter())
    })
}
let columns_from_all_groups = handles.join_all();

After obtaining CompressedDataPages, you must perform three distinct steps to access the raw values:

1. Decompress: Use the decompress function to expand the compressed page buffer.
2. Decode: Convert the decompressed buffer into its logical components (e.g., handling definition levels).
3. Deserialize: Convert the decoded components into your target in-memory format (e.g., Apache Arrow or a custom struct).

Note: Decoding and deserialization are often performed in a single step depending on the target format.

The parquet-tools CLI is used to inspect and extract data from Parquet files. The basic syntax requires providing the path to a Parquet file followed by a subcommand.

Basic Usage:

parquet-tools <file> [SUBCOMMAND]

To run integration tests that validate Parquet2 against files generated by pyarrow, follow these steps to set up the Python environment:

1. Create and prepare a virtual environment.
2. Install pyarrow==7.
3. Run the PyArrow write script.
4. Execute cargo test.

This setup is only required once per change in tests/write_pyarrow.py.

python3 -m venv venv
venv/bin/pip install pip --upgrade
venv/bin/pip install pyarrow==7
venv/bin/python tests/write_pyarrow.py
cargo test

Parquet2 is an unofficial implementation of Parquet IO in Rust. It is organized into several functional modules that handle different aspects of the Parquet format:

• error: Error handling types and macros.
• bloom_filter: Bloom filter support (requires bloom_filter feature).
• compression: Compression algorithms.
• deserialize: Deserialization logic.
• encoding: Data encoding schemes.
• indexes: Indexing structures.
• metadata: Parquet file metadata.
• page: Page-level operations.
• read: Reading operations.
• schema: Schema definitions and handling.
• statistics: Column statistics.
• types: Primitive and complex types.
• write: Writing operations.

The MutStreamingIterator is a specialized fallible streaming iterator where the advance method consumes the iterator to produce the next state. This pattern is used by ColumnIterator and ReadColumnIterator to manage state transitions during sequential reading.

Trait Methods:

• advance(self) -> Result<State<Self>, Self::Error>: Consumes the current iterator and returns either State::Some(next_iterator) or State::Finished(remaining_buffer).
• get(&mut self) -> Option<&mut Self::Item>: Returns a mutable reference to the current item if available.

State Enum:

• State::Some(T): The iterator still has elements.
• State::Finished(Vec<u8>): The iterator has completed its task.