fastparquet Documentation

fastparquet is a Python interface to the Parquet file format designed for high-performance reading and writing of Parquet files. It is specifically optimized for interoperability with Pandas data-frames.

Key Features:

• Columnar Storage: Efficiently read only the data of interest.
• Compression & Encoding: Supports various compression algorithms per-column and optimized encoding schemes.
• Parallelism: Supports single or multiple-file formats and can be used with dask for parallel reading/writing across a cluster.
• Performance: Uses cython to accelerate both reading and writing operations.
• Filesystem Interoperability: Can read from and write to arbitrary file-like objects, including those provided by fsspec (e.g., for S3 access via s3fs).

Project Status: Retiring

As of March 2026, fastparquet is being retired. The release of pandas 3.0 introduced an explicit dependency on pyarrow, reducing the demand for this project.

Recommendations:

• If you are using pandas 3.0 or later, consider using the pyarrow engine instead.
• If you are using pandas 2.x, fastparquet may still be useful, but no further development is anticipated.

Fastparquet supports reading nested schemas using a flattening mechanism. Struct columns are converted into top-level columns using dot notation (e.g., root.visitor.ip).

• Structs: A schema with visitor.ip becomes an ordinary Pandas column named visitor.ip.
• LIST and MAP types: Fastparquet can read some LIST types. A column tags containing a list of strings will be loaded as an object column containing lists.
• Limitations: If a LIST/MAP element is a complex type (struct, map, or list) rather than a primitive, fastparquet cannot read it; the column will either be missing or contain only None values.

Fastparquet supports nanosecond resolution times via an extended "logical" types system. By default, it emits these for the default pandas time type and produces a full parquet schema including both "converted" and "logical" type information.

Important Details:

• All output has isAdjustedToUTC=True (timestamps rather than local time).
• The time-zone is stored in metadata and will be successfully recreated in fastparquet and pyarrow. In other frameworks, times may appear as UTC.
• Spark Compatibility: If you require compatibility with Spark, you may still want to use times="int96" when writing.

When choosing a Parquet engine for your Python workflow, consider the following relationships:

• Apache Arrow (pyarrow): Defines an in-memory data representation and provides its own Parquet interface. If your workflow already utilizes Arrow, it is generally recommended to use the pyarrow engine.
• PySpark: Interfaces Python commands with a Java/Scala execution core. It is a heavy-weight engine used for Spark-based workflows.
• fastparquet: A lightweight, performant Python-native library (using cython) that integrates deeply with the dask ecosystem and fsspec for filesystem access.

Fastparquet supports row-level filtering within row-groups (previously only full row-groups could be excluded via metadata statistics).

Usage: Use the same syntax as before, allowing multiple column expressions to be combined with AND|OR using a list structure.

Note on Performance: This mechanism requires two passes: one to load the columns needed for the boolean mask, and a second to load the actual output columns. This may be slower than a full read but can significantly reduce memory footprint if only a small fraction of rows match the filter and the filter columns are not in the final output. This is not currently supported for reading DataPageV2.

You can install fastparquet using either conda or pip.

Important Note for pip users: It is recommended to install numpy before installing fastparquet via pip to avoid environment resolution failures.

# Using conda
conda install -c conda-forge fastparquet

# Using pip
pip install fastparquet

# Installing latest version from GitHub (main branch)
pip install git+https://github.com/dask/fastparquet

You can install fastparquet using conda, pip, or directly from the GitHub repository.

Conda (Recommended for latest compiled version):

conda install -c conda-forge fastparquet

PyPI:

pip install fastparquet

Note: You may wish to install numpy first to assist the pip resolver.

GitHub (Latest development version):

pip install git+https://github.com/dask/fastparquet

Note: Installing from GitHub requires cython to rebuild C files.

Important Notice (March 2026): Due to changes in pandas 3.0 (which now depends explicitly on pyarrow), fastparquet is being retired. It may still be useful for users on pandas 2.x, but no further development is anticipated.

For binary data (bytestrings) where all lengths are identical or nearly identical, you can use fixed-length byte arrays to gain a modest speed boost. Use the fixed_text keyword in write() to specify the predetermined length.

Warning: This is not recommended for standard text strings, as the overhead of UTF8 encoding/decoding makes it inefficient.

write('out.parq', df, fixed_text={'char_code': 1})

Required Dependencies

• numpy
• pandas
• cramjam
• thrift

Compression Codecs

fastparquet uses cramjam to provide the following compression codecs:

• gzip
• snappy
• lz4
• brotli
• zstd

Optional Codec:

• python-lzo/lzo can be installed for additional compression support.

To write Parquet data to a remote file-system, use the write function and provide:

1. open_with: A callable that accepts (path, mode) to open files on the remote system.
2. mkdirs: A callable used to create necessary directories (required for multi-file mode). For systems like S3 where intermediate directories do not need explicit creation, you can pass a no-op function like noop.

from fastparquet import write

# Example writing to S3 using s3fs
# mkdirs=noop is used because S3 does not require explicit directory creation
write('/mybucket/output_parq', data, file_scheme='hive', 
      row_group_offsets=[0, 500], open_with=myopen, mkdirs=noop)