DOCSEARCHbyAlgolia
Submit Docs

GeoParquet Specification

repository·main·Indexed 22 days ago

https://github.com/opengeospatial/geoparquet

A specification for storing geospatial vector data in Apache Parquet columnar storage. It provides a standardized way to represent geometry and geography to enhance interoperability between cloud data warehouses and analytic tools. The documentation covers the v2.0.0 specification, validation tools like GPQ and GDAL/OGR scripts, implementation guidelines for readers, and best practices for producing high-quality files using tools such as GDAL/OGR, DuckDB, and Apache Sedona.

Tokens
8.7K
Snippets
13
Records
39
Agent score
77%

What's inside GeoParquet

  1. Overview of GeoParquet features and limitations

    main

    GeoParquet is a specification for storing geospatial vector data (points, lines, polygons) in Apache Parquet columnar storage.

    Key Features

    • Multiple Spatial Reference Systems (SRS): Supports native projections with clear default recommendations for interoperability.
    • Multiple Geometry Columns: Supports a default geometry column and additional geometry columns.
    • Efficient Storage: Leverages Parquet's compression for small file sizes and network efficiency.
    • Coordinate Support: Works with both planar and spherical coordinates.
    • Analytic Performance: Optimized for read-heavy workflows via columnar reading and chunk filtering based on statistics.
    • Data Partitioning: Supports geospatial partitioning.

    Limitations

    • Not for Write-Heavy Workflows: GeoParquet is not recommended for systems that require constant updates or frequent data additions; row-based formats are better suited for those use cases.

  2. How Coordinate Reference Systems (CRS) are handled in GeoParquet

    main

    GeoParquet 2.0 uses both the native Parquet GEOMETRY/GEOGRAPHY logical type crs property and a GeoParquet geo metadata layer.

    Key Rules:

    • Source of Truth: The Parquet logical-type crs property is the primary source of truth. The GeoParquet crs field in metadata is a restatement of that CRS.
    • Format Requirement: While the Parquet property can use various forms (like <authority>:<code">), the GeoParquet column-metadata crs field MUST be inline PROJJSON or null.
    • Default CRS: If the crs key is absent, the default is OGC:CRS84 (longitude, latitude based on WGS84).
    • Undefined CRS: To explicitly indicate an undefined or unknown CRS, set the GeoParquet crs field to null. In this case, the Parquet logical-type crs property should be set to "srid:0".

    CRS Mapping Table

    Parquet logical-type crsGeoParquet column-metadata crsMeaning
    absentabsentOGC:CRS84
    inline PROJJSON objectsame CRS as inline PROJJSONCRS fully described in metadata
    <authority>:<code"> stringresolved CRS as inline PROJJSONCRS identified by authority code
    srid:0nullCRS undefined or unknown

  3. Enable Efficient Spatial Access in GeoParquet

    main

    To allow readers to quickly skip irrelevant data during spatial queries, you must ensure the file supports spatial filtering.

    GeoParquet 2.0 (Recommended)

    GeoParquet 2.0 uses native Parquet GEOMETRY/GEOGRAPHY logical types. These columns include built-in geospatial statistics (specifically a bounding box for each column chunk/row group).

    • Benefit: Readers use these statistics to skip row groups that do not intersect the query area.
    • Benefit: You no longer need a separate bbox column, which reduces file size (especially for point datasets).

    GeoParquet 1.1

    If you need to support a wider range of software/versions, use GeoParquet 1.1 with the bbox covering encoding. This adds a Parquet struct of four values to every row to enable spatial filtering.

    Note: For either method to be effective, the data must be spatially ordered and row groups must be sized sensibly.

  4. Understand the GeoParquet specification version and validation

    main
    The GeoParquet specification is currently at version 2.0.0. This version provides guidance for implementing Parquet GEOMETRY and GEOGRAPHY types and includes optional metadata not covered by the core Apache Parquet specification. To validate that your GeoParquet metadata is compliant with this version, you should use the provided schema.json.

  5. Implement Spatial Ordering

    main

    Spatial ordering is essential for row group statistics (native geometry stats or bbox columns) to work effectively. Without ordering, a reader cannot reliably skip large chunks of data.

    GeoParquet does not use a specific internal index (like an R-tree in GeoPackage). Instead, it relies on the physical order of rows. If data is converted from formats like GeoPackage or Shapefile, it may already be ordered.

    How to verify: Open the file in a GIS tool. If the data loads in spatial chunks (e.g., an area loads, then another area loads), it is likely ordered. If the data appears to load globally/randomly across the whole extent, it is not indexed.

  6. How to format Bounding Boxes (bbox) in GeoParquet

    main

    The bbox field defines the spatial extent of a geometry column. It must be an array of numbers in the same CRS as the geometry.

    Formatting by Dimension:

    • XY (2D): [<xmin>, <ymin>, <xmax>, <ymax>]
    • XYZ (3D): [<xmin>, <ymin>, <zmin>, <xmax>, <ymax>, <zmax>]
    • XYZM (3D + Measure): [<xmin>, <ymin>, <zmin>, <mmin>, <xmax>, <ymax>, <zmax>, <mmax>]

    Geographic CRS (Longitude/Latitude): For geographic systems, follow the GeoJSON standard (RFC 7946): list the most southwesterly coordinate followed by the most northeasterly coordinate. This handles antimeridian crossings correctly.

  7. Understand GeoParquet versioning and compatibility

    main

    GeoParquet follows Semantic Versioning (SemVer).

    • Backwards Compatibility: A file written with an older specification version will always be compatible with a newer specification version.
    • Forward Compatibility: An implementation aware of an older specification version MUST be able to either correctly interpret data written with a newer (minor) specification version or recognize that it cannot interpret it.

    Implementation Guidelines for Forward Compatibility:

    • Do not reject metadata containing unknown fields (this allows for new fields like performance indexes to be ignored safely).
    • Explicitly validate all field values you rely on. For example, if the current spec only allows encoding = "WKB", an implementation should validate that specific value rather than assuming it is the only possible value, to allow for future options to be added.

  8. How the `edges` attribute affects geometry interpretation

    main

    The edges attribute defines how to interpret the line between two points. This is critical for geographic data to avoid analytical errors.

    Available Values:

    • "planar" (Default): Uses a flat Cartesian coordinate system.
    • "spherical": Edges follow the shortest distance on a perfect sphere (used by BigQuery/Snowflake).
    • "vincenty", "thomas", "andoyer", "karney": Use specific geodesic formulas based on the ellipsoid specified in the crs.

    Best Practices:

    • If edges is not "planar", it is RECOMMENDED to set orientation="counterclockwise" to ensure predictable polygon winding in spherical systems.

  9. Structure GeoParquet metadata

    main

    GeoParquet uses a two-level metadata approach to store geospatial information:

    1. File metadata: Contains high-level information, such as the version of the GeoParquet specification used.
    2. Column metadata: Contains specific additional metadata for each individual geometry column.

    To implement this, a GeoParquet file MUST include a geo key within the Parquet file metadata (specifically under FileMetaData::key_value_metadata). The value of this geo key must be a JSON-encoded UTF-8 string that validates against the GeoParquet metadata schema (schema.json).

  10. Understand page-level vs. row-group spatial pruning

    main

    Current GeoParquet implementations rely on native Parquet GEOMETRY/GEOGRAPHY types which provide geospatial statistics (bounding boxes) at the column chunk (row group) level.

    Pruning Granularity

    • Row-Group Pruning: A reader can skip an entire row group if its bounding box does not intersect the query area. This is the current standard.
    • Page-Level Pruning: This involves skipping individual pages within a row group. While native geometry statistics do not currently support this, using a GeoParquet 1.1 bbox covering column (a Parquet struct column) allows for finer-grained pruning because the ColumnIndex provides per-page min/max values.

    Performance Impact

    Benchmarks suggest that for highly selective queries, page-level pruning can significantly reduce query time (e.g., halving time from ~93ms to ~48ms in some tests). The effectiveness of page-level pruning depends heavily on the data being spatially ordered so that individual pages remain spatially compact.

  11. Format geometry columns in GeoParquet

    main

    Geometry columns in a GeoParquet file must follow these encoding and structural rules:

    • Logical Types: Columns must be encoded as either GEOMETRY or GEOGRAPHY logical types.
    • Encoding: The underlying data must be a BYTE_ARRAY encoding geospatial features in the WKB (Well-known binary) format.
    • Nesting: Geometry columns MUST be at the root of the schema. They cannot be nested inside complex types like structs, lists, arrays, or map types.
    • Repetition: The repetition level for geometry columns must be either required (exactly one) or optional (zero or one). A geometry column MUST NOT be repeated (e.g., you cannot have a repeated field containing a geometry). While a file may have multiple geometry columns with different names, each individual column must follow these repetition rules.

  12. Implementation considerations for GeoParquet data readers

    main

    When building a reader for Parquet files that may contain geospatial data without official GeoParquet metadata, follow these implementation principles:

    Handling Non-Compliant Data

    If the official GeoParquet metadata is missing, a reader should make the following assumptions to support compatible files:

    • Geometry Types: Assume geometry_types is an empty array (the type is unknown, so the reader should make no assumptions).
    • CRS: Assume the CRS is OGC:CRS84 (or EPSG:4326 with longitude/latitude axis order).
    • Winding Order: Assume no specific orientation/winding order is enforced.
    • Edges: Determine the edge type based on the column name: "geometry" implies "planar", and "geography" implies "spherical".

    Robustness and Fallback Logic

    • Metadata First: A reader must always attempt to read official GeoParquet metadata first. Only fall back to the compatibility assumptions listed above if the metadata is absent.
    • Avoid Exclusive Compatibility: Do not build a reader that only understands these compatible files. A reader that cannot read official GeoParquet but can read compatible files is a regression in interoperability, as it cannot distinguish between a file missing metadata and a file that is intentionally non-compliant.
    • User Hints: Optionally allow users to provide manual hints to override compatibility assumptions, such as specifying a custom column name, providing WKT instead of WKB, specifying a specific geometry_type, or defining the CRS, winding order, edges, bbox, or epoch.