DOCSEARCHbyAlgolia
Submit Docs

pyogrio

repository·main·Indexed 43 days ago

https://github.com/geopandas/pyogrio

A high-performance Python library for bulk-oriented spatial vector file I/O using GDAL/OGR bindings. Optimized for GeoPandas users, it provides significant speedups for reading and writing spatial data compared to row-per-row approaches. It supports common formats like ESRI Shapefile, GeoPackage, GeoJSON, and FlatGeobuf, and offers specialized integration with Apache Arrow for further performance boosts.

Tokens
13.9K
Snippets
51
Records
79
Agent score
56%

What's inside pyogrio

  1. Overview of pyogrio for spatial vector I/O

    main

    pyogrio is a high-performance library designed for bulk-oriented read and write access to GDAL/OGR vector data sources. It is optimized for speed by using pre-compiled bindings to perform bulk operations, avoiding the overhead of row-per-row Python type conversions.

    Key features include:

    • High Performance: Provides significant speedups (often >5-100x for reading and >5-20x for writing) compared to row-per-row approaches like Fiona.
    • GeoPandas Integration: Designed to read from and write to GeoPandas GeoDataFrames.
    • Versatile Data Handling: Supports spatial data (geometries like points, lines, polygons) as well as non-spatial attribute tables (e.g., GeoPackage attribute tables, DBF files, or CSV files).

  2. Overview of pyogrio

    main

    pyogrio is a high-performance, bulk-oriented library for reading and writing spatial vector data using GDAL/OGR bindings. It is designed to work seamlessly with GeoPandas GeoDataFrames.

    Key features include:

    • High Performance: Uses pre-compiled GDAL/OGR bindings to perform bulk operations, avoiding the overhead of row-by-row Python type conversion. It can achieve >5-100x speedups for reading and >5-20x for writing compared to row-per-row approaches like Fiona.
    • Versatile I/O: Supports spatial data (geometries + attributes) and non-spatial data (attribute tables only, such as GeoPackage attribute tables, DBF, or CSV).
    • GeoPandas Integration: Primary use case is reading from and writing to GeoPandas GeoDataFrames.

  3. Handle NULL or unset field values in pyogrio

    main

    When reading data, certain NULL or unset field values cannot be stored directly into an ndarray for specific types. Pyogrio handles them as follows:

    • String types: NULL values are represented as None.
    • Boolean or Integer types (np.int32, np.int64): Data is re-cast to np.float64, and NULL values are represented as np.nan.
    • Date or Datetime types: The field is set as np.datetime64('NaT').

    Warning: Detection of NULL values is limited to the subset of records being read. This means reading different subsets of records might result in different data types for the same column. To determine the original data types of each column, use pyogrio.read_info.

  4. Handling DateTime precision and time zones

    main

    Precision

    GDAL supports datetimes at millisecond resolution.

    • In pandas 2.0+, read_dataframe returns datetime64[ms].
    • For older pandas versions, datetime64[ns] is used.
    • When writing, only precision up to milliseconds is retained.

    Time Zones

    GDAL represents time zones as UTC offsets, whereas pandas uses IANA time zones. Roundtripping through GDAL often loses the original IANA time zone information, leaving only the offset.

    Workarounds:

    • To avoid mixed offsets being converted to UTC during reading, use datetime_as_string=True or mixed_offsets_as_utc=False in read_dataframe.
    • If you need precision > ms or are using formats like ESRI Shapefile that lack datetime support, convert datetimes to strings.

  5. Compare pyogrio to Fiona

    main

    While Fiona is a full-featured, general-purpose library for OGR vector data with extensive functionality, pyogrio is specialized for dataframe-oriented workflows (like GeoPandas).

    Key differences:

    • Approach: Pyogrio uses a stateless, bulk-oriented approach where data is read or written in a single pass.
    • Performance: Pyogrio's bulk-oriented design enables significantly faster I/O operations compared to Fiona, particularly for larger datasets.

  6. Understand the relationship between Pyogrio and GDAL/OGR

    main
    Pyogrio is a high-performance interface powered by the GDAL/OGR library. While the terms are often used interchangeably in documentation, OGR is the specific component of GDAL responsible for vector data interoperability. Vector data sources consist of geometries (such as points, lines, or polygons) paired with associated data columns.

  7. How pyogrio reads and writes data

    main

    Pyogrio operates using two primary modes for reading data, both designed for bulk-oriented I/O:

    1. Standard Mode: Uses a numpy-oriented approach in Cython. It extracts geometries as Well-Known Binary (WKB) objects and reads attributes into numpy arrays before converting them to a GeoPandas GeoDataFrame.
    2. Arrow Mode: Enabled by setting use_arrow=True. This uses the GDAL Arrow Stream interface to read data directly into the Apache Arrow memory format. This mode is highly optimized and provides significant performance boosts, especially for large files.

    Writing Data: Pyogrio writes an entire GeoDataFrame at once. However, it also supports appending data to an existing data source.

    Memory Management: Because all records are read into memory in bulk, reading very large datasets can cause memory issues. To mitigate this, you can use GDAL-offered filtering options during the read process, such as:

    • bbox filters
    • skip_features / max_features
    • sql statements

  8. Understand OGR vector data sources, layers, and drivers

    main

    To use Pyogrio effectively, you should understand the three core OGR abstractions:

    1. OGR vector data source: The container file format. A single source can contain multiple layers or tables. For example, a GeoPackage can hold several spatial layers, whereas an ESRI Shapefile is limited to a single layer.
    2. OGR vector data layer: A specific entity within a data source. A layer contains zero or more records and may or may not include a geometry column.
    3. OGR vector driver: The internal GDAL/OGR component that enables reading or writing a specific format. Availability of drivers depends on your GDAL distribution; common formats are usually enabled by default, while others may require specific builds or manual compilation.

  9. Build pyogrio from source on Windows (OSGeo4W)

    main

    If using GDAL from an OSGeo4W installation (e.g., at C:\GDAL), you must provide the GDAL paths via environment variables or command-line parameters because they cannot be automatically detected.

    Using Environment Variables:

    set GDAL_INCLUDE_PATH=C:\GDAL\include
set GDAL_LIBRARY_PATH=C:\GDAL\lib
set GDAL_VERSION=3.8.3
python -m pip install --no-deps --force-reinstall --no-use-pep517 -e . -v

    Using Command-line Parameters:

    python -m pip install --install-option=build_ext --install-option="-IC:\GDAL\include" --install-option="-lgdal_i" --install-option="-LC:\GDAL\lib" --install-option="--gdalversion=3.8.3" --no-deps --force-reinstall --no-use-pep517 -e . -v

    Note: --no-use-pep517 is required to pass additional options to the build backend. The GDAL DLLs must be on your system PATH.

  10. Set up pyogrio for development

    main

    To develop pyogrio, clone the repository and ensure gdal-config is in your system path, or set the GDAL_INCLUDE_PATH, GDAL_LIBRARY_PATH, and GDAL_VERSION environment variables.

    Building requires Cython, numpy, and pandas.

    Workflow:

    1. Install pre-commit and register hooks: pre-commit install.
    2. Build Cython extensions: python setup.py develop.
    3. Run tests using pytest: pytest pyogrio/tests.
    pre-commit install
python setup.py develop
pytest pyogrio/tests

  11. Optimize reading OpenStreetMap (OSM) data

    main

    Reading OSM data in pyogrio can be slow because feature counts must be calculated by iterating over all features, and data is not structured for fast layer-based reading.

    To improve performance:

    1. Download to local: Download remote OSM data to local files before reading.
    2. Use Arrow: Set use_arrow=True to potentially speed up reading.
    3. Pre-convert: Use ogr2ogr to translate OSM data into a more performant format like GPKG.

    If you encounter errors regarding large files, you may need to enable interleaved reading mode using one of the following methods:

    # Method 1: Set config option globally for all operations
from pyogrio import set_gdal_config_options
set_gdal_config_options({"OGR_INTERLEAVED_READING": True})

# Method 2: Set dataset open option for a specific read call
from pyogrio import read_dataframe
df = read_dataframe(path, INTERLEAVED_READING=True)

  12. Install pyogrio

    main

    pyogrio is available for Linux, MacOS, and Windows via conda-forge and PyPI.

    Requirements

    • Python >= 3.10
    • GDAL >= 3.6
    • For reading to GeoDataFrames: geopandas>=0.12 and shapely>=2.
    • Optimization: Installing pyarrow enables further speed-ups when using the use_arrow=True option.