repository·main·Indexed 43 days ago
https://github.com/holoviz/spatialpandasA library providing Pandas and Dask extensions for high-performance, vectorized spatial and geometric operations. It features GeoDataFrame and GeoSeries for efficient handling of large datasets, including spatial indexing, coordinate indexing via .cx, and spatial joins with sjoin. The Dask integration enables parallel spatial processing with tools like pack_partitions for spatially reordering data along a Hilbert curve and pack_partitions_to_parquet for memory-efficient storage.
PolygonArray.from_exterior_coords or MultiPolygonArray.from_exterior_coords is significantly faster than passing a list of Shapely objects to the standard constructor. This is because it avoids the overhead of creating individual Shapely objects before populating the array.dask is installed in your environment, spatialpandas automatically attempts to register its spatial types with Dask upon import. You do not need to perform additional configuration to enable this integration, but ensure dask is present in your environment for the registration to trigger.For MultiPolygonArray, you can use the from_exterior_coords method. This method expects a nested structure: a list where each element represents a MultiPolygon, and each MultiPolygon is represented by a list of its constituent polygon exterior coordinate arrays.
import numpy as np
from shapely import Polygon, MultiPolygon
from spatialpandas.geometry import MultiPolygonArray
# Setup a single polygon
p = Polygon([[0, 0], [1, 0], [1, 1], [0, 1], [0, 0]])
exterior_coords = np.asarray(p.exterior.coords)
# Method 1: From Shapely MultiPolygon objects
mp = MultiPolygon(polygons=[p, p])
mp_arr_shapely = MultiPolygonArray([mp])
# Method 2: From exterior coordinates (Faster)
# Note the nested list structure: [ [poly1_coords, poly2_coords], ... ]
mp_arr_coords = MultiPolygonArray.from_exterior_coords([[exterior_coords, exterior_coords]])A GeoDataFrame is a pandas DataFrame subclass designed for spatial data. It must contain at least one column with a GeometryDtype. When initializing, the class automatically attempts to identify the first available geometry column and set it as the active geometry. If no geometry columns are present, a ValueError is raised.
Note: The constructor always performs a copy of the input data.
from spatialpandas import GeoDataFrame
import pandas as pd
# Assuming 'df' has a column with GeometryDtype
gdf = GeoDataFrame(df)When working with very large datasets, pack_partitions might exceed memory or disk limits due to the shuffle. pack_partitions_to_parquet provides a more robust way to reorder data spatially and write it directly to a Parquet dataset.
Arguments:
path: Output Parquet dataset path.npartitions: Number of output partitions.p: Hilbert curve parameter.compression: Compression algorithm (e.g., 'snappy').tempdir_format: A string used for temporary files. Must contain a {partition} replacement field (e.g., "tmp/part.{partition}.parquet").storage_options: Key/value pairs for the filesystem backend.overwrite: If True, deletes the existing directory at path before writing.This method writes a Parquet dataset that includes spatial metadata (partition_bounds) in the _common_metadata and _metadata files, allowing for fast spatial filtering when reading the dataset back in.
ddf.pack_partitions_to_parquet(
path="/data/spatial_dataset",
npartitions=100,
p=15,
compression="snappy",
tempdir_format="/tmp/spatial_{partition}_{uuid}.parquet",
overwrite=True
)The primary entry points for the library are GeoDataFrame and GeoSeries. These classes extend Pandas/Dask functionality to support vectorized spatial and geometric operations.
from spatialpandas import GeoDataFrame, GeoSeriesThe .geometry property returns the GeoSeries corresponding to the current active geometry column. If no active geometry is set (i.e., _geometry is not pointing to a valid geometry column), accessing this property will raise a ValueError with the message: "GeoDataFrame has no active geometry column.\nThe active geometry column should be set using the set_geometry method."
# Access the active geometry
active_geom = gdf.geometryThe .cx property provides a coordinate indexer (_CoordinateIndexer) that allows for efficient access to the underlying geometry coordinates within the GeoSeries.
# Access coordinates via the cx indexer
coords = geos.cxYou can create a PolygonArray by passing a list of Shapely Polygon objects to the constructor, or more efficiently, by using the from_exterior_coords class method. The latter accepts a list of coordinate arrays (one for each polygon) and is optimized for performance when dealing with large numbers of polygons.
import numpy as np
from shapely import Polygon
from spatialpandas.geometry import PolygonArray
# Define exterior coordinates for a polygon
polygon_shell = np.array([[89.97223, -58.302593],
[89.98285, -58.29493],
[89.97226, -58.27778],
[89.95511, -58.27485],
[89.92737, -58.28543],
[89.93793, -58.30258],
[89.97223, -58.302593]])
# Method 1: From Shapely objects
p = Polygon(polygon_shell)
p_arr_shapely = PolygonArray([p, p, p])
# Method 2: From exterior coordinates (Faster)
exterior_coords = np.asarray(p.exterior.coords)
p_arr_coords = PolygonArray.from_exterior_coords([exterior_coords, exterior_coords, exterior_coords])A DaskGeoDataFrame is a Dask extension of a GeoDataFrame designed for parallel spatial processing. It supports spatial indexing and efficient repartitioning.
Key properties and methods:
.geometry: Access the geometry column..set_geometry(geometry): Sets the active geometry column..partition_sindex: Returns the spatial index for the geometry column, utilizing partition bounds for efficiency..cx[x0:x1, y0:y1]: Coordinate indexer for spatial slicing..pack_partitions(npartitions=None, p=15, shuffle='tasks'): Reparticles and reorders the dataframe spatially along a Hilbert space-filling curve. This is useful for improving spatial locality..pack_partitions_to_parquet(...): A memory-efficient version of pack_partitions that writes the spatially reordered data directly to a Parquet dataset..to_parquet(fname, ...): Specialized Dask implementation for writing spatial data to Parquet.# Example: Spatial slicing and repartitioning
# Assuming 'ddf' is a DaskGeoDataFrame
# Spatial slice using coordinate indexer
subset = ddf.cx[0:10, 0:10]
# Spatially reorder partitions using Hilbert curve
reordered_ddf = ddf.pack_partitions(npartitions=20, p=15)
# Write spatially optimized parquet
ddf.pack_partitions_to_parquet("path/to/dataset", npartitions=20)The build_sindex() method triggers the construction of a spatial index on the active geometry column. This is used to accelerate spatial queries and joins.
# Build the spatial index
gdf.build_sindex()