TransBigData Documentation

TransBigData is a Python package designed for processing, analyzing, and visualizing transportation spatio-temporal big data (such as Taxi GPS, bicycle sharing, and bus GPS data).

Key functional areas include:

• Data Quality: Methods to obtain dataset information like data volume, time periods, and sampling intervals.
• Data Preprocess: Cleaning methods for various types of data errors.
• Data Gridding: Generation of geographic grids (Rectangular or Hexagonal) and fast algorithms to map GPS data to these grids.
• Data Aggregating: Aggregating GPS and Origin-Destination (OD) data into geographic polygons.
• Data Visualization: Interactive visualization in Jupyter notebooks using the keplergl package.
• Trajectory Processing: Generating trajectory linestrings from GPS points and trajectory densification.
• Basemap Loading: Displaying Mapbox basemaps on matplotlib figures.

The transbigdata module provides a suite of functions for processing trajectory data. These functions allow for cleaning, segmenting, smoothing, and transforming trajectory points into various formats like LineStrings or map-matched paths.

Available trajectory processing functions include:

• traj_clean_drift: Removes drift errors from trajectories.
• traj_clean_redundant: Removes redundant/duplicate points.
• traj_slice: Slices trajectories based on time or distance.
• traj_smooth: Smooths trajectory paths.
• traj_segment: Segments trajectories into smaller parts.
• traj_densify: Increases the density of points in a trajectory.
• traj_sparsify: Decreases the density of points in a trajectory.
• traj_stay_move: Identifies stay and move segments within a trajectory.
• traj_to_linestring: Converts trajectory points into LineString geometries.
• traj_mapmatch: Performs map-matching to snap trajectories to a road network.
• traj_length: Calculates the length of trajectories.

Rasterization involves three steps: generating grid parameters, mapping GPS points to grid IDs, and aggregating data.

1. Generate Grid Parameters

Use tbd.area_to_params to create a parameter dictionary defining the grid system (e.g., resolution, method, rotation).

2. Map GPS to Grids

• For Rectangular Grids: Use tbd.GPS_to_grids. This returns two columns (LONCOL and LATCOL) that uniquely identify a grid cell.
• For Hexagonal (hexa) or Triangular (tri) Grids: Use tbd.GPS_to_grid. This returns three columns (loncol_1, loncol_2, loncol_3) required for these grid types.

3. Aggregate and Generate Geometry

Use tbd.gridid_to_polygon (for rectangular) or tbd.grid_to_polygon (for hexa/tri) to convert grid IDs into GeoPandas geometries.

# --- Rectangular Grid Example ---
params = tbd.area_to_params(bounds, accuracy=1000)
data['LONCOL'], data['LATCOL'] = tbd.GPS_to_grids(data['lon'], data['lat'], params)

# Aggregate
grid_agg = data.groupby(['LONCOL', 'LATCOL'])['VehicleNum'].count().reset_index()
# Convert to geometry
grid_agg['geometry'] = tbd.gridid_to_polygon(grid_agg['LONCOL'], grid_agg['LATCOL'], params)
import geopandas as gpd
grid_agg = gpd.GeoDataFrame(grid_agg)

# --- Hexagonal/Triangular Grid Example ---
params['method'] = 'hexa'  # or 'tri'
params['theta'] = 5       # rotation in degrees

data['loncol_1'], data['loncol_2'], data['loncol_3'] = tbd.GPS_to_grid(data['lon'], data['lat'], params)

# Aggregate
grid_agg = data.groupby(['loncol_1', 'loncol_2', 'loncol_3'])['VehicleNum'].count().reset_index()
# Convert to geometry
grid_agg['geometry'] = tbd.grid_to_polygon([grid_agg['loncol_1'], grid_agg['loncol_2'], grid_agg['loncol_3']], params)
import geopandas as gpd
grid_agg = gpd.GeoDataFrame(grid_agg)

TransBigData allows you to express data distribution using various geographic grids. The process involves three steps: obtaining gridding parameters, mapping GPS points to grid IDs, and generating grid geometries.

1. Obtain Gridding Parameters

Use tbd.area_to_params to define the coordinate system for your grid based on the study area and desired accuracy.

2. Map GPS to Grids

• For Rectangular grids (method='rect'): tbd.GPS_to_grid returns two columns (LONCOL, LATCOL) which uniquely identify a grid.
• For Hexagonal (method='hexa') or Triangle (method='tri') grids: tbd.GPS_to_grid returns three columns (loncol_1, loncol_2, loncol_3) to identify the grid.

3. Generate Geometry

Use tbd.grid_to_polygon to convert the grid IDs into spatial geometries for use with GeoPandas.

# 1. Get parameters
params = tbd.area_to_params(bounds, accuracy=1000)

# 2. Map GPS to Rectangular grids
data['LONCOL'], data['LATCOL'] = tbd.GPS_to_grid(data['lon'], data['lat'], params)

# 3. Aggregate and generate geometry
grid_agg = data.groupby(['LONCOL', 'LATCOL'])['VehicleNum'].count().reset_index()
grid_agg['geometry'] = tbd.grid_to_polygon([grid_agg['LONCOL'], grid_agg['LATCOL']], params)
import geopandas as gpd
grid_agg = gpd.GeoDataFrame(grid_agg)

TransBigData provides Geohash support to encode latitude and longitude into strings, decode them back, and convert them into grid geometries.

Note: Compared to the rectangular grid processing methods in TransBigData, Geohash is slower and does not allow for freely defined grid sizes. Precision is determined by the length of the Geohash string.

Common workflow:

1. Encode: Use geohash_encode to create geohash strings from coordinates.
2. Aggregate: Group data by the geohash string to perform spatial aggregation.
3. Decode & Convert: Use geohash_decode to retrieve center coordinates and geohash_togrid to convert the geohash into a polygon geometry for mapping.

import transbigdata as tbd
import pandas as pd
import geopandas as gpd

# 1. Encode geohash from coordinates
data['geohash'] = tbd.geohash_encode(data['slon'], data['slat'], precision=6)

# 2. Aggregate data by geohash
dataagg = data.groupby(['geohash'])['VehicleNum'].count().reset_index()

# 3. Decode and convert to grid geometry
dataagg['lon_geohash'], dataagg['lat_geohash'] = tbd.geohash_decode(dataagg['geohash'])
dataagg['geometry'] = tbd.geohash_togrid(dataagg['geohash'])

# Convert to GeoDataFrame for spatial operations
dataagg = gpd.GeoDataFrame(dataagg)