Lonboard
repository·main·Indexed 21 days ago
https://github.com/developmentseed/lonboardA Python library for high-performance, interactive geospatial data visualization in Jupyter environments. Built on GeoArrow and GeoParquet, it leverages deck.gl for GPU-accelerated rendering of large datasets. It provides a simple interface via the viz() function for GeoPandas GeoDataFrames and supports efficient data transport using Apache Arrow.
What's inside lonboard
- Lonboard is a library for fast, interactive geospatial data visualization in Jupyter. It is built on top of GeoArrow and GeoParquet and uses deck.gl for GPU-based map rendering. It is designed to enable the interactive visualization of very large geospatial datasets through a simple interface.
Use categorical filtering in DataFilterExtension
mainTheDataFilterExtensionhas been updated to support categorical filtering in addition to numeric range filtering. This allows users to filter geospatial data based on discrete categories rather than just continuous numeric values.How Lonboard's Python and JavaScript components interact
mainLonboard follows an architecture where all TypeScript models are combined into a single entry point. This entry point is compiled by ESBuild into an ES Module.
- Loading: The Python
Mapclass loads this JavaScript bundle via the_esmkey, which informs Jupyter/ipywidgets where to find the bundle. - Synchronization:
anywidgetandipywidgetshandle the serialization of data from Python to JavaScript, ensuring both sides stay in sync automatically.
- Loading: The Python
Compare Lonboard with ipyleaflet
mainUse ipyleaflet if you need a broad range of data types and fine-grained control over rendering for small-to-medium datasets.
Lonboard is preferred for large datasets because ipyleaflet relies on GeoJSON for data transfer, which is slow to write, read, and large in transit. Lonboard avoids these bottlenecks by using more efficient data transport.
Compare Lonboard with kepler.gl-jupyter
mainUse kepler.gl-jupyter if you want a high-level, browser-based visualization and analysis toolkit, or if you need to create standalone static HTML files containing datasets.
Lonboard is preferred if you perform most analysis in Python and require maximum rendering performance. Lonboard lacks the browser-based UI for data exploration found in kepler.gl but offers better performance because it avoids the large GeoJSON-like JavaScript objects and text serialization used by kepler.gl.
Updating layer styling efficiently
mainLonboard's widget architecture allows for efficient updates. Because accessors (styling properties) are stored separately from the main geometry Arrow table, you can update rendering properties (like color or radius) in isolation. This means the core geometry data does not need to be re-synchronized with the frontend when only the styling changes.Manage memory limits in Pyodide
mainPyodide environments have stricter memory limits than standard Python environments. To prevent memory exhaustion when using Lonboard in the browser, explicitly delete Python objects that are no longer needed using the
delstatement.# Example of manual memory management in Pyodide del large_data_objectHow Lonboard's core architecture works
mainLonboard is designed for Python users who want fast exploratory data analysis without writing JavaScript. It uses a widget-based architecture where a central
Mapclass (ananywidget) synchronizes state with a JavaScript ESM bundle.Key architectural principles include:
- Data Provenance: Unlike pydeck, Lonboard primarily supports data originating in Python. This allows Lonboard to bypass complex JavaScript authentication and focus on optimized binary data transfer.
- Binary Data Transfer: Data is serialized to Parquet in Python and sent over the Jupyter WebSocket. On the frontend, it is parsed from Parquet to Arrow in WebAssembly. The data remains in Arrow binary representation and is never converted to JSON or JavaScript objects, ensuring high performance.
- GeoArrow Integration: Lonboard uses the
@geoarrow/deck.gl-layerslibrary to connect GeoArrow data to deck.gl's low-level binary API. This allows all accessors to be passed as binary buffers.
The Lonboard technology stack
mainLonboard's performance and functionality are built upon four foundational technologies:
- deck.gl: A JavaScript geospatial data visualization library that uses the GPU for high-performance rendering of large datasets.
- GeoArrow: A memory format for efficient, uncompressed representation of geospatial vector data.
- GeoParquet: A file format for efficient encoding and decoding of geospatial vector data using compression.
- anywidget: A framework used to build custom Jupyter widgets.
Lonboard uses GeoPandas as the primary user interface, managing the internal conversion from GeoPandas to GeoArrow and GeoParquet to facilitate data transport to the browser.
Understand Lonboard performance characteristics
mainLonboard performance is split into two independent stages: Data Transfer and Rendering Performance.
Data Transfer
This is the process of moving your GeoDataFrame from the Python environment to the browser.
- Local Python session: Extremely fast (usually < 1 second).
- Remote Python session (e.g., Colab, Binder, JupyterHub): Data must be downloaded to your local browser. Transfer speed is heavily impacted by your internet connection and the size of the data.
- Mechanism: Lonboard uses GeoParquet compression to optimize this transfer.
Rendering Performance
Once data is in the browser, it is rendered using your computer's GPU via the
deck.gllibrary.- No Data Minimization: Unlike
datashader, lonboard attempts to render every coordinate provided. If you pass 10 million coordinates, it will attempt to render all 10 million. - Hardware Dependency: The maximum amount of data you can render depends on your GPU. A powerful GPU allows for larger datasets with less lag.
How the render_tile callback works
mainThe
render_tileparameter inRasterLayer.from_geotiffacts as a custom, client-side tile server running in your Python environment. As you pan or zoom the map, Lonboard automatically calls this callback to fetch and process tiles.Callback Requirements:
- Input: An
async_geotiff.Tileobject. - Output: An instance of
lonboard.raster.EncodedImage. - Capabilities: You can use any Python library (like NumPy or PIL) within the callback to manipulate pixel data, apply band math, or use machine learning models before returning the encoded image.
Supported Formats: While you can return various formats, common choices are PNG, JPEG, or WebP.
- Input: An
How data-driven rendering and accessors work
mainLonboard implements data-driven styling by serializing computed data rather than JavaScript functions.
In other libraries (like pydeck), users often pass strings containing JavaScript callbacks (e.g.,
"@@=properties.value") to define styling. In Lonboard, you use familiar Python objects like Numpy arrays, Pandas Series, or PyArrow Arrays to define accessors.Workflow:
- You compute a property in Python (e.g., using a Matplotlib colormap or an ML model output).
- Lonboard stores this as a binary array/Parquet file.
- The binary array is sent to the frontend.
- The frontend uses the binary buffer directly for rendering.
This approach allows you to use the full power of the Python ecosystem (Matplotlib, Scikit-learn, etc.) for styling without learning a JavaScript DSL.