Lonboard

repository·main·Indexed 21 days ago

https://github.com/developmentseed/lonboard

A Python library for high-performance, interactive geospatial data visualization in Jupyter environments. Built on GeoArrow and GeoParquet, it leverages deck.gl for GPU-accelerated rendering of large datasets. It provides a simple interface via the viz() function for GeoPandas GeoDataFrames and supports efficient data transport using Apache Arrow.

Tokens
38.1K
Snippets
134
Records
185
Agent score
75%

What's inside lonboard

  1. How Lonboard's Python and JavaScript components interact

    main

    Lonboard follows an architecture where all TypeScript models are combined into a single entry point. This entry point is compiled by ESBuild into an ES Module.

    • Loading: The Python Map class loads this JavaScript bundle via the _esm key, which informs Jupyter/ipywidgets where to find the bundle.
    • Synchronization: anywidget and ipywidgets handle the serialization of data from Python to JavaScript, ensuring both sides stay in sync automatically.
  2. Compare Lonboard with ipyleaflet

    main

    Use ipyleaflet if you need a broad range of data types and fine-grained control over rendering for small-to-medium datasets.

    Lonboard is preferred for large datasets because ipyleaflet relies on GeoJSON for data transfer, which is slow to write, read, and large in transit. Lonboard avoids these bottlenecks by using more efficient data transport.

  3. Compare Lonboard with kepler.gl-jupyter

    main

    Use kepler.gl-jupyter if you want a high-level, browser-based visualization and analysis toolkit, or if you need to create standalone static HTML files containing datasets.

    Lonboard is preferred if you perform most analysis in Python and require maximum rendering performance. Lonboard lacks the browser-based UI for data exploration found in kepler.gl but offers better performance because it avoids the large GeoJSON-like JavaScript objects and text serialization used by kepler.gl.

  4. Updating layer styling efficiently

    main
    Lonboard's widget architecture allows for efficient updates. Because accessors (styling properties) are stored separately from the main geometry Arrow table, you can update rendering properties (like color or radius) in isolation. This means the core geometry data does not need to be re-synchronized with the frontend when only the styling changes.
  5. Manage memory limits in Pyodide

    main

    Pyodide environments have stricter memory limits than standard Python environments. To prevent memory exhaustion when using Lonboard in the browser, explicitly delete Python objects that are no longer needed using the del statement.

    # Example of manual memory management in Pyodide
    del large_data_object
  6. How Lonboard's core architecture works

    main

    Lonboard is designed for Python users who want fast exploratory data analysis without writing JavaScript. It uses a widget-based architecture where a central Map class (an anywidget) synchronizes state with a JavaScript ESM bundle.

    Key architectural principles include:

    • Data Provenance: Unlike pydeck, Lonboard primarily supports data originating in Python. This allows Lonboard to bypass complex JavaScript authentication and focus on optimized binary data transfer.
    • Binary Data Transfer: Data is serialized to Parquet in Python and sent over the Jupyter WebSocket. On the frontend, it is parsed from Parquet to Arrow in WebAssembly. The data remains in Arrow binary representation and is never converted to JSON or JavaScript objects, ensuring high performance.
    • GeoArrow Integration: Lonboard uses the @geoarrow/deck.gl-layers library to connect GeoArrow data to deck.gl's low-level binary API. This allows all accessors to be passed as binary buffers.
  7. The Lonboard technology stack

    main

    Lonboard's performance and functionality are built upon four foundational technologies:

    • deck.gl: A JavaScript geospatial data visualization library that uses the GPU for high-performance rendering of large datasets.
    • GeoArrow: A memory format for efficient, uncompressed representation of geospatial vector data.
    • GeoParquet: A file format for efficient encoding and decoding of geospatial vector data using compression.
    • anywidget: A framework used to build custom Jupyter widgets.

    Lonboard uses GeoPandas as the primary user interface, managing the internal conversion from GeoPandas to GeoArrow and GeoParquet to facilitate data transport to the browser.

  8. Understand Lonboard performance characteristics

    main

    Lonboard performance is split into two independent stages: Data Transfer and Rendering Performance.

    Data Transfer

    This is the process of moving your GeoDataFrame from the Python environment to the browser.

    • Local Python session: Extremely fast (usually < 1 second).
    • Remote Python session (e.g., Colab, Binder, JupyterHub): Data must be downloaded to your local browser. Transfer speed is heavily impacted by your internet connection and the size of the data.
    • Mechanism: Lonboard uses GeoParquet compression to optimize this transfer.

    Rendering Performance

    Once data is in the browser, it is rendered using your computer's GPU via the deck.gl library.

    • No Data Minimization: Unlike datashader, lonboard attempts to render every coordinate provided. If you pass 10 million coordinates, it will attempt to render all 10 million.
    • Hardware Dependency: The maximum amount of data you can render depends on your GPU. A powerful GPU allows for larger datasets with less lag.
  9. How the render_tile callback works

    main

    The render_tile parameter in RasterLayer.from_geotiff acts as a custom, client-side tile server running in your Python environment. As you pan or zoom the map, Lonboard automatically calls this callback to fetch and process tiles.

    Callback Requirements:

    • Input: An async_geotiff.Tile object.
    • Output: An instance of lonboard.raster.EncodedImage.
    • Capabilities: You can use any Python library (like NumPy or PIL) within the callback to manipulate pixel data, apply band math, or use machine learning models before returning the encoded image.

    Supported Formats: While you can return various formats, common choices are PNG, JPEG, or WebP.

  10. How data-driven rendering and accessors work

    main

    Lonboard implements data-driven styling by serializing computed data rather than JavaScript functions.

    In other libraries (like pydeck), users often pass strings containing JavaScript callbacks (e.g., "@@=properties.value") to define styling. In Lonboard, you use familiar Python objects like Numpy arrays, Pandas Series, or PyArrow Arrays to define accessors.

    Workflow:

    1. You compute a property in Python (e.g., using a Matplotlib colormap or an ML model output).
    2. Lonboard stores this as a binary array/Parquet file.
    3. The binary array is sent to the frontend.
    4. The frontend uses the binary buffer directly for rendering.

    This approach allows you to use the full power of the Python ecosystem (Matplotlib, Scikit-learn, etc.) for styling without learning a JavaScript DSL.