DuckDB Community Extensions

repository·main·Indexed 43 days ago

https://github.com/duckdb/community-extensions

A collection of third-party, community-maintained extensions for the DuckDB analytical database. Includes specialized functionality such as the raster extension for reading and exporting raster files (RT_Read), PDAL integration for point cloud processing (PDAL_Read, PDAL_Pipeline), Eurostat data retrieval (EUROSTAT_Read), and EEA Reference Grid coordinate conversions.

Tokens
17K
Snippets
57
Records
64
Agent score
56%

What's inside DuckDB Community Extensions

  1. Explore and use DuckDB Community Extensions

    main

    The duckdb/community-extensions repository is a collection of third-party extensions for DuckDB. These extensions provide additional functionality not included in the core DuckDB distribution.

    To find and use these extensions:

    1. Browse the list: Visit the list of Community Extensions to see what is available.
    2. Learn more: Visit the DuckDB Community Extensions website for detailed information on specific extensions.
    3. Understand the difference: Refer to the official vs community extensions guide to understand the distinction between core features and community-contributed ones.
  2. How to use dynamic paths with RT_Read and ST_Read

    main

    A critical constraint in the Raster extension is that RT_Read and ST_Read only accept literal path values at bind time. They do not support column references or subqueries directly as arguments.

    To work around this for dynamic workflows (e.g., using paths discovered via a STAC search), use SET VARIABLE and getvariable(). DuckDB resolves variables to constants before invoking the table function, allowing you to pass arrays of paths dynamically.

  3. Understand the Community Extension release cycle

    main

    Community extensions are built using the C++ extension template and are tightly coupled to specific DuckDB versions because the C++ API is not stable.

    There are two primary ways extensions are released:

    1. Descriptor Updates: Whenever an extension descriptor (e.g., description.yml) is updated in the community extensions repository, the extension is rebuilt targeting the latest stable DuckDB version.
    2. DuckDB Releases: When a new DuckDB version is released, all community extensions are automatically rebuilt to ensure availability alongside the new DuckDB release.

    Note: The standard DuckDB community extension install flow does not support installing historical versions of an extension. If you need to pin a specific version, you must host the artifacts in a custom repository or provide direct URLs.

  4. Perform band algebra on raster databands

    main

    You can perform element-wise arithmetic operations on databands using standard operators (+, -, *, /) or built-in functions like RT_CubeAdd. These operations are applied to every pixel within each tile.

    SELECT
        databand_1 + databand_2 AS sum_bands,
        RT_CubeAdd(databand_1, databand_2) - 1.8 AS sum_func_offset,
        databand_1 * 2 AS scaled_band
    FROM
        RT_Read('./test/data/overlay-sample.tiff');
  5. Update an extension before a DuckDB release

    main

    If your extension's duckdb-next-build is failing (meaning it is incompatible with the upcoming DuckDB version), follow these steps to ensure it is ready for release day:

    1. Create a release branch: In your extension repository, create a branch named vx.y-<codename> (matching the upcoming DuckDB version/codename).
    2. Modify the distribution workflow: In your new branch, disable the duckdb-stable-build job because the extension is currently incompatible with the stable release. Set if: false on that job.
    3. Sync submodule: Check out the DuckDB submodule to the corresponding vx.y-codename branch.
    4. Apply fixes: Commit the workflow changes and apply necessary code fixes to your extension in this branch.
    5. Submit to community-extensions: Create a PR to the community-extensions repository. In your extension's descriptor file, add the repo.ref_next field pointing to the latest commit of your new release branch.
    repo:
      github: duckdb/extension-template
      ref: cfaf3e236008e782d27f4341b0ee036002d0a449
      ref_next: <latest commit of vx.y-codename of your repo>
  6. Complete AOI workflow: Dynamic STAC discovery and NDVI computation

    main

    This workflow demonstrates a full pipeline:

    1. Search: Query a STAC catalog and store results in a temporary table.
    2. Variable Setup: Use SET VARIABLE to create arrays of URLs for specific bands (e.g., nir_urls, red_urls).
    3. Read & Mosaic: Use RT_Read(getvariable(...)) to read all matching tiles as a single mosaic.
    4. Clip & Join: Join the NIR and RED mosaics by their grid coordinates (tile_x, tile_y), then use RT_CubeClip to clip the resulting cubes to the Area of Interest (AOI).
    5. Export: Compute NDVI and export the result as a Cloud-Optimized GeoTIFF (COG).
    -- Part A: Search and store results
    CREATE OR REPLACE TEMP TABLE __search_results AS
        -- ... (STAC search logic) ...
        SELECT
            replace(features->'assets'->'B08'->>'href', 'https://sentinel-cogs.s3.us-west-2.amazonaws.com', '/vsis3/sentinel-cogs') AS nir_url,
            replace(features->'assets'->'B04'->>'href', 'https://sentinel-cogs.s3.us-west-2.amazonaws.com', '/vsis3/sentinel-cogs') AS red_url
        FROM __features;
    
    SET VARIABLE nir_urls = (SELECT list(nir_url) FROM __search_results);
    SET VARIABLE red_urls = (SELECT list(red_url) FROM __search_results);
    
    -- Part B: Read, clip, and compute NDVI
    COPY (
        WITH __aoi AS (SELECT geom FROM ST_Read('./test/data/CATAST_Pol_Township-PNA.geojson')),
        __nir AS (
            SELECT tile_x, tile_y, geometry, metadata, databand_1 AS nir
            FROM RT_Read(getvariable('nir_urls'), blocksize_x := 1024, blocksize_y := 1024)
            WHERE ST_Intersects(geometry, (SELECT geom FROM __aoi))
        ),
        __red AS (
            SELECT tile_x, tile_y, geometry, metadata, databand_1 AS red
            FROM RT_Read(getvariable('red_urls'), blocksize_x := 1024, blocksize_y := 1024)
            WHERE ST_Intersects(geometry, (SELECT geom FROM __aoi))
        ),
        __data AS (
            SELECT
                n.tile_x, n.tile_y, n.geometry, n.metadata,
                RT_CubeClip(n.nir, n.tile_x, n.tile_y, n.metadata, (SELECT geom FROM __aoi), (n.metadata->'bands'->0->'nodata')::DOUBLE) AS nir,
                RT_CubeClip(r.red, r.tile_x, r.tile_y, r.metadata, (SELECT geom FROM __aoi), (r.metadata->'bands'->0->'nodata')::DOUBLE) AS red
            FROM __nir n
            JOIN __red r ON n.tile_x = r.tile_x AND n.tile_y = r.tile_y
        )
        SELECT geometry, RT_Cube2TypeFloat((nir - red) / (nir + red)) AS ndvi FROM __data
    ) 
    TO './test/data/sentinel2_to_ndvi.tiff'
    WITH (
        FORMAT RASTER, DRIVER 'COG', CREATION_OPTIONS ('COMPRESS=LZW'), RESAMPLING 'nearest', SRS 'EPSG:32630',
        COMPUTE_VALID_ENVELOPE true, GEOMETRY_COLUMN 'geometry', DATABAND_COLUMNS ['ndvi']
    );
  7. Install and load a community extension

    main

    To use a DuckDB community extension, you must first install it from the community repository and then load it into your current DuckDB session using the following SQL commands. Replace {{ page.extension.name }} with the specific name of the extension you wish to use.

    INSTALL {{ page.extension.name }} FROM community;
    LOAD {{ page.extension.name }};
  8. Export raster tile footprints to vector formats

    main

    Since RT_Read output includes a geometry column for every tile, you can use the spatial extension to export these footprints to vector formats like GeoParquet or GeoPackage using COPY ... TO.

    -- Export tile footprints to GeoParquet
    COPY (
        SELECT id, x, y, tile_x, tile_y, cols, rows, geometry, metadata
        FROM RT_Read(['file1.tif', 'file2.tif'])
        WHERE NOT RT_CubeNullOrEmpty(databand_1)
    )
    TO './output.parquet'
    WITH (
        FORMAT PARQUET, GEOPARQUET_VERSION 'V1'
    );
    
    -- Export tile footprints to GeoPackage
    LOAD spatial;
    COPY (
        SELECT id, x, y, tile_x, tile_y, cols, rows, geometry, metadata
        FROM RT_Read(['file1.tif', 'file2.tif'])
        WHERE NOT RT_CubeNullOrEmpty(databand_1)
    )
    TO './output.gpkg'
    WITH (
        FORMAT GDAL, DRIVER 'GPKG', SRS 'EPSG:32630'
    );
  9. Export query results to raster formats

    main

    Use the COPY ... TO statement with FORMAT RASTER to write data to a file.

    Required/Common options:

    • DRIVER: Specifies the output format (e.g., 'PNG', 'COG', 'GTiff').
    • DATABAND_COLUMNS: An array of column names defining which columns are written as bands and their order.
    • GEOMETRY_COLUMN: The name of the column containing spatial geometry.
    • SRS: The Spatial Reference System (e.g., 'EPSG:25830').
    • CREATION_OPTIONS: Driver-specific options (e.g., 'COMPRESS=LZW').
    -- Export to PNG
    COPY (
        SELECT * FROM RT_Read('./test/data/overlay-sample.tiff')
    )
    TO './test/data/copy_to_raster.png'
    WITH (
        FORMAT RASTER,
        DRIVER 'PNG',
        CREATION_OPTIONS ('WORLDFILE=YES'),
        RESAMPLING 'nearest',
        SRS 'EPSG:25830',
        GEOMETRY_COLUMN 'geometry',
        DATABAND_COLUMNS ['databand_3', 'databand_2', 'databand_1']
    );
    
    -- Export to Cloud-Optimized GeoTIFF (COG)
    COPY (
        SELECT * FROM RT_Read(['file1.tif', 'file2.tif'])
    )
    TO './test/data/output.tif'
    WITH (
        FORMAT RASTER,
        DRIVER 'COG',
        CREATION_OPTIONS ('COMPRESS=LZW'),
        RESAMPLING 'nearest',
        SRS 'EPSG:25830',
        GEOMETRY_COLUMN 'geometry',
        DATABAND_COLUMNS ['databand_1']
    );
  10. Prerequisites for Raster Extension workflows

    main

    To perform Earth Observation analysis or complex raster workflows, you must ensure the json, spatial, and http_client extensions are loaded. The http_client extension may need to be installed from the community repository.

    Run the following commands once per session:

    LOAD json;
    LOAD spatial;
    INSTALL http_client FROM community;
    LOAD http_client;
  11. Filter raster tiles by spatial location

    main

    You can filter the tiles returned by RT_Read using spatial intersection tests. This requires the spatial extension.

    Performance Tip: For large datasets, filter using the bbox column (which contains xmin, ymin, xmax, ymax) instead of the geometry column. Bounding-box filtering is faster because it avoids the overhead of deserializing full geometry objects.

    -- Method 1: Intersection with geometry (requires spatial extension)
    LOAD spatial;
    SELECT x, y, bbox, geometry
    FROM RT_Read('path/to/raster/file.tif')
    WHERE ST_Intersects(geometry, ST_GeomFromText('POLYGON((...)))'));
    
    -- Method 2: Fast bounding-box filtering (preferred for performance)
    SELECT *
    FROM RT_Read('./test/data/overlay-sample.tiff')
    WHERE bbox.xmin > 545500.0 AND bbox.xmax < 545800.0
      AND bbox.ymin > 4724400.0 AND bbox.ymax < 4724600.0;
  12. Read raster files from S3

    main

    You can read remote raster files from S3 using two methods:

    1. GDAL Virtual File System: Use the /vsis3/ prefix. For public buckets, you must first disable signature checking using RT_GdalConfig('AWS_NO_SIGN_REQUEST', 'YES').
    2. Native DuckDB S3: Use the s3:// protocol (requires the httpfs extension for authentication).
    -- Method 1: GDAL /vsis3/ (for public buckets)
    SELECT RT_GdalConfig('AWS_NO_SIGN_REQUEST', 'YES');
    SELECT * FROM RT_Read('/vsis3/sentinel-cogs/sentinel-s2-l2a-cogs/30/T/WN/2021/9/S2B_30TWN_20210930_0_L2A/B02.tif') LIMIT 5;
    
    -- Method 2: Native DuckDB S3
    SELECT * FROM RT_Read('s3://sentinel-cogs/sentinel-s2-l2a-cogs/30/T/WN/2021/9/S2B_30TWN_20210930_0_L2A/B02.tif') LIMIT 5;