GeoTessera

GeoTessera streams manifests, embedding tiles, and landmasks via HTTPS from the Source Cooperative repository. It uses a urllib3 connection pool for efficient downloads and implements several reliability features:

• Per-output-dir mirroring: Tiles are saved to the directory specified by --output (or embeddings_dir) and persist for reuse.
• Connection pooling: Reuses connections for thousands of per-tile GET requests.
• Retries: Uses exponential backoff for rate limiting, server errors, and dropped connections (honouring Retry-After).
• Integrity checking: Validates response integrity via ETag (MD5) and raises errors on truncated responses.
• Conditional caching: Uses If-Modified-Since for manifests to minimize bandwidth.
• Resumable downloads: Skips existing files in the output directory on rerun.
• Progress callbacks: Provides real-time feedback on download speed and size.

The GeoTesseraZarr class provides cloud-native access to embeddings using the geoemb: convention in Zarr v3 format. Data is organized by UTM zone (e.g., utm30/, utm31/).

Key Access Patterns

• Point sampling: Use sample_points() or sample_at() to extract embeddings at specific coordinates across zones.
• Region reading:
  • read_region(): Loads rectangular areas as mosaics with CRS and transform metadata.
  • read_region_quantized(): Returns int8 values and scales (uses 1/4 the bytes).
• Streaming: iter_region() yields a region as row strips, suitable for inference on regions larger than memory.
• Patch reading: read_patch() reads fixed-size squares centered on a point, merging across UTM zones if necessary.
• Matryoshka Depths: Use the depth= parameter on read methods to select a specific prefix array on stores that declare one.
• Zone access: open_zone() returns an xarray Dataset with a .tessera accessor.
• Diagnostics: Use probe() to check a location's status. It returns (embedding, status), where status can be valid, water, nodata, or outside. Use this instead of checking for NaNs to distinguish between water and missing data.

The tile workflow involves two main steps: retrieving raw embeddings and exporting them to a usable format.

1. Retrieve embeddings: GeoTessera downloads quantized int8 embeddings (grid_X.XX_Y.YY.npy) and their corresponding float32 scale files (grid_X.XX_Y.YY_scales.npy) via HTTPS.
2. Dequantization: The quantized values are converted back to float32 using geotessera.dequantize_embedding(quantized_embedding, scales). This helper handles scale broadcasting and missing values.

Tile Grid Details:

• Grid System: 0.1-degree grid.
• Tile Size: ~11km x 11km at the equator.
• Naming: Tiles are named by their center coordinates (e.g., grid_0.15_52.05).
• Bounds: A tile at (lon, lat) covers [lon - 0.05°, lon + 0.05°] and [lat - 0.05°, lat + 0.05°].
• Resolution: 10m per pixel.

import geotessera
# Example of manual dequantization if you have the arrays
# dequantized = geotessera.dequantize_embedding(quantized_array, scales_array)

Tessera embeddings are organized on a fixed 0.1-degree grid system. Understanding this grid is essential for calculating tile boundaries and identifying valid tile centers.

Grid Properties:

• Grid spacing: 0.1° latitude × 0.1° longitude.
• Tile naming: Tiles are named by their center coordinates (e.g., grid_0.15_52.05).
• Coverage: Each tile spans from (center - 0.05°) to (center + 0.05°).
• Resolution: Approximately 11km × 11km at the equator.

Coordinate Calculations: For a tile at center coordinates (lon, lat):

• west = lon - 0.05
• east = lon + 0.05
• south = lat - 0.05
• north = lat + 0.05

Valid Tile Centers: Centers must be aligned to 0.1-degree boundaries. Examples of valid centers include (0.05, 52.05), (0.15, 52.05), and (-0.05, 51.95). Values like (0.07, 52.03) are invalid as they are off-grid.

# For a tile at center coordinates (lon, lat)
west = lon - 0.05
east = lon + 0.05  
south = lat - 0.05
north = lat + 0.05

# Valid tile centers (examples)
valid_centers = [
    (0.05, 52.05),   # Northwest Europe
    (0.15, 52.05),   # Adjacent tile
    (-0.05, 51.95),  # Southwest tile
]

# Invalid centers (not on grid)
invalid_centers = [
    (0.07, 52.03),   # Off-grid
    (0.1, 52.1),     # Off by 0.05°
]

Tessera embeddings are organized into a 0.1-degree grid system.

Grid Geometry

• Tile size: ~0.1° × 0.1° (approx. 11km × 11km at the equator).
• Naming: Tiles are named by their center coordinates (e.g., grid_0.15_52.05).
• Bounds: A tile at center (lon, lat) covers [lon ± 0.05°, lat ± 0.05°].
• Resolution: 10m per pixel.

Data Format and Dequantization

Embeddings are stored in a quantized format to save space. To use them, you must dequantize them:

1. Quantized embeddings (grid_X.XX_Y.YY.npy): int8 type, shape (height, width, 128).
2. Scale files (grid_X.XX_Y.YY_scales.npy): float32 type, contains scale factors.
3. Dequantization formula: final_embedding = quantized_embedding.astype('f4') * scales.

Storage and Manifests

• Manifests: A Parquet-based per-version manifest is used for fast spatial lookups. These are cached in ~/.cache/geotessera/.
• Persistence: Downloaded tiles are stored in your specified --output directory and are resumable (skipped if they already exist).