Mapshaper Documentation

repository·master·Indexed 26 days ago

https://github.com/mbloch/mapshaper

A JavaScript-based tool for editing geospatial data formats including Shapefile, GeoJSON, TopoJSON, and CSV. Mapshaper supports simplifying shapes, attribute editing, clipping, erasing, dissolving, and filtering. It provides a standard CLI, a high-memory version (mapshaper-xl) for large files, and a local web interface (mapshaper-gui).

Tokens
82.7K
Snippets
130
Records
475
Agent score
70%

What's inside mapshaper

  1. Understand raster import behavior and limitations

    master

    Supported Features

    • Single-band (grayscale) and multi-band (RGB/RGBA) rasters.
    • Stripped and tiled storage.
    • Common integer and float sample types.
    • BigTIFF support (via geotiff.js).

    Limitations

    • No Reprojection on Import: Reprojection is handled later via the -proj command or during display preview generation.
    • Unsupported Formats: OME-TIFF, CMYK, CIELab, YCbCr, and certain compressions like ZSTD or LERC are currently limited or unsupported.
    • Sidecars: .aux.xml (GDAL PAM) files are currently ignored; essential metadata should be contained within the GeoTIFF itself.
  2. Understand the polyline buffer construction architecture

    master

    Mapshaper builds polyline buffers in two primary stages:

    1. Construction (src/buffer/):

      • Paths are pre-simplified using Douglas-Peucker based on the tolerance= option (defaults to 1% of the radius). Setting tolerance=0 disables simplification.
      • Two-sided buffers: Uses a fast-path (makeTwoSidedOutlineRing) to create a single closed, potentially self-intersecting ring per path. This is 4-6x faster than per-section rings.
      • One-sided buffers: Defaults to a winding fill with dip-to-vertex concave joins. This replaces the older per-section + audit path for left or right side buffers.
      • Per-section rings: Used for one-sided builds and the band-method escape hatch.
    2. Dissolve (dissolveBufferDataset2 in mapshaper-buffer-common.mjs):

      • Uses addIntersectionCuts, MosaicIndex, and per-shape pathfinding to resolve the union of the constructed rings.
      • For polyline buffers, it runs with per_part_holes to ensure that reverse-wound pockets in self-intersecting rings do not incorrectly block the hole-detection process for other overlapping rings.
  3. Import Shapefiles in the Mapshaper Web App

    master

    Since browsers cannot access the local filesystem like the CLI, you must provide all Shapefile components together using one of two methods:

    1. Select all components: Use Add files to select the .shp, .dbf, .prj, and .shx/.cpg files simultaneously (using Shift+Click or Cmd+Click), or drag the entire selection into the import area.
    2. Drop a .zip file: Drag and drop a .zip archive containing the complete Shapefile bundle.

    Handling encoding errors in the web app: If you see an encoding warning, re-import the files, tick the with advanced options checkbox, and specify the encoding (e.g., encoding=win1252).

  4. GeoJSON Precision and Winding Best Practices

    master

    When working with GeoJSON in Mapshaper, keep these technical details in mind:

    • Precision: Coordinates are emitted at full precision by default. Use precision= to reduce file size. Note that precision=0.0001 equates to roughly 6-11 meters depending on location. High rounding can introduce sliver overlaps; consider pairing precision= with fix-geometry.
    • Winding Order: Mapshaper follows RFC 7946 by default (Counter-Clockwise outer rings, Clockwise holes). Use reverse-winding if you need Clockwise outer rings.
    • Coordinate Systems: While GeoJSON typically uses WGS-84, Mapshaper can also export GeoJSON with projected coordinates.
    • File Size: For datasets with shared boundaries, consider using TopoJSON instead of GeoJSON to significantly reduce file size.
  5. Test a New Command

    master

    When adding a new command, follow these testing patterns based on the level of risk:

    1. Transaction Unit Tests

    Use test/undo-transaction-test.mjs to assert that the captured unit type is correct and that restore() successfully returns the object to its previous state. Tip: Assert what was NOT captured (e.g., a metadata change should produce layer-metadata, not layer).

    2. Payload Store Tests

    • Use test/gui-undo-unit-store-test.mjs to test field storage and packing.
    • Use test/gui-undo-payload-store-test.mjs to test storage limits, cleanup, and lifecycle behavior.

    3. Browser Command Tests

    Add coverage to browser-tests/undo-console.spec.mjs. For standard editing commands, add an entry to COMMAND_CASES:

    {
      name: 'my command',
      command: 'my-command option=value',
      fixture: POLYGON_FIXTURE
    }

    To assert specific granularity, use payloadTypes and noPayloadTypes:

    {
      name: 'field-only command',
      command: 'my-field-command',
      payloadTypes: ['table-fields'],
      noPayloadTypes: ['table']
    }

    Run browser tests manually using:

    npm run test:browser -- browser-tests/undo-console.spec.mjs

    4. Performance Testing

    Use scripts/undo-performance-runner.mjs when adding hooks that capture large tables, layers, or arcs.

  6. Simplify multiple layers consistently using shared topology

    master

    To ensure shared boundaries (like state and county borders) are simplified identically, you must build a shared topology.

    Method 1: Combine files directly Use -i combine-files to import multiple layers. This works if vertices are perfectly aligned.

    Method 2: Dissolve (Recommended) Because source layers are rarely perfectly aligned, the most reliable approach is to import the smallest geographic level (e.g., counties) and derive higher levels (e.g., states) using the -dissolve command before simplifying.

    Web UI Note: To achieve shared topology in the web app, tick with advanced options in the import dialog and add combine-files to the options field.

  7. Edit attributes with -each and -rename-fields

    master

    Modify feature attributes using the following commands:

    • Add/Update fields: Use -each to run a JavaScript expression on every feature. Assigning to a new name creates a new field (e.g., -each 'NEW_FIELD = OLD_FIELD * 2').
    • Rename fields: Use -rename-fields with NEW=OLD pairs.
    • Filter/Reorder fields: Use -filter-fields to keep only specific columns in a specific order.

    Note on CSV leading zeros: To prevent Mapshaper from parsing identifier columns (like FIPS or ZIP) as numbers and dropping leading zeros, use string-fields=FIELD_NAME during import.

    -each 'STATE_FIPS = COUNTY_FIPS.substr(0, 2), AREA_KM2 = round(this.area / 1e6, 1)'
    -rename-fields POPULATION=POP,MEDIAN_INCOME=MEDIAN_INC \
      -filter-fields STATE,COUNTY,POPULATION,MEDIAN_INCOME
    # CLI Import example for leading zeros:
    mapshaper -i counties.csv string-fields=FIPS,STATEFIPS
  8. Checklist for Command Authors

    master

    When implementing a new command, ensure you complete the following steps:

    1. Identify Mutations: List every object the command mutates in place.
    2. Capture State: Capture each object before its first mutation and mark it after mutation.
    3. Optimize Granularity: Prefer narrow units like record, field, schema, order, metadata, simplification, or info over broad table or layer units.
    4. Verify Transactions: Add or update transaction tests for the new granularity.
    5. Browser Testing: Add a browser command case for console-visible editing commands. Include payloadTypes or noPayloadTypes assertions if granularity is critical.
    6. Run Tests: Execute focused unit tests and npm run test:browser -- browser-tests/undo-console.spec.mjs, then run npm test before committing.
  9. Import GeoTIFF, PNG, and JPEG files

    master

    Mapshaper supports importing georeferenced raster files via an asynchronous import path:

    • GeoTIFF: Supports .tif and .tiff files. Large files automatically use an overview or resized rendition unless rendition=full is specified.
    • PNG/JPEG: Supports .png, .jpg, and .jpeg.
    • Georeferencing:
      • For PNG/JPEG, Mapshaper recognizes world-file sidecars (e.g., .pgw, .jgw, .tfw, .wld).
      • For PNG/JPEG, it can read .prj sidecars to populate the dataset's WKT1 projection.
    • CRS: Raster CRS metadata is stored at the dataset level in dataset.info (using crs, crs_string, and wkt1). Per-layer placement metadata (pixel-to-map transform) is stored on the raster layer itself.
  10. Run the Mapshaper web UI locally

    master

    You can run the Mapshaper web interface on your own machine using the mapshaper-gui command (installed with the global mapshaper package).

    # Start the server at http://localhost:5555
    mapshaper-gui
    
    # Start the server on a specific port
    mapshaper-gui --port 8080
    
    # Start the server and pre-load specific files
    mapshaper-gui states.shp rivers.shp
    mapshaper-gui states.shp rivers.shp