mago 3DTiler

repository·main·Indexed 43 days ago

https://github.com/gaia3d/mago-3d-tiler

An open-source OGC 3D Tiles generator that converts spatial data formats—including 3D models, point clouds, and 2D GIS data—into OGC 3D Tiles for Digital Twin services. It supports a wide range of input types such as KML, 3DS, FBX, OBJ, glTF/glb, LAS/LAZ, CityGML, IndoorGML, SHP, GeoJSON, and GPKG, with output types including b3dm, i3dm, and pnts.

Tokens
16K
Snippets
47
Records
72
Agent score
56%

What's inside mago 3DTiler

  1. Run mago 3DTiler directly with Java

    main

    You can execute the tool directly using the .jar file.

    Requirements:

    • JDK 21 is required.

    Basic Execution

    Use the --input (-i) and --output (-o) flags to specify data paths.

    # Using long options
    java -jar mago-3d-tiler.jar --input "/data/input/sample" --output "/data/output/sample"
    
    # Using short options
    java -jar mago-3d-tiler.jar -i "/data/input/sample" -o "/data/output/sample"
    java -jar mago-3d-tiler.jar --input "/data/input/sample" --output "/data/output/sample"
  2. Run mago 3DTiler using Docker

    main

    Use the Docker image to run mago 3DTiler without installing Java or other local dependencies. It is recommended to mount a local workspace directory to the container to persist input and output data.

    Steps:

    1. Pull the image: docker pull gaia3d/mago-3d-tiler
    2. Run the container with volume mounting: Map a local directory (e.g., /workspace) to the container to allow access to your files.
    # Pull the image
    docker pull gaia3d/mago-3d-tiler
    
    # Run the image
    docker run --rm -v "/workspace:/workspace" gaia3d/mago-3d-tiler -input /workspace/3ds-samples -output /workspace/sample-3d-tiles -crs 5186
  3. Integrate mago-common using git subtree

    main

    The mago-common library is designed to be integrated into other Gaia3D projects (like mago-io) using git subtree. This approach allows you to include shared code and resources directly in your repository while maintaining the ability to pull updates or push changes back to the central mago-common repository.

    To manage the integration, follow these steps to add the remote, add the subtree, and synchronize changes.

    ### Add the remote
    ```bash
    git remote add mago-common https://github.com/Gaia3D/mago-common.git

    Add the subtree

    git subtree add --prefix=mago-common https://github.com/Gaia3D/mago-common.git main

    Pull updates from mago-common

    git subtree pull --prefix=mago-common https://github.com/Gaia3D/mago-common.git main

    Push local changes to mago-common

    git subtree push --prefix=mago-common https://github.com/Gaia3D/mago-common.git main
  4. Tune Docker memory for large datasets

    main

    For large meshes, point clouds, CityGML, or photogrammetry, you must set both the container memory limit and the JVM options. Java automatically reads the JAVA_TOOL_OPTIONS environment variable.

    Fixed Memory Allocation

    Use --memory to set the container limit and JAVA_TOOL_OPTIONS to set the heap size (e.g., -Xmx16g).

    Percentage-based Memory Allocation

    Use --memory for the container limit and JAVA_TOOL_OPTIONS with InitialRAMPercentage and MaxRAMPercentage for relative sizing.

    # Example using fixed values
    docker run --rm --memory=24g \
      -e JAVA_TOOL_OPTIONS="-Xms4g -Xmx16g -XX:+UseG1GC -XX:MaxGCPauseMillis=200 -XX:+UseStringDeduplication" \
      -v "/workspace:/workspace" \
      gaia3d/mago-3d-tiler \
      --input /workspace/3ds-samples --output /workspace/sample-3d-tiles --inputType 3ds --crs 5186
    docker run --rm --memory=24g \
      -e JAVA_TOOL_OPTIONS="-Xms4g -Xmx16g -XX:+UseG1GC -XX:MaxGCPauseMillis=200 -XX:+UseStringDeduplication" \
      -v "/workspace:/workspace" \
      gaia3d/mago-3d-tiler \
      --input /workspace/3ds-samples --output /workspace/sample-3d-tiles --inputType 3ds --crs 5186
  5. Tune JVM memory for Java execution

    main

    Large datasets require significant Java heap space. If you see the warning Maximum memory is less than the recommended 16GB, you should increase the heap size using -Xmx.

    For optimal performance with large jobs, use the following flags:

    java -Xms4g -Xmx16g -XX:+UseG1GC -XX:MaxGCPauseMillis=200 -XX:+UseStringDeduplication -Dfile.encoding=UTF-8 -Djava.awt.headless=true -jar mago-3d-tiler.jar --input "/data/input/sample" --output "/data/output/sample"

    Troubleshooting Out-of-Memory (OOM)

    If the process fails due to memory, add these flags to generate a heap dump for analysis: -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=./heap-dumps

  6. Integrate mago-io using git subtree

    main

    The mago-io library is a shared resource used across multiple Gaia3D projects (like mago-io and mago-common) to ensure consistency. Instead of being managed as a standard dependency, it is designed to be integrated into other repositories using git subtree. This allows you to include the library's code directly in your project's file tree while maintaining the ability to pull updates or push changes back to the original repository.

    ### [remote]
    
    ```bash
    git remote add mago-io https://github.com/Gaia3D/mago-io.git

    [add subtree]

    git subtree add --prefix=mago-io https://github.com/Gaia3D/mago-io.git main

    [pull]

    git subtree pull --prefix=mago-io https://github.com/Gaia3D/mago-io.git main

    [push]

    git subtree push --prefix=mago-io https://github.com/Gaia3D/mago-io.git main
  7. Build mago 3DTiler from source

    main

    To build the JAR file from the source code, use Gradle.

    Steps:

    1. Clone the repository: git clone https://github.com/Gaia3D/mago-3d-tiler.git
    2. Navigate to the project directory.
    3. Execute the Gradle jar task: gradle jar
    git clone https://github.com/Gaia3D/mago-3d-tiler.git
    cd mago-3d-tiler
    gradle jar
  8. Run mago 3DTiler with Docker Compose

    main

    Use Docker Compose to manage the container configuration, including volume mappings and memory limits. Create a docker-compose.yml file as shown below. Note that you should replace C:/input and D:/output with your actual local paths and update the --input and --output command arguments to match the container's internal paths.

    version: '3.8'
    name: mago-3d-tiler
    services:
      mago-tiler:
        image: gaia3d/mago-3d-tiler:latest
        pull_policy: always
        platform: linux/amd64
        container_name: mago-tiler
        mem_limit: 24g
        environment:
          JAVA_TOOL_OPTIONS: "-Xms4g -Xmx16g -XX:+UseG1GC -XX:MaxGCPauseMillis=200 -XX:+UseStringDeduplication"
        volumes:
          - C:/input:/input
          - D:/output:/output
        command: >
          --input /input/INPUT_DATA_DIR
          --output /output/OUTPUT_DATA_DIR
  9. Run mago 3DTiler using the JAR file

    main

    You can run mago 3DTiler directly using the compiled JAR file.

    Prerequisite:

    • JDK 21 is required to run the application.

    Usage: Provide the -input and -output arguments to specify the source data directory and the destination for the generated 3D Tiles.

    java -jar mago-3d-tiler-x.x.x.jar -input C:\data\kml-input-dir -output C:\data\kml-output-dir
  10. Photogrammetry Process Flow Overview

    main

    The PhotogrammetryProcessFlow is a specialized tiling pipeline designed for processing photogrammetry meshes. It automates a sequence of preprocessing, tiling, and postprocessing steps to convert raw photogrammetry data into optimized 3D Tiles.

    Key components of this flow include:

    • Input Conversion: Uses an AssimpConverter to handle mesh formats, with support for splitting by node.
    • Terrain Integration: Supports height providers via QuantizedMeshTerrainHeightProvider (if a layer.json is provided) or GeoTiffTerrainHeightProvider (for GeoTIFF files).
    • Geoid Support: Can incorporate geoid data to assist in coordinate transformations.
    • Preprocessing Pipeline: Automatically applies several critical transformations:
      • TileInfoGenerator: Generates metadata for tiles.
      • GaiaTexCoordCorrection: Corrects texture coordinates.
      • GaiaScaler: Scales the model.
      • PhotogrammetryRotation: Handles rotation requirements.
      • GaiaTranslationForPhotogrammetry: Translates the model based on terrain and geoid data.
      • PhotogrammetryMinimization: Minimizes errors/artifacts in the photogrammetry mesh.
    • Tiling: Uses the PhotogrammetryTiler to perform the actual spatial partitioning and tiling.
    • Postprocessing: Currently, the photogrammetry flow does not implement additional post-processing steps.
  11. Configure Terrain and Geoid for Photogrammetry Tiling

    main

    When running the photogrammetry process, you can provide terrain and geoid data via global options to ensure accurate spatial positioning and translation.

    Terrain Configuration

    • Quantized Mesh: If the path provided to the terrain option points to a file named layer.json, the system uses QuantizedMeshTerrainHeightProvider.
    • GeoTIFF: If the path points to other files, the system loads them as GridCoverage2D objects and uses GeoTiffTerrainHeightProvider.

    Geoid Configuration

    • You can provide a path to geoid files to be loaded via the fileLoader, which are then used by the GaiaTranslationForPhotogrammetry preprocessor to ensure the mesh is correctly translated to the appropriate elevation model.
  12. Supported Output Formats and Process Models

    main

    Mago3DTiler selects a specific ProcessFlow based on the requested FormatType. The following mappings determine the internal processing logic:

    Output FormatProcess ModelNotes
    I3DMInstancedProcessFlowinstanced=false
    FORESTInstancedProcessFlowinstanced=true
    PHOTOGRAMMETRYPhotogrammetryProcessFlow
    B3DMPhotogrammetryProcessFlowIf isPhotogrammetry is true
    B3DMBatchedModelProcessFlowIf isPhotogrammetry is false
    PNTSPointCloudProcessFlow

    If an unsupported outputFormat is provided, the application throws an IllegalArgumentException.