DOCSEARCHbyAlgolia
Submit Docs

Virtual Community

repository·master·Indexed 41 days ago

https://github.com/umass-embodied-agi/virtual-community

An open-world simulation platform for multi-agent embodied AI research, designed to study human-robot coexistence using physics-based simulators and real-world 3D scenes. Version 0.1.0 includes a scene generation pipeline for transforming 3D tiles and OSM data into .glb models, a typed Blender API wrapper (BlenderLib) for large-scale development, and tools for object replacement using SAM.

Tokens
13.8K
Snippets
55
Records
66
Agent score
52%

What's inside Virtual Community

  1. Overview of the Scene Generation Pipeline stages

    master

    The pipeline consists of 8 distinct stages (0 through 7) that transform raw 3D tiles and OSM data into exported .glb models:

    • Stage 0 (Preprocessing): Fetches, merges, and aligns Google 3D tiles, street view metadata, and OSM data.
    • Stage 1 (Preliminary Steps): Fetches OSM geometry via the Blosm plugin (Overpass API) and constructs terrain geometry.
    • Stage 2 (Texture Transfer): Transfers textures from masked 3D tiles to the constructed terrain and OSM geometry.
    • Stage 3 (Texture Enhancement): Uses street view images to enhance building textures. This includes solving mappings, fetching images, and performing inpainting (using Lang_SAM to remove trees/pedestrians/vehicles) before projecting images onto buildings.
    • Stage 4 (Combine Building and Terrain): Merges buildings, roofs, and terrain into a single Blender file.
    • Stage 5 (Upscale the scene): Applies super-resolution to building and terrain textures.
    • Stage 6 (Retrieve Map Annotations): Extracts building names and 3D AABB (Axis-Aligned Bounding Box) JSON annotations from OSM.
    • Stage 7 (Postprocess and Export): Converts emissive textures to basic textures and exports the final models as building.glb, terrain.glb, and roofs.glb.

  2. Overview of the Typed Blender API

    master
    The BlenderLib module provides an Object-Oriented Programming (OOP) based wrapper for the Blender API. While the standard Blender API is suitable for small scripts, it lacks type information and does not follow OOP principles, making large-scale projects difficult to maintain. BlenderLib acts as a proxy layer to common Blender objects, providing a structured, type-safe interface designed for large-scale development and easier maintenance.

  4. Prepare Virtual Community assets

    master

    Assets (scenes, robots, objects, avatars, and cars) are hosted on HuggingFace and are automatically downloaded on the first run.

    To download them manually in advance, use the download_all_assets utility from vico.tools.asset_utils.

    If you want to use indoor scenes from GRUTopia, you must manually download commercial_scenes.zip following the GRUtopia instructions and unzip it into vico/assets/ViCo/scene/.

    python -c "from vico.tools.asset_utils import download_all_assets; download_all_assets()"

  5. Run the Scene Generation Pipeline

    master

    Execute the pipeline using the pipeline_unity.sh script. You must provide a radius, a data root directory, a cache root directory, and a specific datapoint file.

    Note on Environment: The Inpainting Street Views stage cannot run in the same environment as the main pipeline. It must be executed separately in a different environment.

    ./pipeline_unity.sh --radius 400 --dataroot data/ --cacheroot cache/ --datapoint datapoint/MIT.txt

  6. Run the simulation tour agent

    master

    To test the simulation with 15 agents touring the New York Scene, run the run_tour_agent.sh script.

    Note on Memory: Loading all indoor scenes and objects requires approximately 40GB of RAM. If you encounter memory issues, add the --no_load_indoor_scene flag to the script to disable indoor scenes.

    ./scripts/run_tour_agent.sh

  7. Install Virtual Community

    master

    Virtual Community requires Python 3.11, CUDA 11.7, and Ubuntu 24.04. It uses uv for dependency management.

    To install, use uv sync to create a virtual environment and install PyTorch (CUDA 11.7) and the Genesis physics engine.

    If you intend to run the example tour agent, you must also install the volume grid library using the provided setup script.

    uv sync
source .venv/bin/activate

# For the tour agent example:
cd vico/agents/sg
./setup.sh

  8. Install the Scene Generation Pipeline dependencies

    master

    To use the scene generation pipeline, you must install Blender, the Blosm plugin, Inpaint Anything, and Upscayl.

    1. Blender: Download and install from blender.org.
    2. Blosm: Install the Blosm plugin within Blender using the script located at install/install_blosm.py.
    3. Inpaint Anything: Install the Inpaint-Anything package from this repository.
    4. Upscayl: Install the upscayl tool for texture upscaling.
    # Note: Installation steps involve manual downloads and running specific scripts
# Install Blosm via Blender script:
# (Run within Blender environment)
# install/install_blosm.py

  9. Use scoped context managers for Blender state

    master

    To prevent side effects when changing Blender's global state (like active objects or edit modes), use the following MeshObject context managers. They automatically restore the previous state upon exiting the block.

    • scoped_active(): Temporarily makes the object the active object in the scene.
    • scoped_mode(mode, scoped_active=True): Temporarily switches the object to a specific mode (e.g., "EDIT", "OBJECT"). If scoped_active is True, it also handles making the object active.
    • scoped_select(select=True): Temporarily changes the selection state of the object.
    • scoped_BMesh(use_modifiers=True): Provides a bmesh object for low-level geometry editing. If use_modifiers is True, it uses the evaluated mesh (including modifiers).
    with mesh.scoped_mode("EDIT", scoped_active=True):
    # Perform edit-mode operations safely
    pass

with mesh.scoped_BMesh(use_modifiers=True) as bm:
    # Work with bmesh geometry
    pass

  10. How texture resolution is determined for materials

    master

    The script uses a Look-Up Table (LUT) to assign texture resolutions based on the surface area of a mesh. This ensures that larger objects receive higher resolution textures while optimizing memory for smaller ones. The resolution is capped at 1024x1024 to accommodate a subsequent 4x superresolution pipeline.

    Resolution Mapping:

    • Area $\le$ 128 $m^2$: 64x64
    • Area $\le$ 500 $m^2$: 128x128
    • Area $\le$ 1875 $m^2$: 256x256
    • Area $\le$ 7500 $m^2$: 512x512
    • Area $\le$ 30000 $m^2$: 1024x1024
    • Area $>$ 30000 $m^2$: 1024x1024 (Max limit)

  11. Troubleshoot GLIBCXX errors

    master

    If you encounter the error libstdcxx.so.6: version GLIBCXX_3.4.32 not found, you need to update your GCC runtime.

    On Ubuntu/Debian, run:

    sudo apt install --only-upgrade libstdc++6

    Alternatively, install an updated runtime via conda-forge:

    conda install -c conda-forge libstdcxx-ng