DOCSEARCHbyAlgolia
Submit Docs

OmniWorld Dataset Documentation

repository·main·Indexed 19 days ago

https://github.com/yangzhou24/omniworld

A massive-scale, multi-domain, and multi-modal dataset for 4D world modeling tasks, including 4D geometric reconstruction, future prediction, and camera-controlled video generation. The repository provides instructions for downloading datasets via Hugging Face, a data structure reference for OmniWorld-Game, and scripts for visualizing scenes as 3D point clouds.

Tokens
994
Snippets
2
Records
4
Agent score
17%

What's inside OmniWorld

  1. Download the OmniWorld dataset

    main

    You can download the entire OmniWorld dataset using the Hugging Face CLI. First, ensure you have the huggingface_hub package installed. Then, use the hf download command to pull the dataset to your local directory.

    For downloading specific files or subsets (e.g., instead of the full OmniWorld-Game dataset), use the scripts/download_specific.py script provided in the repository.

    # 1. Install (if you haven't yet)
pip install --upgrade "huggingface_hub[cli]"

# 2. Full download
hf download InternRobotics/OmniWorld \
           --repo-type dataset \
           --local-dir /path/to/DATA_PATH

  2. Visualize OmniWorld-Game scenes as Point Clouds

    main

    The scripts/visualize_pcd.py script converts a scene from the OmniWorld-Game dataset into a 3D point cloud (.ply format) for inspection.

    Prerequisites

    • You must have the OmniWorld-Game dataset downloaded.
    • The scene directory must follow the expected data structure (containing color/, depth/, camera/, etc.).

    Usage Modes

    1. Metric scale point cloud: Use this for real-world units. You must provide the metadata CSV file so that depth and poses are scaled correctly. The output will be split{N}_points_metric.ply.
    2. Relative scale point cloud: Use this for a non-scaled version. Omit the --metadata_csv flag. The output will be split{N}_points.ply.

    You can view the resulting .ply files using 3D viewers like MeshLab.

    # Metric scale (requires metadata CSV)
python scripts/visualize_pcd.py <your-data-path>/b04f88d1f85a --split_idx 0 --metadata_csv metadata/omniworld_game_metadata.csv

# Relative scale (no metadata)
python scripts/visualize_pcd.py <your-data-path>/b04f88d1f85a --split_idx 0

  3. OmniWorld Dataset availability and modalities

    main

    OmniWorld consists of several datasets across different domains. Below is the status and capability of the released datasets:

    DatasetStatusDomainResolutionDepthCameraTextOpt. flowFg. masks
    OmniWorld-Game✅ ReleasedSimulator1280 × 720🙂🙂🙂🙂🙂
    DROID✅ ReleasedRobot1280 × 720🙂✅🙂🙂🙂
    RH20T✅ ReleasedRobot640 × 360❌✅🙂🙂🙂
    RH20T-Human✅ ReleasedHuman640 × 360❌✅🙂❌❌
    HOI4D✅ ReleasedHuman1920 × 1080🙂🙂🙂🙂✅
    Epic-Kitchens✅ ReleasedHuman1280 × 720❌🙂🙂❌🙂
    Ego-Exo4D✅ ReleasedHuman1024 × 1024❌✅🙂🙂❌
    EgoDex✅ ReleasedHuman1920 × 1080❌✅🙂❌❌
    CityWalk✅ ReleasedInternet1280 × 720❌🙂✅❌❌
    Game-Benchmark✅ ReleasedSimulator1280 × 720🙂🙂🙂🙂🙂

    Note: 🙂 indicates available, ✅ indicates high fidelity/complete, ❌ indicates not available.

  4. OmniWorld-Game data structure reference

    main

    When using visualization or processing scripts, ensure your scene directory (e.g., <your-data-path>/<scene_id>/) contains the following files and subdirectories:

    • color/: RGB frames (.png)
    • depth/: 16-bit depth maps
    • flow/: flow_u_16.png, flow_v_16.png, and flow_vis.png
    • camera/: split_*.json (contains intrinsics + extrinsics)
    • subject_masks/: foreground masks (per split)
    • gdino_mask/: dynamic-object masks (per frame)
    • text/: structured captions (81-frame segments)
    • droidclib/: coarse camera poses
    • fps.txt: source video framerate
    • split_info.json: frame grouping information