video2robot

The Web UI provides a graphical interface for the full pipeline, including video upload, model selection (Veo/Sora), and 3D visualization via viser.

Run the server from the video2robot root directory:

uvicorn web.app:app --host 0.0.0.0 --port 8000

Access the interface at http://localhost:8000.

To use video2robot, you must clone the repository recursively to include the necessary third-party submodules (PromptHMR and GMR).

# Clone repo (with submodules)
git clone --recursive https://github.com/AIM-Intelligence/video2robot.git
cd video2robot

If you have already cloned the repository without submodules, initialize them using:

git submodule update --init --recursive

The pipeline follows this flow: [Prompt] → Veo → [Video] → PromptHMR → [SMPL-X] → GMR → [Robot Motion].

Scripts automatically switch between the gmr and phmr conda environments, so you do not need to activate them manually. Ensure both environments are installed.

Generate motion from an action prompt:

python scripts/run_pipeline.py --action "Action sequence: The subject walks forward with four steps."

Generate motion using Sora provider:

python scripts/run_pipeline.py --action "..." --provider sora

Generate motion from an existing video file:

python scripts/run_pipeline.py --video /path/to/video.mp4

Resume a pipeline from an existing project directory:

python scripts/run_pipeline.py --project data/video_001

The project requires a Google API key for video generation. Set this in a .env file at the root of the repository.

# Create .env file
cp .env.example .env

# Set API key
echo "GOOGLE_API_KEY=your-api-key" >> .env

The GMR environment is required for robot motion conversion. Create a conda environment named gmr with Python 3.10 and install the project in editable mode.

conda create -n gmr python=3.10 -y
conda activate gmr
pip install -e .

Refer to the third_party/GMR/README.md for additional details.

You can execute specific stages of the pipeline using individual scripts:

• Video Generation: python scripts/generate_video.py --action "..."
• Pose Extraction: python scripts/extract_pose.py --project data/video_001
• Robot Conversion: python scripts/convert_to_robot.py --project data/video_001
• Visualization:
  • Basic: python scripts/visualize.py --project data/video_001
  • With Pose: python scripts/visualize.py --project data/video_001 --pose
  • With Robot: python scripts/visualize.py --project data/video_001 --robot

The PromptHMR environment is required for human pose extraction. The installation steps depend on your GPU architecture.

For Blackwell GPU (sm_120) users:

conda create -n phmr python=3.11 -y
conda activate phmr
cd third_party/PromptHMR
bash scripts/install_blackwell.sh

For other GPUs (Ampere, Hopper, etc.):

conda create -n phmr python=3.10 -y
conda activate phmr
cd third_party/PromptHMR
pip install -e .

Refer to the third_party/PromptHMR/README.md for additional details.

When using the 'Prompt' mode, the system can automatically apply a BASE_PROMPT to ensure high-quality motion capture data. This prompt enforces:

• Full-body humanoid subject in tight-fitting clothing.
• Static, eye-level camera in a realistic indoor room (not a studio backdrop).
• Biomechanically accurate, physically realistic motion.
• No camera movement, cuts, or slow motion.

If the toggle-base-prompt is enabled, the user's input is treated as an action to be appended to this base prompt. If disabled, the input is treated as a raw_prompt.

The project uses several predefined path constants for data, configurations, and third-party dependencies. Use ensure_paths() to create the necessary data directories and verify that required third-party modules (PromptHMR and GMR) are present in the third_party directory.

from video2robot.config import ensure_paths, DATA_DIR

# Create data directories and check for PromptHMR/GMR
ensure_paths()

print(f"Data will be stored in: {DATA_DIR}")

The pipeline can be started in two modes: Prompt Mode (generating video from text) or Upload Mode (using an existing video file).

// Prompt Mode: Generates video, then extracts pose, then retargets to robot
// 1. POST /projects
// 2. POST /pipeline/generate-video (params: project, model, duration, action/raw_prompt)
// 3. POST /pipeline/extract-pose (params: project, static_camera)
// 4. POST /pipeline/retarget (params: project, robot_type)

// Upload Mode: Uploads video, then extracts pose, then retargets to robot
// 1. POST /projects
// 2. POST /api/files/upload/{project_name} (FormData with 'file')
// 3. POST /pipeline/extract-pose (params: project, static_camera)
// 4. POST /pipeline/retarget (params: project, robot_type)

The interface provides a visualization mode to view the results in a Viser-based viewer.

1. Click Visualize to start the Viser server for the current project.
2. The system calls POST /viser/start with { "project": "<name>", "all_tracks": true }.
3. The resulting session URL is loaded into an iframe.
4. Use Close Viser to stop the session via POST /viser/stop.

The following robots are supported by the GMR retargeting system:

See third_party/GMR/README.md for the full list.