EnerVerse-AC World Model

To set up the environment, use Conda with Python 3.10.4. The codebase is compatible with CUDA 11.8 (or higher) and PyTorch 2.4.0. Note that a specific prebuilt version of pytorch3d for CUDA 12.1 is used even if your system is on CUDA 11.8.

Follow these steps:

1. Clone the repository.
2. Create and activate a Conda environment.
3. Install requirements via pip.
4. Install the specific pytorch3d wheel provided in the setup commands.

git clone https://github.com/AgibotTech/EnerVerse-AC.git
conda create -n enerverse python=3.10.4
conda activate enerverse

pip install -r requirements.txt

# Install pytorch3d (prebuilt on CUDA 12.1)
pip install --no-index --no-cache-dir pytorch3d -f https://dl.fbaipublicfiles.com/pytorch3d/packaging/wheels/py310_cu121_pyt240/download.html

Before running inference, you must configure two configuration files with downloaded weights:

1. Model Checkpoint: Download the single-view checkpoint from EVAC and update the model.pretrained_checkpoint key in configs/agibotworld/train_configs.yaml to point to your .pt file.
2. CLIP Weights: Download the CLIP weights and update the model.params.img_cond_stage_config.params.abspath key in configs/agibotworld/config.yaml to the absolute path of open_clip_pytorch_model.bin.

Run the main/generate_video_acwm.py script to generate single-view videos based on an image and action sequence.

Arguments:

• -i IMAGE_FILE: Path to the start frame image.
• -a ACTION_FILE: Path to the action file (*.npy or *.h5).
• -ex EXTRINSIC_FILE: Path to the extrinsic file (*.npy or *.json).
• -in INTRINSIC_FILE: Path to the intrinsic file (*.npy or *.json).
• -s SAVE_ROOT: Directory to save the output video.
• --ckp_path PATH_TO_CHECKPOINT: Path to the model checkpoint.
• --config_path PATH_TO_CONFIG: Path to the configuration file.

# Example usage using provided examples
python main/generate_video_acwm.py -i examples/examples0/frame.png -a examples/examples0/action.npy -ex examples/examples0/extrinsics.npy -in examples/examples0/intrinsics.npy -s ./result_video_root --ckp_path PATH_TO_CHECKPOINT --config_path PATH_TO_CONFIG

Use the tools/prepare_infer_data.py script to convert AgiBotWorld data into the format required for inference (start frame, action file, extrinsic file, and intrinsic file).

Arguments:

• -r PATH_TO_AGIBOTWORLD_ROOT: Path to the AgiBotWorld dataset root.
• -t TASK_ID: The specific task ID.
• -e EPISODE_ID: The episode ID.
• -s SAVE_ROOT: Where to save the processed files.
• -j JSON_OF_ACTION_INDEXES_TO_EXTRACT: A JSON object specifying which action indexes to extract.
• -c CAM_NAME: The name of the camera.

python tools/prepare_infer_data.py -r PATH_TO_AGIBOTWORLD_ROOT -t TASK_ID -e EPISODE_ID -s SAVE_ROOT -j JSON_OF_ACTION_INDEXES_TO_EXTRACT -c CAM_NAME