DOCSEARCHbyAlgolia
Submit Docs

LAM (Large Avatar Model)

repository·master·Indexed 21 days ago

https://github.com/aigc3d/lam

A PyTorch implementation for creating ultra-realistic, one-shot animatable 3D Gaussian heads from a single image. Designed for high-speed animation, rendering, and low-latency interactive chatting, it includes a Gradio UI and CLI for inference. The repository also features a landmark detection pipeline utilizing STAR (Self-adaptive Ambiguity Reduction) loss for heatmap regression-based facial landmark detection.

Tokens
5.3K
Snippets
30
Records
35
Agent score
77%

What's inside LAM

  1. What is STAR Loss?

    master
    STAR (Self-adapTive Ambiguity Reduction) loss is a self-adaptive anisotropic direction loss designed for heatmap regression-based facial landmark detection. It addresses semantic ambiguity by using PCA to characterize the predicted distribution. This allows the loss to adaptively suppress prediction errors in directions of high ambiguity, mitigating the impact of ambiguous annotations during training.

  3. Download LAM model weights and assets from HuggingFace

    master

    Use the huggingface-cli to download the necessary assets and model weights. This process involves downloading a temporary directory, extracting the assets and third-party models, and then downloading the specific model weights to the model_zoo directory.

    # Download assets
huggingface-cli download 3DAIGC/LAM-assets --local-dir ./tmp
tar -xf ./tmp/LAM_assets.tar && rm ./tmp/LAM_assets.tar
tar -xf ./tmp/thirdparty_models.tar && rm -r ./tmp/

# Download model weights
huggingface-cli download 3DAIGC/LAM-20K --local-dir ./model_zoo/lam_models/releases/lam/lam-20k/step_045500/
    huggingface-cli download 3DAIGC/LAM-assets --local-dir ./tmp
tar -xf ./tmp/LAM_assets.tar && rm ./tmp/LAM_assets.tar
tar -xf ./tmp/thirdparty_models.tar && rm -r ./tmp/
huggingface-cli download 3DAIGC/LAM-20K --local-dir ./model_zoo/lam_models/releases/lam/lam-20k/step_045500/

  4. Install Torch and Xformers for Windows

    master

    Install the specific versions of Torch and xformers required for the environment using the CUDA 11.8 index URL.

    pip install torch==2.3.0 torchvision==0.18.0 torchaudio==2.3.0 --index-url https://download.pytorch.org/whl/cu118
pip install -U xformers==0.0.26.post1 --index-url https://download.pytorch.org/whl/cu118

  5. Install LAM on Linux

    master

    To set up the LAM environment on Linux, clone the repository and run the installation script corresponding to your CUDA version (12.1 or 11.8).

    git clone  https://github.com/aigc3d/LAM.git
cd LAM
# Install with Cuda 12.1
sh  ./scripts/install/install_cu121.sh
# Or Install with Cuda 11.8
sh ./scripts/install/install_cu118.sh
    git clone  https://github.com/aigc3d/LAM.git
cd LAM
# Install with Cuda 12.1
sh  ./scripts/install/install_cu121.sh
# Or Install with Cuda 11.8
sh ./scripts/install/install_cu118.sh

  6. Prepare datasets for STAR Loss

    master

    To use the landmark detection implementation, you must organize your data into specific directory structures for WFLW, 300W, and COFW datasets.

    1. Download Raw Images: Obtain images from COFW, 300W, and WFLW.
    2. Preprocess: Follow the preprocessing steps used in ADNet and download metadata from the ADNet repository.
    3. Directory Structure: Organize files as follows:
    |-- ${image_dir}
   |-- WFLW
      | -- WFLW_images
   |-- 300W
      | -- afw
      | -- helen
      | -- ibug
      | -- lfpw
   |-- COFW
      | -- train
      | -- test
|-- ${annot_dir}
   |-- WFLW
      |-- train.tsv, test.tsv
   |-- 300W
      |-- train.tsv, test.tsv
   |--COFW
      |-- train.tsv, test.tsv

  7. Download LAM Model Weights and Assets

    master

    You can download the necessary assets and model weights using either huggingface-cli or modelscope.

    Using HuggingFace

    Requires huggingface-cli installed.

    Using ModelScope

    Requires pip install modelscope.

    Note: After downloading, you must extract the assets and third-party models as shown in the commands below.

    # HuggingFace Method
huggingface-cli download 3DAIGC/LAM-assets --local-dir ./tmp
tar -xf ./tmp/LAM_assets.tar && rm ./tmp/LAM_assets.tar
tar -xf ./tmp/thirdparty_models.tar && rm -r ./tmp/
huggingface-cli download 3DAIGC/LAM-20K --local-dir ./model_zoo/lam_models/releases/lam/lam-20k/step_045500/

# ModelScope Method
pip3 install modelscope
modelscope download --model "Damo_XR_Lab/LAM-assets" --local_dir "./tmp/"
tar -xf ./tmp/LAM_assets.tar && rm ./tmp/LAM_assets.tar
tar -xf ./tmp/thirdparty_models.tar && rm -r ./tmp/
modelscope download "Damo_XR_Lab/LAM-20K" --local_dir "./model_zoo/lam_models/releases/lam/lam-20k/step_045500/"

  8. Install LAM on Windows

    master

    For Windows users, a one-click installation package (supporting CUDA 12.8) is available. For detailed instructions, refer to the official Windows Install Guide in the repository.

    • One-click package: Download Link
    • Detailed Guide: scripts/install/WINDOWS_INSTALL.md

  9. Set up the environment for Exporting Chatting Avatars

    master

    To export chatting avatars, you must install the Python FBX SDK (version 2020.2+) and Blender (version > 4.0.0).

    Follow these steps:

    1. Install FBX SDK and dependencies: You can either install the FBX SDK from a tarball using the provided installation script or install a pre-built FBX-SDK Wheel. Additionally, install pathlib and patool.
    2. Install Blender: Download and extract Blender version 4.0.0 or higher to a local directory.
    3. Download Assets: Download and extract the chatting avatar template files into the assets/ directory.
    # Option A: Install FBX SDK via tarball and script
wget https://virutalbuy-public.oss-cn-hangzhou.aliyuncs.com/share/aigc3d/data/LAM/fbxsdk_linux.tar
tar -xf fbxsdk_linux.tar
sh tools/install_fbx_sdk.sh

# Option B: Install pre-built FBX-SDK Wheel
wget https://virutalbuy-public.oss-cn-hangzhou.aliyuncs.com/share/aigc3d/data/LAM/fbx-2020.3.4-cp310-cp310-manylinux1_x86_64.whl
pip install fbx-2020.3.4-cp310-cp310-manylinux1_x86_64.whl

# Install other requirements
pip install pathlib
pip install patool

# Download and extract sample files
wget https://virutalbuy-public.oss-cn-hangzhou.aliyuncs.com/share/aigc3d/data/LAM/sample_oac.tar
tar -xf sample_oac.tar -C assets/

  10. Download LAM model weights and assets from ModelScope

    master

    If HuggingFace is inaccessible, use ModelScope to download the assets and weights. You must first install the modelscope library.

    # Install modelscope
pip3 install modelscope

# Download assets
modelscope download --model "Damo_XR_Lab/LAM-assets" --local_dir "./tmp/"
tar -xf ./tmp/LAM_assets.tar && rm ./tmp/LAM_assets.tar
tar -xf ./tmp/thirdparty_models.tar && rm -r ./tmp/

# Download model weights
modelscope download "Damo_XR_Lab/LAM-20K" --local_dir "./model_zoo/lam_models/releases/lam/lam-20k/step_045500/"
    pip3 install modelscope
modelscope download --model "Damo_XR_Lab/LAM-assets" --local_dir "./tmp/"
tar -xf ./tmp/LAM_assets.tar && rm ./tmp/LAM_assets.tar
tar -xf ./tmp/thirdparty_models.tar && rm -r ./tmp/
modelscope download "Damo_XR_Lab/LAM-20K" --local_dir "./model_zoo/lam_models/releases/lam/lam-20k/step_045500/"

  11. Compile and Install Pytorch3d on Windows

    master

    Installing Pytorch3d on Windows requires manual configuration of the setup.py file to prevent compilation errors.

    1. Set environment variables DISTUTILS_USE_SDK=1 and PYTORCH3D_NO_NINJA=1.
    2. Clone the pytorch3d repository.
    3. Crucial Step: Modify setup.py by adding "-DWIN32_LEAN_AND_MEAN" to the nvcc_args list.
    4. Run the installation command.
    set DISTUTILS_USE_SDK=1
set PYTORCH3D_NO_NINJA=1
git clone https://github.com/facebookresearch/pytorch3d.git
cd pytorch3d
# [Manual Step] Modify setup.py: add "-DWIN32_LEAN_AND_MEAN" in nvcc_args
python setup.py install

  12. Run the Gradio app for Chatting Avatar Export

    master

    Run the app_lam.py script using Gradio. You must provide the path to your Blender executable using the --blender_path flag so the application can perform the export operations.

    # Example command
python app_lam.py --blender_path ~/software/blender-4.0.2-linux-x64/blender