Animatable Gaussians

repository·master·Indexed 22 days ago

https://github.com/lizhe00/animatablegaussians

A framework for modeling high-fidelity, animatable human avatars using 3D Gaussian Splatting and 2D CNNs. It learns pose-dependent Gaussian maps to capture dynamic appearances and garment details from RGB videos. The project includes a modified diff-gaussian-rasterization extension for depth and alpha rendering, as well as support for datasets such as AvatarReX, ActorsHQ, and THuman4.0.

Tokens
15K
Snippets
37
Records
57
Agent score
78%

What's inside Animatable Gaussians

  1. Overview of GLM (OpenGL Mathematics)

    master

    GLM is a header-only C++ mathematics library designed for graphics software. It is based on the OpenGL Shading Language (GLSL) specifications, meaning it uses the same naming conventions and functionality as GLSL, making it highly intuitive for developers familiar with shader programming.

    Key features include:

    • GLSL Compatibility: Classes and functions mirror GLSL.
    • Extension System: Provides advanced capabilities like matrix transformations, quaternions, data packing, random numbers, and noise.
    • Interoperability: Works with OpenGL and is suitable for software rendering (raytracing/rasterization), image processing, and physics simulations.
    • Header-only: Easy to integrate into C++ projects without complex build steps.
  2. Understand the Gaussian-Splatting License terms

    master

    The diff-gaussian-rasterization-depth-alpha component (part of the gaussian-splatting software) is licensed for non-commercial research and evaluation purposes only.

    Key Usage Terms:

    • Permitted Use: Academic and industrial research users may use, test, and evaluate the software free of charge.
    • Commercial Use: Explicitly prohibited without prior and written consent from the Licensors (Inria and MPII). For commercial inquiries, contact stip-sophia.transfert@inria.fr.
    • Derivative Works: You may create derivative works, but any new terms you apply to them must still respect the non-commercial limitation of the original license.
    • Redistribution: If you redistribute the software or derivative works, you must include a complete copy of this license and retain all original copyright, patent, trademark, or attribution notices.
    • Citation: When using this software for publications or research results, users are strongly encouraged to cite the corresponding original publications as specified in the software's documentation.
  3. How to use GLM extensions

    master

    GLM extends its core GLSL feature set through dedicated header files. To use an extension, include its specific header file; once included, the features are added to the glm namespace. Including an extension also automatically includes all its dependent core functionalities and other extensions.

    #include <glm/glm.hpp>
    #include <glm/gtc/matrix_transform.hpp>
    
    int foo()
    {
        glm::vec4 Position = glm::vec4(glm:: vec3(0.0f), 1.0f);
        glm::mat4 Model = glm::translate(glm::mat4(1.0f), glm::vec3(1.0f));
    
        glm::vec4 Transformed = Model * Position;
        return 0;
    }
  4. Understand the AvatarReX Dataset structure

    master

    Each subject directory contains the following files and folders:

    • Multi-view images: Located in ./<subject>/<camera_name>/<frame_id>.jpg.
    • Foreground segmentation (masks): Located in ./<subject>/<camera_name>/mask/pha/<frame_id>.jpg. These are obtained via BackgroundMattingV2.
    • calibration_full.json: Contains camera calibration data (Rotation R, Translation T, and Intrinsic K).
    • smpl_params.npz: Contains SMPL-X fitting parameters.
    • missing_img_files.txt: A list of filenames for frames that were lost during capture.
  5. Handle GLM precision qualifiers

    master

    GLM supports GLSL precision qualifiers (lowp, mediump, highp) using prefixes instead of qualifiers. By default, all types use high precision. Use these prefixes to trade precision for performance.

    Example mapping:

    • lowp vec4 $\rightarrow$ lowp_vec4
    • mediump vec4 $\rightarrow$ medium_vec4
    • highp vec4 $\rightarrow$ highp_vec4
    #include <glm/glm.hpp>
    
    ivec3 foo(const vec4 & v)
    {
        highp_vec4 a = v;
        medium_vec4 b = a;
        lowp_ivec3 c = glm::ivec3(b);
        return c;
    }
  6. Enable or Disable Swizzle Operators

    master

    Swizzling allows selecting and rearranging vector components (e.g., v.xyz()).

    To enable swizzling, define GLM_FORCE_SWIZZLE.

    Warning: Enabling swizzling can significantly increase binary size and compilation time.

    Implementation Modes:

    1. Standard C++98 (R-value only): Uses member functions like .bgr(). These return a copy and cannot be used as L-values (you cannot assign to them).
    2. Language Extensions (L-value and R-value): On compilers supporting anonymous struct as union members (like Visual C++, GCC, or Clang), swizzling allows GLSL-like syntax (e.g., v.bgra = ...).

    Note for Extensions: Swizzle objects returned by extensions are not direct vector types; they must be converted via constructors vec4(v.rgba) or the operator v.rgba() to be used in functions like clamp().

    #define GLM_FORCE_SWIZZLE
    #include <glm/glm.hpp>
    
    // C++98 style (R-value only)
    glm::vec3 bgr = color.bgr();
    
    // Extension style (L-value)
    color.bgra = other_color;
    
    // Using extension results in functions
    vec4 clamped = clamp(vec4(color.rgba), 0.f, 1.f);
  7. Optimize build times using separated headers in GLM

    master

    To minimize compilation times, avoid global headers and instead include only the specific headers for the features you need. You can use two approaches:

    1. Separated Core Headers: Include specific headers for vectors, matrices, and math functions (e.g., <glm/vec3.hpp>, <glm/mat4x4.hpp>, <glm/trigonometric.hpp>).
    2. Extension Headers: Include specific extension headers for transformations or specialized vector types (e.g., <glm/ext/matrix_transform.hpp>).
    // Include GLM core features
    #include <glm/vec2.hpp>           // vec2
    #include <glm/vec3.hpp>           // vec3
    #include <glm/mat4x4.hpp>         // mat4
    #include <glm/trigonometric.hpp>  // radians
    
    // Include GLM extension
    #include <glm/ext/matrix_transform.hpp> // perspective, translate, rotate
  8. Getting started with GLM

    master

    GLM (OpenGL Mathematics) can be integrated into your C++ project using several header inclusion strategies:

    • Global headers: Include a single header that provides access to the entire library.
    • Separated headers: Include only specific headers for the components you need (e.g., vectors, matrices) to improve compilation times.
    • Extension headers: Use specific headers for experimental or non-core features (often found in the gtx directory).

    To integrate GLM into a CMake-based build system, use the CMake find module to locate the library.

    # Example of finding GLM with CMake
    find_package(glm REQUIRED)
    target_link_libraries(your_target PRIVATE glm::glm)
  9. Best practices for using GLM

    master

    To avoid common issues when integrating GLM into your project, follow these guidelines:

    1. Avoid using namespace glm;: GLM uses many common tokens for types and functions. Using the namespace globally can cause name collisions with other libraries or the standard library. Always use the glm:: prefix.
    2. Define NOMINMAX on Windows: Windows headers often define min and max as macros, which can conflict with GLM. Define NOMINMAX before including Windows headers.
    3. Angle Units: All GLM functions use radians for angles (consistent with GLSL). Do not pass degrees to functions like glm::rotate or glm::perspective.
    4. Handle Domain Errors: Functions like glm::normalize may crash if passed a null vector (all zeros). This is treated as a domain error, similar to standard C++ math functions.