live2d-py

Supported Features

• Model loading (Cubism 2.1, 3.0+)
• Eye tracking
• Click interaction
• Motion playback callbacks
• Lip sync (audio synchronization)
• Parameter control for various model parts
• Transparency control for individual parts
• Precise click detection for specific parts

UI Library Compatibility

live2d-py is designed to be compatible with any UI library that can perform drawing via OpenGL. Supported libraries include:

• Pygame
• PyQt5 / PySide2 / PySide6
• GLFW
• pyopengltk
• FreeGlut
• Qfluentwidgets

Supported Features

• Model Loading (Cubism 2.1 and 3.0+)
• Eye tracking
• Click interaction and precise part-level click detection
• Motion playback callbacks
• Lip-sync synchronization
• Parameter and opacity control for model parts

UI Library Compatibility

live2d-py is compatible with any UI library that can provide an OpenGL context for rendering, including:

• Pygame
• PyQt5 / PySide2 / PySide6
• GLFW
• pyopengltk
• FreeGlut
• Qfluentwidgets

The live2d.v2 module (for Cubism 2.1 and below) is implemented entirely in Python. It was derived from deobfuscating and converting live2d.min.js to Python.

Key characteristics:

• Features: Includes enhanced functionality like precise click detection on parts and part color settings.
• Performance: Performance may be suboptimal due to the preservation of certain JavaScript-like characteristics during conversion.

The clipping mask system manages multiple clipping areas by arranging them into different channels and regions of a single texture (typically 256x256).

Mask Layout Strategy

• 1-2 Masks: Uses Channel 0 and Channel 1.
• 3-4 Masks: Subdivides channels into a 2x2 grid.
• 5-9 Masks: Uses a 3x3 grid layout.

Mask Rendering Workflow

1. Calculate Bounds: Determine the total bounds for each clipping group.
2. Bind Off-Screen FBO: Switch to an off-screen framebuffer.
3. Setup Viewport: Set the viewport to the CLIPPING_MASK_SIZE.
4. Clear Texture: Clear the mask texture with (0, 0, 0, 0).
5. Layout Bounds: Arrange the mask regions.
6. Render Shapes: For each clipping group, calculate the mask and draw matrices, then render the mask meshes to the texture.
7. Restore Main FBO: Switch back to the original framebuffer and viewport.

The library provides different modules depending on the version of the Live2D Cubism model you are using. Choose the module that matches your model files:

• live2d.v2: For Cubism 2.1 and earlier. This is a pure Python implementation.
• live2d.v2cpp: For Cubism 2.1 and earlier. This is a high-performance C++ port. Recommended for better performance; it is API-compatible with live2d.v2.
• live2d.v3: For Cubism 3.0 and later. This is a Python C Extension wrapping the Native SDK.

Model File Identification:

• Cubism 2.X: Files typically end in .moc, .model.json, and .mtn.
• Cubism 3.0+: Files typically end in .moc3, .model3.json, and .motion3.json.

# To switch from pure Python to high-performance C++ for v2 models:
import live2d.v2cpp as live2d

Rotation deformers use affine transformations. The process involves:

1. Affine Interpolation: Each component of the affine transform (originX, originY, scaleX, scaleY, rotationDeg, reflectX, reflectY) undergoes multi-dimensional interpolation independently.
2. Matrix Transformation: The interpolated affine components are converted into a 3x3 matrix and applied to vertices.

The transformation formula for a vertex $(x, y)$ is:

• $dstX = scaleX imes x + scaleY imes y + originX$
• $dstY = scaleTX imes x + scaleTY imes y + originY$

Where scaleTX and scaleTY are derived from the rotation angle and scale factors.

The rendering pipeline follows a specific sequence of updates and draws:

1. Update Phase (ModelContext.update()):

   • Parameter Update: Detects changes in paramValues.
   • Deformer Calculation: For each deformer, it performs setupInterpolate() (interpolation) and setupTransform() (applying transformations).
   • Mesh Calculation: For each mesh, it performs setupInterpolate() and setupTransform().

2. Pre-Draw Phase (ModelContext.preDraw()):

   • Prepares drawing parameters and calls clipManager.setupClip() to calculate clipping boundaries and render mask layers to an off-screen texture.

3. Draw Phase (ModelContext.draw()):

   • Iterates through drawDataList based on the drawing order.
   • Calculates final opacity by combining drawData.getOpacity(), partsOpacity, and baseOpacity.
   • Executes drawData.draw() for each mesh.

The Live2D v2 rendering system utilizes two primary shader programs to handle standard rendering and masking operations:

1. Main Shader (shaderProgram): Used for standard model rendering and mask rendering.
2. Off-Screen Shader (shaderProgramOff): Used for mask blending operations.

Color Blending Formulas

The shaders implement specific color composition logic:

• Pre-multiplied Alpha: rgb *= a
• Multiply Blending: rgb = rgb * multiplyColor
• Screen Blending: rgb = rgb + screen - (rgb * screen) (equivalent to rgb + (1-rgb)*screen)

The live2d.v3 module (for Cubism 3.0+) is a Python C extension that wraps the Cubism Native SDK. It is composed of four layers:

• Core: The Cubism Native Core (official). Reads .moc3 files.
• Framework: The Cubism Native Framework (official). Handles JSON reading, physics, and graphics rendering.
• Main: A simplified application layer that implements the LAppModel C++ class on top of the Framework.
• Wrapper: The Python binding layer. This is the only part that introduces Python dependencies. It wraps the LAppModel class to make it accessible in Python via files like Live2D.cpp and PyLAppModel.cpp.

The modules Core, Framework, and Main are independent of Python and could theoretically be used with other programming languages.

The Live2D v2 rendering system is organized into a hierarchy of classes that manage model state, rendering parameters, and clipping.

• ALive2DModel (Abstract Base Class): The primary interface for model operations.
• Live2DModelOpenGL (OpenGL Implementation): The concrete implementation for OpenGL-based rendering.
• ModelContext: Manages the internal state of the model, including:
  • deformerList: List of deformers.
  • drawDataList: List of drawing data.
  • partsDataList: List of parts data.
  • paramValues: Current parameter values.
  • drawContextList: List of drawing contexts.
• DrawParamOpenGL: Holds rendering parameters.
• ClippingManagerOpenGL: Manages clipping masks.