Talking Head (3D)

Moods in head.animMoods define complex, looping animation sequences. A mood consists of a baseline (default shapekey values), speech settings (delta rates for pitch/volume), and an anims array.

Each animation in the anims array is a template that the class iterates through. The class follows a hierarchy of keys to select the next animation based on state (e.g., idle vs talking), body form (M vs F), or view (full, upper, etc.).

Key Properties in anims:

• name: The animation component name (e.g., 'blink', 'mouth').
• delay: How long the pose is held (can be a number or an array for random ranges).
• dt: Durations (ms) for each part in the sequence.
• vs: Shapekey targets. Values can be a single number [0, 1] or an array [min, max] for random values (Gaussian/Uniform).
• alt: Used for probabilistic branching (using the p key for probability).

Use head.setMood("mood-name") to apply a mood.

head.animMoods["custom-mood-1"] = {
  baseline: { eyesLookDown: 0.1 },
  speech: { deltaRate: 0, deltaPitch: 0, deltaVolume: 0 },
  anims: [
    { name: 'breathing', delay: 1500, dt: [ 1200,500,1000 ], vs: { chestInhale: [0.5,0.5,0] } },
    { name: 'head',
      idle: { delay: [0,1000], dt: [ [200,5000] ], vs: { headRotateX: [[-0.04,0.10]], headRotateY: [[-0.3,0.3]], headRotateZ: [[-0.08,0.08]] } },
      talking: { dt: [ [0,1000,0] ], vs: { headRotateX: [[-0.05,0.15,1,2]], headRotateY: [[-0.1,0.1]], headRotateZ: [[-0.1,0.1]] } }
    }
  ]
};
head.setMood("custom-mood-1");

Follow these steps to prepare a VRoid model for use in TalkingHead:

1. Export from VRoid: Create your character in VRoid Studio and export it as a VRM 1.0 file.
2. Import to Blender: In Blender, use File | Import | VRM (.vrm) to load your file.
3. Cleanup: In the Outliner, right-click the "Colliders" collection and select Delete Hierarchy to reduce file size.
4. Run Processing Scripts: Open the "Scripting" workspace and execute the following scripts in order:
   • rename-vroid-bones.py (renames bones to compatible names)
   • build-vroid-eyes.py (creates eye movement shape keys)
   • build-vroid-shapekeys.py (generates ARKit and Oculus viseme shape keys)
5. Fix Rigging: Select the armature, then go to TalkingHead add-on | Operations | Fix bone axes (T-pose). This is critical to prevent twisted body parts or clothing.
6. Optional Scaling: Select the armature, then go to TalkingHead add-on | Operations | Scale character.
7. Material Adjustment: For all materials, set the "Metallic" value to 0 to ensure correct GLB export.
8. Apply Transforms: Select all objects and go to Object | Apply | All Transforms.

If you have an existing Three.js scene, you can use avatarOnly mode to prevent TalkingHead from creating its own renderer and lights. In this mode, you are responsible for calling the head.animate(dt) update function within your own animation loop.

To set this up:

1. Initialize TalkingHead with avatarOnly: true and provide your existing camera via avatarOnlyCamera.
2. Call await head.showAvatar(...).
3. Add head.armature to your scene.
4. Call head.animate(delta * 1000) in your animation loop (where delta is in seconds).

// Create avatarOnly instance and load
const head = new TalkingHead( container, {
  /* ... */
  avatarOnly: true, // set avatarOnly mode
  avatarOnlyCamera: camera // Your camera avatar talks to
});
await head.showAvatar({ /* ... */ });

// Add to your own scene
head.armature.position.set(1,0,0);
head.armature.rotation.set(0,0.5,0);
scene.add(head.armature);

// You own animation loop
const clock = new THREE.Clock();
function animate() {
  const delta = clock.getDelta();
  head.animate(delta * 1000); // Update avatar
  renderer.render( scene, camera );
}
renderer.setAnimationLoop(animate);

To use TalkingHead with MPFB, follow these installation steps:

1. Install MPFB Extension

1. Download and install Blender.
2. In Blender, go to Edit | Preferences | Get extensions and allow online access.
3. Search for "MPFB" and click Install.
4. Verify installation by checking the View | Sidebar for an "MPFB v2.0.15" (or later) tab.

2. Install MPFB Asset Packs

1. In the "MPFB" tab, go to System and resources | Web resources | Asset packs.
2. Download "MakeHuman system assets" and any other desired packs.
3. Required: From the "Functional asset packs" section, you must download:
   • "Visemes 02" (Meta/Oculus style visemes)
   • "Faceunits 01" (ARKit style face units)
4. Install packs via Apply assets | Library Settings | Load pack from zip file.

3. Install TalkingHead Add-on and Assets

1. Download talkinghead-addon.py, talkinghead.mpfbskel (rig), and talkinghead.mhw (weights) to a local directory.
2. Install the add-on: Edit | Preferences | Add-ons | Install from Disk....
3. In Add-on preferences, set the "Data Directory" to the folder containing the downloaded files.
4. Initialize the Rig in MPFB:
   • Create a dummy character: New human | From scratch | Create human.
   • Load rig: Create assets | MakeRig | Load/Save rig | Load rig | select talkinghead.mpfbskel.
   • Load weights: Create assets | MakeRig | Load/Save rig | Load weights | select talkinghead.mhw.
   • Save to library: Create assets | MakeRig | Load/Save rig | Set library rig name to talkinghead and identifying bones to e.g. LeftToe_End | Save rig to library.
   • Delete the dummy character and restart Blender.