DOCSEARCHbyAlgolia
Submit Docs

ChatdollKit Documentation

repository·master·Indexed 19 days ago

https://github.com/uezo/chatdollkit

A 3D virtual assistant SDK for Unity that transforms 3D models (such as VRM) into voice-enabled, AI-driven chatbots. It integrates LLMs (ChatGPT, Claude, Gemini Pro, Dify), speech-to-text (STT), and text-to-speech (TTS) services with 3D character animation, lip-sync, and facial expression control. Supports multi-platform deployment including Windows, Mac, Linux, iOS, Android, VR, AR, and WebGL.

Tokens
17.9K
Snippets
40
Records
70
Agent score
78%

What's inside ChatdollKit

  1. Overview of ChatdollKit features

    master

    ChatdollKit is a framework for creating voice-interactive chatbots using 3D models.

    Key Features:

    • Generative AI Support: Supports multiple LLMs including ChatGPT, Anthropic Claude, Google Gemini Pro, and Dify. Includes support for function calling (ChatGPT/Gemini) and multimodal capabilities.
    • 3D Model Expression: Supports synchronization of speech and motion, autonomous control of expressions and animations, and lip-sync/blinking.
    • Dialogue Control: Integrates speech recognition (STT) and speech synthesis (TTS) (supporting OpenAI, Azure, Google, Watson, VOICEVOX / AivisSpeech, Style-Bert-VITS2, Nijibois, etc.), manages dialogue context, intent extraction, topic routing, and wake word detection.
    • Multi-platform: Supports Windows, Mac, Linux, iOS, Android, and Unity-supported platforms including VR, AR, and WebGL.

  2. ChatdollKit Overview

    master

    ChatdollKit is a 3D virtual assistant SDK designed to transform 3D models into voice-enabled chatbots. It is a generative AI-native toolkit that integrates LLMs, speech processing, and 3D model animation.

    Key Capabilities

    • Generative AI Native: Supports LLMs like ChatGPT, Anthropic Claude, Google Gemini Pro, and Dify, including function calling and multimodal capabilities.
    • 3D Model Expression: Synchronizes speech with motion, controls facial expressions and animations autonomously, and supports blinking and lip-sync.
    • Dialog Control: Integrates Speech-to-Text (STT) and Text-to-Speech (TTS) (e.g., OpenAI, Azure, Google, VOICEVOX, AivisSpeech), manages dialog context, extracts intents, and supports wakeword detection.
    • Multi-platform Support: Compatible with Windows, Mac, Linux, iOS, Android, and Unity-supported platforms including VR, AR, and WebGL.

  3. Control animations via LLM tags

    master

    Characters can perform gestures (animations) by including [anim:animation_name] tags in their responses. This requires two steps:

    1. System Prompt: Instruct the AI to use specific animation names.
    2. Registration: You must register these names in the ModelController via code to map them to your Animator Controller parameters.

    Registration Examples:

    Base Animations:

    modelController.RegisterAnimation("waving_arm", new Model.Animation("BaseParam", 10, 3.0f));

    Additive Animations:

    modelController.RegisterAnimation("look_away", new Model.Animation("BaseParam", 6, 3.0f, "AGIA_Layer_look_away_01", "Additive Layer"));

    Using AGIARegistry (for Animation Girl Idle Animations):

    modelController.RegisterAnimations(AGIARegistry.GetAnimations(animationCollectionKey));
    // Registering a base animation
modelController.RegisterAnimation("waving_arm", new Model.Animation("BaseParam", 10, 3.0f));

  4. Configure Wake Words, Cancel Words, and Interrupt Words

    master

    You can trigger conversation states using specific phrases via the AIAvatar Inspector.

    • Wake Words: Phrases that start a conversation. You can register multiple words. Use Prefix / Suffix Allowance to allow extra characters around the word (e.g., if the word is "Hello" and allowance is 4, "Hey, Hello!" will trigger it).
    • Cancel Words: Phrases that end a conversation.
    • Interrupt Words: Phrases that cause the character to stop speaking and listen to the user (e.g., "Wait").
      • Note: To use this, set Microphone Mute By to Threshold in the AIAvatar Inspector so the system can hear you while the character is talking.
    • Ignore Words: Strings to be ignored when checking for Wake/Cancel/Interrupt words (e.g., punctuation).
    • Wake Length: Instead of specific words, you can trigger conversation based on the length of recognized text. Set to 0 to disable.

  5. Handle Chain of Thought (CoT) prompting

    master

    ChatdollKit supports Chain of Thought (CoT) prompting by treating content inside <thinking> ~ </thinking> tags as non-spoken text. This allows the AI to 'think' internally without the character reading the thoughts aloud.

    You can customize the tag used for thinking (e.g., using reason instead of thinking) via the ThinkTag field in the LLMContentProcessor inspector.

  6. WebGL Improvements for Low Latency

    master
    For WebGL builds, ChatdollKit uses AudioWorkletNode for microphone input capture to achieve low latency and high reliability. It includes enhanced error handling to detect HTTP errors immediately and prevent the application from hanging, along with stabilized mute/unmute processing.

  7. Control facial expressions via LLM tags

    master

    You can make the character's facial expressions change autonomously based on the conversation. To do this, instruct the AI in its system prompt to include [face:expression_name] tags in its responses.

    Requirements:

    • The expression names must match the names defined in your VRM model (including case sensitivity).
    • The AI must understand the meaning of the names provided.

    System Prompt Example:

    You have four expressions: 'Joy', 'Angry', 'Sorrow', 'Fun' and 'Surprised'.
If you want to express a particular emotion, please insert it at the beginning of the sentence like [face:Joy].

Example
[face:Joy]Hey, you can see the ocean! [face:Fun]Let's go swimming.
    [face:Joy]Hey, you can see the ocean!

  8. Configure LLM and Speech services

    master

    ChatdollKit uses modular components for intelligence and sensory input:

    LLM Services

    Attach components from ChatdollKit/Scripts/LLM to provide the character's brain. Supported frameworks include:

    • ChatGPT
    • Claude
    • Gemini
    • Dify

    Speech Services

    • SpeechListener (ChatdollKit/Scripts/SpeechListener): Handles voice input. Requires API keys and language codes. Enable PrintResult to output recognized text to the Unity log for debugging.
    • SpeechSynthesizer (ChatdollKit/Scripts/SpeechSynthesizer): Handles voice output (TTS).

  9. AIAvatarKit Backend for AI Agent Logic

    master
    Introduced in version 0.8.11/0.8.12, the AIAvatarKit backend allows you to offload AI agent logic to a server. This improves frontend maintainability and enables you to plug in various agent SDKs (like AutoGen) to extend functionality infinitely.

  10. Insert speech pauses using [pause:seconds] tags

    master

    To make conversation feel more natural, you can instruct the AI to insert pauses. Use the [pause:float] tag where the float represents the number of seconds to wait.

    System Prompt Example:

    You can insert pauses in the character's speech to make conversations feel more natural and human-like.

Example:
Hey, it's a beautiful day outside! [pause:1.5] What do you think we should do?
    [pause:1.5]

  11. Configure Camera with SimpleCamera prefab

    master
    The SimpleCamera prefab packages camera image acquisition, preview display, and camera switching. Note that camera specification handling is experimental and may vary by device. Refer to the prefab and its attached scripts for specific details.

  12. Configure Speech and LLM services

    master

    ChatdollKit uses modular components for intelligence and voice:

    LLM Services

    Attach a component from ChatdollKit/Scripts/LLM to handle the brain of the avatar. Supported services include:

    • ChatGPT
    • Claude
    • Gemini
    • Dify

    Required configuration typically includes API keys and system prompts.

    Speech Services

    • SpeechListener: Handles speech-to-text (recognition). Attach this from ChatdollKit/Scripts/SpeechListener. Enabling PrintResult is recommended for debugging as it logs recognized text.
    • SpeechSynthesizer: Handles text-to-speech (synthesis). Attach this from ChatdollKit/Scripts/SpeechSynthesizer and configure language codes and API keys.