AITuberKit Documentation

AITuberKit is an all-in-one toolkit for building AI character web applications. Key capabilities include:

1. AI Character Interaction

• LLM Integration: Use various API keys for dialogue.
• Multimodal: Supports camera vision and image uploads.
• Memory: Includes short-term conversation memory and long-term memory via RAG.

2. AITuber Live Streaming

• Comment Interaction: Automatically responds to YouTube live comments (via YouTube API or OneComme).
• Autonomous Mode: Characters can speak spontaneously even without new comments.

3. Demo Terminals & Digital Signage

• Signage Mode: Full-screen display with password protection, NG word filtering, and input limits.
• Human Detection: Uses camera face detection to trigger greetings or farewells.
• Idle Mode: Automatic speech during inactivity using fixed phrases or AI-generated content.

4. Advanced Modes

• Realtime API: Low-latency dialogue and function execution via OpenAI Realtime API.
• Audio Mode: Natural voice dialogue via OpenAI Audio API.
• Reasoning Mode: Displays the AI's thought process.
• Game Commentary: Periodically analyzes screenshots to provide live commentary.

5. Integration & Extensions

• External Integration: Connect via WebSocket for advanced text/image exchange.
• Slide Mode: AI-driven presentation mode.
• Message API: Receive external text/images to trigger AI speech.

AITuberKit is an all-in-one open-source toolkit for building web applications that allow users to interact with AI characters. It supports various AI services, character models, and speech synthesis engines. Key capabilities include:

• AI Character Interaction: Chatting via LLM APIs, multimodal support (camera/images), short-term memory, and RAG-based long-term memory.
• AITuber Broadcasting: Automatic responses to YouTube or OneComme comments, continuous dialogue modes, and customizable comment fetching intervals.
• Demo Terminal & Digital Signage: Full-screen modes with passcode authentication, NG word filtering, face detection for visitor greeting, and 'Idle Mode' for automatic character speech when inactive.
• Advanced Modes: Low-latency interaction via OpenAI Realtime API, natural voice dialogue via Audio API, Reasoning mode for displaying AI thought processes, and Game Commentary mode via periodic screen capture analysis.
• Integration & Extension: WebSocket support for external server communication, Slide mode for automated presentations, and a dedicated API for receiving external text/images to trigger character speech.

AITuberKit is an all-in-one toolkit for building web applications that allow users to interact with AI characters. It supports various AI services, character models, and speech synthesis engines.

Core Capabilities

1. AI Character Interaction

• LLM Integration: Chat with characters using various LLM API keys.
• Multimodal Support: Characters can recognize camera feeds or uploaded images.
• Memory: Maintains recent conversation history and utilizes RAG-based long-term memory for context.

2. AITuber Streaming

• Live Comment Integration: Automatically responds to YouTube comments using either the YouTube API or OneComme (わんコメ).
• Continuous Conversation Mode: Allows the character to speak spontaneously even when no comments are received.

3. Demo Terminals & Digital Signage

• Demo Terminal Mode: Full-screen display optimized for signage, featuring passcode authentication, NG word filtering, and input length limits.
• Human Detection: Uses camera face detection to automatically trigger greetings or farewells.
• Idle Mode: Automatically triggers speech when conversation stops, using fixed phrases, time-of-day greetings, or AI-generated content.

4. Advanced Interaction Modes

• Realtime API: Low-latency interaction and function calling via OpenAI's Realtime API.
• Audio Mode: Natural voice interaction using OpenAI's Audio API.
• Reasoning Mode: Displays the AI's thought process and allows tuning of reasoning parameters.
• Game Commentary Mode: Periodically analyzes screen captures to generate and speak commentary based on gameplay.

5. Integration & Extension

• External Integration Mode: Connects to server applications via WebSocket for advanced text/image exchange.
• Slide Mode: AI characters can automatically present slides.
• Message Reception API: Allows external systems to send text or images to trigger character speech via a dedicated API.

The logo used in the aituber-kit repository is subject to specific usage terms. Copyright belongs to the logo creator (Ruka Design).

Permitted Uses

• Non-commercial use related to this repository.
• Using the logo to introduce this repository.
• Using the logo for linking purposes (only for links to websites related to this repository).

Prohibited Uses

• Commercial use without prior written permission.
• Modifying the logo or creating derivative works.
• Redistributing the logo to third parties.
• Using the logo in a way that damages the reputation or credit of the repository or its owners.

Disclaimer

• The logo is provided "as is" without warranties of any kind. The repository owner is not liable for any damages arising from the use of the logo.
• Terms are subject to change without notice; users are responsible for checking updates regularly.

The streaming response pipeline is designed as a state machine to process AI token streams. It decomposes the complex task of handling AI responses into a pure, side-effect-free transformation process using a SpeechSegmenter.

Core Architecture

Instead of a single monolithic loop, the system uses a "Pure State Machine + Injectable Side Effects" pattern:

1. SpeechSegmenter (The State Machine): A push-based, pure state machine that performs sequential transformations. It handles token stream reading, emotion/motion tag parsing, sentence splitting, and code block detection without triggering any external actions.
2. Side Effects (The Consumers): Once the segmenter produces events (like a completed sentence or a detected tag), specialized modules consume these events to perform actual work:
   • messageLogWriter: Handles chatLog upserts, message ID boundary management, and accumulating thinking content.
   • speechDispatcher: Manages session arbitration, follows stop signals, and calls speakCharacter to trigger audio.
3. Orchestration: The processAIResponse function acts as a lightweight orchestrator (under 100 lines) that reads the stream, classifies chunks (e.g., THINKING), feeds them into the segmenter, and distributes the resulting events to the writers and dispatchers.

Key Benefits

• Reliable Cancellation: Fixes issues where SpeakQueue.stopAll() failed to prevent subsequent speech by ensuring the pipeline respects session stop tokens.
• Code Block Integrity: Prevents text from leaking into or out of code blocks (``) during chunk boundaries.
• Tag Consistency: Ensures emotion tags (e.g., [happy]) and motion tags (e.g., [motion:xxx]) are handled with consistent semantics across different input sources.

AITuberKit supports two WebSocket protocols for external integration: v1 legacy and the new v2 protocol.

• v1 legacy: Uses a simple JSON structure.
  • Client sends: {"content": "...", "type": "chat", "image": "..."}
  • Server sends: {"text": "...", "role": "assistant", "emotion": "...", "type": "", "image": "..."}
• v2 protocol: Uses a standardized envelope for all messages, providing better reliability through session management, heartbeats, and request tracking.

Compatibility Logic:

• The client detects the protocol by checking if the server sends a session.ready event. If not, it falls back to v1.
• The server detects the protocol by checking for the version: "2" field. If missing, it treats the message as v1 and normalizes it to v2 internally.

// v2 Envelope Example
{
  "version": "2",
  "id": "msg_...",
  "type": "chat.message",
  "sessionId": "session_...",
  "timestamp": "2026-06-20T00:00:00.000Z",
  "payload": {},
  "metadata": {}
}