DOCSEARCHbyAlgolia
Submit Docs

xiaozhi-esp32-server

repository·main·Indexed 27 days ago

https://github.com/xinnan-tech/xiaozhi-esp32-server

A backend service for the xiaozhi-esp32 intelligent hardware project, enabling voice-driven AI interactions using LLMs, TTS, and ASR. It supports MQTT, UDP, and Websocket protocols and includes a digital-human module for interaction debugging, a Java Spring Boot manager-api for device and user administration, and management terminals for web and mobile (App/WeChat Mini Program). It also features a Voice Box Theme Customizer for generating assets.bin files for ESP32-S3, C3, P4, and C6 chips.

Tokens
52.2K
Snippets
104
Records
283
Agent score
95%

What's inside xiaozhi-esp32-server

  1. Overview of manager-web (Web Management Frontend)

    main

    The manager-web is a Vue.js 2 Single Page Application (SPA) designed for administrators to manage the system via a desktop browser.

    Key Features:

    • AI Configuration: Manage API keys and settings for ASR, LLM, and TTS providers.
    • Device Control: Register and monitor ESP32 devices.
    • System Management: Manage users, roles, and permissions.
    • OTA Management: Handle firmware update processes.
    • Customization: Adjust TTS timbres and system parameters.

  2. Overview of manager-api (Management Backend)

    main

    The manager-api is a Java Spring Boot service that acts as the central administrative hub for the ecosystem. It provides RESTful APIs for the web and mobile management interfaces and serves as the configuration source for xiaozhi-server.

    Core Functions:

    • User & Device Management: Managing user accounts, device registration, and permissions.
    • Configuration Centralization: Storing AI service API keys, model selections, TTS parameters, and OTA firmware versions.
    • Data Persistence: Uses MySQL for relational data and Redis for high-performance caching.
    • API Documentation: Accessible via Knife4j (Swagger) at /xiaozhi/doc.html.

  3. Overview of xiaozhi-esp32-server

    main

    The xiaozhi-esp32-server provides backend services for the xiaozhi-esp32 open-source hardware project. It is designed to support intelligent terminal hardware using human-machine symbiotic intelligence. The server is implemented using Python, Java, and Vue and follows the Xiaozhi-Kommunikationsprotokoll (Xiaozhi Communication Protocol).

    Key supported protocols and features include:

    • MQTT + UDP
    • Websocket
    • MCP endpoints
    • Voiceprint recognition

  4. Overview of manager-mobile (Mobile Management App)

    main

    The manager-mobile is a cross-platform mobile management tool built with uni-app v3, Vue 3, and Vite. It allows administrators to manage the system on the go.

    Supported Platforms:

    • iOS
    • Android
    • WeChat Mini Program
    • H5

    Core Capabilities:

    • Mobile-optimized user login and device management.
    • AI service configuration.
    • Real-time system monitoring.

  5. Overview of Xiaozhi Backend Service

    main

    The xiaozhi-esp32-server provides backend services for the open-source xiaozhi-esp32 intelligent hardware project. It is designed based on human-machine symbiotic intelligence theory and implements the Xiaozhi Communication Protocol using Python, Java, and Vue.

    Key capabilities include:

    • Support for MQTT+UDP, Websocket, and MCP access points.
    • Voiceprint recognition and knowledge base integration.
    • Integration with various LLM, VLLM, TTS, and VAD providers.

  6. Overview of Xiaozhi implemented features

    main

    Xiaozhi provides a comprehensive suite of intelligent interaction features:

    • Core Architecture: Based on MQTT+UDP gateway, WebSocket, and HTTP servers.
    • Voice Interaction: Supports streaming ASR, streaming TTS, and VAD (Voice Activity Detection).
    • Voiceprint Recognition: Real-time speaker identity recognition for personalized LLM responses.
    • Multimodal Perception: Supports VLLM (Vision LLM) for visual interaction.
    • Intelligence & Memory: Supports multiple LLMs, Function Calling, and various memory systems (local short-term, mem0ai, PowerMem).
    • Knowledge Base: Supports RAGFlow for RAG-based responses.
    • Tool & Command Support: Supports IOT protocols, MCP (Model Context Protocol) for both client and server, and sending commands to ESP32 devices via MQTT.
    • Management: Web-based management interface for users, devices, and system configuration.

  7. Overview of Xiaozhi features

    main

    Xiaozhi provides a comprehensive smart interaction ecosystem:

    • Core Architecture: Based on MQTT+UDP, WebSocket, and HTTP server with full dashboard management and authentication.
    • Voice Interaction: Supports ASR/TTS streaming, VAD (Voice Activity Detection), and multi-language support.
    • Voiceprint Recognition: Multi-user registration and real-time identity recognition for personalized LLM responses.
    • Intelligence: Supports multiple LLMs, VLLMs (vision), and intent recognition via function calling or plugins.
    • Memory & Knowledge: Supports local short-term memory, mem0ai interface, PowerMem smart memory, and RAGFlow for knowledge base retrieval.
    • Tool & Command Execution: Supports IOT/MCP protocols (client/server), custom tool functions, and MQTT-based command sending to ESP32 devices.
    • Management: Web-based backend for user, system, and device management (supports Simplified Chinese, Traditional Chinese, and English).
    • Extensibility: Plugin system with hot-loading capabilities.

  8. Overview of xiaozhi-esp32-server Architecture

    main

    The xiaozhi-esp32-server is a distributed backend system designed to support ESP32-based intelligent hardware. It acts as a bridge between physical hardware, AI services (ASR, LLM, TTS), and user management interfaces. The system is split into a core AI engine and a management suite to separate real-time processing from administrative tasks.

    Core Components

    • ESP32 Hardware (Client): Captures voice, sends raw audio via WebSocket, receives synthesized audio, and controls IoT peripherals.
    • xiaozhi-server (Core AI Engine): A Python-based server that handles real-time bidirectional WebSocket communication with ESP32. It manages Voice Activity Detection (VAD), ASR, LLM interaction, TTS, and a plugin system for IoT control.
    • manager-api (Management Backend): A Java Spring Boot service providing RESTful APIs for user authentication, device registration, and system configuration. It uses MySQL for persistence and Redis for high-speed caching.
    • manager-web (Web Control Panel): A Vue.js SPA for administrators to configure AI services, manage users, and monitor devices.
    • manager-mobile (Mobile Console): A uni-app (Vue 3 + Vite) application for Android, iOS, and WeChat Mini Programs to manage the system on the go.
    • digital-human (Test Module): A standalone Python+Web module for debugging digital human interaction flows, featuring a browser-based test page and Sherpa-ONNX wake word detection.

  9. Overview of Xiaozhi ESP32 Backend Service

    main

    The xiaozhi-esp32-server is a backend service designed for smart hardware projects, specifically intended to work with xiaozhi-esp32 ESP32-based hardware. It provides the necessary infrastructure for voice recognition, knowledge bases, and communication between hardware and AI models.

    Key Technical Details:

    • Implementation Languages: Python, Java, and Vue.
    • Communication Protocols: Supports MQTT+UDP, Websocket, and MCP (Model Context Protocol) endpoints.
    • Core Capabilities: Voice recognition, knowledge base integration, and smart device control.

  10. Understand the `manager-mobile` Mobile Console

    main

    The manager-mobile is a cross-platform mobile application built with uni-app v3, Vue 3, and Vite.

    Platform Support:

    • Android: Supported
    • iOS: Supported
    • WeChat Mini Program: Supported
    • H5: Not supported

    Key Features:

    • Mobile Management: Optimized interface for user login, device management, and AI configuration on the go.
    • Unified Codebase: Uses uni-app to target multiple platforms from a single source.
    • State Management: Uses Pinia with pinia-plugin-persistedstate for persistent storage.

  11. Understand the `digital-human` Test Module

    main

    The digital-human module is an independent testing environment used to debug digital human interaction flows without needing the full system.

    Key Components:

    • Local Test Page: An HTML/CSS/JS interface to validate playback, reception, and interaction logic.
    • Wake Word Runtime: A Python-based runtime using Sherpa-ONNX for local keyword detection.
    • Event Bridge: A WebSocket/HTTP bridge that synchronizes wake word states between the local runtime and the web page.
    • Usage: Start the module using start.py to launch the local HTTP service and test page.

  12. Understand the `xiaozhi-server` Core AI Engine

    main

    The xiaozhi-server is a Python-based intelligent core that processes voice interactions from ESP32 devices. It manages the dialogue flow, integrates AI services, and executes custom functions.

    Key Capabilities:

    • AI Service Integration: Uses an AI Service Provider Pattern (core/providers/) to switch between different ASR, TTS, LLM, and VAD vendors via abstract base classes.
    • WebSocket Communication: Manages ESP32 connections using websockets, providing each client with an isolated ConnectionHandler to maintain session state.
    • Modular Message Handling: Dispatches tasks to specialized handlers in core/handle/ (e.g., receiveAudioHandle.py for audio, intentHandler.py for NLU, functionHandler.py for plugins).
    • Plugin System: Supports custom 'skills' (like weather or IoT control) via the plugins_func/ directory. Plugins define functions and schemas that the LLM can trigger via function calling.
    • Dynamic Configuration: Merges local config.yaml settings with remote configurations fetched from manager-api via manage_api_client.py.