ESP-SR Speech Recognition Framework

ESP-SR is a speech recognition framework designed to help users build AI speech solutions on Espressif hardware. It is provided as a component for easy integration into projects. The framework consists of several specialized modules:

• Audio Front-end (AFE): Integrates AEC (Acoustic Echo Cancellation), VAD (Voice Activity Detection), BSS (Blind Source Separation), NS (Noise Suppression), and NSNET (Deep noise suppression).
• Wake Word Engine (WakeNet): Provides high-performance, low-memory footprint wake word detection (e.g., "Alexa", "Hi, ESP").
• VAD (VADNet): A voice activity detection model designed to replace WebRTC VAD for improved performance.
• Speech Command Recognition (MultiNet): A flexible offline model supporting up to 300 Chinese or English commands without requiring retraining.
• Speech Synthesis: Module for generating speech from text.

WakeNet is a neural network-based wake word engine designed for low-power embedded MCUs. It supports identifying up to 5 different wake words.

Technical Specifications

• Speech Feature Extraction: Uses the MFCC (Mel-frequency cepstrum) method.
• Audio Input Requirements:
  • Sampling Rate: 16 KHz
  • Channels: Mono
  • Encoding: signed 16-bit
  • Frame Window Width: 30 ms
  • Frame Step: 30 ms
• Trigger Mechanism: To ensure accuracy in continuous audio streams, the engine calculates an average value M of recognition results over several frames. A trigger command is issued only when M exceeds a specified threshold.

MultiNet is a lightweight offline model designed for the ESP32 series to recognize up to 200 custom speech commands.

Key Features:

• Supported Chips: ESP32-S3 (supports Chinese and English), ESP32 (supports Chinese).
• Customization: Supports adding, deleting, or modifying commands during runtime.
• Modes: Supports both single-recognition and continuous-recognition modes.
• Performance: Low latency (within 500ms) and low resource consumption.
• OTA Support: Models are stored in separate partitions, allowing for application OTA.

Input Requirements: MultiNet requires audio processed by an Audio Front End (AFE). The input format must be:

• Sample Rate: 16 KHz
• Bit Depth: 16 bit
• Channels: Mono (single channel)

MultiNet is a lightweight, offline speech command recognition model. It supports up to 200 commands, including user-defined ones, and allows for adding, deleting, or modifying commands during operation.

Key Features:

• Hardware Support:
  • esp32s3: Supports both Chinese and English speech commands. Supports online model switching.
  • esp32: Supports Chinese speech commands.
• Recognition Modes: Supports single recognition and continuous recognition.
• Performance: Low latency (within 500ms) and low resource consumption.
• Deployment: Models can be partitioned to support OTA (Over-the-Air) updates.

Input Requirements: MultiNet requires audio processed by an Audio Front End (AFE) algorithm in 16 KHz, 16 bit, mono format.

ESP Chinese TTS is a lightweight speech synthesis (Text-to-Speech) system specifically designed for embedded systems.

Limitations:

• Currently, only the Chinese language is supported.
• It is optimized for use with the ESP32-S3 platform.

ESP-SR is a speech recognition framework providing several specialized modules for audio processing and speech intelligence on ESP32 chips. The framework includes:

• Audio Front-end (AFE): Pre-processing modules for audio signals.
• Wake Word Engine (WakeNet): For detecting specific wake words.
• VAD Model (vadnet): Voice Activity Detection to identify speech presence.
• Acoustic Echo Cancellation (AEC): To remove echo from the microphone signal.
• Speech Command Word Recognition (MultiNet): For recognizing specific spoken commands.
• Speech Synthesis: Text-to-speech capabilities (currently only supports Chinese language).
• Model Selection and Loading: Tools for managing and loading speech models from flash memory.

Espressif's Text-to-Speech (TTS) is a lightweight speech synthesis system designed for embedded systems.

Key Features:

• Language Support: Currently supports Chinese only.
• Encoding: Input text must use UTF-8 encoding.
• Output Format: Uses a streaming output mechanism to reduce latency. The default output is mono, 16-bit @ 16000Hz.
• Capabilities: Automatic polyphone (multi-pronunciation) recognition, adjustable speech speed, and optimized digital number broadcasting.

System Components:

• Parser: Converts UTF-8 input text into a Pinyin list based on dictionaries and grammar rules.
• Synthesizer: Combines the Pinyin list with a predefined voice set to synthesize waveform files.

ESP-SR is a speech recognition framework for ESP32 and ESP32-S3 chips. It consists of the following core modules:

• Audio Front-end (AFE): Pre-processing for audio signals.
• Wake Word Engine (WakeNet): Detects specific wake words to trigger the system.
• Speech Command Word Recognition (MultiNet): Recognizes specific spoken commands.
• Speech Synthesis: Converts text to speech (currently supports Chinese language only).

The Espressif TTS (Text-to-Speech) model is a lightweight speech synthesis system optimized for embedded systems.

Key Features:

• Language Support: Currently supports only Chinese.
• Input Format: Text must be encoded in UTF-8.
• Output Characteristics: Supports streaming output to reduce latency, polyphonic pronunciation, and adjustable speech rates.
• Output Format: Default output is mono, 16-bit @ 16000 Hz.

Core Components:

1. Parser: Converts UTF-8 Chinese text into phonemes.
2. Synthesizer: Generates raw wave data from phonemes using a sound set.

Workflow: Text Input (UTF-8) $\rightarrow$ Parser (Phonemes) $\rightarrow$ Synthesizer (Wave Data) $\rightarrow$ Audio Output.

The Audio Front-end (AFE) framework processes audio data for speech recognition and voice communication. It provides several specialized algorithms:

• AEC (Acoustic Echo Cancellation): Removes echo from microphone input (supports up to two mics).
• NS (Noise Suppression): Suppresses non-human/stationary noise in single-channel audio.
• BSS (Blind Source Separation): Separates target sound sources from interference using dual-channel processing.
• MISO (Multi Input Single Output): Selects a high signal-to-noise ratio channel from dual-channel input when WakeNet is not enabled.
• VAD (Voice Activity Detection): Provides real-time output of voice activity states.
• AGC (Automatic Gain Control): Dynamically adjusts output amplitude (amplifies weak signals, compresses strong signals).
• WakeNet: A neural-network-based wake word engine designed for low-power MCUs.