DOCSEARCHbyAlgolia
Submit Docs

Espressif Audio Development Framework (ESP-ADF)

repository·release/v2.x·Indexed 25 days ago

https://github.com/espressif/esp-adf

An advanced framework for developing audio and video applications on ESP32-series SoCs. It includes the esp_codec_dev driver for unified audio codec interfacing, ESP_COZE for bidirectional streaming dialogue via WebSockets, and the ESP_Dispatcher framework for decoupling product features into event input, logic control, and execution output modules. The framework supports various audio formats, media protocols, and hardware codecs such as ES8311, ES8388, and ZL38063.

Tokens
124.7K
Snippets
184
Records
740
Agent score
79%

What's inside ESP-ADF

  1. Overview of ESP32-S3-Korvo-2 V3.1

    release/v2.x

    The ESP32-S3-Korvo-2 V3.1 is a multimedia development board based on the ESP32-S3 chip. It is designed for voice recognition and near/far-field voice wake-up applications.

    Key features include:

    • Microphone Array: Two-microphone array suitable for AIoT applications like wake word detection and speech command recognition.
    • Audio Peripherals: Integrated ES7210 4-channel ADC, ES8311 mono audio codec, and NS4150 3W mono Class D power amplifier.
    • Multimedia Support: Supports LCD extension boards, camera modules (JPEG video stream processing), and microSD cards (1-bit mode) for audio storage/playback.
    • Connectivity: Wi-Fi and Bluetooth LE via the ESP32-S3-WROOM-1 module.

  2. Overview of RTMP Example Applications

    release/v2.x

    The RTMP example demonstrates implementing live streaming scenarios on ESP32 series boards using the esp_rtmp module. It provides three distinct application modes:

    • Pusher Application: Gathers video (from camera) and audio (from microphone) and sends it to an RTMP Server. Supports configurable resolution, frame rate, audio format, sample rate, and channels.
    • Puller Application: Fetches audio and video data from an RTMP server and saves it to an SDCard in FLV format for offline playback.
    • Server Application: Sets up an RTMP server directly on the ESP32 board. It can connect to multiple clients and broadcast stream data from a Pusher to a Puller.

    Note: You can run the Pusher and Server applications simultaneously on the same board to create a local live streaming server.

  3. Overview of A2DP Sink and HFP Example

    release/v2.x

    This example demonstrates a dual-role Bluetooth application. It uses the A2DP protocol for receiving audio streams and the AVRCP protocol for media information and control notifications. Additionally, it can function as an HFP (Hands-Free Profile) Hands Free Unit to receive incoming phone calls.

    Data Pipeline Flow: [Bluetooth] ---> bt_stream_reader ---> i2s_stream_writer ---> codec_chip ---> speaker

  4. What is ESP_COZE?

    release/v2.x
    ESP_COZE is a bidirectional streaming dialogue component designed for the Coze platform. It enables real-time voice and text interaction with a Coze agent using WebSockets. It is optimized for low latency and lightweight operation on embedded devices like the ESP32, making it suitable for voice assistants and intelligent Q&A systems.

  5. Overview of the Core Dump Service example

    release/v2.x
    This example demonstrates how to configure and use the core dump service within the ESP-ADF. It simulates a system crash by referencing a non-existent assembly instruction. Once the crash occurs, the service automatically uploads the resulting core dump data to a specified HTTP server, which then receives and parses the information.

  6. Overview of the Audio Sonic Example

    release/v2.x

    The Audio Sonic example demonstrates the ADF sonic feature using two distinct pipelines:

    1. Recording Pipeline: Records sound to a microSD card. Triggered by long-pressing the [REC] key.

      • Pipeline Flow: [mic] ---> codec_chip ---> i2s_stream ---> wav_encoder ---> fatfs_stream ---> [sdcard]

    2. Playback Pipeline: Reads rec.wav from the microSD card, applies sonic processing (speed and pitch adjustment), and plays it back. Triggered by releasing the long-pressed [REC] key.

      • Pipeline Flow: [sdcard] ---> fatfs_stream ---> wav_decoder ---> sonic ---> i2s_stream ---> [codec_chip]

    Sonic Features: Users can adjust the following audio output effects using the [Mode] key:

    • Audio speed
    • Audio pitch

  7. Overview of ESP-ADF Peripherals

    release/v2.x

    The ESP-ADF provides a unified API for managing various hardware peripherals, including buttons, LEDs, SD Cards, and Wi-Fi. Peripherals are implemented using a common base API that is extended with specific functionality for each component.

    Availability of peripherals depends on the development board being used. The following table summarizes peripheral support for the primary ESP-ADF boards:

    PeripheralESP32-LyraTESP32-LyraTD-MSC
    Wi-FiYESYES
    SD CardYESYES
    SpiffsYESYES
    ConsoleYESYES
    TouchYES
    ButtonYES
    LEDYES
    ADC ButtonYES
    LED Controller (IS31FL3216)YES

  8. Overview of Voice Interaction with DuerOS 3.0

    release/v2.x

    This example demonstrates voice interaction using the DuerOS 3.2 cloud (developed by Baidu). It is designed for applications like smart speakers, smart toys, and voice control devices. The example utilizes several highly-encapsulated ADF APIs to manage complex audio and connectivity tasks:

    • esp audio (esp_audio.h): Manages audio playback.
    • wifi service: Manages provisioning and Wi-Fi connection.
    • audio recorder (audio_recorder.h): Manages wake-up detection and voice data.
    • display service (display_service.c): Manages system indicators.
    • Dueros service: Manages the connection to the DuerOS cloud.
    • esp_periph_set_register_callback (esp_peripherals.h) or Key service (input_key_service.c): Manages key events.

    Prerequisite: You must apply for a DuerOS profile on the DuerOS Open Platform and copy its content to ADF_PATH/components/dueros_service/duer_profile.

  9. Overview of AEC (Acoustic Echo Cancellation) Example

    release/v2.x

    The AEC example demonstrates how to play music while simultaneously performing echo cancellation on the audio recorded by a microphone. The processed audio is then encoded into WAV format and saved to a microSD card for later comparison with the original audio.

    Pipeline Architecture

    The example utilizes two distinct audio pipelines:

    1. Playing MP3 pipeline: Decodes an MP3 file from flash and sends it to the codec chip. [flash] ---> mp3_decoder ---> filter ---> i2s_stream ---> [codec_chip]

    2. Recording WAV pipeline: Captures audio from the codec chip, applies the AEC algorithm, encodes it to WAV, and writes it to the microSD card via FATFS. [codec_chip] ---> i2s_stream ---> filter ---> AEC ---> wav_encoder ---> fatfs_stream ---> [sdcard]

    Key Limitations

    • Unlike the complete algorithm mode, AEC mode does not allow the activation of AGC (Automatic Gain Control) and NS (Noise Suppression) features.
    • It supports audio input at an 8 kHz sampling rate.

  10. Overview of the VoIP Example

    release/v2.x

    The ESP VoIP example is a telephone client implementation based on the standard SIP (Session Initiation Protocol) protocol. It is designed for P2P or audio conference scenarios.

    Memory Consumption

    esp32-lyrat-mini-v1.2:

    • Total: 392,008 Bytes
    • In RAM: 236,328 Bytes
    • PSRAM: 155,680 Bytes

    Other boards:

    • Total: 252,900 Bytes
    • In RAM: 111,076 Bytes
    • PSRAM: 141,824 Bytes

  11. Overview of Wi-Fi, A2DP, HFP, and Bluetooth LE Coexistence Example

    release/v2.x

    This example demonstrates the coexistence of multiple wireless protocols on a single device. It creates several GATT services and starts advertising (ADV) with the name BLUFI_DEVICE to allow configuration of Wi-Fi and BluFi services.

    Key Features:

    • Wi-Fi/HTTP Stream: Once Wi-Fi is connected, the example demonstrates an HTTP STREAM module capable of playing music from an HTTP URI.
    • Bluetooth Classic (A2DP Sink): Implements an A2DP SINK. Bluetooth devices (like smartphones) can discover the device as ESP_ADF_COEX_EXAMPLE. Once connected, AVRCP can be used to control music playback status.
    • Mode Switching: Use the [Mode] key to switch between HTTP and A2DP audio.
    • Provisioning: Long press the [Set] key to enter network provisioning mode.

  12. Overview of Speech Recognition in ESP-ADF

    release/v2.x

    ESP-ADF provides a suite of speech recognition and processing capabilities, including:

    • Front-end speech processing: Signal conditioning and enhancement.
    • TTS (Text-to-Speech): Converting text into audible speech.
    • Voice Wake-up: Detecting specific wake words to trigger device actions.
    • Command Word Recognition: Identifying specific spoken commands.

    The framework utilizes an Element-based audio_recorder which integrates speech recognition and audio signal processing into an event-driven High-level API. This architecture allows for flexible configuration while providing a simplified interface for developers.