DOCSEARCHbyAlgolia
Submit Docs

esp-webrtc-solution

repository·main·Indexed 18 days ago

https://github.com/espressif/esp-webrtc-solution

A comprehensive framework for implementing WebRTC applications on Espressif hardware. It provides core peer connection logic, media capture, and an AV Render System for synchronized audio (I2S) and video (LCD/MJPEG) output. The solution includes the codec_board component for managing media peripherals like ES8311/ES8388 codecs, SDcards, and cameras, as well as an ESP Peer implementation for real-time P2P communication using G.711A audio and data channels.

Tokens
16.5K
Snippets
45
Records
89
Agent score
64%

What's inside esp-webrtc-solution

  1. Overview of esp_webrtc_solution components

    main

    The esp_webrtc_solution repository provides a complete stack for building WebRTC applications on Espressif hardware. The core functionality is built upon several key components:

    • esp_webrtc: The core WebRTC implementation.
    • esp_peer: Realization of the WebRTC PeerConnection.
    • esp_capture: Component used for capturing media data (available on the Espressif component registry).
    • av_render: Component used for playing back media data.

    Developers can use these components directly or refer to the provided solutions in the solutions/ directory to see how they integrate into functional applications.

  2. Overview of the WebRTC USB Camera Bridge Demo

    main

    The WebRTC USB Camera Bridge Demo transforms an ESP32 device into a WebRTC-to-USB UVC bridge.

    Workflow:

    1. A browser captures webcam and microphone media.
    2. The browser sends media to the ESP32 via WebRTC using AppRTC signaling.
    3. The ESP32 receives the stream via esp_peer.
    4. The ESP32 forwards video frames to its USB UVC device interface.
    5. A host PC recognizes the ESP32 as a standard, plug-and-play USB camera (UVC device).

    Note: Currently, only video is forwarded to the UVC output. Audio is negotiated in WebRTC but is ignored by the ESP32.

  3. Overview of OpenAI Real-Time Chat Demo

    main

    This demo uses esp_webrtc to establish a real-time chat connection with OpenAI, enabling voice-based interaction and device control via function calls. It implements the OpenAI Realtime GA WebRTC ephemeral token flow:

    1. Requests a short-lived client secret via POST /v1/realtime/client_secrets using an API key.
    2. Exchanges SDP via POST /v1/realtime/calls using the ephemeral token.

    Key advantages over the standard OpenAI Realtime Embedded SDK include integrated media components, optimized OPUS encoding/decoding via esp_audio_codec, Acoustic Echo Cancellation (AEC), and an enhanced esp_peer implementation.

  4. Overview of codec_board

    main

    The codec_board component provides board support for various ESP32 series boards, specifically focusing on media-related peripherals. It simplifies development by:

    • Gathering board-specific peripheral settings from a configuration file.
    • Automatically adding drivers for supported peripherals.
    • Providing high-level APIs to interact with peripherals via manageable handles.

    This module is a lightweight alternative to esp-bsp when you only need support for media devices (Codec, SDcard, Camera, LCD) rather than full board support.

  6. Overview of ESP_WebRTC Architecture

    main

    The esp_webrtc component is an integrated solution for building WebRTC connections, based on libpeer with several enhancements including full TURN support, optimized connection speeds, and separate tasks for sending and receiving data. It is composed of three main layers:

    1. Signaling (esp_peer_signaling): Responsible for peer detection and exchanging SDP information or control commands. Users can implement custom signaling or use existing samples like esp_signaling_get_apprtc_impl.
    2. PeerConnection (esp_peer): Handles ICE to find connectable peers and establishes the connection. It manages both media (RTP) and user data (SCTP DataChannel). A default implementation is available via esp_peer_get_default_impl.
    3. WebRTC Solution (esp_webrtc): A high-level API that combines signaling, PeerConnection, and a media system. It automates audio/video capture, transmission, and rendering (using av_render), allowing developers to focus on codec configuration.

  7. Customize AV_Render with custom renderers

    main

    If the default implementations are insufficient, you can implement custom renderers.

    av_render abstracts renderers using:

    • audio_render_ops_t for audio.
    • video_render_ops_t for video.

    To implement your own, use the existing implementations as references:

    • I2S Audio: render_impl/i2s_render.c
    • LCD Video: render_impl/lcd_render.c (uses esp_lcd)

  8. How WHIP signaling works in the demo

    main

    The demo implements WHIP signaling using a custom implementation called esp_signaling_get_whip_impl to exchange SDP with the server.

    The signaling lifecycle follows these steps:

    1. Initial SDP: The client sends an initial SDP to the WHIP server.
    2. ICE Update: If the server response contains a STUN server URL, the client calls esp_peer_update_ice_info and then sends a PATCH request with new candidates to notify the server.
    3. Session Termination: When signaling is closed, the client sends a DELETE request to remove the session from the server.

  9. How AV_Render works

    main

    AV_Render is a lightweight player designed for audio and video playback using a push-based playback model.

    Instead of pulling data, the user pushes encoded stream data into the player. The workflow is:

    1. The user provides stream information.
    2. The user pushes encoded data via av_render_add_audio_data and av_render_add_video_data.
    3. The player selects the appropriate codec and decodes the stream.
    4. The decoded output is synchronized and rendered (e.g., audio via I2S, video via LCD).

    Architecture

    av_render processes audio and video in parallel pipelines with separate decode and render threads. A synchronization module ensures alignment based on the selected sync_mode.

    Sync Modes

    • None: Audio and video run independently.
    • Audio: Video is synchronized to the audio clock.
    • Time: Both streams follow system time or timestamps.

  10. Manual Peer Connection Control in esp_webrtc

    main

    The DoorBell Demo modifies the standard esp_webrtc behavior to allow signaling to be established independently of the WebRTC peer connection. This is necessary to support custom signaling flows (like the doorbell 'ring' event) before a call is actually accepted.

    Key modifications in esp_webrtc used by this demo:

    • no_auto_reconnect Configuration: Disables the automatic building of the peer connection immediately upon establishing a signaling connection.
    • esp_webrtc_enable_peer_connection API: Allows the developer to manually trigger the connection or disconnection of the peer connection based on signaling messages.

  11. Typical Call Sequence for esp_webrtc

    main

    The esp_webrtc high-level API follows specific lifecycles for building and stopping connections.

    Connection Build Flow

    1. APP calls esp_webrtc_start.
    2. esp_webrtc starts signaling via esp_signaling_start.
    3. Once on_ice_info() is received, esp_webrtc calls esp_peer_open.
    4. Upon on_connected(), esp_webrtc calls esp_peer_new_connection.
    5. esp_peer triggers on_msg(ESP_PEER_MSG_TYPE_SDP).
    6. esp_webrtc sends this via esp_signaling_send_msg.
    7. When on_msg(ESP_PEER_SIGNALING_MSG_SDP) is received from signaling, esp_webrtc calls esp_peer_send_msg.
    8. Connection is established when on_state(ESP_PEER_STATE_CONNECTED) is reported.

    Stop Flow (User Initiated)

    1. APP calls esp_webrtc_stop.
    2. esp_webrtc calls esp_peer_disconnect.
    3. esp_webrtc calls esp_peer_close.
    4. esp_webrtc calls esp_peer_signaling_stop.

    Disconnect Flow (Peer Leaves Normally)

    If the signaling layer receives on_msg(ESP_PEER_SIGNALING_MSG_BYE):

    1. esp_webrtc calls esp_peer_disconnect.
    2. esp_webrtc calls esp_peer_new_connection (to allow waiting for the peer to rejoin).
    sequenceDiagram
APP ->> esp_webrtc: esp_webrtc_start
esp_webrtc ->> esp_signaling: esp_signaling_start
esp_signaling -->> esp_webrtc: on_ice_info()
esp_webrtc ->> esp_peer: esp_peer_open
esp_signaling -->> esp_webrtc: on_connected()
esp_webrtc ->> esp_peer: esp_peer_new_connection
esp_peer -->> esp_webrtc: on_msg(ESP_PEER_MSG_TYPE_SDP)
esp_webrtc ->> esp_signaling: esp_peer_signaling_send_msg
esp_signaling -->> esp_webrtc: on_msg(ESP_PEER_SIGNALING_MSG_SDP)
esp_webrtc ->> esp_peer: esp_peer_send_msg
esp_peer -->> esp_webrtc: on_state(ESP_PEER_STATE_CONNECTED)

  12. Technical Implementation of Peer Demo

    main
    The Peer Demo utilizes the apprtc signaling implementation via the esp_signaling_get_apprtc_signaling function. The connection process follows the standard WebRTC connection build flow provided by the esp_webrtc component. It serves as a reference for implementing the esp_peer protocol API to set up WebRTC-based applications.