DOCSEARCHbyAlgolia
Submit Docs

Elixir WebRTC

repository·master·Indexed 19 days ago

https://github.com/elixir-webrtc/ex_webrtc

An implementation of the W3C WebRTC API in the Elixir programming language, designed to provide a standard-compliant WebRTC interface. It mirrors the W3C and JavaScript APIs and can be used independently or via the membrane_webrtc_plugin for the Membrane multimedia framework. Features include support for PeerConnections, DataChannels (via ex_sctp), and WHIP/WHEP broadcasting.

Tokens
23.1K
Snippets
75
Records
115
Agent score
66%

What's inside ex_webrtc

  1. Identify use cases for WebRTC and Elixir WebRTC

    master

    WebRTC is ideal for applications requiring low-latency (targeting low hundreds of milliseconds) for video, audio, or generic data. Common use cases include:

    • Videoconferencing: One-on-one meetings or large meeting rooms (e.g., Google Meet).
    • Media Ingress: Sending media from a presenter to a server for broadcasting via other protocols.
    • AI/ML Inference: Capturing voice and video from web users to process via machine learning models on the backend.
    • Peer-to-Peer Communication: Connecting two browsers or two Elixir peers directly.

  2. What is SDP munging?

    master
    SDP munging is the practice of manually modifying the SDP string to enable or disable specific WebRTC features. This process occurs in the window between calling createOffer/createAnswer and calling setLocalDescription. It is commonly used to enable experimental support for new codecs or to adjust session parameters that are not exposed via high-level APIs.

  3. What is a transceiver and why use it?

    master

    A transceiver is an entity responsible for both sending and receiving media data, consisting of an RTP sender and an RTP receiver. Each transceiver maps to one m-line in the SDP offer/answer.

    Using transceivers instead of just operating on tracks provides several advantages:

    • Connection before media access: You can establish a P2P connection before obtaining access to media devices (using the Warmup technique).
    • SDP Control: Transceivers map directly to the SDP offer/answer, allowing high control over what is sent on which m-line.
    • Consistent Offering: They allow you to offer to receive media in a manner consistent with offering to send media, avoiding the asymmetric behavior of older APIs.

    Key Concepts

    • direction: Your local preferred direction. It is created with sendrecv by default when using add_transceiver. It can be changed via add_track or remove_track.
    • currentDirection: The direction actually negotiated between the local and remote sides, which changes when applying local or remote SDP.
    • Transceiver Creation: Applying a remote offer that contains new m-lines creates a new transceiver with the recvonly direction. Applying a remote offer never steals an explicitly created transceiver (one added via add_transceiver), but using add_track might.

  4. What is Simulcast and how does it work in Elixir WebRTC?

    master

    Simulcast is a technique where a client sends multiple video encodings (differing in resolution and/or frame rate) to a server. The server then selects the appropriate encoding for each peer based on their available bandwidth, preferences, or UI layout.

    In the context of Elixir WebRTC:

    • Inbound Simulcast: Supported. The library automatically accepts incoming simulcast tracks without extra configuration.
    • Outbound Simulcast: Not currently supported.
    • Automatic Switching: Not currently supported (must be implemented manually using RTP munging).
    • Bandwidth Estimation: Not currently supported.

    To support 3 simulcast encodings, the minimal starting resolution must be 960x540.

  5. Understand WebRTC SDP negotiation

    master

    WebRTC uses Session Description Protocol (SDP) offer/answer exchanges to negotiate session parameters such as the number of audio/video tracks, their directions, and supported codecs. Because the exchange mechanism (signaling) is not standardized, developers often use WebSockets to transport these SDP strings between peers.

    Key concepts for negotiation:

    • mline: Represents a transceiver or data channel. It starts with m= and defines the media type, direction, and port.
    • Direction: Audio/video mlines use directions like sendrecv, recvonly, sendonly, or inactive.
    • Codecs: mlines include a list of supported codecs sorted by preference. Each codec is identified by a numeric payload type.
    • ICE Candidates: SDP can include ICE candidates. Note that the first offer created via createOffer typically won't have ICE candidates immediately; you may need to wait a few seconds and then read peerconnection.localDescription to find the gathered candidates.

  6. How WebRTC encryption flow works

    master

    WebRTC communication is always encrypted. The end-to-end (E2E) flow follows these steps:

    1. Establish ICE connection: Set up the network path.
    2. Perform DTLS handshake: Authenticate the peers.
    3. Create SRTP/SRTCP context: Derive encryption keys from the DTLS master secret.
    4. Media Flow: Encapsulate media into RTP $\rightarrow$ encrypt using SRTP $\rightarrow$ send via ICE (UDP).
    5. Data Flow: Encapsulate arbitrary data into SCTP $\rightarrow$ encrypt using DTLS $\rightarrow$ send via ICE (UDP).

    Note on Media Encryption: Media is encapsulated into RTP packets but not directly into DTLS datagrams. DTLS is used solely to obtain the keying material required for SRTP.

  7. Analyze Quality of Experience (QoE) and Jitter

    master

    QoE debugging is difficult due to environmental variables. Use chrome://webrtc-internals to monitor:

    • Packet Loss & Retransmission: Monitor nackCount, retransmittedPacketsSent, and packetsLost. Retransmissions (RTX) are critical; without them, even 1% packet loss significantly impacts quality.
    • Bitrate: Check the stability of incoming and outgoing bitrates.
    • Jitter: Monitor jitterBufferDelay and jitterBufferEmittedCount_in_ms (the average time a frame spends in the jitter buffer before playout). The jitter buffer is adjusted dynamically.

  8. Handle complex negotiation with multiple PeerConnections

    master

    When building a media forwarder or a multi-peer application, you must manage multiple PeerConnection instances. Each connection requires its own negotiation process.

    Negotiation Strategies

    When a new peer joins and needs to both send and receive tracks, you have two main strategies:

    1. Two-step negotiation:

      • First, the new peer acts as the offerer to add its own tracks to the server.
      • Second, the server acts as the offerer to add the existing tracks from other peers to the new peer's connection.

    2. Single-step negotiation (Transceiver pre-allocation):

      • If you know how many tracks the new peer will provide, you can use PeerConnection.add_transceiver on the server side to create empty transceivers before the peer joins.
      • This allows a single negotiation to handle both the incoming tracks from the peer and the outgoing tracks from the server.

    Detecting when negotiation is required

    Instead of manually tracking track changes, listen for the negotiationneeded event. In ex_webrtc, this is received as a message: {:ex_webrtc, _from, :negotiation_needed}.

    Note: If adding multiple tracks, batch them and perform a single renegotiation at the end to avoid unnecessary overhead.

  9. Understand the difference between Elixir WebRTC and Membrane

    master

    It is important to distinguish between these two technologies when choosing a multimedia solution:

    • Elixir WebRTC: A W3C WebRTC standard implementation written in almost pure Elixir. It aims to mirror the W3C (and JavaScript) API as closely as possible. It does not use Membrane internally.
    • Membrane: A broader multimedia framework supporting various protocols, codecs, and containers. Membrane uses Elixir WebRTC via its membrane_webrtc_plugin to enable WebRTC integration within the Membrane ecosystem.

  10. Understand Keying Material derivation

    master

    Keying material is derived from the master secret established during the DTLS-SRTP handshake using a Pseudo-Random Function (PRF).

    In WebRTC, the derivation formula is: keying_material = PRF(master_secret, client_random + server_random + context_value_length + context_value, label)

    Key details:

    • Label: To prevent key reuse across different contexts, WebRTC uses the specific string "EXTRACTOR-dtls_srtp" as the label.
    • Context Value: While optional in the general spec, context_value and context_value_length are not used in WebRTC.
    • Structure: The resulting keying material is divided into four parts:
      • ClientMasterKey
      • ServerMasterKey
      • ClientMasterSalt
      • ServerMasterSalt

    These parts are then fed into an SRTP KDF (Key Derivation Function) to produce the final encryption keys. The client uses the Client components, and the server uses the Server components.

    keying_material = PRF(master_secret, client_random + server_random + context_value_length + context_value, label)
// In WebRTC, label is "EXTRACTOR-dtls_srtp"

  11. Validate SDP Session Negotiation

    master

    To ensure the SDP offer/answer is correct, check the following:

    • m-lines: Verify the number of audio and video mlines. The number of mlines must remain constant between the offer and the answer.
    • Rejections: Look for mlines that are rejected, indicated by a port of 0 in the m= line or an a=inactive attribute. A port of 0 often occurs when a transceiver is stopped via stop().
    • Directions: Check the media direction attributes (a=sendrecv, a=sendonly, a=recvonly, or a=inactive).
    • Codecs: Verify codecs, their profiles, and payload types.

    Note: If one side offers a single track and the other side wants to send additional tracks, additional negotiation must be performed.

  12. Transceiver association: add_track vs add_transceiver

    master

    When a remote offer contains a new m-line, the PeerConnection attempts to associate it with an existing transceiver. The behavior depends on how the transceiver was originally created:

    • add_track: If a transceiver was created via add_track, the connection will attempt to 'steal' or associate that transceiver with the new m-line. This is common when users add tracks without explicitly managing transceivers.
    • add_transceiver: If a transceiver was explicitly created using add_transceiver, the connection is less likely to perform this automatic association/stealing.
    # Using add_transceiver (Explicit management)
{:ok, pc1_tr1} = PeerConnection.add_transceiver(pc1, :audio)

# Using add_track (Implicit management - allows 'stealing' association)
track = MediaStreamTrack.new(:audio)
{:ok, _pc2_sender} = PeerConnection.add_track(pc2, track)