WebRTC JavaScript Code Samples

To build a video transformation workflow using Insertable Streams, you can use the Pipeline class to orchestrate three main components: a Source, a Transform, and a Sink.

Core Components

1. MediaStreamSource: Responsible for providing the raw input. Implement getMediaStream() to return a Promise<MediaStream>.
2. FrameTransform: Responsible for the actual video manipulation. Implement init() for setup and transform(frame, controller) to process each VideoFrame. Use the controller to queue the output frame.
3. MediaStreamSink: Responsible for consuming the processed stream. Implement setMediaStream(stream) to receive the transformed MediaStream.

Orchestration with Pipeline

The Pipeline class manages the lifecycle and connection of these components. You update the pipeline by calling updateSource(source), updateTransform(transform), and updateSink(sink). The pipeline automatically handles initialization, resource cleanup via destroy(), and stream re-routing when components change.

// Example conceptual usage
const pipeline = new Pipeline();

// 1. Set the source (e.g., a camera)
await pipeline.updateSource(myMediaStreamSource);

// 2. Set the transform (e.g., a grayscale filter)
await pipeline.updateTransform(myFrameTransform);

// 3. Set the sink (e.g., a video element)
await pipeline.updateSink(myMediaStreamSink);

You can intercept and analyze encoded video frames in a WebRTC stream by using the createEncodedStreams() method on an RTCRtpReceiver. This method returns an object containing a readable stream and a writable stream.

To analyze frames without disrupting the playback, you can use a TransformStream to process the readable side and then pipe the results to the writable side. This allows you to inspect metadata like keyframe indicators, timestamps, and synchronization sources.

In the provided implementation, the videoAnalyzer function is used as a transform logic to detect keyframes by checking the lowest value bit in the first byte of the encoded data (per RFC 6386).

function gotRemoteTrack(e) {
  // Access the encoded streams from the receiver
  const frameStreams = e.receiver.createEncodedStreams();

  // Pipe the readable stream through a TransformStream for analysis,
  // then pipe to the writable stream to continue the flow.
  frameStreams.readable.pipeThrough(new TransformStream({
    transform: videoAnalyzer
  }))
  .pipeTo(frameStreams.writable);

  remoteVideo.srcObject = e.streams[0];
}

The video processing pipeline follows a producer-transformer-consumer model using the WebRTC Insertable Streams API:

1. Producer: A MediaStreamTrackProcessor reads frames from a source MediaStreamTrack and exposes them via a ReadableStream of VideoFrame objects.
2. Transformer: A TransformStream is created using a FrameTransformFn. This stream sits between the processor and the generator, applying logic (like WebGL, Canvas2D, or WebCodecs) to each frame.
3. Consumer: A MediaStreamTrackGenerator provides a WritableStream that receives the processed frames. This generator then produces a new MediaStreamTrack which can be used in a RTCPeerConnection or displayed in a <video> element.

Data Flow: Source Track $\rightarrow$ MediaStreamTrackProcessor $\rightarrow$ TransformStream $\rightarrow$ MediaStreamTrackGenerator $\rightarrow$ Processed Track

To establish a connection, you must exchange ICE candidates between peers.

1. On the local side: Listen to the onicecandidate event on the RTCPeerConnection. When a candidate is generated, send its details (candidate string, sdpMid, and sdpMLineIndex) to the remote peer via your signaling server.
2. On the remote side: When a candidate is received via signaling, add it to the local peer connection using pc.addIceCandidate(candidate).

Note: If the signaling message contains a null candidate, call pc.addIceCandidate(null) to signal the end of the candidate gathering process.

// Local side: capturing candidates
pc.onicecandidate = e => {
  if (e.candidate) {
    const message = {
      type: 'candidate',
      candidate: e.candidate.candidate,
      sdpMid: e.candidate.sdpMid,
      sdpMLineIndex: e.candidate.sdpMLineIndex
    };
    signaling.postMessage(message);
  }
};

// Remote side: adding candidates
async function handleCandidate(candidate) {
  if (!candidate.candidate) {
    await pc.addIceCandidate(null);
  } else {
    await pc.addIceCandidate(candidate);
  }
}

You can calculate the real-time bitrate of a WebRTC connection by inspecting the getStats() report.

To calculate the bitrate:

1. Call pc.getStats().
2. Iterate through the reports to find a transport type report.
3. Use the selectedCandidatePairId from the transport report to find the corresponding candidate-pair report.
4. Calculate the difference in bytesReceived and the difference in timestamp between two consecutive polling intervals.
5. Formula: bitrate = (bytesNow - bytesPrev) * 8 / (timestampNow - timestampPrev) (result in bits per second).

Note: The timestamp used for calculation should be the one provided in the RTC stats report to ensure accuracy relative to the network stack's timing.

async function displayStats() {
  const stats = await pc2.getStats();
  let activeCandidatePair;
  
  stats.forEach(report => {
    if (report.type === 'transport') {
      activeCandidatePair = stats.get(report.selectedCandidatePairId);
    }
  });

  if (activeCandidatePair) {
    const bytesNow = activeCandidatePair.bytesReceived;
    const timestampNow = activeCandidatePair.timestamp;
    
    const bitrate = Math.round((bytesNow - bytesPrev) * 8 / (timestampNow - timestampPrev));
    
    bytesPrev = bytesNow;
    timestampPrev = timestampNow;
  }
}