WebCodecs API

WebCodecs provides a symmetric API for encoding and decoding media using specialized interfaces. The workflow involves transforming raw frames into encoded chunks (encoding) or transforming encoded chunks back into raw frames (decoding).

Encoding Workflow

1. Input: Provide AudioFrame or VideoFrame objects to an encoder.
2. Process: Use AudioEncoder or VideoEncoder to process the frames.
3. Output: The encoder produces EncodedAudioChunks or EncodedVideoChunks containing codec-specific bytes.

Decoding Workflow

1. Input: Provide EncodedAudioChunks or EncodedVideoChunks to a decoder.
2. Process: Use AudioDecoder or VideoDecoder to process the chunks.
3. Output: The decoder produces AudioFrame or VideoFrame objects.

Data Types

• AudioFrame: Contains decoded audio data. Can provide an AudioBuffer for rendering via AudioWorklet.
• VideoFrame: Contains decoded video data. Can provide an ImageBitmap for manipulation in WebGL or rendering to a <canvas>.

While MSE is widely used for low-latency streaming, WebCodecs is designed to address several limitations of the MSE model:

• Low-latency control: MSE's low-latency mode is often implicit and lacks standardization across browsers, whereas WebCodecs provides direct control.
• Buffer management: MSE requires applications to work around default 'stop on underrun' behaviors and requires manual containerization of input before buffering.
• Seeking/History: For use cases like cloud gaming that require no history for seeking, MSE requires manual workarounds to disable history, while WebCodecs allows for more direct control.
• Access to decoded data: MSE is primarily designed for rendering; WebCodecs provides easy access to decoded output, enabling use cases like transcoding.
• Encoding: MSE only addresses the decoding side; WebCodecs provides both encoding and decoding capabilities.

To upload live audio and video, convert MediaStreamTrack objects into ReadableStreams of unencoded media using MediaStreamTrackProcessor. Then, feed these streams into AudioEncoder and VideoEncoder instances.

Workflow:

1. Use MediaStreamTrackProcessor(track).readable to get a stream of raw frames.
2. Configure encoders with a specific codec and tuning parameters (like bitrate, framerate, width, and height).
3. The output callback of the encoder receives byte arrays (encoded chunks) which are not in a container (like MP4); your application is responsible for muxing/containerizing these chunks before transmission.
4. Use a loop or stream reader to continuously call encoder.encode(frame).

// The tracks are converted to ReadableStreams of unencoded audio and video.
const audio = (new MediaStreamTrackProcessor(audioTrack)).readable;
const video = (new MediaStreamTrackProcessor(videoTrack)).readable;

// Build and configure the encoders.
const audioEncoder = new AudioEncoder({
  output: muxAndSend,
  error: onEncoderError,
});
const audioPromise = audioEncoder.configure({
  codec: 'opus',
  tuning: {
    bitrate: 60_000,
  }
});

const videoEncoder = new VideoEncoder({
  output: muxAndSend,
  error: onEncoderError,
});
const videoPromise = videoEncoder.configure({
  codec : 'vp8',
  tuning: {
    bitrate: 1_000_000,
    framerate: 24,
    width: 1024,
    height: 768
  }
});

try {
  await Promise.all([audioPromise, videoPromise]);
} catch (exception) {
  // Configuration not supported.
  return;
}

// Helper to feed raw media to encoders as fast as possible.
function readAndEncode(reader, encoder) {
  reader.read().then((result) => {
    if (result.done) return;
    encoder.encode(result.value);
    readAndEncode(reader, encoder);
  });
}

// Feed the encoders data from the track readers.
readAndEncode(audio.getReader(), audioEncoder);
readAndEncode(video.getReader(), videoEncoder);

To achieve extremely low latency (e.g., for cloud gaming), you can decode video chunks and paint the resulting VideoFrame objects directly to a <canvas> element.

Key requirements:

• Use VideoDecoder with an output callback that handles the VideoFrame.
• VideoFrame is a CanvasImageSource, so you can use canvasContext.drawImage(frame, 0, 0).
• IMPORTANT: You must call frame.close() immediately after painting to release the frame and avoid stalling the decoder.
• Monitor decoder backpressure to ensure the application keeps up with the stream.

// The document contains a canvas for displaying VideoFrames.
const canvasElement = document.getElementById("canvas");
const canvasContext = canvas.getContext('2d', canvasOptions)

// Paint every video frame ASAP for lowest latency.
function paintFrameToCanvas(videoFrame) {
  // VideoFrame is a CanvasImageSource.
  canvasContext.drawImage(videoFrame, 0, 0);

  // IMPORTANT: Release the frame to avoid stalling the decoder.
  videoFrame.close();
}

const videoDecoder = new VideoDecoder({
  output: paintFrameToCanvas,
  error: onDecoderError
});

videoDecoder.configure({codec: 'vp8'}).then(() => {
  // The app fetches VP8 chunks, feeding each chunk to the decode
  // callback as fast as possible.
  streamEncodedChunks(videoDecoder.decode.bind(videoDecoder));
}).catch(() => {
  // App provides fallback logic when config not supported.
  ...
});