DOCSEARCHbyAlgolia
Submit Docs

WebAV SDK Documentation

repository·main·Indexed 24 days ago

https://github.com/webav-tech/webav

An SDK for creating and editing video files on the web platform using WebCodecs. It includes @webav/av-cliper for audio/video processing and synthesis, @webav/av-recorder for recording MediaStreams into MP4 files, and @webav/av-canvas for composing media sources including text, images, video, audio, and user/display media into a single MediaStream.

Tokens
22.3K
Snippets
37
Records
158
Agent score
83%

What's inside WebAV

  1. Overview of WebAV modules

    main

    WebAV consists of three primary modules, each serving a specific role in audio/video processing:

    • @webav/av-cliper: The foundational API for audio/video data processing. It handles tasks such as decoding, synthesizing, concatenating, and clipping audio/video data.
    • @webav/av-recorder: Used for recording a MediaStream and outputting it as an MP4 video. In a browser environment, you can obtain a MediaStream from various sources like <canvas>, <video>, cameras, microphones, or screen sharing.
    • @webav/av-canvas: Provides an AVCanvas (a programmable "canvas") that allows users or code to add and control various assets. It supports outputting a MediaStream, which enables streaming the canvas content to a server or recording it as a local video.

    Supported Assets in @webav/av-canvas:

    • Camera
    • Microphone
    • Screen
    • Audio/Video files
    • Images
    • Text

  3. Overview of WebAV

    main

    WebAV is a web-based video editing SDK built on top of the WebCodecs API. It is designed for high-performance client-side video processing, allowing developers to perform video editing tasks directly in the browser without server-side costs or privacy concerns.

    Key Features

    • Cross-platform: Runs in Edge, Chrome, and Electron.
    • Zero Server Cost: All computations are performed on the client/browser.
    • Privacy-focused: User data is never uploaded to a server.
    • High Performance: Achieves performance close to native solutions, approximately 10–20x faster than ffmpeg.wasm.
    • Extensible: Designed to work seamlessly with the Canvas API and WebAudio API for custom functionality.
    • Lightweight: The package size is approximately 50kb (minified + gzipped, without tree-shaking).

  4. Introduction to WebAV

    main

    WebAV is an SDK designed for creating and editing video files directly on the Web platform. It is built on top of the WebCodecs API, enabling high-performance, client-side media processing.

    Key Characteristics

    • Client-side Computation: All processing happens in the browser, ensuring zero server costs and enhanced user privacy as no data is uploaded.
    • High Performance: Optimized to be 10 to 20 times faster than ffmpeg.wasm.
    • Lightweight: The package size is approximately 50kb (minified + gzipped).
    • Extensible: Designed to work seamlessly with standard Web APIs like Canvas and WebAudio for custom media effects.
    • Cross-platform: Compatible with Edge, Chrome, and Electron.

  5. Use AVCanvas to generate MediaStream

    main

    AVCanvas is a tool used to combine various media sources—including Text, Image, Video, Audio, UserMedia (camera/microphone), and DisplayMedia (screen sharing)—into a single MediaStream.

    This MediaStream can then be used with AVRecorder to output MP4 streams, which can be saved as local files or pushed to a server.

  6. Access Web Audio and Video resources

    main

    For developers starting with media in the browser, the following resources are available:

    • Web Audio and Video Knowledge Graph: A collection of related APIs, SDKs, articles, and external products to help beginners.
    • WebCodecs Explainer: Detailed documentation on the WebCodecs API, which enables high-performance media processing in the browser.
    • WebGL Chromakey: Information on real-time green matting using WebGL.

  7. Core concepts of AVCliper

    main

    AVCliper is a WebCodecs-based engine used to synthesize video by combining video, audio, images, and text with animation support. It relies on three main architectural abstractions:

    1. Combinator: The video synthesizer. You add OffscreenSprite instances to a Combinator to produce a video stream. Currently, it only supports outputting binary streams in the MP4 format.
    2. IClip: An abstraction for raw resources. It allows reading raw data at specific time slices via IClip.tick(time). Supported implementations include:
      • MP4Clip (Video)
      • AudioClip (Audio)
      • ImgClip (Images)
      • EmbedSubtitlesClip (Embedded SRT subtitles)
    3. BaseSprite: An abstraction for manipulable elements. It attaches spatial and temporal information (Rect: position, width, height, layer, and animation) to resources. There are two types:
      • OffscreenSprite: Wraps an IClip and adds it to a Combinator for background offscreen video synthesis.
      • AVCanvas: A canvas-based approach that includes VideoSprite, AudioSprite, ImgSprite, and TextSprite to allow programmatic or user-driven control over where resources are drawn.

  8. Core concepts of @webav/av-cliper

    main

    The @webav/av-cliper package is the foundational SDK for audio/video data processing. It provides the core abstractions for handling media assets and composing them into video.

    Key abstractions include:

    • IClip: An abstraction for audio/video assets. It parses video, audio, images, and subtitle resources to provide data to other modules.
    • Sprite<IClip>: Attaches spatial and temporal properties to an IClip. It allows you to control the position, time offset, and animation of media assets, enabling multi-asset collaboration.
    • Combinator: A composition engine that accepts multiple Sprite instances. It uses their position, layer (zIndex), and time offset information to synthesize and output a video file.

  9. Common Use Cases for WebAV

    main

    WebAV is suitable for several media processing workflows:

    • Batch Audio/Video Processing: Tasks such as adding watermarks, adding voiceovers, or embedding subtitles.
    • Media Product Development: Building applications for video editing, live streaming, or video animation production.

  10. Create a Custom IClip for programmatic content

    main

    If you need to generate video content programmatically rather than using existing files, use a Custom IClip. This is useful for:

    • Combining images and audio from third-party sources like canvas or pixi.js.
    • Merging programmatically generated assets (e.g., a countdown timer rendered on a canvas) into a video file.
    • Applying complex control logic to assets that are not standard video/audio files.

  11. Use @webav/av-canvas for complex media composition

    main

    The @webav/av-canvas module provides a programmable canvas that allows users or code to manipulate, add, and control various assets to produce a MediaStream. This stream can then be used for recording local video or pushing content to a server.

    Supported Materials:

    • Camera
    • Microphone
    • Screen
    • Audio and video files
    • Images
    • Text

  12. Decode and play video using WebCodecs, Canvas, and Web Audio

    main

    For scenarios requiring advanced playback control that the native <video> tag cannot provide, you can use a combination of WebCodecs, canvas, and Web Audio to decode and play video. This approach allows you to access the raw image and audio data of a video stream.

    Key capabilities of this approach:

    • Buffer control: Manage buffers for low-latency scenarios.
    • Frame dropping: Implement logic to drop frames based on device pressure.
    • Error recovery: Automatically recover when decoding exceptions occur.
    • Advanced playback speeds: Play frame-by-frame or at super-fast double speeds.
    • Audio handling: Play audio at double speed without pitch scaling.
    • Custom FPS: Control the playback frames per second manually.

    Note: This solution is more complex than using a standard <video> tag. Use the native <video> element whenever possible unless these specific low-level controls are required.