libav.js

repository·master·Indexed 19 days ago

https://github.com/yahweasel/libav.js

A WebAssembly and asm.js compilation of FFmpeg libraries (libavformat, libavcodec, libavfilter, libavutil, and libswresample) for low-level audio and video processing in the browser and Node.js. Version 6.9.8 provides various build variants to manage FFmpeg's size, support for Web Workers, and integration options for the WebCodecs API.

Tokens
11.1K
Snippets
34
Records
44
Agent score
67%

What's inside libav.js

  1. Understand libav.js configuration file structure

    master

    A libav.js configuration (or a configuration fragment) consists of several files that are concatenated during the build process. When you run make build-<variant>, the build system expects to find these files in configs/configs/<variant>.

    Configuration Files

    • deps.mk: Library dependencies used to direct the building of libraries (generated from deps.txt within fragments).
    • ffmpeg-config.txt: The specific configuration options passed to FFmpeg.
    • libs.txt: Libraries to be linked.
    • license.js: The license header prepended to built files.
    • link-flags.txt: Extra link flags required during the build.

    Fragment Behavior

    • Explicit Fragments: Some fragments exist as physical directories in configs/fragments containing the files listed above.
    • Implicit Fragments: For most FFmpeg features (protocols, formats, codecs, etc.), no physical fragment file is required. If you specify a name like codec-h263p, the system will automatically include the relevant FFmpeg configuration flag even if configs/fragments/codec-h263p does not exist.
  2. Integrate libav.js with WebCodecs

    master

    To leverage hardware-accelerated decoding via the WebCodecs API while using libav.js for demuxing/muxing, use these companion projects:

    • libavjs-webcodecs-polyfill: Provides a consistent WebCodecs API by falling back to libav.js for codecs not supported by the browser's native WebCodecs.
    • libavjs-webcodecs-bridge: A bridge that converts data formats between libav.js and WebCodecs. This allows you to use libav.js for demuxing and WebCodecs for decoding (or vice versa).
    • TransAVormer: A stream-based frontend that orchestrates libav.js and WebCodecs for media transformation.
  3. Access struct members using accessor functions

    master

    Most libav structs are exposed as raw pointers (numbers). To interact with their fields, use accessor functions following the pattern Struct_member (to read) and Struct_member_s (to write).

    For example, to interact with an AVCodecContext:

    • Read frame_size: await AVCodecContext_frame_size(ctx)
    • Write frame_size: await AVCodecContext_frame_size_s(ctx, frame_size)
  4. Read data using Streaming Reader Devices

    master

    Streaming reader devices (character devices) are ideal for data received sequentially from start to finish. They have no fixed size and cannot be seeked. When libav.js runs out of data, it will block and trigger a callback.

    Implementation Styles

    1. Callback Style (Reactive)

    Set the libav.onread callback. When libav needs data, it calls this function. You then provide data using libav.ff_reader_dev_send.

    2. Push Style (Polling)

    When using libav.ff_read_multi, you can pass the device name as the third argument. In this mode, ff_read_multi will return -libav.EAGAIN if it needs more data. You then call libav.ff_reader_dev_send to provide it.

    API Reference

    • libav.mkreaderdev(<name>): Creates the device.
    • libav.onread = function(filename, position, length) { ... }: The callback triggered when data is needed.
    • libav.ff_reader_dev_send(<name>, <data>): Sends a Uint8Array to the device. Send null as <data> to indicate EOF.
    • await libav.unlink(<name>): Deletes the device.
    // Example: Push Style with ff_read_multi
    await libav.mkreaderdev("input");
    
    // Set up callback to handle requests
    libav.onread = function(name, pos, len) {
        // ... logic to get data ...
        libav.ff_reader_dev_send("input", data);
    };
    
    while (true) {
        const [result, packets] = await libav.ff_read_multi(fmt_ctx, pkt, "input");
        if (result === -libav.EAGAIN) {
            // Data is being handled via the onread callback or manual push
        }
        // ... process packets ...
    }
  5. Read data using Block Reader Devices

    master

    Block reader devices are best for input data with a fixed, known size. They are simpler to use than streaming devices because they use a direct callback for specific positions.

    Implementation Details

    • Creation: Use await libav.mkblockreaderdev(<name>, <size>). The <size> in bytes is mandatory.
    • Data Delivery: When libav requests data, libav.onblockread is invoked with (<name>, <position>, <length>). You must respond by calling libav.ff_block_reader_dev_send.
    • Constraints: You cannot send extra data in advance; the device only 'remembers' the most recently sent data. If you are not using a worker, libav.js owns the data, so you may need to duplicate it if you intend to use it elsewhere.
    • Deletion: Use await libav.unlink(<name>).

    API Reference

    • libav.onblockread = function(name, pos, length) { ... }: The callback for read requests.
    • libav.ff_block_reader_dev_send(<name>, <position>, <data>): Sends a Uint8Array to a specific position.
    libav.onblockread = async function(name, pos, length) {
        const ab = await file.slice(pos, pos + length).arrayBuffer();
        libav.ff_block_reader_dev_send(name, pos, new Uint8Array(ab));
    };
    
    await libav.mkblockreaderdev("input", totalSize);
    
    while (true) {
        const [result, packets] = await libav.ff_read_multi(
            fmt_ctx, pkt, null, {limit: 32*1024}
        );
        // ... process packets ...
    }
  6. Handle data transfer with Frame and Packet objects

    master

    libav.js uses Frame and Packet objects to represent media data. While these can be represented as raw pointers, most functions copy data into these high-level JavaScript objects.

    Important: Data Transfer When running in worker mode, libav.js uses libavjsTransfer to move data between the main thread and workers. If you send a Frame or Packet into libav.js, any ArrayBuffer included in the libavjsTransfer array will be transferred (moved). This means you will lose access to the underlying data in the original thread. This behavior works in both directions.

  7. Write simple files in memory

    master

    For small files, you can use libav's default in-memory filesystem. Any libav function that writes to a filename that doesn't exist will create a 'simple file' by default. Once the file is finalized, you can retrieve its contents as a Uint8Array using libav.readFile(<name>).

    // libav writes to 'output.webm' automatically if no device is specified
    await libav.ffmpeg('-i', 'input', '-f', 'webm', 'output.webm');
    
    // Read the resulting file
    const data = await libav.readFile('output.webm');
    // 'data' is a Uint8Array owned by the caller
  8. Configure libav.js for bundlers

    master

    Because libav.js manages its own loading procedure based on the environment, bundling it can be problematic. If you must bundle it, you can override the frontend, factory, or backend using the LibAV.LibAV options object.

    Overriding Components

    • Frontend: Load a different frontend file manually.
    • Factory:
      • Pass toImport (string) to specify a different factory file.
      • Pass factory (function) to provide your own factory function.
    • Backend (Wasm): Pass wasmurl (URL or object URL) to specify the WebAssembly file.

    Handling Non-ES6 Modules

    If your bundler transforms the ES6 module into a non-ES6 format, you must pass noes6: true to LibAV.LibAV. If you are also using toImport, ensure you specify the non-ES6 factory.

    // Overriding the factory via toImport
    LibAV.LibAV({ toImport: "libav-but-better.wasm.js" });
    
    // Overriding the factory via a function
    LibAV.LibAV({ factory: LibAVFactory });
    
    // Overriding the Wasm backend
    LibAV.LibAV({ wasmurl: URL.createObjectURL(wasmBlob) });
    
    // Handling non-ES6 environments
    LibAV.LibAV({ noes6: true });
  9. Read data using Readahead Files

    master

    Readahead files allow you to use large Blob or File objects as if they were simple files. They are transparent and attempt to anticipate libav's next read to improve performance.

    • Creation: await libav.mkreadaheadfile(<name>, <content>) where <content> is a Blob or File.
    • Deletion: You must use await libav.unlinkreadaheadfile(<name>) instead of the standard libav.unlink to clear the readahead cache.
    await libav.mkreadaheadfile("large_video.mp4", blobObject);
    // ... use "large_video.mp4" ...
    await libav.unlinkreadaheadfile("large_video.mp4");
  10. Read data using WorkerFS Files

    master

    WorkerFS is a filesystem specifically for WebWorkers. It behaves similarly to readahead files but does not perform read-ahead and presents a blocking file interface.

    • Creation: name = await libav.mkworkerfsfile(<name>, <content>).
      • Note: This returns a new filename. You must use this returned name for subsequent libav operations.
      • <content> must be a Blob or File.
    • Deletion: Use await libav.unlinkworkerfsfile(<original_name>).
      • Note: Unlike creation, you pass the original name you provided, not the one returned by the creation function.

    When to use

    Use WorkerFS if you require a blocking file interface while reading in libav (e.g., when using interfaces other than the standard libavformat reader).

    const workerFileName = await libav.mkworkerfsfile("my_file", blob);
    // Use workerFileName for libav operations
    
    await libav.unlinkworkerfsfile("my_file");
  11. How to write tests for libav.js

    master

    Tests are JavaScript files located in tests/tests/. They are executed as async functions and receive a harness object h as an argument.

    Test Structure

    • Success/Failure: A test should perform its logic and throw an exception to indicate a failure.
    • Output: Use h.print or h.printErr for logging/outputting information.
    • Naming Convention: Files are typically named with a three-digit number followed by a description (e.g., 001-description.js).
    • Registration: New tests must be added to suite.json to be included in the run.

    Loading libav.js in a test

    You can obtain a libav.js instance from the harness:

    • await h.LibAV(): Reuses the same instance used by other tests.
    • await h.LibAV({}): Creates a separate instance (you can pass loading options in the object).

    The --include-slow flag is accessible within the test via h.options.includeSlow.

    async function testName(h) {
      const libav = await h.LibAV();
      // ... test logic ...
      if (failed) {
        throw new Error("Test failed");
      }
    }
  12. Use libav.js with TypeScript

    master

    To get type support, use the @libav.js/types package or the libav.types.d.ts file. This allows you to declare the LibAV global with correct types.

    import type LibAVJS from "libav.js";
    declare let LibAV: LibAVJS.LibAVWrapper;