DOCSEARCHbyAlgolia
Submit Docs

ez-ffmpeg

repository·main·Indexed 43 days ago

https://github.com/yeautyye/ez-ffmpeg

A safe and ergonomic Rust interface for FFmpeg integration (v0.19.0). It provides a high-level API for video processing, transcoding, and streaming, featuring an optional async mode for non-blocking operations via Tokio, hardware device discovery, and built-in WGPU-powered video effects such as chroma keying and color grading.

Tokens
48.7K
Snippets
141
Records
237
Agent score
56%

What's inside ez-ffmpeg

  1. Overview of ez-ffmpeg

    main

    ez-ffmpeg

    ez-ffmpeg provides a safe and ergonomic Rust interface for FFmpeg integration. It is designed to follow FFmpeg's original logic and parameter structures closely, making it easy to migrate CLI commands to Rust code.

    Key Capabilities:

    • Safe API: Exposes a safe public API while handling internal FFmpeg FFI with audited unsafe code.
    • FFmpeg Parity: The transcoding pipeline is ported from FFmpeg's fftools/ffmpeg sources, maintaining the same stage semantics and function names.
    • Advanced Features: Supports custom Rust filters, GPU-accelerated filters (via wgpu), an embedded RTMP server, and typed detection APIs (e.g., black, silence, scene, loudness, and crop detection).
    • Recipes: Includes one-shot recipes for common tasks like generating thumbnails, GIFs, and HLS, as well as an HDR-to-SDR tone-mapping cookbook.

  2. Using animated effects with play_time

    main

    Animated effects (such as soul, sway, wave, and swirl) do not use a frame counter for their phase. Instead, they read ez.play_time, which is derived from pts × time_base.

    This ensures that animation speed remains stable across different frame rates and remains consistent during seeks. If a frame lacks a timestamp (AV_NOPTS_VALUE), play_time defaults to 0. All animated effects are designed so that the t = 0 phase is either the identity state or a mild pose.

  4. Choosing between `set_framerate` and the `fps` filter

    main

    When modifying video framerate in ez-ffmpeg, choose your method based on the complexity of your task:

    MethodBest Use Case
    set_framerate on OutputQuick and efficient simple adjustments where only the output framerate needs to change.
    fps FilterComplex workflows involving input stream manipulation or when the framerate change must be part of a larger filter sequence.

  5. How to use time parameters on Input vs Output

    main

    In ez-ffmpeg, time-based clipping methods can be called on different parts of the pipeline:

    1. On the Input object: Controls how the input file is read. This is typically used to skip data at the source.
    2. On the Output object: Controls the output stream parameters. This is used to define how the resulting file is written or processed.

    Method Reference

    MethodPurposeExample (1s)
    set_start_time_usSkip initial video.set_start_time_us(1_000_000)
    set_recording_time_usLimit total duration.set_recording_time_us(1_000_000)
    set_stop_time_usStop at absolute timestamp.set_stop_time_us(1_000_000)

  6. Use stream specifiers for stream-specific metadata

    main

    When applying metadata to specific streams (rather than the global file level), ez-ffmpeg uses FFmpeg's stream specifier syntax. Common specifiers include:

    • v:0: The first video stream
    • a:0: The first audio stream
    • s: All subtitle streams
    • v: All video streams

  7. Understand the wgpu_filter::effects test strategy

    main

    The wgpu_filter::effects module employs several strategies to ensure correctness across shader validation, memory layout, and visual output:

    1. Shader Validation

    Shaders are validated offline using naga (matching the wgpu major version). This ensures that shader-text errors are caught during cargo test even in environments without a physical GPU.

    2. Memory Layout (Layout Sentinels)

    To prevent runtime errors caused by WGSL's uniform struct alignment requirements, all parameter structs are designed such that their size_of is pinned to a multiple of 16 bytes. This ensures the host-side data matches the GPU's expected alignment.

    3. Visual Correctness (GPU Runtime Oracles)

    Visual effects are verified using 'oracles'—specific input/output expectations that validate the mathematical and visual behavior of the shaders. These include:

    • Identity/Neutral Checks: Verifying that neutral parameters (e.g., zero amplitude for sway, zero strength for fisheye, or t = 0 for swirl) result in an identity transformation.
    • Directional Contracts: Ensuring effects like rotation or mirroring move pixels in the expected directions.
    • Edge/Boundary Behavior: Testing how effects like sharpen, soft_blur, fisheye, and sway behave at the frame edges or during overscan.
    • Chroma Keying Specifics: Validating that chroma_key correctly handles similarity tolerance, skin-tone preservation, non-black backgrounds, and linear ramp blending (rather than hard thresholds).
    • Temporal Consistency: Using a fixed 1/30 time base to drive play_time (e.g., PTS 15 = 0.5s) to ensure animated effects (like swirl or wave) are predictable and reproducible.

  8. Understanding the chroma_key effect math

    main

    The chroma_key effect follows FFmpeg/OBS reference mathematics rather than a novel implementation.

    Key characteristics:

    • Color Space: Both the pixel and the key color are projected to CbCr using FFmpeg's chromakey RGB→UV coefficients.
    • Filtering: The distance (not the color) is box-filtered over a 3×3 neighborhood to handle chroma blocking.
    • Matte: Uses FFmpeg's linear (d - similarity)/blend ramp.
    • Spill Suppression: Uses FFmpeg despill's self-gating clamp.
    • Output: Because the pipeline's output is alpha-less YUV, the matte composites the result over a provided solid background color.

    Note: While it follows FFmpeg math, it may not be bit-identical to FFmpeg when using non-BT.601 input ranges due to how the shader decodes pixels to RGB.

  9. How RTMP backpressure management works

    main

    The RTMP implementation uses a tiered backpressure system in its write queue to handle slow clients. When the write queue reaches certain thresholds, the server changes its behavior to prioritize stream stability and keyframes over full data delivery:

    LevelThresholdBehavior
    Normal< 1MBAll frames enqueued
    Warning1-2MBDrop non-keyframe video, keep audio + keyframes
    High2-4MBKeep keyframes and sequence headers only
    Critical≥ 4MBDisconnect client

    If a client is disconnected unexpectedly, it is likely because they hit the Critical threshold.

  10. Understand the Packet Sink (WebCodecs-Style AVC Feed) pattern

    main

    The Packet Sink pattern allows you to encode synthetic video using libx264 (or other H.264 encoders) and map the resulting encoded packets directly to a WebCodecs-style decoder feed. This bypasses the need to write to a container or demux bytes back from a file.

    Key characteristics of this pattern:

    • Configuration: The on_stream_info callback provides a VideoDecoderConfig equivalent, including the RFC 6381 codec string (e.g., avc1.PPCCLL), the avcC record (description), coded dimensions, and sample aspect ratio.
    • Data Delivery: The on_packet callback provides samples shaped like EncodedVideoChunk, containing:
      • AVCC payload (4-byte NAL length prefixes).
      • Microsecond timestamps and durations (via *_us helpers).
      • Key/delta flags (IDR-accurate).
    • Lifecycle: The on_end callback signals a clean finish.
    • Concurrency & Backpressure: Callbacks run serially on a dedicated delivery thread within a stateful PacketSinkHandler. Because they run on this thread, the handler does not require locking. However, callbacks block the encoding pipeline while running; users should refer to the packet_sink module documentation to respect the backpressure contract.

  11. Understand the Packet Sink (A/V Transport Packager Stub) example

    main

    The packet_sink_av_transport example demonstrates how to encode synchronized video (using libx264) and audio (using AAC) into a single packet sink. It simulates the behavior of an fMP4 segmenter or an RTP/SRT sender session by routing both streams through a single stateful PacketSinkHandler.

    Key behaviors of this implementation include:

    • Stream Announcement: The on_stream_info method announces every track once with consumer-ready configuration (e.g., avc1.PPCCLL + avcC for video, mp4a.40.X + AudioSpecificConfig for audio, dimensions, sample rate, and channel layout).
    • Stream Routing: Packets are routed by stream_index. Because no cross-stream interleaving order is guaranteed, each track maintains its own independent timing state.
    • Time Synchronization: All tracks share a single zero-based time origin. The applied_offset is used to recover each stream's original encoder timeline, ensuring true A/V offsets are preserved in the packaged output.
    • Validation: Each track ensures strictly increasing DTS (Decoding Time Stamps), positive durations, and IDR-accurate sync flags, which are validated by a strict tier before delivery.

  12. Choose the right thumbnail extraction method

    main

    Select an extraction strategy based on your performance and accuracy requirements:

    MethodUse CaseAccuracy
    Single ThumbnailA single preview image for a video.High
    Multiple ThumbnailsSeveral preview images (consecutive or sampled via fps).High
    Keyframe-OnlyScrub previews, batch thumbnail grids, or high-speed processing.Low (snaps to nearest keyframe)