DOCSEARCHbyAlgolia
Submit Docs

typed-ffmpeg

repository·main·Indexed 22 days ago

https://github.com/lucemia/typed-ffmpeg

A type-safe interface for FFmpeg supporting Python and TypeScript. It provides IDE auto-completion, detailed typing, and filter graph serialization. The library offers version-specific packages for FFmpeg v5, v6, v7, and v8, and includes a shared runtime (ffmpeg-core) for managing Directed Acyclic Graph (DAG) models and command-line generation. Additionally, it features a visual, node-based FFmpeg Flow Editor for creating command pipelines.

Tokens
116K
Snippets
534
Records
633
Agent score
76%

What's inside typed-ffmpeg

  1. Overview of the FFmpeg Flow Editor

    main

    The FFmpeg Flow Editor is a visual, node-based interface designed for creating and managing FFmpeg command pipelines. It allows users to build complex FFmpeg commands, including complex filter chains and multiple inputs/outputs, through an interactive flow diagram.

    Key capabilities include:

    • Real-time command preview.
    • Type-safe FFmpeg command generation.
    • Support for various FFmpeg filters and options.

  2. What is ffmpeg-core?

    main

    ffmpeg-core is the shared runtime for all typed-ffmpeg multi-version packages (v5, v6, v7, v8). It contains the hand-written logic used to bridge the high-level API with the FFmpeg command line.

    It consists of four main layers:

    • DAG Layer (ffmpeg_core.dag): Handles filter graph representation and manipulation.
    • Compile Layer (ffmpeg_core.compile): Responsible for FFmpeg command-line generation.
    • IR Layer (ffmpeg_core.ir): Provides an Intermediate Representation for multi-backend support.
    • Common Utilities (ffmpeg_core.common): Handles serialization, caching, and schemas.

  3. Identify feature availability via docstrings

    main

    All typed-ffmpeg packages annotate version-specific availability directly in their docstrings.

    • Items available only from a certain version are marked with "New in FFmpeg X.0".
    • Items removed in a later version are marked with "Removed in FFmpeg X.0".

    Use these annotations to determine if a specific filter, codec, or format is compatible with your target FFmpeg version.

  4. Understand the Loopback Decoder (FFmpeg 7.0+)

    main

    A loopback decoder (-dec of:ost) is a specialized FFmpeg mechanism that decodes the encoded output of an existing output stream and exposes those decoded frames as a filtergraph input labeled [dec:N].

    Key Concepts:

    • Input: It references an already-defined output stream via its node and stream index.
    • Output: It produces a filterable stream usable within a filter graph.
    • Static Typing: The library attempts to determine the type (video or audio) of the tapped stream to provide type-safe access to .video and .audio properties.

  6. How loopback decoders and labels work

    main

    A loopback decoder (LoopbackDecoderNode) taps an existing OutputNode. In the generated FFmpeg command, the decoder's options are placed immediately after the tapped output's arguments, followed by the -dec <output_index>:<stream_index> flag.

    Key behaviors:

    • Labeling: FFmpeg labels these streams as [dec:N]. The index N is determined by the order in which -dec flags appear in the command line.
    • Stream Reuse: If you reuse a stream produced by a loopback decoder (e.g., using dec.video in two different filters), the library will automatically insert split or asplit filters to handle the multiple consumers. However, the tapped output stream itself (the one being decoded) is treated as a shared resource and is never split.
    • Multiple Decoders: You can have multiple distinct loopback decoders tapping the same output stream; each will receive its own -dec flag and unique [dec:N] label.
    source = ffmpeg.input("INPUT")
    
# Two distinct decoders tapping the same output stream
encoded = ffmpeg.input("INPUT").video.output(
        filename="-", f="null", vcodec="libx264"
    )
    
dec_a = encoded.loopback(0)
    dec_b = encoded.loopback(0, extra_options={"threads": 2})
    
# The resulting command will contain two -dec flags and [dec:0], [dec:1] labels
out = dec_a.video.hstack(dec_b.video).output(filename="OUT.mkv")

  7. How code generation works in typed-ffmpeg

    main

    Typed-ffmpeg generates its filter, codec, and format bindings by introspecting real FFmpeg binaries. Each version package (e.g., v5, v6, v7, v8) is derived from its corresponding FFmpeg major version.

    The generation pipeline uses three inputs per version:

    1. FFmpeg binary: Executed to extract available filters, codecs, muxers, and AV options.
    2. FFmpeg source code: A release tarball parsed for command-line option definitions.
    3. FFmpeg documentation: Version-correct doc/filters.texi parsed for filter descriptions and parameters.

    The output consists of Python modules located in packages/v{N}/src/ffmpeg/ containing typed filter functions, codec/format classes, and stream methods.

  8. Understanding Loopback Decoders (-dec / [dec:N])

    main

    A Loopback Decoder allows you to tap into an output stream and feed it back into the FFmpeg filtergraph as a new input.

    In a command line, this is represented by the -dec flag and the of:ost argument (where of is the output file index and ost is the stream index). When parsed, these create a LoopbackDecoderNode which provides new stream labels (e.g., [dec:0], [dec:1]) that can be used as inputs in a -filter_complex graph.

    Key Concepts:

    • -dec of:ost: The command-line syntax to declare a decoder. of refers to the ordinal index of the output file (starting at 0), and ost is the stream index within that output.
    • [dec:N]: The label generated for the $N^{th}$ decoder occurrence. These labels are used to reference the decoded stream in subsequent filters.
    • Stream Type: The decoder automatically determines if it is a dec.video or dec.audio stream by inspecting the tapped output's stream type at the specified index ost.

  9. Handle removed hardware-acceleration filters in FFmpeg 8

    main

    FFmpeg 8 removed many hardware-accelerated filters (OpenCL, Vulkan, VAAPI, CUDA) and third-party library wrappers (e.g., drawtext, subtitles, ladspa, rubberband) from the default build.

    Important: These removals often reflect the absence of these features in the standard Docker-based test build used to generate the typed-ffmpeg bindings. These filters may still be available if you compile your own FFmpeg binary with the relevant flags (e.g., --enable-opencl, --enable-vulkan, --enable-vaapi, etc.).

  10. How loopback decoders (-dec / [dec:N]) work

    main

    Loopback decoders allow a command line to reference an output stream as an input for further processing.

    In the command line, these are represented by the -dec flag followed by an index (e.g., -dec 0:0). In filter graphs, they are referenced using labels like [dec:N], where N is the occurrence index of the -dec flag.

    Key behaviors:

    • Labels: Loopback decoder labels (dec:N) are treated as whole labels and are not type-suffixed.
    • Resolution: The parser uses an iterative worklist to resolve the dependency cycle between outputs, loopback decoders, and filter graphs.
    • Constraints:
      • -dec cannot be used with stream copying (-c:v copy).
      • The output index referenced by -dec must exist.
      • Circular or unresolvable references will raise an FFMpegValueError.

  11. Loopback Decoder Compilation Logic

    main

    When compiling a graph containing LoopbackDecoderNode to a CLI command, the following rules apply:

    1. Filter Complex Labels: The decoder streams are referenced in -filter_complex using the [dec:N] syntax.
    2. Argument Ordering: The compiler emits the OutputNode arguments first, followed immediately by any LoopbackDecoderNode that taps that output. This ensures the -dec flag correctly references the previously defined output stream.
    3. Decoder Options: Any keyword arguments provided to the loopback() method are converted into FFmpeg flags (e.g., { threads: 2 } becomes -threads 2) and placed before the -dec flag.
    4. Index Assignment: The N in [dec:N] is assigned based on the occurrence order of the -dec flag in the generated command line.

  12. Implement FFmpeg Loopback Decoder support

    main

    The Loopback Decoder (-dec / [dec:N]) implementation provides support for FFmpeg 7.0+ loopback decoding. This allows an output stream to be fed back into a decoder as a new input stream within a filtergraph.

    Core API Concept

    • out.loopback(stream_index): This method returns a LoopbackDecoderNode.
    • LoopbackDecoderNode: A first-class node whose single input is an OutputStream(node=OutputNode, index=ost) reference. Its output is a typed filterable stream.
    • Filterable Streams: Once looped back, the streams are accessible via dec.video and dec.audio for further filtering.
    • Compilation: The node compiles to [decoder opts] -dec of:ost with [dec:N] filtergraph labels.

    Implementation Details

    • Python: Support is provided for FFmpeg v7/v8 via codegen templates located in src/scripts/code_gen/templates/. The loopback() method is only generated for FFmpeg major versions $\ge$ 7.
    • TypeScript: The ts-core implementation is hand-written in packages/ts-core.
    // Conceptual Python usage
# out is an OutputStream
# stream_index is the index of the stream to loop back
dec = out.loopback(stream_index)

# Access typed streams for further filtering
video_stream = dec.video
audio_stream = dec.audio