DeFFcode

DeFFcode provides a variety of basic recipes designed for beginners to interact with video decoding, transcoding, and metadata extraction.

Key Concepts:

• Frames: In Python, frames are represented as three-dimensional NumPy ndarray objects composed of 3 nested levels of arrays.
• Prerequisites: Proficiency with OpenCV-Python is highly recommended as many recipes utilize OpenCV for capturing and previewing frames.

Available Basic Recipe Categories:

1. Decoding Recipes: Includes decoding video files (RGB/BGR/FFmpeg formats), camera devices via indexes, network streams (HTTPs, RTSP/RTP), and image sequences.
2. Transcoding Recipes: Includes transcoding live frames (using OpenCV VideoWriter or WriteGear for lossless video) and applying simple filtergraphs (trimming, reversing, cropping, rotating, flipping, and scaling).
3. Metadata Recipes: Includes extracting video metadata using the Sourcer API or the FFdecoder API.
4. Image Processing: Includes saving key-frames as PNG images or generating thumbnails using fancy filters.

DeFFcode is a cross-platform, high-performance video frames decoder. It acts as a highly-extensible wrapper around the FFmpeg multimedia framework, executing FFmpeg pipelines inside a subprocess pipe. This allows for generating real-time, low-overhead, and fast video frames with robust error-handling using a few lines of Python code.

Key characteristics include:

• Highly Adaptive: Supports diverse input sources including live USB/Virtual/IP camera feeds, multimedia files, screen recordings, image sequences, and network URL schemes (HTTP(s), RTP/RTSP, etc.).
• Highly Flexible: Provides complete control over the underlying FFmpeg pipeline, allowing access to specifications like framerate, resolution, hardware decoders, filtergraphs, and pixel-formats.
• Highly Convenient: Uses a coding syntax similar to OpenCV-Python, making it easy for developers to transition to FFmpeg-based applications. It supports index-based camera device capturing similar to OpenCV.

The Advanced Recipes section provides complex implementation patterns for DeFFcode, focusing on high-performance video processing, hardware acceleration, and complex filtergraphs. These recipes are intended for users who have already mastered the Basic Recipes and have proficiency with OpenCV-Python.

Advanced capabilities include:

• Decoding: Live virtual sources (patterns, gradients, visualizations), live feed devices (webcams, desktop screen recording), and hardware-accelerated decoding (CUVID, CUDA).
• Transcoding: Live complex filtergraphs (watermarks, image sequences), video art generation (YUV bitplane, Jetcolor, Ghosting, Pixelation), hardware-accelerated transcoding (CUDA, NVENC), and multi-input source configurations (side-by-side, parallel RTSP, Picture-in-Picture).
• Metadata: Updating video metadata via the FFdecoder API and per-frame metadata extraction (smart keyframe-only decoding, VFR synchronization via pts_time).

By default, FFdecoder.generateFrame() yields only raw numpy frames, losing temporal context. To access metadata like timestamps and keyframe status, enable the -extract_metadata parameter during instantiation. When enabled, generateFrame() yields (frame, meta) tuples, where meta is a dictionary containing FFmpeg showinfo data.

Metadata Dictionary Keys:

• frame_num (int): Monotonic frame index.
• pts_time (float): Presentation timestamp in seconds.
• is_keyframe (bool): True if the frame is an I-frame.
• frame_type (str): One of "I" (keyframe), "P" (predictive), "B" (bi-predictive), or "?" (unknown).

Constraints & Warnings:

• FFmpeg Required: A valid FFmpeg executable must be installed on your system; otherwise, a RuntimeError is raised.
• Incompatible with -filter_complex: You cannot use -extract_metadata alongside -filter_complex. If both are provided, metadata extraction is silently disabled. However, it is compatible with existing -vf (video filters), as showinfo is automatically appended to the filter chain.
• Naming Conflict: Do not name your Python script deffcode.py, as this will cause a ModuleNotFound error.

from deffcode import FFdecoder

# Enable metadata extraction using the exclusive parameter
decoder = FFdecoder(
    "video.mp4",
    frame_format="bgr24",
    **{"-extract_metadata": True},
).formulate()

for frame, meta in decoder.generateFrame():
    if frame is None:
        break
    print(f"Frame: {meta['frame_num']}, Time: {meta['pts_time']}, Keyframe: {meta['is_keyframe']}")

decoder.terminate()

DeFFcode's Sourcer and FFdecoder APIs allow you to pass Python lists to the source and source_demuxer parameters. This enables ingesting multiple media streams simultaneously within a single FFmpeg instance for tasks like side-by-side composites, Picture-in-Picture (PiP), or multi-camera mixing.

Requirements

• FFmpeg Executable: A valid FFmpeg installation is mandatory. Failure to detect it will raise a RuntimeError.
• Explicit Routing: When using multiple inputs, you must provide either -map or -filter_complex in your parameters. If neither is provided, formulate() will raise a ValueError because FFmpeg's auto-selection is often ambiguous.
• OpenCV: Recommended for previewing frames via cv2.imshow().

Limitations

1. -vcodec Scope: A single -vcodec parameter only applies to the first input (source[0]). To apply different codecs per input, use -filter_complex or per-input options via -ffprefixes.
2. Metadata Incompatibility: The -extract_metadata option is incompatible with -filter_complex. If you use a filter graph, FFdecoder will disable metadata extraction.
3. List Length Matching: If you provide -ffprefixes or source_demuxer as a list, its length must match the length of the source list. Use [] or None for inputs that require no extra parameters.

YUV pixel formats (like yuv420p, yuv444p, nv12, etc.) are generally incompatible with standard OpenCV APIs. To make them compatible, you must pass the -enforce_cv_patch boolean attribute within the ffparam dictionary when formulating the decoder.

When using this patch, you can then convert the YUV frame to BGR (e.g., using cv2.COLOR_YUV2BGR_I420) to allow for previewing with cv2.imshow() or writing to standard video containers.

from deffcode import FFdecoder
import json, cv2

# Enable the OpenCV patch for YUV compatibility
ffparams = {"-enforce_cv_patch": True}

decoder = FFdecoder(
    "input_foo.mp4", 
    frame_format="yuv420p", 
    verbose=True, 
    **ffparams
).formulate()

metadata_dict = json.loads(decoder.metadata)
writer = cv2.VideoWriter(
    "output_foo.avi", 
    cv2.VideoWriter_fourcc("M", "J", "P", "G"), 
    metadata_dict["output_framerate"], 
    tuple(metadata_dict["output_frames_resolution"])
)

for frame in decoder.generateFrame():
    if frame is None: break
    # Convert YUV to BGR for compatibility
    bgr = cv2.cvtColor(frame, cv2.COLOR_YUV2BGR_I420)
    writer.write(bgr)

decoder.terminate()
writer.release()