DOCSEARCHbyAlgolia
Submit Docs

ffmpegcv

repository·main·Indexed 42 days ago

https://github.com/chenxinfeng4/ffmpegcv

A high-performance Python alternative to OpenCV for video reading and writing built on FFmpeg. Optimized for deep learning pipelines, it supports GPU-accelerated decoding/encoding (NVIDIA and Intel QSV) and direct-to-CUDA memory transfers via the toCUDA function to provide frames as PyCUDA arrays or PyTorch tensors.

Tokens
6K
Snippets
17
Records
19
Agent score
4%

What's inside ffmpegcv

  1. Accelerate deep learning pipelines with toCUDA

    main

    The toCUDA function translates video/streams from HWC-uint8 (CPU) to CHW-float32 (CUDA device). This reduces CPU load and is >2x faster than manual conversion.

    Requirements: NVIDIA CUDA environment and pycuda (installed via pip install ffmpegcv[cuda]).

    Usage Pattern

    import ffmpegcv

# Recommended for GPU: use nv12 pix_fmt
cap = ffmpegcv.toCUDA(
    ffmpegcv.VideoCaptureNV(file, pix_fmt='nv12'),
    tensor_format='chw'
)

for frame_CHW_cuda in cap:
    # frame_CHW_cuda is a PyCUDA array or PyTorch tensor
    # Perform normalization and model inference directly on GPU
    result = model(frame_CHW_cuda)

    Key Methods for CUDA frames

    • cap.read(): Returns a pycuda array.
    • cap.read_cudamem(): Returns a pycuda memory allocation object.
    • cap.read_torch(): Returns a pytorch tensor.
    • cap.read_torch(output_memory): Performs a zero-copy read into existing memory.
    cap = ffmpegcv.toCUDA(
    ffmpegcv.VideoCaptureNV(file, pix_fmt='nv12'),
    tensor_format='chw'
)

for frame_CHW_cuda in cap:
    result = model(frame_CHW_cuda)

  2. How ffmpegcv and OpenCV compare

    main

    ffmpegcv is a high-performance alternative to OpenCV for video I/O, specifically optimized for deep learning pipelines.

    Key Advantages:

    • Performance: Faster video reading/writing and supports GPU acceleration (NVIDIA/Intel).
    • Codecs: Supports more codecs than OpenCV, including h264 and h265 (HEVC).
    • ROI Operations: Native support for cropping, resizing, and padding during the read process, which is significantly faster than OpenCV.
    • Deep Learning Integration: Can export frames directly to CUDA devices in CHW or HWC float32 formats.
    • Simplicity: Requires only numpy and FFmpeg (no complex OpenCV installation required).

  3. Use FFmpegReaderNoblock for high-performance reading/writing

    main

    To maximize efficiency and prevent CPU blocking during video I/O, use the noblock wrapper. This utilizes multiprocessing to prepare frames in the background.

    How it works: When you wrap a VideoCapture or VideoWriter with ffmpegcv.noblock(), the next frame is automatically buffered in a separate process while your main Python thread performs other tasks (like heavy CPU processing). This can lead to significant performance gains (up to double efficiency) compared to the standard blocking readers.

    Usage Pattern: ffmpegcv.noblock(ffmpegcv.VideoCapture, *args) or ffmpegcv.noblock(ffmpegcv.VideoWriter, *args).

    import ffmpegcv

# Wrap VideoCapture to enable non-blocking background frame preparation
vid_noblock = ffmpegcv.noblock(ffmpegcv.VideoCapture, 'video.mp4', pix_fmt='rbg24')

for _ in range(1000):
    # This read is extremely fast because the next frame is already buffered in the background
    ret, img = vid_noblock.read() 
    
    # Perform heavy CPU work here; the background process prepares the next frame
    do_heavy_processing(img)

  4. Install ffmpegcv

    main

    FFMPEGCV requires FFmpeg to be installed on your system before installing the Python package.

    1. Install FFmpeg

    • Linux: sudo apt install ffmpeg
    • macOS: brew install ffmpeg (Note: NVIDIA GPU acceleration is not supported on macOS)
    • Windows: Download FFmpeg and add it to your system PATH
    • Conda: conda install ffmpeg=6.0.0 (Avoid the default 4.x.x version)

    2. Install ffmpegcv

    Install the stable version via pip:

    pip install ffmpegcv

    Or install the latest version from GitHub:

    pip install git+https://github.com/chenxinfeng4/ffmpegcv

    For advanced functions like toCUDA acceleration, install with the cuda extra:

    pip install ffmpegcv[cuda]
    pip install ffmpegcv

  5. Use toCUDA to accelerate deep learning pipelines

    main

    The toCUDA function converts video/streams from HWC-uint8 (CPU) to CUDA-device CHW-float32. This significantly reduces CPU load and is up to 2x faster than manual conversion.

    Requirements:

    • NVIDIA CUDA environment.
    • pycuda installed (pip install pycuda).

    Usage Pattern: Wrap your reader with toCUDA. You must specify the correct pix_fmt for the underlying reader (yuv420p for CPU, nv12 for GPU).

    Available Methods on the returned object:

    • .read(): Returns a pycuda array.
    • .read_cudamem(): Returns a pycuda memory object.
    • .read_torch(): Returns a PyTorch tensor.
    import ffmpegcv

# Accelerated pipeline
cap = ffmpegcv.toCUDA(
    ffmpegcv.VideoCaptureNV(file, pix_fmt='nv12', resize=(W, H)),
    tensor_format='chw'
)

for frame_CHW_cuda in cap:
    # frame_CHW_cuda is already on GPU
    frame_CHW_cuda = (frame_CHW_cuda - mean) / std
    result = model(frame_CHW_cuda)

  6. Use VideoWriter to write video files

    main

    The VideoWriter class allows writing video files with support for various codecs and GPU acceleration.

    Key Features:

    • Default Codec: h264 on CPU, h265 on GPU.
    • Format: Use .mp4 or .mkv instead of .avi.
    • GPU Acceleration: Use VideoWriterNV for NVIDIA or VideoWriterQSV for Intel.
    • Pixel Format: Use pix_fmt='rgb24' for input frames.
    • Resizing: You can specify a target resize=(w, h) during initialization.
    import ffmpegcv

# Standard CPU writing
with ffmpegcv.VideoWriter('out.mp4', None, 10) as out:
    out.write(frame)

# NVIDIA GPU accelerated writing
out_gpu = ffmpegcv.VideoWriterNV('out.mp4', 'h264', 10)

# Writing with specific pixel format and resize
out_custom = ffmpegcv.VideoWriter('out.mp4', None, 10, pix_fmt='rgb24', resize=(640, 480))

  7. Write video files with ffmpegcv

    main

    FFMPEGCV provides high-performance video writing. The default codec is h264 for CPU and h265 for GPU. Frame size is automatically determined by the first frame written.

    Basic CPU Writing

    import ffmpegcv

with ffmpegcv.VideoWriter('output.mp4', None, 30) as out:
    out.write(frame)

    GPU Accelerated Writing (NVIDIA)

    import ffmpegcv

# Write h264 using GPU 0
out = ffmpegcv.VideoWriterNV('output.mp4', 'h264', 30)

# Write hevc using GPU 1
out = ffmpegcv.VideoWriterNV('output.mp4', 'hevc', 30, gpu=1)

    Advanced Options

    • Pixel Format: Use pix_fmt='rgb24' if your input frames are RGB (default is BGR).
    • Resize: Resize output frames on the fly.
    out = ffmpegcv.VideoWriter('output.mp4', None, 30, resize=(640, 480))
    with ffmpegcv.VideoWriter('output.mp4', None, 30) as out:
    out.write(frame)

  8. Perform ROI (Region of Interest) operations during video reading

    main

    You can perform cropping, resizing, and padding directly within the VideoCapture constructor. This is much faster than reading the full frame and processing it with OpenCV.

    • Crop: Use crop_xywh=(x, y, w, h).
    • Resize: Use resize=(width, height).
    • Resize with Aspect Ratio: Use resize=(width, height) and resize_keepratio=True to add black padding.
    • Combined: You can combine operations, e.g., crop_xywh followed by resize.
    # Crop video
cap = ffmpegcv.VideoCapture(file, crop_xywh=(0, 0, 640, 480))

# Resize video
cap = ffmpegcv.VideoCapture(file, resize=(640, 480))

# Resize while keeping aspect ratio (with padding)
cap = ffmpegcv.VideoCapture(file, resize=(640, 480), resize_keepratio=True)

# Crop then resize
cap = ffmpegcv.VideoCapture(file, crop_xywh=(0, 0, 640, 480), resize=(512, 512))

  9. Use VideoCapture to read video files

    main

    The VideoCapture class provides several ways to read video. It is compatible with the OpenCV API but offers more Pythonic usage patterns.

    Basic Usage (Recommended): Use a context manager (with statement) to ensure the resource is released automatically.

    Supported Reader Types:

    • VideoCapture: Standard CPU reader.
    • VideoCaptureNV: NVIDIA GPU accelerated reader.
    • VideoCaptureQSV: Intel Integrated Graphics (QSV) reader.
    • VideoCaptureCAM: Read from a camera (via device ID or name).
    • VideoCaptureStream: Read RTP/RTSP/RTMP/HTTP streams.
    • VideoCaptureStreamRT: Low-latency RTSP stream reading (for IP cameras).
    import ffmpegcv

# Recommended Pythonic way
with ffmpegcv.VideoCapture('input.mp4') as cap:
    for iframe, frame in enumerate(cap):
        if iframe > 100: break
        # process frame

# Reading from a camera by ID
cap = ffmpegcv.VideoCaptureCAM(0)

# Reading from a camera by name
cap = ffmpegcv.VideoCaptureCAM("Integrated Camera")

  10. Core Video IO Functions

    main

    FFMPEGCV provides several classes for reading and writing video, offering alternatives to OpenCV for different hardware backends and stream types:

    Video Readers

    • VideoCapture: Standard CPU-based video reading.
    • VideoCaptureNV: NVIDIA GPU-accelerated decoding.
    • VideoCaptureQSV: Intel QuickSync Video decoding (experimental).
    • VideoCaptureCAM: Read from a camera (supports device ID or device name).
    • VideoCaptureStream: Read RTP/RTSP/RTMP/HTTP streams.
    • VideoCaptureStreamRT: Real-time low-latency RTSP stream reading (ideal for IP Cameras).

    Video Writers

    • VideoWriter: Standard CPU-based video writing.
    • VideoWriterNV: NVIDIA GPU-accelerated encoding (e.g., using h264 or hevc).
    • VideoWriterQSV: Intel QuickSync Video encoding (experimental).

    Utilities

    • noblock: Read/Write video in the background using multiprocessing.
    • toCUDA: Translates video/streams directly to CUDA device in CHW/HWC-float32 format for deep learning pipelines.

  11. Boost performance with noblock

    main

    The ffmpegcv.noblock function is a proxy that uses multiprocessing to prepare frames in the background. This prevents the Python program from blocking during read() or write() operations, which can provide up to a 2x performance boost in CPU-intensive applications.

    Use it to wrap ffmpegcv.VideoCapture or ffmpegcv.VideoWriter arguments.

    import ffmpegcv

# Wrap VideoCapture to enable background frame preparation
vid_noblock = ffmpegcv.noblock(ffmpegcv.VideoCapture, vfile, pix_fmt='rbg24')

for _ in range(1000):
    # This read is non-blocking because the next frame is already being buffered in the background
    ret, img = vid_noblock.read() 
    # Perform heavy CPU tasks here
    do_heavy_processing(img)

  12. Perform ROI operations (Crop, Resize, Pad)

    main

    You can perform Region of Interest (ROI) operations during video reading. These operations are significantly faster when using ffmpegcv-GPU compared to OpenCV.

    Crop

    Crop the video using crop_xywh=(x, y, w, h). The top-left corner is (0, 0).

    cap = ffmpegcv.VideoCapture(file, crop_xywh=(0, 0, 640, 480))

    Resize

    Resize the video to a specific resolution.

    cap = ffmpegcv.VideoCapture(file, resize=(640, 480))

    Resize with Padding

    Resize while keeping the aspect ratio by adding black borders.

    cap = ffmpegcv.VideoCapture(file, resize=(640, 480), resize_keepratio=True)

    Combined Crop and Resize

    cap = ffmpegcv.VideoCapture(file, crop_xywh=(0, 0, 640, 480), resize=(512, 512))
    cap = ffmpegcv.VideoCapture(file, crop_xywh=(0, 0, 640, 480), resize=(512, 512))