TorchCodec

TorchCodec is a Python library designed to decode video and audio data directly into PyTorch tensors on both CPU and CUDA GPU. It also provides encoding capabilities for audio, video, and images.

Key features include:

• PyTorch Integration: Returns data as PyTorch tensors, making it ready for use in PyTorch transforms or ML model training.
• FFmpeg Backend: Leverages the FFmpeg version installed on your system to handle complex decoding/encoding tasks while providing a simplified, Pythonic API.
• Hardware Acceleration: Supports decoding and encoding on both CPU and CUDA GPUs.

The torchcodec.transforms module provides tools for applying transformations during the video decoding process. These transforms can be used to perform operations like resizing, cropping, or custom decoding logic directly as part of the decoding pipeline.

Key transform types include:

• DecoderTransform: A base class or interface for defining custom decoding logic.
• CenterCrop: Crops the center portion of the video frames.
• RandomCrop: Applies a random crop to the video frames.
• Resize: Resizes the video frames to specified dimensions.

For a full walkthrough of how to apply these transforms during decoding, refer to the decoding transforms tutorial.

The torchcodec.samplers module provides utility functions to extract specific segments or frames from video files using regular or random sampling patterns. You can sample based on either frame indices or timestamps.

Available sampling functions:

• clips_at_regular_indices: Extracts clips at fixed, regular frame index intervals.
• clips_at_random_indices: Extracts clips at randomly selected frame indices.
• clips_at_regular_timestamps: Extracts clips at fixed, regular time intervals (seconds).
• clips_at_random_timestamps: Extracts clips at randomly selected time intervals (seconds).

For a full tutorial on decoding and sampling, refer to the generated examples in the documentation.

A scan is a full pass over a video file used to retrieve metadata about streams and frames. Crucially, a scan does not involve decoding, making it significantly more efficient than decoding the entire file.

In torchcodec.decoders.VideoDecoder, the scanning behavior is controlled by the seek_mode parameter:

• seek_mode="exact": Triggers a scan.
• seek_mode="approximate": Does not trigger a scan.

The torchcodec package provides several key data structures for handling decoded media content:

• Frame: Represents a single decoded frame of video or audio.
• FrameBatch: A collection of Frame objects, typically used for efficient batch processing of media.
• AudioSamples: A structure containing decoded audio data.

A clip is a sequence of frames, typically ordered by their pts. These frames do not have to be consecutive.

In terms of data structures:

• A single clip is represented as a 4D torchcodec.FrameBatch.
• A group of clips (such as those returned by samplers) is represented as a 5D torchcodec.FrameBatch.

To encode both audio and video streams simultaneously, use the multi-stream encoder classes. This approach is suitable for creating synchronized media files. For a complete walkthrough, refer to the encoding_multi_stream_encoding.py tutorial.

Key components include:

• Encoder: The base class for multi-stream encoding.
• VideoStream: Manages the video component of the multi-stream encoder.
• AudioStream: Manages the audio component of the multi-stream encoder.

# Refer to the tutorial for a full implementation:
# :ref:`sphx_glr_generated_examples_encoding_multi_stream_encoding.py`

To install TorchCodec, follow the instructions provided in the official repository README.

Note: TorchCodec relies on FFmpeg being installed on your system. It will use the version of FFmpeg that is currently available in your environment.

https://github.com/meta-pytorch/torchcodec?tab=readme-ov-file#installing-torchcodec

To use CUDA-enabled decoding on Linux or Windows, ensure you have a GPU with NVDEC hardware support and the libnvrtc CUDA library installed.

To install a specific CUDA Toolkit version (e.g., 13.0), use the --index-url flag. Ensure the installed torch version matches the CUDA version:

pip install torch torchcodec --index-url=https://download.pytorch.org/whl/cu130

To verify your FFmpeg installation supports NVDEC, run:

ffmpeg -decoders | grep -i nvidia

It should return a line like V..... h264_cuvid Nvidia CUVID H264 decoder (codec h264).