OmniShotCut

OmniShotCut processes long videos by splitting them into overlapping chunks (windows) to manage memory and maintain temporal context. The process follows these steps:

1. Splitting: split_videos divides the video into chunks of chunk_size with a specified overlap_size. It handles padding for the final chunk and calculates valid_start_idx and valid_end_idx to identify regions where windows overlap with neighbors.
2. Local Decoding: For each window, decode_window_segments converts the model's raw query outputs into contiguous local segments (start_local, end_local, intra_label, inter_label).
3. Global Cut Collection: collect_cuts_from_window converts local segments into global cut points. To prevent mislabeling, the first segment of every window (the opening shot) is skipped, as its true identity is determined by the overlapping window.
4. Stitching: assemble_ranges sorts the collected global cuts, removes duplicates (using duplicate_tolerance), and builds the final shot ranges [start, end] and their corresponding labels. The very first shot of the video is automatically assigned the GENERAL_INTRA_ID and NEW_START_INTER_ID.

You can perform shot boundary detection using the omnishotcut Python package. The API supports loading models from local paths or HuggingFace repositories and running inference on video files, numpy arrays, or torch tensors.

Loading a Model

Use omnishotcut.load() to initialize the model. You can specify a HuggingFace repo and a specific filename.

Running Inference

Call .inference() on the loaded model. The input can be a video file path, a numpy array, or a torch tensor with shape (T, H, W, 3) (RGB, uint8 or float [0, 1]).

Inference Modes

• mode="clean_shot": Returns only clean cuts (no transitions). Returns a list of [start_frame, end_frame] pairs.
• mode="default": Returns clean cuts AND transitions (dissolves, wipes, fades) along with their labels. Returns (ranges, intra_labels, inter_labels).

import omnishotcut

# Load model from HuggingFace
cut_model = omnishotcut.load("uva-cv-lab/OmniShotCut_v1.5", filename = "OmniShotCut_ckpt.pth")

# Option 1: Clean shots only (returns list of [start, end])
ranges = cut_model.inference("video.mp4", mode="clean_shot")

# Option 2: Detailed detection (returns ranges, intra_labels, inter_labels)
ranges, intra_labels, inter_labels = cut_model.inference("video.mp4", mode="default")

# Option 3: Inference on raw frames (numpy or torch tensor)
# frames_thwc shape: (T, H, W, 3)
ranges = cut_model.inference(frames_thwc, mode="clean_shot")

The single_video_inference function is the core engine for processing a video file to detect shot boundaries. It returns predicted shot ranges, intra-label IDs, inter-label IDs, the full video as a NumPy array, and the video's FPS.

Parameters:

• video_path (str): Path to the input video file.
• model: The loaded OmniShotCut model instance.
• model_args: Arguments associated with the loaded model.
• overlap_window_length (int): The number of context frames to use during inference (defaults to DEFAULT_NUM_CONTEXT_FRAMES in the demo).

Returns:

• pred_ranges (List[List[int]]): A list of [start_frame, end_frame] pairs for each detected shot.
• pred_intra_labels (List[int]): A list of intra-label IDs for each shot.
• pred_inter_labels (List[int]): A list of inter-label IDs for each shot.
• video_np_full (np.ndarray): The full video content as a NumPy array.
• fps (float): The frames per second of the video.

from omnishotcut.engine import single_video_inference

pred_ranges, pred_intra_labels, pred_inter_labels, video_np_full, fps = single_video_inference(
    video_path=video_path,
    model=model,
    model_args=model_args,
    overlap_window_length=10
)

The single_video_inference function is the primary entry point for processing a video file from disk. It handles video decoding, windowed inference with overlapping segments, and stitching the results into global shot ranges.

Parameters:

• video_path: Path to the video file.
• model: The loaded OmniShotCut model.
• model_args: The model configuration object returned by load_model.
• overlap_window_length: The number of frames to overlap between consecutive windows to ensure continuity.

Returns:

• pred_ranges_full: A list of shot ranges as [start_frame, end_frame].
• pred_intra_labels_full: List of intra-relation labels for each shot.
• pred_inter_labels_full: List of inter-relation labels for each shot.
• video_np_full: The decoded video as a NumPy array (T, H, W, C).
• fps: The video's frames per second.

from omnishotcut.engine import load_model, single_video_inference

model, model_args = load_model("checkpoint.pth")

ranges, intra, inter, video_np, fps = single_video_inference(
    "video.mp4", 
    model, 
    model_args, 
    overlap_window_length=16
)

The inference method performs shot boundary detection on a video. It supports multiple input formats and processing modes.

Input Types (video)

• str: A file path to a video file.
• np.ndarray: A NumPy array with shape (T, H, W, 3) in RGB format.
• torch.Tensor: A PyTorch tensor with shape (T, H, W, 3) in RGB format.

Note on Data Types: If the input is not uint8, it is treated as a float in the range [0, 1] and automatically scaled to uint8 internally. The input is automatically resized to the model's internal process resolution (e.g., 128x96) before processing.

Arguments

• mode:
  • "clean_shot": Returns only general cuts (ignores transitions like dissolves or wipes).
  • "default": Returns all detected shots with full labels (intra and inter labels).
• overlap: The number of overlap frames between adjacent inference windows (default is 10).

Returns

• If mode="clean_shot": Returns a list of [start_frame, end_frame] ranges.
• If mode="default": Returns a tuple of (ranges, intra_labels, inter_labels):
  • ranges: A list of [start_frame, end_frame] shot boundaries.
  • intra_labels: A list of string labels representing the type of shot (e.g., 'General', 'Dissolve', 'Wipes').
  • inter_labels: A list of string labels representing the relationship between shots (e.g., 'New_Start', 'Hard_Cut').

from omnishotcut import load

model = load("username/repo-name")

# Example 1: Clean shot detection (returns only ranges)
ranges = model.inference("video.mp4", mode="clean_shot")

# Example 2: Full detection (returns ranges and labels)
ranges, intra, inter = model.inference("video.mp4", mode="default")

The visualize_concated_frames function generates visualization images (concatenated frames) that highlight the detected shot boundaries.

Parameters:

• frames (np.ndarray): The full video as a NumPy array.
• out_dir (str): Directory where the visualization images will be saved.
• highlight_ranges_closed (List[List[int]]): The list of [start_frame, end_frame] shot ranges to highlight.
• max_frames_per_img (int): Maximum number of frames to include in each concatenated image.
• end_range_exclusive (bool): Whether the end frame of a range is exclusive.
• fps (float): The video's frames per second.
• start_index (int): The starting frame index for visualization.

from omnishotcut.util.visualization import visualize_concated_frames

page_paths = visualize_concated_frames(
    frames=video_np_full,
    out_dir="demo_video_results",
    highlight_ranges_closed=pred_ranges,
    max_frames_per_img=132,
    end_range_exclusive=True,
    fps=fps,
    start_index=0
)