DOCSEARCHbyAlgolia
Submit Docs

VBench Benchmark Suite

repository·master·Indexed 23 days ago

https://github.com/vchitect/vbench

A comprehensive benchmark suite for the multi-dimensional evaluation of video generative models. It assesses technical quality, long-form content, image-to-video tasks, trustworthiness, and intrinsic faithfulness. The suite includes tools for automatic evaluation via run_eval.py, integrity checks for corrupted files, and prompt optimization using GPT-4o.

Tokens
209.3K
Snippets
448
Records
771
Agent score
81%

What's inside VBench

  1. Overview of YOLOv8 Models and Results

    master

    YOLOv8 by Ultralytics is a state-of-the-art model designed for object detection, image segmentation, and image classification. The available models in this repository are optimized for the COCO dataset and include various scales (n, s, m, l, x) with different configurations.

    Key Configuration Terms

    • Backbone: The architecture type (e.g., P5).
    • Mask Refine: Indicates if the model uses mask-based bounding box refinement during training.
    • SyncBN: Indicates the use of Synchronized Batch Normalization.
    • AMP: Indicates training with Automatic Mixed Precision.
    • TTA: Test Time Augmentation. To use this during testing, you must specify the --tta flag. It performs multi-scaling and flipping transformations.

    Model Selection Guide

    Models are categorized by size and performance (box AP):

    • YOLOv8-n (Nano): Smallest, fastest, ~37.2 AP.
    • YOLOv8-s (Small): ~44.2 AP.
    • YOLOv8-m (Medium): ~49.8 AP.
    • YOLOv8-l (Large): ~52.1 AP.
    • YOLOv8-x (Extra Large): ~52.7 AP.

    Note that performance may fluctuate by approximately 0.3 mAP due to training instability.

  2. Overview of YOLOX Models and Results

    master

    YOLOX is a high-performance detector that uses an anchor-free approach, a decoupled head, and the SimOTA label assignment strategy. It is available in several scales ranging from YOLOX-tiny to YOLOX-x.

    Key features include:

    • Scalability: Models range from lightweight (Nano/Tiny) to high-performance (L/X).
    • Deployment Support: Deployable via ONNX, TensorRT, NCNN, and OpenVino.
    • Training Optimization: For faster training without mAP degradation, it is recommended to use RTMDet's training hyperparameters (e.g., using AdamW instead of SGD and adjusting base_lr and weight_decay).

  3. Overview of YOLOv6 Object Detection Models

    master

    YOLOv6 is a single-stage object detection framework designed for industrial applications, focusing on high speed and accuracy for deployment-ready use cases. It offers various model scales (n, t, s, m, l) to accommodate different hardware constraints and performance requirements.

    Key performance highlights on the COCO dataset include:

    • YOLOv6-N: 35.9% AP, ~1234 FPS (NVIDIA Tesla T4)
    • YOLOv6-S: 43.5% AP, ~495 FPS
    • YOLOv6-M/L: Higher accuracy (49.5% - 52.3% AP) with competitive inference speeds.

  4. Overview of MMYOLO

    master

    MMYOLO is an open-source toolbox for YOLO series algorithms built on PyTorch and MMDetection. It is part of the OpenMMLab project and is designed for real-time object recognition tasks.

    Key Features:

    • Unified Benchmark: Unifies module implementations across various YOLO algorithms for fair comparison and analysis.
    • Modular Design: Decomposes the framework into components, allowing users to customize models by combining different modules, training strategies, and testing strategies.
    • Rich Documentation: Provides guidance on getting started, model deployment, advanced usage, and algorithm analysis.

    Performance Note: MMYOLO provides significant training acceleration compared to the MMDetection version, with training speeds reported to be up to 2.6 times faster.

  5. Overview of MMYOLO Model Conversion

    master

    This project is a standalone deployment project for MMYOLO, designed to operate independently of the MMDeploy ecosystem. Its primary goal is to reduce the learning and engineering overhead for users by providing dedicated support for converting and deploying models after training.

    Currently, the project supports conversion to the following formats:

    • ONNX
    • TensorRT

    Future updates are expected to include support for additional inference platforms.

  6. Introduction to MMYOLO

    master

    MMYOLO is an open-source toolbox for YOLO series algorithms built on PyTorch and MMDetection. It is part of the OpenMMLab project and is designed for real-time object recognition tasks.

    Key Features:

    • Unified Benchmark: Unifies module implementations across various YOLO algorithms for fair comparison and analysis.
    • Modular Design: Decomposes the framework into components, allowing users to customize models by combining different modules, training strategies, and testing strategies.
    • Detailed Documentation: Provides guidance on getting started, model deployment, advanced usage, and algorithm analysis.

    Technical Requirements:

    • The master branch requires PyTorch 1.6+.

  7. Overview of MMYOLO Model Easy-Deployment

    master

    MMYOLO Model Easy-Deployment is a tool designed to convert MMYOLO models to various inference backends without requiring MMDeploy. This reduces the complexity and time needed to prepare models for production environments.

    Currently, the project supports conversion to:

    • ONNX
    • TensorRT

    Future support for backends like ncnn is planned.

  8. Introduction to MMDeploy for Model Deployment

    master
    MMDeploy is an open-source deep learning model deployment toolset from the OpenMMLab project. It provides a unified experience for exporting various models to different platforms and devices. It is particularly useful for exporting specific compiled SDKs from training results, reducing manual effort in the deployment pipeline.

  9. Overview of YOLOv6 Object Detection

    master
    YOLOv6 is a single-stage object detection framework designed specifically for industrial applications, focusing on high efficiency and deployment readiness. It offers various model scales (Nano, Tiny, Small, Medium, Large) to balance the trade-off between inference speed and detection accuracy. For example, the YOLOv6-N model achieves 35.9% AP on the COCO dataset with a high throughput of 1234 FPS on an NVIDIA Tesla T4 GPU.

  11. Overview of RTMDet Algorithm

    master

    RTMDet (Real-time Models for Object Detection) is a high-performance, low-latency one-stage object detector developed by OpenMMLab. It is designed for 'Release to Manufacture' scenarios, offering a range of model sizes (tiny, s, m, l, x) to balance accuracy and inference speed.

    Key characteristics:

    • Scalability: Models range from RTMDet-tiny (4M parameters, <1ms inference) to RTMDet-x (300+ FPS, 52.6 mAP).
    • Performance: Achieves high accuracy through optimized components like CSPNeXt backbone, PAFPN, and SepBNHead.
    • Training Strategy: Uses a two-stage training approach with strong augmentation (Mosaic + MixUp) for the majority of training, followed by a 20-epoch fine-tuning stage with weak augmentation and EMA (Exponential Moving Average) to improve stability and accuracy.

  12. Overview of the VBench series

    master

    The VBench repository provides unified implementations for three generations of video generative model evaluation suites:

    1. VBench: A comprehensive benchmark suite for evaluating video generation quality across multiple dimensions, including evaluation methods, prompt suites, and human alignment. The core code is located in the root directory (.).

    2. VBench++: An extension of VBench that adds support for:

      • VBench-I2V: Image-to-video evaluation (located in vbench2_beta_i2v).
      • VBench-Long: Long video evaluation (located in vbench2_beta_long).
      • VBench-Trustworthiness: Evaluation of fairness, bias, and safety (located in vbench2_beta_trustworthiness).

    3. VBench-2.0: An advanced suite focused on intrinsic faithfulness, evaluating complex capabilities like commonsense reasoning, physics-based realism, human motion, and creative composition. The code is located in the VBench-2.0 directory.

    Note: While VBench++ modules use specific directory names for clarity, they are maintained within this repository to ensure backward compatibility for existing users.