InternImage

InternImage is a large-scale vision foundation model based on Convolutional Neural Networks (CNN). Unlike Transformer-based networks, it utilizes DCNv3 (Deformable Convolution v3) as its core operator, providing a dynamic effective receptive field and adaptive spatial aggregation suitable for downstream tasks like detection and segmentation.

Key Capabilities:

• Supports various downstream tasks including classification, object detection, and semantic segmentation.
• Supports autonomous driving perception (e.g., BEVFormer series).
• Supports low-cost training via DeepSpeed.
• Supports TensorRT inference for classification, detection, and segmentation.
• Provides model intermediate feature extraction via classification/extract_feature.py.

The OpenLane-V2/ directory follows this structure:

• train/, val/, test/: Data splits.
  • [segment_id]/: A sequence of frames.
    • image/[camera]/[timestamp].jpg: Images captured by specific cameras.
    • info/[timestamp].json: Metadata and annotations for a single frame.
• data_dict_[xxx].json: JSON files defining the split of train/val/test for specific subsets (e.g., subset_A).
• preprocess.py: Preprocessing script.
• openlanev2.md5: MD5 checksums.

The OpenLane-V2 Score (OLS) is a consolidated metric used to evaluate performance across multiple aspects of the autonomous driving task. It is calculated as the average of four specific metrics:

• $\text{DET}_{l}$: mAP on directed lane centerlines.
• $\text{DET}_{t}$: mAP on traffic elements.
• $\text{TOP}_{ll}$: mAP on topology among lane centerlines.
• $\text{TOP}_{lt}$: mAP on topology between lane centerlines and traffic elements.

The ADE20K model zoo provides pre-trained InternImage backbones for semantic segmentation using two different architectures: UperNet and Mask2Former. Models vary by backbone size (T, S, B, L, XL, H) and input resolution.

Key Performance Metrics:

• mIoU (ss/ms): Semantic segmentation mIoU for single-scale and multi-scale.
• Training Speed: Measured using A100 GPUs.
• Memory Management: If you encounter out-of-memory issues during training, set with_cp=True to save memory.

InternImage is part of a broader ecosystem of vision and multimodal research. Related projects include:

Multimodal Foundation Models

• Uni-Perceiver: A unified pre-training framework for general perception tasks, supporting zero-shot and few-shot tasks.
• Uni-Perceiver v2: A general model for image and image-text tasks.
• M3I-Pretraining: A single-stage pre-training paradigm based on maximizing mutual information between inputs and targets.
• InternVL: A leading multimodal large language model (MLLM) excelling in OCR, multimodal reasoning, and dialogue.

Autonomous Driving

• BEVFormer: A next-generation pure vision surround perception solution based on Bird's-Eye-View (BEV).
• BEVFormer v2: A two-stage detector fusing BEV perception and perspective view detection.

The COCO model zoo provides pre-trained weights and configurations for various object detection architectures using InternImage backbones. Models are categorized into:

• Mask R-CNN + InternImage: Standard instance segmentation models (Backbones: T, S, B).
• Cascade Mask R-CNN + InternImage: High-performance instance segmentation models (Backbones: L, XL).
• DINO + InternImage: Transformer-based detection models (Backbones: T, L).
• DINO + CB-InternImage: Large-scale models using CB-InternImage (Backbone: H).

Each model entry includes a configuration file, a checkpoint (.pth), and training logs.