MMRazor Documentation

MMRazor is a model compression toolkit designed for model slimming. It supports four mainstream technologies:

• Neural Architecture Search (NAS)
• Pruning
• Knowledge Distillation (KD)
• Quantization

It is part of the OpenMMLab project and is designed for high compatibility with other OpenMMLab repositories (like MMClassification, MMDetection, and MMSegmentation) through a decoupled architecture of slimming algorithms and vision tasks.

MMRazor is a model compression toolbox and benchmark designed for model slimming and AutoML. It is part of the OpenMMLab project and provides four main categories of compression technologies:

• Neural Architecture Search (NAS): Automating the design of network structures.
• Pruning: Removing redundant parameters to reduce model size.
• Knowledge Distillation (KD): Transferring knowledge from a large teacher model to a smaller student model.
• Quantization: Reducing the precision of weights and activations (e.g., PTQ, QAT).

Key features include high compatibility with other OpenMMLab projects due to similar architectural designs, flexibility through plug-and-play algorithm combinations, and ease of extension via modular configuration files.

MMRazor is a model compression toolkit designed for model slimming and AutoML. It is part of the OpenMMLab project and provides a unified framework for four mainstream model compression technologies:

• Neural Architecture Search (NAS): Automating the search for optimal network architectures.
• Pruning: Removing redundant parameters to reduce model size.
• Knowledge Distillation (KD): Transferring knowledge from a large teacher model to a smaller student model.
• Quantization: Reducing the precision of model weights and activations.

Key advantages include compatibility with various OpenMMLab projects due to decoupled algorithm design, flexibility through a plug-and-play system for different algorithms, and convenience for developers to implement new methods via configuration files or minimal code changes.

MMRazor is a general-purpose model compression toolkit that supports both conventional Computer Vision (CV) models and large-scale models. It provides implementations for pruning and quantization algorithms applied to various architectures such as LLaMA, OPT, and Stable Diffusion.

The project structure for large model support is organized as follows:

• mmrazor/implementations: Contains core algorithm components for pruning and quantization.
• projects/mmrazor_large/algorithms: Contains usage introductions for specific algorithms.
• projects/mmrazor_large/examples: Contains model-specific implementation examples, categorized by model type (e.g., language_models/LLaMA, language_models/OPT, ResNet).

Group Fisher Pruning is a general channel pruning approach designed for networks with complicated structures such as residual connections, group/depthwise convolutions, and feature pyramid networks (FPN).

Key features include:

• Layer Grouping Algorithm: Automatically identifies coupled channels that must be pruned simultaneously.
• Fisher Information Metric: Uses a unified metric based on Fisher information to evaluate the importance of individual and coupled channels.
• Memory-Aware Importance: Normalizes importance based on memory reduction rather than just FLOPs, as memory reduction is more highly correlated with GPU inference speedup.

This method is applicable to various backbones including ResNet, ResNeXt, MobileNetV2, and RegNet for both image classification and object detection tasks.

The mmrazor.engine module contains the core execution logic for model compression and knowledge distillation tasks. It is organized into the following submodules:

• hooks: Provides mechanisms to intercept and modify the training or evaluation process (e.g., logging, checkpointing).
• optimizers: Contains optimization algorithms used to update model parameters.
• runner: The central execution engine that manages the lifecycle of training, validation, and testing.

The mmrazor.models module defines the building blocks for compression algorithms and model architectures. Key submodules include:

• algorithms: High-level logic for specific compression or distillation methods.
• architectures: Base neural network architectures.
• distillers: Components specifically designed for knowledge distillation.
• losses: Loss functions used during training.
• mutables and mutators: Components used for structural modifications or parameter manipulation.
• ops: Specialized operations used within models.
• task_modules: Modules that encapsulate specific tasks within the model pipeline.

Knowledge Distillation (KD) is a technique used to compress the knowledge from a large, high-performance 'Teacher' model into a smaller, more efficient 'Student' model. This allows for deploying models that maintain high accuracy while being computationally cheaper for real-world use.

In MMrazor, KD configurations for classification tasks (using mmcls backends) typically involve distilling information from the logits of a teacher model to the student model. This repository provides pre-configured experiments and pre-trained models for common teacher-student pairs on the ImageNet dataset.

Contrastive Representation Distillation (CRD) is a knowledge distillation method that uses contrastive learning to transfer representational knowledge from a teacher network to a student network. Unlike standard knowledge distillation which minimizes KL divergence between probabilistic outputs, CRD aims to capture the structural knowledge of the teacher's representations. It is applicable to tasks such as single model compression, ensemble distillation, and cross-modal transfer.

This implementation is based on the paper Contrastive Representation Distillation.