MMDeploy Documentation

MMDeploy provides a unified deployment experience for a wide range of OpenMMLab algorithm libraries, including:

• mmpretrain
• mmdet
• mmseg
• mmagic
• mmocr
• mmpose
• mmdet3d
• mmrotate
• mmaction2

MMDeploy is divided into two primary functional parts: Model Conversion and the SDK.

• Model Conversion logic is primarily located in tools/, mmdeploy/, and parts of csrc/.
• SDK core components are located in csrc/, third_party/, and demo/.

Key directories:

• configs/: Algorithm library configurations for model conversion.
• csrc/: Source code for the SDK and custom operators.
• demo/: FFI interface application examples (C#, Java, Python, etc.).
• mmdeploy/: Python package used for model conversion.
• service/: Server implementations for Client/Server mode (used when target frameworks like SNPE do not support Python).
• tools/: Entry points for all functionalities, including deploy.py, onnx2xx.py, profiler.py, and test.py.

MMDeploy integrates TVM into both its model conversion tools and its SDK. The following TVM features are supported:

• AutoTVM tuner: For automated performance tuning.
• Ansor tuner: For automated performance tuning.
• Graph Executor runtime: For executing the model graph.
• Virtual Machine runtime: For executing the model via the TVM VM.

When deploying deep learning models, developers typically encounter three main categories of difficulties:

1. Model Dynamization: Most inference frameworks default to static input/output shapes and structures for performance reasons. Making a model dynamic (allowing variable input sizes or structures) without breaking original logic is a key challenge.
2. New Operator Implementation: New deep learning operators are often released faster than inference engines (like ONNX or TensorRT) can support them. This requires engineers to implement custom operator support.
3. IR and Inference Engine Compatibility: Different inference engines have varying levels of support for Intermediate Representations (IR) like ONNX. Ensuring consistent behavior across different engines often requires engine-specific model customization.

Model deployment is the process of running a trained model in a specific environment, addressing issues like framework incompatibility and slow execution speeds. The standard deployment pipeline follows this flow:

1. Deep Learning Framework: Where the model is trained (e.g., PyTorch).
2. Intermediate Representation (IR): A common format used to bridge frameworks and engines, most notably ONNX.
3. Inference Engine: The runtime that executes the model (e.g., ONNX Runtime, TensorRT, ncnn, OpenVINO).

Key concepts:

• Static Computation Graphs: Deployment often involves converting models into static graphs that lack control flow (like branches or loops).
• MMDeploy Role: MMDeploy facilitates the deployment of OpenMMLab models (detection, segmentation, super-resolution, etc.) across multiple inference engines like ONNX Runtime, TensorRT, ncnn, OpenPPPL, and OpenVINO.

The MMDeploy SDK is highly customizable and allows developers to control three main stages of the inference pipeline:

1. Transform: Data preprocessing.
2. Net: Model inference.
3. Module: Post-processing.

MMDeploy facilitates the deployment of OpenMMLab algorithms through a three-stage pipeline:

1. Model Converter: Converts input models (e.g., PyTorch) into device-agnostic IR models (ONNX, TorchScript) or directly into backend-specific formats (e.g., TensorRT, ONNX Runtime).
2. MMDeploy Model (SDK Model): A collection of the converted backend model and its associated metadata required for inference.
3. Inference SDK: A multi-language interface that encapsulates pre-processing, network inference, and post-processing.

ONNX models are stored using Protobuf (Protocol Buffers), a serialization mechanism by Google. The .onnx file is a binary instance of the data definitions (Protobuf files) that specify the types for models, nodes, and tensors.

An ONNX model is structured as a computational graph where:

• ModelProto: The top-level container representing the model. It includes metadata (version, creator) and the graph.
• GraphProto: Contained within ModelProto, it defines the computational structure, including inputs, outputs, and nodes.
• ValueInfoProto: Describes tensor information, including the tensor name, data type, and shape.
• NodeProto: Describes an operator node, including the operator type (op_type), and the names of its input and output tensors.

MMDeploy is divided into two independent parts: Model Conversion and the SDK.

• Model Conversion: Primarily uses tools, mmdeploy (Python package), and parts of csrc. It is used to rewrite model code (e.g., forward methods) and convert models to specific inference backends (like ncnn or TensorRT).
• SDK: Consists of csrc, third_party, and demo. It is a C++ based framework designed to execute converted models on various platforms, providing abstractions for preprocessing, inference, and postprocessing, along with FFI (Foreign Function Interface) for multiple languages.