MODNet Portrait Matting

To convert a PyTorch checkpoint to an ONNX model, run the onnx.export_onnx module. Note that a higher version of PyTorch than the official MODNet (which uses torch==1.7.1) is recommended for this export function.

python -m onnx.export_onnx \
    --ckpt-path=pretrained/modnet_photographic_portrait_matting.ckpt \
    --output-path=pretrained/modnet_photographic_portrait_matting.onnx

Official pre-trained models for MODNet are hosted on Google Drive. You must download these models manually and place them in the pretrained/ directory of the repository to use them for portrait matting tasks.

https://drive.google.com/drive/folders/1umYmlCulvIFNaqPjwod1SayFmSRHziyR?usp=sharing

To convert a standard MODNet checkpoint into a TorchScript version, ensure you are in the project root directory and have PyTorch version 1.2.0 or higher installed. Use the torchscript.export_torchscript module to perform the export.

python -m torchscript.export_torchscript \
    --ckpt-path=pretrained/modnet_photographic_portrait_matting.ckpt \
    --output-path=pretrained/modnet_photographic_portrait_matting.torchscript

Execute the matting process using the demo.video_matting.custom.run module. You must provide the path to your input video.

Command Syntax:

python -m demo.video_matting.custom.run --video YOUR_VIDEO_PATH

Arguments:

• --video YOUR_VIDEO_PATH: (Required) The specific path to the video file you want to process.
• --result-type: (Optional, default=fg) Specifies what to save. Options are fg (save the alpha matte) or fg (save the foreground).
• --fps: (Optional, default=30) Sets the frames per second for the resulting video.

To achieve the best results with the MODNet video matting demo, ensure your input videos meet the following criteria:

• Distinguishable Subjects: The portrait and the background should be clearly distinguishable (not similar).
• Lighting: Videos should be captured in soft and bright ambient lighting.
• Motion: Contents should not move too fast.

Note on Limitations:

• The model may make errors in portrait semantics estimation under challenging scenes due to limited labeled training data.
• The demo does not currently support the OFD trick.

You can obtain pre-trained models for MODNet through the following sources:

• Official Image Matting Model (TorchScript version): Available via Baidu Pan using extraction code dm9e.
• Pre-trained Image Matting Model (for manual export): Available via Google Drive. Place the downloaded file into the MODNet/pretrained/ directory.

Follow these steps to set up a conda environment and install dependencies for the WebCam demo:

1. Clone the repository and enter the directory:

   git clone https://github.com/ZHKKKe/MODNet.git
   cd MODNet

2. Download the pre-trained model from this link and place it in the MODNet/pretrained/ folder.
3. Create and activate a conda environment (e.g., using Python 3.6):

   conda create -n modnet python=3.6
   source activate modnet

4. Install dependencies (ensure your CUDA version is compatible with your PyTorch installation):

   pip install -r demo/video_matting/webcam/requirements.txt

5. Run the demo:

   python -m demo.video_matting.webcam.run

For higher performance (FPS) or custom video processing, use the offline demos:

• WebCam (Ubuntu recommended): Use the offline demo located at demo/video_matting/webcam.
• Custom Videos: Use the offline demo located at demo/video_matting/custom to process your own video files.

To use the ONNX export and inference tools, install the required dependencies using the provided requirements file:

pip install -r onnx/requirements.txt

The community has developed several optimized versions and applications of MODNet:

• ONNX Version: Convert pre-trained models to ONNX using this code or try the ONNX Colab demo.
• TorchScript Version: Convert models to TorchScript using this code.
• TensorRT Version: Access TensorRT optimized versions via this repository.
• Docker Container: Use a containerized environment via this repository.
• Background Blur: Use this Colab demo to apply Bokeh effects using MODNet.

To run the custom portrait video matting demo, follow these steps to prepare your environment and assets:

1. Clone the repository:

   git clone https://github.com/ZHKKKe/MODNet.git
   cd MODNet

2. Download the pre-trained model: Download the model from this link and place it in the MODNet/pretrained/ directory.

3. Create a Conda environment:

   conda create -n modnet python=3.6
   source activate modnet

4. Install dependencies: Ensure your CUDA version is compatible with your installed PyTorch version before running:

   pip install -r demo/video_matting/custom/requirements.txt

You can test MODNet's portrait matting capabilities through several online interfaces:

• Portrait Image Matting: Use the Colab demo to upload images and download alpha mattes.
• Portrait Video Matting: Use the online Colab demo for WebCam-based real-time matting.
• Fast Web Demo: For high-speed (2K resolution) processing on mobile or PC, visit the personal homepage.