MatAnyone 2

To install MatAnyone2 using Conda, clone the repository, create a new environment with Python 3.10, and install the package in editable mode. You can optionally install dependencies for the Gradio demo.

# Clone Repo
git clone https://github.com/pq-yang/MatAnyone2
cd MatAnyone2

# Create Conda Environment and Install Dependencies
conda create -n matanyone2 python=3.10 -y
conda activate matanyone2

# install python dependencies
pip install -e .

# [optional] install python dependencies for gradio demo
pip3 install -r hugging_face/requirements.txt

Download the pretrained model matanyone2.pth and place it in a pretrained_models directory. The model will also be automatically downloaded during the first inference if not found.

pretrained_models
   |- matanyone2.pth

To reproduce quantitative results for the CRGNN real benchmark, follow these steps for data preparation, batch inference, and evaluation.

### 1. Preparation
Arrange your files as follows:

data |- crgnn
|- alpha |- image_allframe |- mask # first frame seg mask

### 2. Batch Inference
```shell
bash evaluation/infer_batch_crgnn.sh

3. Evaluation

Arrange results and true data:

data
   |- crgnn
        |- alpha

   |- results
        |- crgnn

python evaluation/eval_crgnn.py \
    --pred-dir ./data/results/crgnn \
    --true-dir ./data/crgnn/alpha 

You can add MatAnyone2 to a new project using uv by initializing a project and adding the package directly from the GitHub repository.

uv init my-matting-project && cd my-matting-project
uv add matanyone2@git+https://github.com/pq-yang/MatAnyone2.git

To run an interactive GUI that allows you to upload videos and assign masks via clicks, launch the Gradio app located in the hugging_face directory. Note that FFmpeg is required on your system.

cd hugging_face

# install GUI dependencies
pip3 install -r requirements.txt

# launch the demo
python app.py

The interactive_state is a dictionary used to track the progress of user interactions and mask accumulation. It is essential for multi-mask matting.

Key Fields:

• inference_times: Counter for total inferences.
• negative_click_times: Count of negative point prompts.
• positive_click_times: Count of positive point prompts.
• mask_save: Boolean/config for saving masks.
• multi_mask: A dictionary containing:
  • mask_names: List of names for the added masks.
  • masks: The actual mask data/arrays.
• track_end_number: The frame number where tracking should stop.

You can perform video matting programmatically using the MatAnyone2 and InferenceCore classes. The model can be loaded from Hugging Face using from_pretrained.

from matanyone2 import MatAnyone2, InferenceCore

# Load model from Hugging Face
model = MatAnyone2.from_pretrained("PeiqingYang/MatAnyone2")

# Initialize processor
processor = InferenceCore(model, device="cuda:0")

# Run inference
processor.process_video(
    input_path="inputs/video/test-sample2.mp4",
    mask_path="inputs/mask/test-sample2.png",
    output_path="results",
)

The interface supports adding and removing multiple masks to handle complex objects.

Add a mask

Use add_multi_mask to commit the current SAM refinement as a new mask layer.

• Inputs: state, interactive_state, mask_dropdown.
• Outputs: interactive_state, mask_dropdown, template_frame, click_state.

Remove a mask

Use remove_multi_mask to delete a mask layer.

• Inputs: interactive_state, mask_dropdown.
• Outputs: interactive_state, mask_dropdown.

The MaskGenerator class is used to manage the Segment Anything Model (SAM) controller. It requires a SAM checkpoint and an argument object containing configuration like sam_model_type and device.

class MaskGenerator():
    def __init__(self, sam_checkpoint, args):
        self.args = args
        self.samcontroler = SamControler(sam_checkpoint, args.sam_model_type, args.device)

    def first_frame_click(self, image: np.ndarray, points:np.ndarray, labels: np.ndarray, multimask=True):
        mask, logit, painted_image = self.samcontroler.first_frame_click(image, points, labels, multimask)
        return mask, logit, painted_image