EfficientVMamba

Mask R-CNN is a framework for object instance segmentation that extends Faster R-CNN by adding a branch for predicting object masks in parallel with bounding box recognition. This repository provides configurations and pre-trained models for various Mask R-CNN architectures, including ResNet-50 (R-50), ResNet-101 (R-101), and ResNeXt (X-101) backbones with Feature Pyramid Networks (FPN).

Key performance metrics available in the results table include:

• box AP: Average Precision for bounding boxes.
• mask AP: Average Precision for segmentation masks.
• Inf time (fps): Inference speed in frames per second.
• Mem (GB): Memory usage in Gigabytes.

Models are available in both caffe and pytorch styles, with different learning rate schedules (1x, 2x) and precision modes (e.g., FP16).

For detection and segmentation tasks, use the training command structure similar to classification, but select the appropriate configuration files from the configs/vssm directory. Ensure you adjust the following parameters:

• --cfg: Path to the task-specific config file.
• --data-path: Path to your specific dataset.
• --output: Desired output location for logs and checkpoints.

If you intend to use EfficientVMamba for detection or segmentation tasks, you must install the following OpenMMLab-related dependencies:

pip install mmengine==0.10.1 mmcv==2.1.0 opencv-python-headless ftfy
pip install mmdet==3.3.0 mmsegmentation==1.2.2 mmpretrain==1.2.0

Follow these steps to set up the EfficientVMamba environment. The installation process follows the same steps as the original VMamba repository.

1. Clone the repository
2. Create a Conda environment
3. Install Python dependencies
4. Install selective_scan and its dependencies (Required)

# Step 1: Clone the repository
git clone https://github.com/TerryPei/EfficientVMamba.git
cd EfficientVMamba

# Step 2: Create and activate a new conda environment
conda create -n vmamba
conda activate vmamba

# Step 3: Install Dependencies
pip install -r requirements.txt

# Step 4: Install selective_scan and its dependencies
cd selective_scan && pip install . && pytest

To install the selective_scan package along with its CUDA core components, clear any existing build artifacts and install the current directory using pip. After installation, you can verify the installation by running the provided test suite.

rm -r dist/ build/; pip install .
pytest test_selective_scan.py # expected all pass

To use ConvNeXt backbones for downstream tasks in this repository, you must first install mmpretrain. This package provides the necessary backbone implementations.

pip install mmpretrain

When combining a Swin Transformer backbone with a one-stage detector (e.g., RetinaNet), you must account for the Layer Normalization layer at the output of the Swin Transformer.

To ensure correct feature extraction in the Feature Pyramid Network (FPN), you must:

1. Set start_level to 0 in the FPN configuration.
2. Set the out_indices of the backbone to [1, 2, 3].

Refer to the retinanet_swin-t-p4-w7_fpn_1x_coco.py configuration file as a template for this setup.

The SS2D module utilizes cross_selective_scan to perform efficient spatial scanning. Instead of a standard 1D scan, it uses EfficientScan and EfficientMerge (from efficient_scan.py) to process spatial patches.

1. EfficientScan: Reduces the spatial resolution by a step_size factor, creating a multi-scale representation.
2. Cross-Selective Scan: Performs the SSM operation on these reduced representations using projected weights (x_proj_weight, dt_projs_weight).
3. EfficientMerge: Reconstructs the original spatial resolution from the scanned outputs.

This approach allows the model to capture long-range dependencies across the 2D plane more efficiently than standard 1D scans applied to flattened images.

The training pipeline supports Exponential Moving Average (EMA) to improve model generalization.

1. Initialization: If --model_ema is enabled, an EMA model is created after the base model is moved to the GPU but before DDP wrapping.
2. Update: During train_one_epoch, after every ACCUMULATION_STEPS gradient updates, the EMA model is updated using the current model weights: model_ema.update(model).
3. Evaluation: During validation, the script evaluates both the standard model and the EMA model, logging Acc@1 and Acc@5 for both to track which version performs better.

When applying Mixup, the targets are not just swapped; they are interpolated based on the mixing coefficient $\lambda$ (lam).

If you use Mixup or mixup_target, the resulting target is a soft label (one-hot encoded) where:

• The original class has a weight of lam * (1 - smoothing) + smoothing/num_classes.
• The mixed class has a weight of (1 - lam) * (1 - smoothing) + smoothing/num_classes.

This ensures that the labels reflect the proportion of each image present in the mixed sample, while also incorporating label smoothing.