CLIPSeg

Certain datasets require additional repositories. These should be cloned into the third_party folder of the project.

# Run these commands in the third_party folder
git clone https://github.com/cvlab-yonsei/JoEm
git clone https://github.com/Jia-Research-Lab/PFENet.git
git clone https://github.com/ChenyunWu/PhraseCutDataset.git
git clone https://github.com/juhongm999/hsnet.git

Model weights are provided in a zip file. You can download them using wget and unzip them into a weights directory. Note that the MIT license does not apply to these weights.

wget https://owncloud.gwdg.de/index.php/s/ioHbRzFx6th32hn/download -O weights.zip
unzip -d weights -j weights.zip

To use CLIPSeg, you need to install pytorch, torchvision, and the OpenAI clip library. You can install the CLIP library directly from GitHub using pip.

pip install git+https://github.com/openai/CLIP.git

Images must be loaded and transformed before being passed to the model. The required preprocessing includes converting to a tensor, normalizing with ImageNet statistics, and resizing to (352, 352).

You can load images from a local file using PIL.Image.open or from a URL using requests.

from PIL import Image
from torchvision import transforms
import torch

# Load image
input_image = Image.open('example_image.jpg')

# Define transformation pipeline
transform = transforms.Compose([
    transforms.ToTensor(),
    transforms.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225]),
    transforms.Resize((352, 352)),
])

# Apply transform and add batch dimension
img = transform(input_image).unsqueeze(0)

To use CLIPSeg, you must first download and unzip the model weights. The weights provided are decoder weights only, so when loading the state dict, you must use strict=False because the CLIP weights are not included in the checkpoint.

1. Download the weights: wget https://owncloud.gwdg.de/index.php/s/ioHbRzFx6th32hn/download -O weights.zip
2. Unzip the weights into a directory named weights.
3. Import CLIPDensePredT from models.clipseg.

! wget https://owncloud.gwdg.de/index.php/s/ioHbRzFx6th32hn/download -O weights.zip
! unzip -d weights -j weights.zip

from models.clipseg import CLIPDensePredT
import torch

# Initialize model
model = CLIPDensePredT(version='ViT-B/16', reduce_dim=64)
model.eval()

# Load weights with strict=False
model.load_state_dict(torch.load('weights/rd64-uni.pth', map_location=torch.device('cpu')), strict=False)

To use the segmentation features, you need to load both the standard CLIP model (for text encoding and similarity) and the CLIPDensePredTMasked model (for visual feature extraction).

import clip
from models.clipseg import CLIPDensePredTMasked

clip_device = 'cuda'
clip_model, preprocess = clip.load("ViT-B/16", device=clip_device)
clip_model.eval();

clip_mask_model = CLIPDensePredTMasked(version='ViT-B/16').to(clip_device)
clip_mask_model.eval();

For more refined predictions (avoiding square-like artifacts), use the rd64-uni-refined.pth weights. When initializing the model, you must set complex_trans_conv=True to match the architecture of these weights.

model = CLIPDensePredT(version='ViT-B/16', reduce_dim=64, complex_trans_conv=True)
model.load_state_dict(torch.load('weights/rd64-uni-refined.pth'), strict=False)

COCO experiment results can be aggregated by calculating the mean of specific slices of the DataFrame. This is useful for comparing different model variants like CLIPSeg (COCO) and CLIPSeg (COCO+N).

Example:

coco = experiment('experiments/coco.yaml', nums=':29').dataframe()
tab1 = coco[['coco_h2_miou_0.1', 'coco_h2_biniou_0.1', 'coco_h2_ap']]
print('CLIPSeg (COCO) & 0.1 & CLIP &  ' + ' & '.join(f'{x*100:.1f}' for x in tab1[:4].mean(0).values), '\\')

Generalization metrics from experiments/generalize.yaml include aff_best_fgiou, aff_ap, ability_best_fgiou, ability_ap, part_best_fgiou, and part_ap.

Example:

gen = generalization[['aff_best_fgiou', 'aff_ap', 'ability_best_fgiou', 'ability_ap', 'part_best_fgiou', 'part_ap']].values
print(
    'CLIPSeg (PC+) & ' + ' & '.join(f'{x*100:.1f}' for x in gen[1]) + ' \\ \\n' +
    'CLIPSeg (LVIS)  & ' + ' & '.join(f'{x*100:.1f}' for x in gen[0]) + ' \\ \\n' +
    'CLIP-Deconv & ' + ' & '.join(f'{x*100:.1f}' for x in gen[2]) + ' \\ \\n' +
    'VITSeg & ' + ' & '.join(f'{x*100:.1f}' for x in gen[3]) + ' \\'
)

For One-shot Pascal experiments, you can aggregate mean values across specific index ranges in the DataFrame to create LaTeX rows.

Example for CLIPSeg (PC+) at 0.3 threshold:

pas = experiment('experiments/pascal_1shot.yaml', nums=':8').dataframe()
tab1 = pas[['pas_h2_miou_0.3', 'pas_h2_biniou_0.3', 'pas_h2_ap']]
print('CLIPSeg (PC+) & 0.3 & CLIP & ' + ' & '.join(f'{x*100:.1f}' for x in tab1[0:4].mean(0).values), '\\')

Ablation results from experiments/ablation.yaml can be filtered and formatted. Common metrics include pc_miou_best, pc_ap, pc-vis_miou_best, and pc-vis_ap.

Example:

ablation = experiment('experiments/ablation.yaml', nums=':8').dataframe()
tab1 = ablation[['name', 'pc_miou_best', 'pc_ap', 'pc-vis_miou_best', 'pc-vis_ap']]
for k in ['pc_miou_best', 'pc_ap', 'pc-vis_miou_best', 'pc-vis_ap']:
    tab1.loc[:, k] = (100 * tab1.loc[:, k]).round(1)
tab1.loc[:, 'name'] = ['CLIPSeg', 'no CLIP pre-training', 'no-negatives', '50% negatives', 'no visual', '$D=16$', 'only layer 3', 'highlight mask']
print(tab1.loc[[0,1,4,5,6,7],:].to_latex(header=False, index=False))

Zero-shot results from experiments/pascal_0shot.yaml can be extracted using the pas_zs_seen and pas_zs_unseen columns.

Example:

zs = experiment('experiments/pascal_0shot.yaml', nums=':11').dataframe()
tab1 = zs[['pas_zs_seen', 'pas_zs_unseen']]
print('CLIPSeg (PC+) & CLIP &  ' + ' & '.join(f'{x*100:.1f}' for x in tab1[8:9].values[0].tolist() + tab1[10:11].values[0].tolist()), '\\')