DOCSEARCHbyAlgolia
Submit Docs

MobileCLIP

repository·main·Indexed 23 days ago

https://github.com/apple/ml-mobileclip

Fast and efficient image-text models optimized for mobile and edge devices, featuring MobileCLIP and MobileCLIP2 architectures. The repository includes support for real-time zero-shot scene classification via the MobileCLIPExplore iOS app, OpenCLIP integration for inference, and tools for synthetic caption generation using CoCa models. It provides pretrained checkpoints and scripts for ImageNet evaluation and training on DFNDR datasets.

Tokens
4.1K
Snippets
12
Records
17
Agent score
68%

What's inside MobileCLIP

  1. Download CoCa models for synthetic caption generation

    main

    CoCa models used to generate synthetic captions in DFNDR-2B are available on HuggingFace.

    Recommendations:

    • For balanced performance, use mscoco38k_s12m_context77.
    • Models with context77 are generally preferred. Models with context256 have a higher chance of generating repeated output.

    Use the hf download command to retrieve specific model variants from the apple/mobileclip2_coca_dfn2b_s13b_ prefix.

    # context=77 models
for model in \
  mscoco38k_s12m_context77 \
  gbc1m-short_context77 \
  docci_s12m_context77 \
  dci-short_s12m_context77 \
  dci-complete_s12m_context77 \
  dci-extended_s12m_context77 \
  recap-coco-30k_s12m_context77 \
do
  hf download apple/mobileclip2_coca_dfn2b_s13b_$model
done

# Context=256 models. These models have a higher chance of generating repeated output
for model in \
  docci_s12m_context256 \
  dci-complete_s12m_context256 \
  dci-extended_s12m_context256 \
do
  hf download apple/mobileclip2_coca_dfn2b_s13b_$model
done

# Base CoCa-DFN2B model that can be used for fine-tuning new CoCa models on 
# high-quality datasets
hf download apple/mobileclip2_coca_dfn2b_s13b_context77

  2. Install MobileCLIPExplore on iOS

    main

    To use the MobileCLIPExplore demonstration app, you must provide the CoreML models manually.

    1. Download the required CoreML models from Hugging Face.
    2. Place the downloaded models into the following directory within your project:

    .../MobileCLIPExplore/MobileCLIPExplore/Models

    Requirements:

    • iOS 17.2 or later.

  3. Create and edit custom prompts in MobileCLIPExplore

    main

    You can customize how the model classifies scenes by editing prompts and classnames.

    1. Edit Prompt: Select the prompt or the pencil icon (✎).
    2. Customize Prefix/Suffix: Tap the start or end of a prompt to modify it.
    3. Modify Detection Targets: Use the Classnames field to add or remove specific items you want the app to detect.

  4. Train models on DataCompDR datasets

    main

    Use the provided shell scripts within the open_clip directory to train ViT-B/16 models on DataComp datasets, with options for both standard and DR (Reinforcement) training.

    cd open_clip/
bash configs/run_datacomp12m.sh  # Train a ViT-B/16 on DataComp-12M without DR
bash configs/run_datacompdr12m.sh  # Train a ViT-B/16 on DataComp-12M with DR
bash configs/run_datacompdr1B.sh  # Train a ViT-B/16 on DataComp-1B with DR

  7. Run zero-shot ImageNet evaluation

    main

    To reproduce evaluation results on the ImageNet-1k dataset, use the eval/zeroshot_imagenet.py script. You must specify the --model-arch and --model-path.

    # Run evaluation with single GPU
python eval/zeroshot_imagenet.py --model-arch mobileclip_s0 --model-path /path/to/mobileclip_s0.pt

  8. Install MobileCLIP dependencies (v1 reproducibility)

    main

    To reproduce the v1 OpenCLIP code, you must first revert specific path references in the MobileCLIP repository to ensure compatibility with an older OpenCLIP commit, then apply the v1 patch.

    # Revert changes for compatibility with older OpenCLIP commit
sed -i 's/open_clip_train/training/g' ../dr/transforms.py
find . --name "*.sh" | xargs sed -i 's/open_clip_train/training/g'

# Clone OpenCLIP repository, apply patch, and install
git clone https://github.com/mlfoundations/open_clip.git
cd open_clip
git checkout cf86ee7ec4658845f640858ecd34d0f15588271a  # Wed May 29 21:57:08 2024 +0700
git apply ../open_clip_v1.patch
cp ../configs/ ./ -r
cp ../dr/ ./src/training/ -r

  9. Use MobileCLIPExplore for real-time scene classification

    main

    MobileCLIPExplore allows for real-time zero-shot scene classification using the device camera.

    Workflow:

    1. Select Model: Choose one of the four available MobileCLIP models from the top menu.
    2. Configure Prompt: Choose a prompt preset or create a custom one.
    3. Classify: Point the camera at a subject. Prediction results will appear in a table.

    Metrics:

    • FPS (frames per second): Displayed in the top left of the landing view.
    • Response Time (ms): Displayed in the top left of the landing view.

  10. Train models on DFNDR datasets

    main

    Use the provided shell scripts within the open_clip directory to train MobileCLIP models on DFNDR datasets.

    cd open_clip/
bash configs/run_dfndr12m.sh  # Train a MobileCILP-B on DFN-2B12M with DR
bash configs/run_dfndr1B.sh  # Train a MobileCLIP-B on DFN-2B with DR

  11. Prepare DataCompDR-12M datasets

    main

    The DataCompDR datasets on HuggingFace do not include original images and captions. To use DataCompDR-12M, you must download both the DataComp-12M dataset (for original captions) and the DataCompDR-12M-noimage dataset, then join them using the following script. Images must be downloaded separately (see hf_dataset_example.py for a single image download example).

    #!/bin/bash
DATACOMP12M_PATH="./datasets/DataComp-12M/" # Download path of DataComp-12M from HF
DATACOMPDR12M_NOIMG_PATH="./datasets/DataCompDR-12M-noimage/" # Download path of DataCompDR-12M from HF
DATACOMPDR12M_PATH="./datasets/DataCompDR-12M/"
for  i in {00000000..00001023}
do
  mkdir tmp
  tar -xf $DATACOMP12M_PATH/${i}.tar -C tmp
  tar -xf $DATACOMPDR12M_NOIMG_PATH/${i}.tar -C tmp
  tar -cf $DATACOMPDR12M_PATH/${i}.tar -C tmp *.*
  rm -rf tmp
done

  12. Install MobileCLIP dependencies (v2)

    main

    To support loading reinforcements and training loss, a patched version of OpenCLIP is required. Follow these steps to clone the MobileCLIP repository, checkout the specific OpenCLIP commit, apply the v2 patch, and install the necessary configuration and source files.

    # Clone MobileCLIP repository
git clone git@github.com:apple/ml-mobileclip.git
cd ml-mobileclip/

# Clone OpenCLIP repository, apply patch, and install
git clone https://github.com/mlfoundations/open_clip.git
cd open_clip
git checkout 7260a46e7b4bcf518f5200fea06da5bc85aae025  # Mon Mar 17 18:18:30 2025 -0400
git apply ../open_clip_v2.patch
cp ../configs/ ./ -r
cp ../dr/ ./src/open_clip_train/ -r
cp ../../mobileclip2/* ./src/open_clip/ -r