OpenCLIP

CLIPA uses token length reduction to implement an inverse scaling law for CLIP training. You can reduce image token length using the following strategies:

• resize: Uses the --force-image-size flag to specify a target image size. This is generally the most effective strategy as it retains full image information.
• random mask: Randomly masks out image patches. Use the --force-patch-dropout flag to specify the desired mask ratio.
• grid mask (Experimental): Preserves one patch in each 2x2 grid window. Implementation is not provided as resizing is preferred.
• block mask (Experimental): Keeps a single block and removes other patches. Implementation is not provided as resizing is preferred.

# Example usage for resizing
--force-image-size 224

# Example usage for random masking
--force-patch-dropout 0.5

The main branch of OpenCLIP uses a post-refactor training stack. This stack is organized around TrainingTask wrappers, uses dict-based batches, supports FSDP2, and includes NaFlex image/audio pipelines.

Warning for users:

• If you require the older, release-stable training API, you must pin to the v3 branch or the latest 3.x release on PyPI.
• While inference for pretrained image/text models remains compatible, training scripts and downstream integrations should be reviewed for breaking changes when upgrading to the main branch.

To simulate larger batch sizes without increasing GPU memory usage linearly, use the --accum-freq k flag.

If your per-GPU batch size (--batch-size) is m, the effective batch size becomes k * m * num_gpus.

Important Considerations:

• Increasing --accum-freq above 1 will keep samples/s approximately constant, but time-per-batch will double.
• It is recommended to use --grad-checkpointing, --local-loss, or --gather-with-grad to reduce batch size before relying solely on accumulation.
• Using accumulation requires additional GPU memory to store features and data from all m batches in memory, and results in m loss computations instead of one.

You can train on multiple datasets by separating paths with :: in the --train-data flag.

• Sampling: Use --dataset-resampled to enable sampling with replacement.
• Weighting: Use --train-data-upsampling-factors to control the relative frequency of each source. For example, --train-data-upsampling-factors=1::2 upsamples the second source twice as much as the first.

# Example: Training on two different datasets
--train-data "/data/cc12m/cc12m-train-{0000..2175}.tar::/data/LAION-400M/{00000..41455}.tar" \
--train-data-upsampling-factors=1::2

If you have existing image/text training scripts that rely on the pre-task loop (e.g., calling train_one_epoch directly), you can use the compatibility shim:

python -m open_clip_train.legacy_main

Limitations of legacy_main:

• Does not support FSDP2, EMA, CLAP audio training, NaFlex, or length bucketing.
• Uses a frozen decode-first data pipeline.
• Should be treated as a compatibility shim rather than a path for new development.

To fine-tune CoCa models (e.g., on MSCOCO), use the open_clip_train.main script. To focus specifically on the generative side rather than the contrastive side, set --coca-contrastive-loss-weight 0 and --coca-caption-loss-weight 1.

python -m open_clip_train.main \
    --dataset-type "csv" \
    --train-data "path/to/data/dir/train2014.csv" \
    --warmup 1000 \
    --batch-size 128 \
    --lr 1e-5 \
    --wd 0.1 \
    --epochs 1 \
    --workers 3 \
    --model "coca_ViT-L-14" \
    --report-to "wandb" \
    --coca-contrastive-loss-weight 0 \
    --coca-caption-loss-weight 1 \
    --log-every-n-steps 100

LAION-400M models were trained to replicate OpenAI's ViT results using the LAION-400M dataset. Available architectures include:

• ViT-B/32 224x224: Top-1 ImageNet-1k zero-shot accuracy of 62.96%.
• ViT-B/16 224x224: Top-1 ImageNet-1k zero-shot accuracy of 67.07%.
• ViT-B/16+ 240x240: Increased vision width (896), text width (640), and resolution (240x240). Top-1 ImageNet-1k zero-shot accuracy of 69.21%.
• ViT-L/14 224x224: Top-1 ImageNet-1k zero-shot accuracy of 72.77%.

Trained weights can be found in release v0.2.