DOCSEARCHbyAlgolia
Submit Docs

LLM2CLIP

repository·main·Indexed 20 days ago

https://github.com/microsoft/llm2clip

A framework that leverages Large Language Models (LLMs) as textual teachers to enhance CLIP's visual encoder. Using a Caption-to-Caption Contrastive Learning strategy, LLM2CLIP enables CLIP-style models to handle longer text inputs and complex, dense captions. The training process involves a two-stage approach consisting of Masked Next Token Prediction (MNTP) and Supervised SimCSE Loss, utilizing mixed datasets of pure text and image-captions.

Tokens
3.4K
Snippets
11
Records
21
Agent score
70%

What's inside LLM2CLIP

  1. Overview of LLM2CLIP

    main
    LLM2CLIP is a framework that uses Large Language Models (LLMs) as textual teachers to enhance CLIP's visual encoder. It addresses common CLIP limitations such as short context windows (77 tokens) and weak text comprehension by using a Caption-to-Caption Contrastive Learning strategy. This approach improves text-image alignment, especially for long, dense, and complex captions, and provides better open-world knowledge for multimodal feature alignment.

  2. Understand the LLM2CLIP training process

    main

    The training process for LLM2CLIP follows a two-stage approach to integrate Large Language Models (LLMs) with CLIP:

    1. Masked Next Token Prediction (MNTP): An initial stage used to prepare the model.
    2. Supervised SimCSE Loss: The primary stage where supervised SimCSE loss is applied to make different captions of the same image positive samples for each other, while captions of different images serve as negative samples.

    Note: While MNTP is used for optimal performance, it has been observed that removing it does not significantly affect the final conclusions or results.

    Dataset Strategy: To maintain the LLM's original distribution and mitigate caption bias, the training process mixes and shuffles two types of datasets:

    • Pure text datasets (e.g., Wikitext-103)
    • Image-caption datasets (e.g., CC-3M)

    Batches are sampled normally from this mixed and shuffled pool during training.

  3. Understand the LLM2CLIP fine-tuning process

    main

    LLM2CLIP uses a two-stage fine-tuning process to align the LLM with the CLIP vision encoder:

    1. Stage 1: Masked Next Token Prediction (MNTP): An initial stage used to prepare the model. While it has little impact on final results if removed, it is recommended for optimal performance.
    2. Stage 2: Supervised SimCSE Loss: The core stage where different captions of the same image are treated as positive samples, and captions of different images are treated as negative samples. This allows the LLM to provide meaningful supervisory signals to the image.

    Dataset Strategy: The training mixes pure text datasets (e.g., Wikitext-103) with image-caption datasets (e.g., CC-3M). The pure text data helps retain the LLM's original distribution and mitigates biases introduced by captions. The datasets are mixed, shuffled, and sampled normally during training.

  4. LLM2CLIP integration with Diffusion Models

    main

    LLM2CLIP can be integrated into image-text generative models (like Stable Diffusion) to enable more complex and precise control over text-to-image generation.

    Initial experiments indicate that using the LLM2CLIP Llama 3 model performs significantly better than a standard Llama 3 model when integrated with Stable Diffusion 3.

  5. Understanding LLM gradient freezing in LLM2CLIP

    main

    In the LLM2CLIP training methodology, the LLM gradients are typically kept frozen (or only specific layers are opened based on GPU memory constraints).

    Why the LLM is frozen:

    1. Batch Size: CLIP training is highly dependent on large batch sizes. Opening the LLM for full fine-tuning significantly increases memory requirements, which would force a reduction in batch size, potentially harming performance.
    2. Alignment Goal: The primary objective is to align the visual model with the correct textual modality provided by the LLM, rather than changing the LLM itself.

    Users attempting to replace the original text encoder with a tuned Llama 3 model should refer to the provided fine-tuned weights rather than attempting to manually reconfigure standard CLIP text encoder parameters in the configuration files.

  6. Train using Supervised learning with CC3M

    main

    To perform supervised training using CC3M, follow these steps:

    1. Data Preparation:
      • Prepare the E5 data used in llm2vec.
      • Prepare the CC3M dataset in CSV format (containing short_caption and long_caption).
    2. Configuration: Add the LoRA weights (pretrained during the MNTP phase) into the train_configs/supervised/MetaLlama3_cc3m.json configuration file.
    3. Execution: Run the supervised training script.

    Note: You will need a Hugging Face token (HF_TOKEN) for authentication.

    HF_TOKEN=xxxx accelerate launch --config_file ./ac_zero2.yaml run_supervised.py train_configs/supervised/MetaLlama3_cc3m.json

  8. Install llm2vec and dependencies

    main

    To set up the environment for LLM2CLIP (which builds on llm2vec), create a Conda environment with Python 3.10 and install the required packages. Note that a specific version of accelerate is required to ensure compatibility with the provided scripts.

    conda create -n llm2vec python=3.10 -y
conda activate llm2vec
pip install llm2vec
pip install flash-attn --no-build-isolation
pip install deepspeed
pip install accelerate==0.34.2

  9. Train using MNTP with CC3M

    main

    To perform Masked Next Token Prediction (MNTP) training using the CC3M dataset, follow these steps:

    1. Data Preparation: Prepare the CC3M dataset in CSV format. The CSV must contain two specific keys: short_caption and long_caption.
    2. Configuration: Update the path to your cc3m.csv file within the MetaLlama3_cc3m.json configuration file.
    3. Execution: Run the training script using accelerate launch with the provided configuration.

    Note: You will need a Hugging Face token (HF_TOKEN) for authentication.

    cd llm2vec
HF_TOKEN=xxxx accelerate launch --config_file ./ac_zero2.yaml run_mntp.py train_configs/mntp/MetaLlama3_cc3m.json

  10. Use pre-tuned LLM2CLIP models for retrieval and training

    main

    To avoid manual configuration of text encoders, you can use pre-tuned models available on Hugging Face:

    • For retrieval experiments: Use the caption contrastive fine-tuned version of Llama3-8B-CC: microsoft/LLM2CLIP-Llama-3-8B-Instruct-CC-Finetuned.
    • For training your own CLIP models: Use the adapter and projector parameters available in the OpenAI ViT-L repository: microsoft/LLM2CLIP-Openai-L-14-336.

    Detailed retrieval testing methods can be found in the respective model cards on Hugging Face.

    # Llama 3-8B-Instruct-CC-Finetuned (for retrieval)
https://huggingface.co/microsoft/LLM2CLIP-Llama-3-8B-Instruct-CC-Finetuned

# OpenAI ViT-L (for adapter/projector parameters)
https://huggingface.co/microsoft/LLM2CLIP-Openai-L-14-336

  11. Use fine-tuned Llama 3 models for retrieval and training

    main

    To assist with retrieval experiments or to train your own CLIP models, you can use the caption contrastive fine-tuned version of Llama 3.

    • For retrieval experiments: Use the LLM2CLIP-Llama-3-8B-Instruct-CC-Finetuned model available on Hugging Face.
    • For adapter and projector parameters: Use the parameters provided in the LLM2CLIP-Openai-L-14-336 repository on Hugging Face.

    Note: If you are using the EVA series of models, be aware that precision mismatches during Hugging Face conversion have been reported and are being addressed in progressive updates.