DOCSEARCHbyAlgolia
Submit Docs

FastVLM

repository·main·Indexed 27 days ago

https://github.com/apple/ml-fastvlm

FastVLM provides efficient vision encoding for Vision Language Models (VLMs) using the FastViTHD hybrid vision encoder to reduce encoding time and token output for high-resolution images. It supports high-performance inference on servers and Apple Silicon devices (iOS 18.2+ and macOS 15.2+). The project includes pretrained models in 0.5B, 1.5B, and 7B sizes, tools for exporting models to MLX format via coremltools, and a llava package for managing multimodal conversations and image processing.

Tokens
5K
Snippets
15
Records
38
Agent score
92%

What's inside ml-fastvlm

  1. Use custom or fine-tuned FastVLM models

    main
    You can use custom quantized or fine-tuned versions of FastVLM. To integrate a custom model, follow the model export documentation to [export the model](../model_export#export-vlm). Ensure the custom model files are placed in app/FastVLM/model after clearing any existing models in that directory.

  2. Export the VLM to MLX format

    main

    Convert a FastVLM checkpoint to the MLX format using mlx_vlm.convert. You can export only the LLM component and optionally apply quantization.

    # Standard export (LLM only)
python -m mlx_vlm.convert --hf-path  /path/to/fastvlm-checkpoint \
                          --mlx-path /path/to/exported-fastvlm \
                          --only-llm

# Export with 8-bit quantization
python -m mlx_vlm.convert --hf-path  /path/to/fastvlm-checkpoint \
                          --mlx-path /path/to/exported-fastvlm \
                          --only-llm \
                          -q \
                          --q-bits 8

  3. Setup FastVLM environment

    main

    To set up the FastVLM environment, create a new Conda environment with Python 3.10 and install the package in editable mode.

    conda create -n fastvlm python=3.10
conda activate fastvlm
pip install -e .

  4. Export the Vision Encoder

    main

    To ensure all states needed for auto-inference (especially for third-party libraries like mlx-vlm) are available, you must export the vision encoder using coremltools and patch the checkpoint. This saves additional metadata to the model checkpoint directory.

    python export_vision_encoder.py --model-path /path/to/fastvlm-checkpoint

  6. Run FastVLM on iOS and macOS

    main

    FastVLM is designed for on-device visual question answering. To run the application:

    1. Download the desired model using the get_pretrained_mlx_model.sh script.
    2. Open the project in Xcode.
    3. Build and Run the app.

    Compatibility Requirements:

    • iOS: 18.2 or later
    • macOS: 15.2 or later

    Features:

    • View Time-To-First-Token (TTFT) for every inference.
    • Private and secure on-device processing.
    • Flexible prompting via the Prompts button (top-right corner). Use Customize... to edit or create new prompts.

  7. Inference on Apple Silicon

    main

    To run FastVLM on Apple Silicon, PyTorch checkpoints must be exported to a compatible format. Detailed instructions and code are located in the model_export/ subfolder.

    Pre-converted models are available for convenience:

    • fastvlm_0.5b_stage3 (FP16)
    • fastvlm_1.5b_stage3 (INT8)
    • fastvlm_7b_stage3 (INT4)

  9. Download pretrained FastVLM models

    main

    Use the get_pretrained_mlx_model.sh script to download pretrained FastVLM models for use in the app. The script downloads the model from the web and places it in the correct directory.

    Available model sizes:

    • FastVLM 0.5B: Small and fast, optimized for mobile speed.
    • FastVLM 1.5B: Balanced speed and accuracy for larger devices.
    • FastVLM 7B: High accuracy, optimized for performance over speed.

    Important: Before downloading or copying a new model, ensure you clear the existing model in app/FastVLM/model.

  10. Install the patched mlx-vlm for FastVLM support

    main

    FastVLM requires a specific patch for mlx-vlm to support inference. Follow these steps to clone, checkout the compatible commit, apply the patch, and install in editable mode:

    1. Clone the mlx-vlm repository.
    2. Checkout commit 1884b551bc741f26b2d54d68fa89d4e934b9a3de.
    3. Apply the fastvlm_mlx-vlm.patch file located in the parent directory.
    4. Install the package.
    git clone https://github.com/Blaizzy/mlx-vlm.git
cd mlx-vlm 
git checkout 1884b551bc741f26b2d54d68fa89d4e934b9a3de
git apply ../fastvlm_mlx-vlm.patch
pip install -e .

  11. Troubleshoot ValueError during conversion

    main

    If you encounter the error ValueError: Received parameters not in model: language_model.lm_head.weight. during conversion, it is likely caused by an incorrect tie_word_embeddings value in the LLaVA model's config.json.

    Solution: Manually update the tie_word_embeddings value in your config.json to the correct setting for your model architecture.

  12. Run inference with PyTorch checkpoints

    main

    Use the predict.py script to run inference on a PyTorch checkpoint by providing the model path, an image file, and a text prompt.

    python predict.py --model-path /path/to/checkpoint-dir \
                  --image-file /path/to/image.png \
                  --prompt "Describe the image."