DOCSEARCHbyAlgolia
Submit Docs

mlx-vlm

repository·main·Indexed 26 days ago

https://github.com/blaizzy/mlx-vlm

A package for the inference and fine-tuning of Vision Language Models (VLMs) and Omni Models (supporting audio and video) optimized for Apple Silicon using the MLX framework. It includes features for computer control via GUI and voice agents, as well as specialized support for DeepSeek-OCR and DeepSeek-OCR-2 for document understanding, text localization, and OCR tasks.

Tokens
62.1K
Snippets
173
Records
295
Agent score
90%

What's inside mlx-vlm

  1. Overview of Mage-Flow models

    main

    Mage-Flow is a family of Microsoft's 4B native-resolution image generation and editing models. mlx-vlm supports loading original Hugging Face Diffusers-style checkpoints. All variants support native resolutions from 512 to 2048 pixels per side, including aspect ratios up to 4:1. Note that width and height must be multiples of 16.

    Available Models and Recommended Settings

    Hugging Face modelAliasTaskRecommended settings
    microsoft/Mage-Flow-Basemage-flow-baseGeneration30 steps, guidance 5
    microsoft/Mage-Flowmage-flowGeneration20 steps, guidance 5
    microsoft/Mage-Flow-Turbomage-flow-turboGeneration4 steps, guidance 1
    microsoft/Mage-Flow-Edit-Basemage-flow-edit-baseEditing30 steps, guidance 5
    microsoft/Mage-Flow-Editmage-flow-editEditing30 steps, guidance 5
    microsoft/Mage-Flow-Edit-Turbomage-flow-edit-turboEditing4 steps, guidance 1

  2. LocateAnything Model Specifications

    main

    LocateAnything is a 3B parameter vision-language grounding model designed for visual grounding, open-vocabulary object localization, and referring expression localization.

    • Model ID: nvidia/LocateAnything-3B
    • Architecture: MoonViT vision encoder + MLP connector + Qwen2.5 language model
    • Modalities: Image + text

  3. Use DeepSeek-OCR-2 for Document Understanding and OCR

    main
    DeepSeek-OCR-2 is an OCR model optimized for document understanding, text extraction, and visual grounding. It supports converting documents to Markdown, general OCR, figure parsing, image description, and text localization (grounding). For best results, use the <|grounding|> prefix and set temperature=0.0 for deterministic output.

  4. MolmoPoint Model Specifications

    main

    MolmoPoint is an ~8B parameter vision-language model designed for visual pointing and grounding. It uses a SigLIP ViT vision encoder and a Qwen2-style decoder.

    Key Capabilities:

    • Visual pointing/grounding (pixel-precise coordinates).
    • Image description.
    • Visual Question Answering (VQA).

    Hardware/Performance Notes:

    • Memory: Peak memory is approximately 39 GB for the full bf16 model.
    • Speed: ~6 tokens/sec on Apple Silicon (M-series).
    • Dependencies: The custom image processor uses PIL and numpy only (no torch dependency).

  6. Use Nemotron Labs Diffusion via CLI

    main

    The Nemotron Labs Diffusion model (nvidia/Nemotron-Labs-Diffusion-8B) supports three primary generation modes via the mlx_vlm.generate CLI command. Use --gen-kwargs to specify the generation_mode.

    ### Autoregressive generation (Default)
```sh
mx_vlm.generate \
  --model nvidia/Nemotron-Labs-Diffusion-8B \
  --prompt "Write a short story about a clockmaker." \
  --max-tokens 256 \
  --temperature 0.0

    Diffusion generation

    Pass generation_mode="diffusion" (or alias "dlm") via --gen-kwargs. Use --verbose to see live block visualization.

    mlx_vlm.generate \
  --model nvidia/Nemotron-Labs-Diffusion-8B \
  --prompt "Write a short story about a clockmaker." \
  --max-tokens 256 \
  --max-denoising-steps 16 \
  --temperature 0.0 \
  --gen-kwargs '{"generation_mode": "diffusion"}' \
  --verbose

    Linear self-speculative generation

    Pass generation_mode="linear_speculative" (or alias "linear_spec") via --gen-kwargs. This automatically loads the linear_spec_lora adapter if available.

    mx_vlm.generate \
  --model nvidia/Nemotron-Labs-Diffusion-8B \
  --prompt "Write a short story about a clockmaker." \
  --max-tokens 256 \
  --temperature 0.0 \
  --gen-kwargs '{"generation_mode": "linear_speculative"}'

    Thinking mode

    Enable chat-template support for step-by-step reasoning using the --enable-thinking flag.

    mx_vlm.generate \
  --model nvidia/Nemotron-Labs-Diffusion-8B \
  --prompt "Solve this step by step: if a train travels 180 km in 2.5 hours, what is its average speed?" \
  --enable-thinking \
  --max-tokens 512 \
  --temperature 0.0

  7. Install Computer Use with MLX-VLM

    main

    To use the computer control features, you must have a Mac with Apple Silicon (M series), Python 3.10+, and pip installed. Follow these steps to set up the environment:

    1. Install the core mlx-vlm package.
    2. Clone the repository.
    3. Navigate to the computer_use directory.
    4. Install the required dependencies from requirements.txt.

  8. Convert SAM 3D Body Weights from PyTorch to MLX

    main

    To use the model in MLX, you must convert the original Meta PyTorch .ckpt and TorchScript JIT .pt files into MLX-compatible safetensors and config.json files.

    Use the convert_weights module via the CLI:

    python -m mlx_vlm.models.sam3d_body.convert_weights \
    --checkpoint /path/to/model.ckpt \
    --mhr-model /path/to/assets/mhr_model.pt \
    --output /path/to/sam3d-mlx-weights/

  9. Generate text, image, and video with MiniMax M3

    main

    MiniMax M3 supports multimodal generation via the mlx_vlm.generate CLI.

    Text Generation:

    mlx_vlm.generate \
  --model ~/MiniMax-M3-4bit \
  --prompt "Write a poem on LLMs" \
  --max-tokens 256

    Image Generation (Vision):

    mlx_vlm.generate \
  --model ~/MiniMax-M3-4bit \
  --image ./image.jpg \
  --prompt "Describe this image." \
  --max-tokens 256

    Video Generation: Use the --video flag and optionally specify --fps.

    mlx_vlm.generate \
  --model ~/MiniMax-M3-4bit \
  --video ./clip.mp4 \
  --fps 2.0 \
  --prompt "Summarize this video." \
  --max-tokens 256

  10. Use DeepSeek-OCR for Document to Markdown conversion

    main

    Convert document images into structured Markdown format using the <|grounding|> prefix. This is ideal for preserving document structure.

    Prompt Format:

    <image>
<|grounding|>Convert the document to markdown.
    mlx_vlm.generate \
    --model mlx-community/DeepSeek-OCR-bf16 \
    --image document.png \
    --prompt "<|grounding|>Convert the document to markdown." \
    --max-tokens 2000

  11. Use Gemma 4 Base models with Python API

    main

    For Base (pretrained) models (e.g., google/gemma-4-e4b), you do not need a chat template. Pass the raw prompt string directly to generate().

    from mlx_vlm import load, generate

# Base: raw prompt is fine
model, processor = load("google/gemma-4-e4b")
result = generate(model, processor, prompt="The capital of France is", max_tokens=32)

  12. Expand plain prompts locally using a text model

    main

    Ideogram 4 is optimized for JSON captions. You can use a local mlx-vlm text/VLM model (passed via --prompt-expansion-model in CLI or prompt_expansion_model in Python) to automatically expand a plain text prompt into a structured JSON caption. The resulting expanded prompt is returned in the revised_prompt field.

    python -m mlx_vlm generate_image \
  --model ideogram-ai/ideogram-4-fp8 \
  --prompt "A cinematic photo of a glass teapot on a rainy London cafe table" \
  --prompt-expansion-model mlx-community/gemma-4-12B-it-4bit \
  --size 1024x1024 \
  --seed 42 \
  --output outputs/ideogram4-expanded.png \
  --gen-kwargs '{"sampler_preset":"V4_DEFAULT_20"}'