DOCSEARCHbyAlgolia
Submit Docs

mflux

repository·main·Indexed 25 days ago

https://github.com/filipstrand/mflux

MLX native implementations of state-of-the-art generative image models for local execution on macOS. mflux provides high-performance implementations of models from Hugging Face Diffusers and Transformers, supporting model families such as Z-Image, Krea 2, FLUX.2, Ideogram 4, and others. Key features include quantization, LoRA support and training, text-to-image and image-to-image generation, and a Python API for programmatic use.

Tokens
37.8K
Snippets
75
Records
186
Agent score
81%

What's inside mflux

  1. Overview of FIBO text-to-image model

    main

    FIBO is an 8B-parameter DiT-based, flow-matching model from Bria.ai. It is a JSON-native text-to-image model trained on long structured captions (up to 1,000+ words), which allows for precise, reproducible control over lighting, composition, color, and camera settings.

    Key technical components:

    • Text Encoder: SmolLM3-3B
    • Conditioning: DimFusion architecture for long-caption training
    • VAE: Wan 2.2
    • VLM-assisted prompting: Uses a fine-tuned Qwen3-VL to expand short user intents into structured JSON prompts.

    FIBO is designed for 'disentangled control', meaning you can tweak specific attributes (like camera angle) without causing 'prompt drift' in the rest of the scene.

  2. MFLUX Core Features

    main

    MFLUX provides several core capabilities across its models:

    General Features:

    • Quantization and local model loading.
    • LoRA support (multi-LoRA, scales, library lookup).
    • Metadata export/reuse and prompt file support.

    Model-Specific Capabilities:

    • Text-to-image and image-to-image generation.
    • LoRA finetuning.
    • In-context editing, multi-image editing, and virtual try-on.
    • ControlNet (Canny), depth conditioning, fill/inpainting, and Redux.
    • Upscaling (via SeedVR2 and Flux ControlNet).
    • Depth map extraction and FIBO prompt tooling.

  3. Use Qwen Image Edit for natural language image editing

    main

    Qwen Image Edit allows you to modify images using text instructions while maintaining original structure and context. It uses the Qwen/Qwen-Image-Edit-2509 model.

    Key capabilities include:

    • Natural Language Editing: Use descriptive text to request changes.
    • Structure Preservation: Maintain original poses and body positions by explicitly requesting it in the prompt.
    • Multi-Image Composition: Support for multiple input images for complex transformations (e.g., replacing an object in Image 1 with an object from Image 2).
    • LoRA Support: Works with LoRA adapters for specialized transformations like camera angles or styles.

    Important Considerations:

    • Model Size: Requires downloading Qwen/Qwen-Image-Edit-2509 weights (~58GB).
    • Quantization: Use caution with 6-bit quantization or below, as it can significantly degrade image quality.
    • Image Quality: Images may appear softer compared to Flux models.
    • Variation: Seed changes typically have minimal impact; vary the prompt to achieve different results.

  4. Krea 2 Model Requirements and Training

    main

    Model Size

    Krea 2 Turbo requires significant storage. The first run will download approximately 33 GB of weights (including turbo.safetensors, the text encoder, and the VAE). To save space, use -q 8 at inference or save a quantized copy using mflux-save.

    Training

    mflux-train does not currently support Krea 2. To use custom styles, train LoRAs on krea/Krea-2-Raw using external tooling, then apply them during inference with mflux-generate-krea2 using the --lora-paths flag.

  6. Important notes for Depth Pro

    main

    When using the Depth Pro model (Apple's monocular depth model implementation), keep the following in mind:

    • Automatic Download: The model requires an additional ~1.9GB download from Apple, which occurs automatically on the first use.
    • Quantization: While quantization is supported, output quality can vary significantly depending on the input image.

  7. Ideogram 4 JSON Caption Schema

    main

    For optimal typography and layout control, Ideogram 4 expects structured JSON captions. A complete caption typically includes:

    • high_level_description: A summary of the scene.
    • style_description: Details on aesthetics, lighting, medium, art_style, and color_palette.
    • compositional_deconstruction:
      • background: Description of the backdrop.
      • elements: An array of objects defining specific parts of the image. Each element includes:
        • type: (e.g., text, obj)
        • bbox: Bounding box coordinates [ymin, xmin, ymax, xmax]
        • text: (For type text) The actual string to render.
        • desc: A description of the element or the text style.
    {
  "high_level_description": "A white ceramic teapot on a simple studio table.",
  "style_description": {
    "aesthetics": "clean, calm, minimal",
    "lighting": "soft diffuse studio lighting",
    "photo": "eye-level, 50mm lens, shallow depth of field",
    "medium": "photograph",
    "color_palette": ["#FFFFFF", "#E5E0D8", "#2E2E2E"]
  },
  "compositional_deconstruction": {
    "background": "A neutral studio tabletop with a pale wall behind it.",
    "elements": [
      {
        "type": "obj",
        "bbox": [250, 320, 780, 690],
        "desc": "A glossy white ceramic teapot with a curved handle and short spout."
      }
    ]
  }
}

  8. Generate images directly from a JSON prompt file

    main

    To bypass the VLM expansion step and have full control over the prompt structure, you can provide a pre-existing JSON prompt file directly to the generation command. This is useful if you have manually edited the JSON or used other LLMs/VLMs to create the structure.

    CLI Usage: Use mflux-generate-fibo with the --prompt-file flag instead of --prompt.

    mflux-generate-fibo \
    --prompt-file animal_bakers.json \
    --width 1200 \
    --height 540 \
    --steps 50 \
    --guidance 4.0 \
    --seed 42 \
    --output animal_bakers.png

  9. Use FIBO Lite for fast generation

    main

    FIBO Lite is a distilled variant of the base model that uses CFG distillation and SCFM for faster, few-step generation. It is approximately 10x faster than the base model.

    When using FIBO Lite:

    • Use --steps 8 (or similar low step counts).
    • Set guidance=1.0.
    • No negative prompt is required.
    • Note: There is a slight quality tradeoff compared to the base FIBO model.
    mflux-generate-fibo \
  --model fibo-lite \
  --prompt "A tiny watercolor robot in a garden" \
  --steps 8 \
  --seed 42

  11. Use local model paths

    main

    To use a model stored locally instead of downloading from a repository, pass the local directory path to the --model argument. When using a local path, you must also specify the --base-model name (e.g., z-image-turbo) to ensure the correct configuration is applied.

    mflux-generate-z-image-turbo \
  --model "/Users/me/models/z-image-turbo" \
  --base-model z-image-turbo \
  --steps 9 \
  --prompt "Luxury food photograph"

  12. Fine-tune FLUX.2 models

    main

    Fine-tuning is supported for flux2-klein-base-4b and flux2-klein-base-9b models using the mflux-train command.

    Edit Fine-tuning (Image-conditioned)

    To perform edit-style training (image in + prompt $\rightarrow$ image out), prepare your data using paired *_out.* and *_in.* files along with *_in.txt prompt files. This mode is supported for Flux2 Klein base models.

    Configuration

    Training is controlled via a JSON configuration file. You must specify the model, data directory, steps, and lora_layers targets.

    {
  "model": "flux2-klein-base-9b",
  "data": "images/",
  "seed": 42,
  "steps": 40,
  "guidance": 1.0,
  "quantize": null,
  "low_ram": false,
  "max_resolution": 1024,
  "training_loop": { "num_epochs": 100, "batch_size": 1, "timestep_low": 25, "timestep_high": 40 },
  "optimizer": { "name": "AdamW", "learning_rate": 1e-4 },
  "checkpoint": { "output_path": "train", "save_frequency": 25 },
  "monitoring": {
    "plot_frequency": 1,
    "generate_image_frequency": 20
  },
  "lora_layers": {
    "targets": [
      { "module_path": "transformer_blocks.{block}.attn.to_q", "blocks": { "start": 0, "end": 5 }, "rank": 16 },
      { "module_path": "transformer_blocks.{block}.attn.to_k", "blocks": { "start": 0, "end": 5 }, "rank": 16 },
      { "module_path": "transformer_blocks.{block}.attn.to_v", "blocks": { "start": 0, "end": 5 }, "rank": 16 },
      { "module_path": "transformer_blocks.{block}.attn.to_out", "blocks": { "start": 0, "end": 5 }, "rank": 16 },
      { "module_path": "transformer_blocks.{block}.attn.add_q_proj", "blocks": { "start": 0, "end": 5 }, "rank": 16 },
      { "module_path": "transformer_blocks.{block}.attn.add_k_proj", "blocks": { "start": 0, "end": 5 }, "rank": 16 },
      { "module_path": "transformer_blocks.{block}.attn.add_v_proj", "blocks": { "start": 0, "end": 5 }, "rank": 16 },
      { "module_path": "transformer_blocks.{block}.attn.to_add_out", "blocks": { "start": 0, "end": 5 }, "rank": 16 },
      { "module_path": "transformer_blocks.{block}.ff.linear_in", "blocks": { "start": 0, "end": 5 }, "rank": 16 },
      { "module_path": "transformer_blocks.{block}.ff.linear_out", "blocks": { "start": 0, "end": 5 }, "rank": 16 },
      { "module_path": "transformer_blocks.{block}.ff_context.linear_in", "blocks": { "start": 0, "end": 5 }, "rank": 16 },
      { "module_path": "transformer_blocks.{block}.ff_context.linear_out", "blocks": { "start": 0, "end": 5 }, "rank": 16 },
      { "module_path": "single_transformer_blocks.{block}.attn.to_qkv_mlp_proj", "blocks": { "start": 0, "end": 20 }, "rank": 16 },
      { "module_path": "single_transformer_blocks.{block}.attn.to_out", "blocks": { "start": 0, "end": 20 }, "rank": 16 }
    ]
  }
}
    mflux-train --config /path/to/train.json