DOCSEARCHbyAlgolia
Submit Docs

comfyui-gguf

repository·main·Indexed 26 days ago

https://github.com/city96/comfyui-gguf

GGUF Quantization support for native ComfyUI models. This project provides tools to convert and quantize large model checkpoints, such as Flux, Hunyuan Video, and Wan 2.1, into GGUF format. It includes scripts for converting safetensors/ckpt models to FP16/BF16, instructions for compiling a patched version of llama.cpp for image model quantization, and a utility to fix 5D tensors for Hunyuan Video and Wan 2.1 models.

Tokens
848
Snippets
4
Records
4
Agent score
37%

What's inside comfyui-gguf

  1. Set up and compile custom llama.cpp for quantization

    main

    To support image models, you must use a patched version of llama.cpp. Follow these steps to prepare and compile the llama-quantize binary.

    1. Fix line endings: Run python fix_lines_ending.py to ensure the patch file is valid (converts CRLF to LF).
    2. Clone llama.cpp: git clone https://github.com/ggerganov/llama.cpp
    3. Apply Patch:
      cd llama.cpp
git checkout tags/b3962
git apply ..\lcpp.patch
    4. Compile:

    For Visual Studio 2019, Linux, etc.:

    mkdir build
cmake -B build
cmake --build build --config Debug -j10 --target llama-quantize
cd ..

    For Visual Studio 2022:

    First, configure with C++17 support:

    mkdir build
cmake -B build -DCMAKE_CXX_STANDARD=17 -DCMAKE_CXX_STANDARD_REQUIRED=ON -DCMAKE_CXX_FLAGS="-std=c++17"

    Then, edit llama.cpp\common\log.cpp to insert these lines after the first line:

    #include "log.h"

#define _SILENCE_CXX23_CHRONO_DEPRECATION_WARNING
#include <chrono>

    Finally, build:

    cmake --build build --config Debug -j10 --target llama-quantize
cd ..
    git clone https://github.com/ggerganov/llama.cpp
cd llama.cpp
git checkout tags/b3962
git apply ..\lcpp.patch

  2. Fix 5D tensors for Hunyuan Video and Wan 2.1

    main

    After converting and quantizing Hunyuan Video or Wan 2.1 models, you must run fix_5d_tensor.py to add back missing keys that were lost during the conversion/quantization process.

    Usage: Run the script with --src (the quantized model) and --dst (the final destination path).

    Note: The script saves a temporary fix_5d_tensors_[arch].safetensors file in the ComfyUI-GGUF/tools folder; you should delete this once finished.

    fix_5d_tensors.py --src E:\models\video\raw\wan2.1-t2v-1.3b-Q8_0.gguf --dst E:\models\video\wan2.1-t2v-1.3b-Q8_0.gguf

  3. Convert safetensors/ckpt models to FP16/BF16 GGUF

    main

    Use convert.py to convert initial model checkpoints (safetensors or ckpt) to FP16 or BF16 GGUF format.

    Requirements:

    • gguf>=0.13.0 must be installed.

    Important Notes:

    • Flux Models: Do not use the diffusers UNET format for Flux; it will fail because q/k/v keys are merged. Use the default/reference checkpoint key format. If your model is in diffusers format, load it in ComfyUI and save it using the built-in ModelSave node first.
    • Hunyuan Video / Wan 2.1: You will see a warning about 5D tensors. The script saves a non-functional model to disk first. You must quantize this file and then run fix_5d_tensor.py manually to restore the missing keys.
    python convert.py --src E:\models\unet\flux1-dev.safetensors

  4. Quantize GGUF models using llama-quantize

    main

    Use the compiled llama-quantize binary to quantize your FP16/BF16 GGUF model to a specific format (e.g., Q4_K_S).

    Warning:

    • Do not use this binary with LLMs.
    • Do not quantize SDXL, SD1, or other Conv2D heavy models directly. If necessary, extract the UNET model first.
    • For Hunyuan Video / Wan 2.1, you MUST run fix_5d_tensor.py after the quantization step is complete.
    lama.cpp\build\bin\Debug\llama-quantize.exe E:\models\unet\flux1-dev-BF16.gguf E:\models\unet\flux1-dev-Q4_K_S.gguf Q4_K_S