DOCSEARCHbyAlgolia
Submit Docs

stable-diffusion.cpp Documentation

repository·master·Indexed 27 days ago

https://github.com/leejet/stable-diffusion.cpp

A lightweight, pure C/C++ implementation for diffusion model inference based on ggml. It supports a wide range of image models (SD 1.x/2.x/3.5, FLUX, Wan) and video models (Wan2.1/2.2, LTX-2.3, HunyuanVideo 1.5). Key features include ADetailer for image repair, AnimateDiff for motion, and support for multiple hardware backends including CPU, CUDA, Vulkan, Metal, OpenCL, and SYCL. It supports weight formats such as .ckpt, .safetensors, and .gguf.

Tokens
40K
Snippets
94
Records
211
Agent score
92%

What's inside stable-diffusion.cpp

  1. Supported Model Types and Features

    master

    stable-diffusion.cpp supports a wide range of diffusion models and features:

    Image Models

    • Standard SD: SD1.x, SD2.x, SD-Turbo, SDXL, SDXL-Turbo, SD3/SD3.5.
    • FLUX: FLUX.1-dev, FLUX.1-schnell, FLUX.2-dev, FLUX.2-klein.
    • Specialized: Lens, Chroma, Qwen Image, PiD, Krea2, Ideogram4, and more.
    • Editing: FLUX.1-Kontext-dev, Qwen Image Edit series, Mage-Flow-Edit.

    Video Models

    • Wan2.1/Wan2.2, LTX-2.3, HunyuanVideo 1.5, LingBot-Video.

    Key Features

    • Extensions: LoRA support, Control Net (SD 1.5), IP-Adapter (SD 1.5/SDXL), ADetailer, and PhotoMaker.
    • Optimization: Flash Attention, VAE tiling, and TAESD for efficient latent decoding.
    • Upscaling: ESRGAN support.
    • Sampling Methods: Euler A, Euler, Heun, DPM2, DPM++ 2M, DPM++ 2M v2, DPM++ 2S a, ER-SDE, and LCM.

  2. Download weights for Ovis-Image-7B

    master

    To use Ovis-Image-7B, you need to download the model weights in either safetensors or gguf format, a VAE, and the Ovis 2.5 text encoder weights.

    Model Weights (Ovis-Image-7B):

    VAE:

    Text Encoder (Ovis 2.5):

  4. Run LongCat Image with sd-cli

    master

    LongCat Image uses a LongCat diffusion transformer, the FLUX VAE, and Qwen2.5-VL as the LLM text encoder.

    Text Rendering Tip: LongCat uses quoted text for character-level text rendering. To render specific text, place the target text inside single quotes ('), double quotes ("), or Chinese quotes.

    .\bin\Release\sd-cli.exe --diffusion-model  ..\models\diffusion_models\LongCat-Image-Q4_K_M.gguf --vae ..\models\vae\ae.sft --llm ..\models\text_encoders\Qwen2.5-VL-7B-Instruct-Q8_0.gguf -p "a lovely cat holding a sign says 'longcat.cpp'" --cfg-scale 5.0 --sampling-method euler --flow-shift 3 -v --offload-to-cpu --diffusion-fa

  5. Convert SD1.x tiny U-Net models for sd.cpp

    master

    SD1.x models with tiny U-Nets require a specific conversion process because some tensors are stored in a non-contiguous manner. You must ensure tensors are made contiguous in Python before saving and then run the conversion script.

    1. Prepare the model: Use Python to load the model and call .contiguous() on the U-Net parameters.
    2. Convert: Use the convert_diffusers_to_original_stable_diffusion.py script to generate the final .safetensors file.

    Required Script: convert_diffusers_to_original_stable_diffusion.py

  7. Download PiD weights and dependencies

    master

    To use PiD, you need to download three components:

    1. PiD Diffusion Model: Download the .safetensors files from Comfy-Org/PixelDiT.
    2. Text Encoder (Gemma 2 2B): Download the .safetensors files from Comfy-Org/PixelDiT.
    3. Matching VAE: Download the VAE that matches your PiD checkpoint backbone from nvidia/PiD.

    VAE Format Mapping: You must specify the correct --vae-format flag based on the backbone used:

    • Flux / Z-Image PiD: Use Flux VAE and pass --vae-format flux
    • SD3 PiD: Use SD3 VAE and pass --vae-format sd3
    • Flux.2 PiD: Use Flux.2 VAE and pass --vae-format flux2
    • Qwen-Image PiD: Use Qwen-Image 2D VAE and pass --vae-format wan

  8. Use LoRA with stable-diffusion.cpp

    master

    You can apply LoRA weights to your generation by including them in the prompt using the syntax <lora:NAME:WEIGHT>.

    To use LoRA, you must ensure the weights are in a directory accessible to the application. You can specify this directory using the --lora-model-dir flag. If this flag is omitted, the application looks in the current working directory.

    Supported file extensions for LoRA weights include .safetensors and .ckpt.

    ./bin/sd-cli -m ../models/v1-5-pruned-emaonly.safetensors -p "a lovely cat<lora:marblesh:1>" --lora-model-dir ../models

  9. Scale with multiple RPC servers

    master

    You can connect to multiple RPC servers simultaneously by providing a comma-separated list of addresses to the --rpc-servers flag. The client maps these servers to sequential device IDs (e.g., RPC0, RPC1, etc.).

    Example Setup:

    • Server 1 (192.168.1.10) running two instances on ports 50052 and 50053.
    • Server 2 (192.168.1.11) running one instance on port 50052.

    Client Command:

    ./sd-cli --rpc-servers 192.168.1.10:50052,192.168.1.10:50053,192.168.1.11:50052

  10. Download Z-Image weights and components

    master

    To use Z-Image with stable-diffusion.cpp, you need to download the diffusion model, the VAE, and the LLM (text encoder). You can choose between Turbo and Base versions, and between safetensors or gguf formats.

    Z-Image-Turbo

    Z-Image (Base)

    Required Components

  11. Build the llama.cpp RPC server

    master

    The RPC server acts as the worker. When building, you must enable a hardware backend (e.g., GGML_VULKAN=ON, GGML_METAL=ON, or GGML_CUDA=ON) or it will default to CPU.

    CRITICAL: You must include -DGGML_MAX_NAME=128 in both CMAKE_C_FLAGS and CMAKE_CXX_FLAGS to ensure compatibility with the stable-diffusion.cpp client. Without this, data transfers will fail.

    Target the rpc-server build target specifically to speed up compilation.

    ### Linux / WSL (Vulkan)
```bash
mkdir build
cd build
cmake .. -DGGML_RPC=ON \
    -DGGML_VULKAN=ON \
    -DGGML_BUILD_SHARED_LIBS=OFF \
    -DLLAMA_CURL=OFF \
    -DCMAKE_C_FLAGS=-DGGML_MAX_NAME=128 \
    -DCMAKE_CXX_FLAGS=-DGGML_MAX_NAME=128
cmake --build . --config Release --target rpc-server -j $(nproc)

    macOS (Metal)

    mkdir build
cd build
cmake .. -DGGML_RPC=ON \
    -DGGML_METAL=ON \
    -DGGML_BUILD_SHARED_LIBS=OFF \
    -DLLAMA_CURL=OFF \
    -DCMAKE_C_FLAGS=-DGGML_MAX_NAME=128 \
    -DCMAKE_CXX_FLAGS=-DGGML_MAX_NAME=128
cmake --build . --config Release --target rpc-server

    Windows (Visual Studio 2022, Vulkan)

    mkdir build
cd build
cmake .. -G "Visual Studio 17 2022" -A x64 `
    -DGGML_RPC=ON `
    -DGGML_VULKAN=ON `
    -DGGML_BUILD_SHARED_LIBS=OFF `
    -DLLAMA_CURL=OFF `
    -DCMAKE_C_FLAGS=-DGGML_MAX_NAME=128 `
    -DCMAKE_CXX_FLAGS=-DGGML_MAX_NAME=128
cmake --build . --config Release --target rpc-server

  12. Upscale generated images using ESRGAN

    master

    You can improve the resolution and clarity of generated images by using an ESRGAN model during the generation process. Use the --upscale-model PATH parameter to specify the path to your ESRGAN model file (e.g., RealESRGAN_x4plus_anime_6B.pth).

    sd-cli -m ../models/v1-5-pruned-emaonly.safetensors -p "a lovely cat" --upscale-model ../models/RealESRGAN_x4plus_anime_6B.pth