SD.Next Documentation

SD.Next is an all-in-one WebUI for AI image and video generation. Key features include:

• Automatic model download: Select reference models to download them automatically, or add your own for auto-detection.
• SDNQ: A quantization engine providing up to 4x VRAM reduction.
• Balanced Offload: Dynamically balances CPU and GPU memory for large models on limited hardware.
• Caption and Enhance: Built-in support for 25+ LLM and VLM models, OpenCLiP, and Taggers (WaifuDiffusion/DeepDanbooru).
• Image Processing: Includes a color-grading suite of tools.
• Multi-platform: Optimized for various hardware via platform-specific auto-detection during installation.

To run MiDaS in a containerized environment, ensure you have Docker and the NVIDIA Docker runtime installed.

1. Build the image: docker build -t midas .

2. Run inference (mounts input, output, and weights directories): docker run --rm --gpus all -v $PWD/input:/opt/MiDaS/input -v $PWD/output:/opt/MiDaS/output -v $PWD/weights:/opt/MiDaS/weights midas

To load a custom pipeline in diffusers, pass the custom_pipeline argument to the DiffusionPipeline.from_pretrained method. The value should correspond to one of the files located in diffusers/examples/community.

pipe = DiffusionPipeline.from_pretrained(
    "Alpha-VLLM/Lumina-DiMOO",
    vqvae=vqvae,
    tokenizer=tokenizer,
    torch_dtype=torch.bfloat16,
    custom_pipeline="lumina_dimoo",
)

To use MiDaS for monocular depth estimation, follow these steps to prepare your environment and weights:

1. Download Weights: Pick one or more models and download the corresponding .pt or .xml/.bin files into a weights folder.
2. Install Dependencies: Use Conda to create and activate the environment.

Optional Steps:

• For Next-ViT models: Add the submodule https://github.com/isl-org/Next-ViT to midas/external/next_vit.
• For OpenVINO models: Install via pip install openvino.

conda env create -f environment.yaml
conda activate midas-py310

To install and run SD.Next quickly, clone the repository and execute the appropriate startup script for your operating system. The built-in installer handles dependency management and automatic updates.

git clone https://github.com/vladmandic/sdnext
cd sdnext
./webui.sh # Linux/Mac
webui.bat  # Windows
webui.ps1  # PowerShell

The api-grid.py script is a CLI tool that automates the generation of image grids by interacting with the SD.Next API. It reads parameter variations from two text files (representing the X and Y axes) and generates a grid of images, optionally including labels and saving the results to a specified folder.

To use it, provide files containing key:value pairs. For example, a line like prompt:a cat will append that prompt to the generation request, while lora:my_lora will append <lora:my_lora> to the prompt. Other keys will attempt to set corresponding attributes in the API request (e.g., steps:30).

After running the benchmark, you can apply the recommended settings in the SD.Next WebUI under Compute Settings -> SDNQ Attention.

Key configuration options include:

• sdnq_attention_matmul_type: Controls the quantization type for attention MatMuls (e.g., enabled resolves to int8, float8_e4m3fn for FP8 support).
• sdnq_attention_pv_matmul_type: Controls the quantization type for PV MatMuls.
• sdnq_attention_smooth_k: Enables/disables smooth K buybacks to reduce error.
• sdnq_attention_use_hadamard: Enables/disables Hadamard buybacks.
• sdnq_attention_hadamard_group_size: Sets the group size for Hadamard (values above head dim are clamped; non-pow2 values floor to the nearest power of 2).

Important: Changes to SDNQ attention SDP overrides require a restart of the application to take effect.

Use the sdnq-attention-benchmark.py script to benchmark attention mechanisms and dequantization settings for SD.Next. The tool evaluates performance (latency) and accuracy (weight error) across various configurations to provide recommended settings for the WebUI.

Key Benchmark Sections:

• attention: Benchmarks attention kernels and shapes (e.g., wan22, ltx2, krea2).
• dequant: Benchmarks weight dequantization, including MatMul backends, group sizes, SVD ranks, and Hadamard groups.
• block: Benchmarks block-level configurations.
• conv: Benchmarks convolutional layer quantization.

Note on VRAM: Benchmarks require significant free VRAM. The script will skip sections if insufficient VRAM is detected (e.g., ~2GB for dequant, ~3GB for blocks, and up to 12GB for certain attention shapes like wan22-cfg).