DOCSEARCHbyAlgolia
Submit Docs

ControlNetPlus (ControlNet++)

repository·main·Indexed 24 days ago

https://github.com/xinsir6/controlnetplus

An all-in-one architecture for SDXL that supports over 10 control types within a single network. It enables high-fidelity, multi-condition image generation and editing, including tasks like inpainting, outpainting, and super-resolution. Key features include a unified architecture for combining multiple conditions (e.g., Openpose + Canny) without increasing computation, bucket training for high-resolution support, and compatibility with other SDXL and LoRA models.

Tokens
3.7K
Snippets
5
Records
18
Agent score
30%

What's inside ControlNetPlus

  1. Overview of ControlNet++

    main

    ControlNet++ is an all-in-one architecture designed for SDXL that supports over 10 different control types in conditional text-to-image generation. It is optimized for both high-resolution image generation and detailed image editing.

    Key features include:

    • Unified Architecture: Supports multiple image conditions using the same network parameters without increasing computation.
    • High Resolution: Uses bucket training to support any aspect ratio and high-quality outputs.
    • Multi-Condition Support: Allows combining different control types (e.g., Openpose + Canny) through learned condition fusion, requiring no extra hyperparameters or special prompt engineering.
    • Compatibility: Works with other open-source SDXL models (like BluePencilXL, CounterfeitXL) and LoRA models.

  2. What is ControlNet++?

    main

    ControlNet++ is an all-in-one ControlNet architecture designed for SDXL that supports over 10 different control types (e.g., Openpose, Depth, Canny, Lineart) within a single network.

    Key features include:

    • Multi-Condition Support: Unlike standard ControlNets, it can process multiple image conditions simultaneously (e.g., Openpose + Canny) without increasing computational overhead or requiring complex hyperparameter tuning.
    • High Resolution: Uses bucket training (similar to NovelAI) to support any aspect ratio and high-resolution image generation.
    • Prompt Following: Utilizes re-described prompts (similar to DALL-E 3) via CogVLM to improve adherence to detailed descriptions.
    • Compatibility: Fully compatible with other SDXL models (e.g., BluePencilXL, CounterfeitXL) and LoRA models.

  3. Advanced Editing Features in Promax Model

    main

    The Promax version of ControlNet++ includes specialized modules for advanced image manipulation:

    • Tile Deblur: For removing blur from images.
    • Tile Variation: For introducing controlled variations to a tile.
    • Tile Super Resolution: For upscaling images (e.g., from 1M to 9M resolution).
    • Image Inpainting: For filling in or modifying specific parts of an image.
    • Image Outpainting: For extending the boundaries of an image.

  4. Best practices for multi-condition generation

    main

    ControlNet++ allows combining multiple control types for more precise image editing. Recommended combinations include:

    • Openpose + Canny: Use the pose skeleton to control body position and Canny (fine lines) to define hands and feet details to prevent deformities.
    • Openpose + Depth: Use the depth map for background information and the pose skeleton for foreground character control.
    • Openpose + Scribble: Use Scribble for loose, non-strict outlines to allow more creative freedom during initial generation, then use fine lines (like Canny) to edit details.
    • Openpose + Segment: Combine pose control with segmentation maps for precise object/area control.

  5. How ControlNet++ Architecture Works

    main

    ControlNet++ introduces several architectural improvements to support unified single and multi-condition generation:

    • Control Encoder: Uses a control type id (e.g., (1, 0, 0, 0, 0, 0) for openpose) converted into sinusoidal positional embeddings. These are projected to match the time embedding dimension and added to it, helping the network distinguish between different control types.
    • Condition Transformer: Enables multiple control inputs by sharing a single condition encoder across all types. It uses a transformer layer to exchange information between the original image and condition images, predicting a condition bias rather than using the transformer output directly (similar to a ResNet residual connection).
    • Modified Condition Encoder: Uses a "fat" encoder with increased convolution channels to provide higher representation ability, which is necessary since the same encoder is shared across many different condition types.
    • Unified Training Strategy: The model is trained in a single stage to ensure that single-condition optimization and multi-condition fusion converge simultaneously, improving robustness and fusion smoothness.

  6. Adjusting condition scales in Condition Transformer

    main

    By default, each input condition has a scale of 1.0, which is designed for natural fusion during multi-condition training.

    If you need to increase the influence of a specific input condition, you can adjust the condition scale within the Condition Transformer module. In this module, input conditions are added to the source image features along with a bias prediction. Multiplying the condition by a specific scale will significantly increase its impact, though it may lead to unpredictable results.

  9. Optimize Openpose performance

    main

    To achieve optimal performance when using the Openpose model, you must replace the draw_pose function in the controlnet_aux package.

    Note for ComfyUI users: ComfyUI uses its own version of the controlnet_aux package; ensure the replacement is applied correctly within that environment. Detailed instructions can be found in the project's inference scripts.

  10. Optimize OpenPose Preprocessing

    main

    To achieve the best performance for openpose control, you must replace the default draw_bodypose function in the controlnet_aux package's util.py with a custom implementation. This custom function handles canvas scaling and limb drawing more effectively for various image resolutions.

    def draw_bodypose(canvas: np.ndarray, keypoints: List[Keypoint]) -> np.ndarray:
    """\n    Draw keypoints and limbs representing body pose on a given canvas.\n\n    Args:\n        canvas (np.ndarray): A 3D numpy array representing the canvas (image) on which to draw the body pose.\n        keypoints (List[Keypoint]): A list of Keypoint objects representing the body keypoints to be drawn.\n\n    Returns:\n        np.ndarray: A 3D numpy array representing the modified canvas with the drawn body pose.\n\n    Note:\n        The function expects the x and y coordinates of the keypoints to be normalized between 0 and 1.\n    """
    H, W, C = canvas.shape

    
    if max(W, H) < 500:
        ratio = 1.0
    elif max(W, H) >= 500 and max(W, H) < 1000:
        ratio = 2.0
    elif max(W, H) >= 1000 and max(W, H) < 2000:
        ratio = 3.0
    elif max(W, H) >= 2000 and max(W, H) < 3000:
        ratio = 4.0
    elif max(W, H) >= 3000 and max(W, H) < 4000:
        ratio = 5.0
    else:
        ratio = 7.0

    stickwidth = 4

    limbSeq = [
        [2, 3], [2, 6], [3, 4], [4, 5], 
        [6, 7], [7, 8], [2, 9], [9, 10], 
        [10, 11], [2, 12], [12, 13], [13, 14], 
        [2, 1], [1, 15], [15, 17], [1, 16], 
        [16, 18],
    ]

    colors = [[255, 0, 0], [255, 85, 0], [255, 170, 0], [255, 255, 0], [170, 255, 0], [85, 255, 0], [0, 255, 0], \
              [0, 255, 85], [0, 255, 170], [0, 255, 255], [0, 170, 255], [0, 85, 255], [0, 0, 255], [85, 0, 255], \
              [170, 0, 255], [255, 0, 255], [0, 170, 255], [255, 0, 85]]

    for (k1_index, k2_index), color in zip(limbSeq, colors):
        keypoint1 = keypoints[k1_index - 1]
        keypoint2 = keypoints[k2_index - 1]

        if keypoint1 is None or keypoint2 is None:
            continue

        Y = np.array([keypoint1.x, keypoint2.x]) * float(W)
        X = np.array([keypoint1.y, keypoint2.y]) * float(H)
        mX = np.mean(X)
        mY = np.mean(Y)
        length = ((X[0] - X[1]) ** 2 + (Y[0] - Y[1]) ** 2) ** 0.5
        angle = math.degrees(math.atan2(X[0] - X[1], Y[0] - Y[1]))
        polygon = cv2.ellipse2Poly((int(mY), int(mX)), (int(length / 2), int(stickwidth * ratio)), int(angle), 0, 360, 1)
        cv2.fillConvexPoly(canvas, polygon, [int(float(c) * 0.6) for c in color])

    for keypoint, color in zip(keypoints, colors):
        if keypoint is None:
            continue

        x, y = keypoint.x, keypoint.y
        x = int(x * W)
        y = int(y * H)
        cv2.circle(canvas, (int(x), int(y)), int(4 * ratio), color, thickness=-1)

    return canvas

  11. Download ControlNet++ Model Weights

    main

    Model weights for ControlNet++ can be downloaded from Hugging Face. The primary weight repository is:

    https://huggingface.co/xinsir/controlnet-union-sdxl-1.0

    For the latest model updates, follow https://huggingface.co/xinsir.

  12. Install ControlNetPlus

    main

    To set up the environment for ControlNetPlus, use Python >= 3.8. It is recommended to use a Conda virtual environment.

    Follow these steps:

    1. Create a new Conda environment named controlplus with Python 3.8.
    2. Activate the environment.
    3. Install the required dependencies using pip install -r requirements.txt.
    conda create -n controlplus python=3.8
conda activate controlplus
pip install -r requirements.txt