[New Models] ControlNet++ [SD1.5]

[huchenlei]

ControlNet++ offers better alignment of output against input condition by replacing the latent space loss function with pixel space cross entropy loss between input control condition and control condition extracted from diffusion output during training.

Model comparision

Input condition

control_v11p_sd15_canny

controlnet++_canny_sd15

You can observe that there is extra hair not in the input condition generated by official ControlNet model, but the extra hair is not generated by the ControlNet++ model.

Generation infotext:

1girl, absurdres, highres, (masterpiece:1.2), (best quality, highest quality),
Negative prompt: (worst quality:2), (low quality:2), (normal quality:2), lowres, normal quality, ((monochrome)), ((grayscale)), skin spots, acnes, skin blemishes, age spot, backlight,(ugly:1.331), (duplicate:1.331), (morbid:1.21), (mutilated:1.21), (tranny:1.331), mutated hands, (poorly drawn hands:1.331), blurry, (bad anatomy:1.21), (bad proportions:1.331), extra limbs, (more than 2 nipples:1.331), (missing arms:1.331), (extra legs:1.331), (fused fingers:1.61051), (too many fingers:1.61051), (unclear eyes:1.331), bad hands, missing fingers, extra digit, (futa:1.1), bad body, pubic hair, glans, easynegative,more than 2 tits, ng_deepnegative_v1_75t,(big fee:1),more than 2 feet,incorrect feet, easynegative,
Steps: 20, Sampler: DPM++ 2M, Schedule type: Karras, CFG scale: 7, Seed: 1220524552, Size: 512x512, Model hash: b42b09ff12, Model: cetusMix_v4, Clip skip: 2, ControlNet 0: "Module: canny, Model: control_v11p_sd15_canny [d14c016b], Weight: 1, Resize Mode: Crop and Resize, Low Vram: False, Processor Res: 512, Threshold A: 100, Threshold B: 200, Guidance Start: 0, Guidance End: 1, Pixel Perfect: False, Control Mode: Balanced, Hr Option: Both, Save Detected Map: True", TI hashes: "easynegative: c74b4e810b03, ng_deepnegative_v1_75t: 54e7e4826d53", Version: v1.9.0-46-gd1998d74

More comparison examples from ControlNet++ demo site:

Model download

Currently there are 5 models available:

• canny
• lineart
• softedge
• seg
• depth

You can download models from following sources:

• LARGE (fp32) - https://huggingface.co/spaces/limingcv/ControlNet-Plus-Plus/tree/main/checkpoints
• MEDIUM (fp16) - https://huggingface.co/huchenlei/ControlNet_plus_plus_collection_fp16/tree/main

[CCpt5]

Asked about SDXL models on the official repo and seem possible they just need the resources to train:

[crapthings]

hello is this work with diffusers?

[huchenlei]

You need to convert state_dict keys to diffusers format to the model to work for diffusers.

[sabinakaminska95]

Could you show how? in diffiusers converts scripts are problem cause are too big
safetensors_rust.SafetensorError: Error while deserializing header: HeaderTooLarge

[dadadies]

In the example given ControlNet++ is more blurry compared to the official ControlNet. Is that part of the design style?