cross normalization

[nzhang258]

Thanks a lot for such an amazing work and here are some questions about training code.

1. Could you plz tell me which part of the training code in SVD-v2 is about cross Normalization?
2. if i have some attn params like ip-adapter, it's seem does not work well if I only update the "to_out" in unet. Did you meet this problem?

Looking forward to your reply~

[AlonzoLeeeooo]

Hi @nzhang258 ,

Maybe I can help you with this. In ControlNeXt-SVD-v2-Training/models/unet_spatio_temporal_condition_controlnext.py, line 456 to 464, you can see:

if idx == 0 and conditional_controls is not None:
                scale = conditional_controls['scale']
                conditional_controls = conditional_controls['output']
                mean_latents, std_latents = torch.mean(sample, dim=(1, 2, 3), keepdim=True), torch.std(sample, dim=(1, 2, 3), keepdim=True)
                mean_control, std_control = torch.mean(conditional_controls, dim=(1, 2, 3), keepdim=True), torch.std(conditional_controls, dim=(1, 2, 3), keepdim=True)
                conditional_controls = (conditional_controls - mean_control) * (std_latents / (std_control + 1e-5)) + mean_latents
                conditional_controls = F.adaptive_avg_pool2d(conditional_controls, sample.shape[-2:])
                #  0.2: This superparameter is used to adjust the control level: increasing this value will strengthen the control level.
                sample = sample + conditional_controls * scale * 0.2

This is how the cross normalization is computed. Hope this may help.

[jackyyang9]

It seems not like the formulation presented in paper eq (10)?