SparseCtrl-RGB causes video interpolation to flash due to bad colors

[aihopper]

This affects both video interpolation and video prediction

Here is a minimal repro case that attempts to interpolate every other frame. Notice the flashing frames:

readme_00002.mp4

Repro steps

1. First I used Animatediff to generate a video of a running man: video

2. Then I take the generated video (which should be well understood by AnimateDiff), remove all the odd frames

3. Use AnimateDiff to generate the frames I removed by interpolating the present frames, here is the yaml file:

# 3-prediction
- adapter_lora_scale: 1.0
  adapter_lora_path: "models/Motion_Module/v3_sd15_adapter.ckpt"
  dreambooth_path:   ""

  inference_config: "configs/inference/inference-v3.yaml"
  motion_module:    "models/Motion_Module/v3_sd15_mm.ckpt"

  controlnet_config: "configs/inference/sparsectrl/latent_condition.yaml"
  controlnet_path:   "models/SparseCtrl/v3_sd15_sparsectrl_rgb.ckpt"
  
  H: 512
  W: 512
  seed:           45
  steps:          25
  guidance_scale: 8.5

  controlnet_image_indexs: [0,2,4,6,8,10,12,14]

  controlnet_images:
    - "running/readme_001.png"
    - "running/readme_003.png"
    - "running/readme_005.png"
    - "running/readme_007.png"
    - "running/readme_009.png"
    - "running/readme_011.png"
    - "running/readme_013.png"
    - "running/readme_015.png"
    
  prompt:
    - "man, full shot, running in a white suit, brown shoes, gray background, high quality, detailed"

  n_prompt:
    - "worst quality, low quality, letterboxed"

Notes:

• this same issue is present in ComfyUI, it was noticed there first (SparseCtrl-RGB, flashing in generated frames  Kosinkadink/ComfyUI-AnimateDiff-Evolved#476) and now I am able to repro the issue in this repo
• this failure is not caused by, as the paper mentions, an out-of-domain input (such as anime image animation), that is why we used a video that was generated by AnimateDiff itself

[aihopper]

The main differences seem to be along the edges, the generated image seem to have softer edges

Also histograms show notable differencies, the below image is the generated one (brighter/blurrier)

[aihopper]

The flashing is very noticeable in the outputs the repo generates using 3_1_sparsectrl_i2v.yaml. Here it tries to interpolate between 5 frames, so each 5 frames you get a flash.