-
Notifications
You must be signed in to change notification settings - Fork 60
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add Stable Diffusion ControlNet support #622
Conversation
The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update. |
what happen? I hope it will be merge it soon, aslo sdxl control model too :) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks for the pull-request. Looks good to me, but I realize I missed the input flattening from previous SD pipelines pull-requests: could you explain a little bit what happens and when ?
Also, perhaps consider raising an exception when someone tries to call the SDXL ControlNet pipeline that is present but not implemented.
optimum/neuron/pipelines/diffusers/pipeline_controlnet_sd_xl.py
Outdated
Show resolved
Hide resolved
If you push your pull-request back, consider cherry-picking this commit from my branch to fix the TGI docker build. |
@dacorvo for the tracing, the compiler only accepts tensors, but not a list or a tuple of tensors which could be the case in transformers. So we flatten (actually we create directly non-nested dummy) inputs during the tracing, and during the inference runtime, we need to flatten inputs generated by the preprocessor (or the output of another model in the pipe like the case of stable diffusion) before feeding it into the compiled model. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Left a few nits, otherwise lgtm from what I can understand.
Co-authored-by: Michael Benayoun <[email protected]>
Co-authored-by: Michael Benayoun <[email protected]>
Co-authored-by: Michael Benayoun <[email protected]>
What does this PR do?
Fixes #575
optimum-cli export neuron -m runwayml/stable-diffusion-v1-5 --task stable-diffusion --batch_size 1 --height 512 --width 512 --controlnet_ids lllyasviel/sd-controlnet-canny --num_images_per_prompt 1 sd_neuron_controlnet/
optimum-cli export neuron -m hf-internal-testing/tiny-stable-diffusion-torch --task stable-diffusion --batch_size 1 --height 64 --width 64 --controlnet_ids hf-internal-testing/tiny-controlnet --num_images_per_prompt 1 sd_neuron_tiny_controlnet/
Next Steps
Before submitting