Cannot use .from_pipe for NeuronStableDiffusionXLPipeline

[Suprhimp]

System Info

v0.0.24
ubuntu, inf2.2xlarge instance with dlami

Who can help?

@philschmid @JingyaHuang

Information

• The official example scripts
• My own modified scripts

Tasks

• An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
• My own task or dataset (give details below)

Reproduction (minimal, reproducible, runnable)

I want to use neuron pipeline for txt2img and img2img.

So, I made two pipeline for that. NeuronStableDiffusionXLImg2ImgPipeline and NeuronStableDiffusionXLPipeline

And I load the stablediffusion compiled model for each pipeline but I want to share memory so I tried to use .from_pipe()method.

Reproduction code:

from optimum.neuron import NeuronStableDiffusionXLImg2ImgPipeline,NeuronStableDiffusionXLPipeline

def model_fn(model_dir):

    # load local converted model into pipeline

    txt2img_pipeline = NeuronStableDiffusionXLPipeline.from_pretrained("my-sd-compiled-model-dir", device_ids=[0,1])

    pipeline = NeuronStableDiffusionXLImg2ImgPipeline.from_pipe(txt2img_pipeline)

    return pipeline,txt2img_pipeline

when I run model_fn
it gives me error.

error message:

Loading only U-Net into both Neuron Cores...
INFO:Neuron:Dynamic batching is disabled for torch_neuronx.DataParallel
cannot get type annotation for Parameter config of <class 'optimum.neuron.modeling_diffusion.NeuronStableDiffusionXLImg2ImgPipeline'>.
cannot get type annotation for Parameter data_parallel_mode of <class 'optimum.neuron.modeling_diffusion.NeuronStableDiffusionXLImg2ImgPipeline'>.
ERROR:app:Error loading model: 'NeuronStableDiffusionXLPipeline' object has no attribute 'components'
Traceback (most recent call last):
 ...
 ...
  File "/home/ubuntu/diffusers-ultimate-upscale/code/inference.py", line 52, in model_fn
    pipeline = NeuronStableDiffusionXLImg2ImgPipeline.from_pipe(txt2img_pipeline)
  File "/home/ubuntu/.local/lib/python3.10/site-packages/diffusers/pipelines/pipeline_utils.py", line 1800, in from_pipe
    for name, component in pipeline.components.items():
  File "/home/ubuntu/.local/lib/python3.10/site-packages/diffusers/configuration_utils.py", line 143, in __getattr__
    raise AttributeError(f"'{type(self).__name__}' object has no attribute '{name}'")
AttributeError: 'NeuronStableDiffusionXLPipeline' object has no attribute 'components'

Expected behavior

can use two pipeline with single sd compiled model.

Maby I can use img2img pipeline with denoising 100%. But I think this is not perfect solution of this...(maby not )

[JingyaHuang]

Hi @Suprhimp, thanks for bringing this. I did not consider this method while overwriting the pipelines, so there is no components attributes in the pipelines, and thus the error was raised.

We shall add the components property to the NeuronStableDiffusionPipeline to enable this method:
https://github.com/huggingface/diffusers/blob/28f9d84549c0b1d24ef00d69a4c723f3a11cffb6/src/diffusers/pipelines/pipeline_utils.py#L1525-L1559

Are you interested in opening a PR?

[Suprhimp]

Yes im interested. But actually I've been tried to add that property in NeuronStableDiffusionPipeline. But I failed.
I'll try one more time.
But still I can't understand why NeuronStableDiffusionPipeline doesn't have components property even diffusers pipeline include components property. I thought NeuronStableDiffusionPipeline is child of DiffuserPipeline isn't it?

Is there any hint to add components..? Or Does it easily solved when I add components property in NeuronStableDiffusionPipeline? I think I have to change higher hierarchy but the higher hierachy is in diffusers..

[Suprhimp]

I used diffusers v0.30.3

I logged all property in the pipeline with this code

from optimum.neuron import NeuronStableDiffusionXLPipeline,NeuronStableDiffusionXLImg2ImgPipeline

pipe = NeuronStableDiffusionXLPipeline.from_pretrained("/home/ubuntu/optimum-compile/Phoito_SD_4.0_Lightning_compel_components")
print(dir(pipe))

and it gives this output

['__abstractmethods__', '__call__', '__class__', '__delattr__', '__dict__', '__dir__', '__doc__', '__eq__', '__format__', '__ge__', '__getattr__', '__getattribute__', '__gt__', '__hash__', '__init__', '__init_subclass__', '__le__', '__lt__', '__module__', '__ne__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__sizeof__', '__slots__', '__str__', '__subclasshook__', '__weakref__', '_abc_impl', '_attributes_init', '_best_guess_weight_name', '_callback_tensor_inputs', '_check_text_inv_inputs', '_deprecated_kwargs', '_dict_from_json_file', '_exclude_from_cpu_offload', '_execution_device', '_export', '_extend_tokens_and_embeddings', '_fetch_state_dict', '_from_pretrained', '_from_transformers', '_get_add_time_ids', '_get_init_keys', '_get_signature_keys', '_get_signature_types', '_internal_dict', '_is_onnx', '_load_config', '_load_connected_pipes', '_lora_loadable_modules', '_maybe_convert_prompt', '_neuron_config_init', '_optional_components', '_optionally_disable_offloading', '_pad_to_compiled_shape', '_path_tempdirectory_instance', '_raise_if_invalid_padding', '_retrieve_tokens_and_embeddings', '_save_config', '_save_pretrained', '_upload_folder', '_validate_static_shape', 'auto_model_class', 'base_model_prefix', 'check_inputs', 'check_num_images_per_prompt', 'clip_skip', '**components**', 'config', 'config_class', 'config_name', 'configs', 'cross_attention_kwargs', 'data_parallel_mode', 'delete_adapters', 'denoising_end', 'device', 'disable_attention_slicing', 'disable_freeu', 'disable_lora', 'disable_vae_slicing', 'disable_vae_tiling', 'disable_xformers_memory_efficient_attention', 'do_classifier_free_guidance', 'download', 'dtype', 'dynamic_batch_size', 'enable_attention_slicing', 'enable_freeu', 'enable_lora', 'enable_model_cpu_offload', 'enable_sequential_cpu_offload', 'enable_vae_slicing', 'enable_vae_tiling', 'enable_xformers_memory_efficient_attention', 'encode_image', 'encode_prompt', 'extract_init_dict', 'feature_extractor', 'forward', 'from_config', 'from_pipe', 'from_pretrained', 'from_single_file', 'fuse_lora', 'fuse_qkv_projections', 'get_active_adapters', 'get_config_dict', 'get_guidance_scale_embedding', 'get_input_static_shapes', 'get_list_adapters', 'git_config_username_and_email', 'guidance_rescale', 'guidance_scale', 'has_compatibles', 'hf_device_map', 'ignore_for_config', 'image_processor', 'interrupt', 'is_lcm', 'is_weights_neff_separated', 'library_name', 'load_config', 'load_ip_adapter', 'load_lora_into_text_encoder', 'load_lora_into_unet', 'load_lora_weights', 'load_model', 'load_textual_inversion', 'load_tf_weights', 'lora_scale', 'lora_state_dict', 'maybe_convert_prompt', 'maybe_free_model_hooks', 'model_and_config_save_paths', 'model_cpu_offload_seq', 'model_save_dir', 'model_type', 'name_or_path', 'neuron_configs', 'neuron_padding_manager', 'num_fused_loras', 'num_images_per_prompt', 'num_timesteps', 'numpy_to_pil', 'pack_weights', 'prepare_extra_step_kwargs', 'prepare_ip_adapter_image_embeds', 'prepare_latents', 'preprocessors', 'progress_bar', 'push_to_hub', 'register_modules', 'register_to_config', 'remove_all_hooks', 'remove_padding', 'replace_weights', 'reset_device_map', 'run_safety_checker', 'safety_checker', 'save_config', 'save_lora_weights', 'save_pretrained', 'scheduler', 'set_adapters', 'set_attention_slice', 'set_default_dp_mode', 'set_ip_adapter_scale', 'set_lora_device', 'set_progress_bar_config', 'set_use_memory_efficient_attention_xformers', 'sub_component_config_name', 'text_encoder', 'text_encoder_2', 'text_encoder_name', 'to', 'to_json_file', 'to_json_string', 'tokenizer', 'tokenizer_2', 'unet', 'unet_name', 'unfuse_lora', 'unfuse_qkv_projections', 'unload_ip_adapter', 'unload_lora_weights', 'unload_textual_inversion', 'upcast_vae', 'vae_decoder', 'vae_encoder', 'vae_scale_factor', 'watermark', 'write_lora_layers']

As you can see, there is components property already.

and still it has error on from_pipe

cannot get type annotation for Parameter config of <class 'optimum.neuron.modeling_diffusion.NeuronStableDiffusionXLImg2ImgPipeline'>.
Traceback (most recent call last):
  File "test.py", line 8, in <module>
    pipeline = NeuronStableDiffusionXLImg2ImgPipeline.from_pipe(pipe)
  File "/opt/aws_neuron_venv_pytorch/lib/python3.8/site-packages/diffusers/pipelines/pipeline_utils.py", line 1800, in from_pipe
    for name, component in pipeline.components.items():
  File "/opt/aws_neuron_venv_pytorch/lib/python3.8/site-packages/diffusers/configuration_utils.py", line 143, in __getattr__
    raise AttributeError(f"'{type(self).__name__}' object has no attribute '{name}'")
AttributeError: 'NeuronStableDiffusionXLPipeline' object has no attribute 'components'

PS. When I didn't use latest diffusers, NeuronStableDiffusionXLImg2ImgPipeline has no from_pipe method.

[JingyaHuang]

Yes im interested. But actually I've been tried to add that property in NeuronStableDiffusionPipeline. But I failed. I'll try one more time. But still I can't understand why NeuronStableDiffusionPipeline doesn't have components property even diffusers pipeline include components property. I thought NeuronStableDiffusionPipeline is child of DiffuserPipeline isn't it?

Is there any hint to add components..? Or Does it easily solved when I add components property in NeuronStableDiffusionPipeline? I think I have to change higher hierarchy but the higher hierachy is in diffusers..

Hi @Suprhimp, not exactly, so far in optimum neuron NeuronStableDiffusionPipeline inherits NeuronStableDiffusionXLPipelineBase and NeuronStableDiffusionXLPipelineMixin. NeuronStableDiffusionXLPipelineBase can be seen as an equivalent of DiffusionPipeline pipeline in diffusers but there are some methods and attributes not implemented for Neuron, among which the components property and the .from_pipe() method.

The initial concern was if we just inherit from diffusers without overwriting, the instability could quickly leads to breaking changes in Optimum Neuron. But as diffusers stabilize and Optimum Neuron now pin diffusers version, I think it's time to implement neuron pipelines more dynamically, reuse what's in diffusers and override only when it's necessary.

I opened a PR which address this, and the from_pipe method shall be supported: #711.

[Suprhimp]

Thank you. I'd like to expect it to have the scalability of the diffuser of the optimum-neuron.
(Maby like ip-adapter... I leave other issue in this. )