[Bug]: Error: Unexpected type of example_input

[kolabearafk]

OpenVINO Version

openvino 2023.2.0

Operating System

Ubuntu 20.04 (LTS)

Device used for inference

CPU

Framework

PyTorch

Model used

StableDiffusion UNet - https://github.com/huggingface/diffusers/blob/645a62bf3bcba47f97bf2a36a8689f40bd6b04fd/src/diffusers/models/unet_2d_condition.py#L843

Issue description

Describe the bug

Can't use dictionary type for example_input when trying to convert pytorch model

Expected behavior

Not giving this - "Error: Unexpected type of example_input" and successfully convert the pytorch model to openvino

Screenshots

Additional context

I'm trying to convert the UNet model from diffusers library's StableDiffusionPipeline but with an additional parameter called "added_cond_kwargs" that is supposed to be Dict[str, torch.Tensor]. But openvino's ov.convert_model(unet, example_input) function is giving the error "Error: Unexpected type of example_input".

It seems openvino's pytorch_frontend_utils.py script raises an error if the type of example_input doesn't match the expected type (see

openvino/tools/mo/openvino/tools/mo/moc_frontend/pytorch_frontend_utils.py

Line 158 in 9dad09a

raise Error("Unexpected type of example_input. Supported types torch.Tensor, np.array or ov.Tensor. "

) but the diffusers library's UNet model requires the added_cond_kwargs parameter to be a Dict (see https://github.com/huggingface/diffusers/blob/645a62bf3bcba47f97bf2a36a8689f40bd6b04fd/src/diffusers/models/unet_2d_condition.py#L843).

Please help! Thank you.

Step-by-step reproduction

No response

Relevant log output

See attached screenshot above.

Issue submission checklist

• I'm reporting an issue. It's not a question.
• I checked the problem with the documentation, FAQ, open issues, Stack Overflow, etc., and have not found a solution.
• There is reproducer code and related data files such as images, videos, models, etc.

[mvafin]

@kolabearafk What exactly you provide as example_input argument? The dictionary case is supported if you pass dictionary with inputs it should work. It might not work if you pass dictionary in tuple, for example.

[kolabearafk]

Hi @mvafin,

Thanks for the reply!

I'm trying to put a dictionary inside the example_input dictionary. Please see below for more information:

example_input = {
    "sample": torch.rand((2, 4, 64, 64)), 
    "timestep": torch.tensor(1), 
    "encoder_hidden_states": torch.randn((2, 77, 768)), 
    "added_cond_kwargs": {"image_embeds": torch.rand((2, 1280))}

I was able to get it working after following @eaidova's advice by trying with openvino-nightly instead of openvino 2023.2.0. It was able to convert the unet with openvino-nightly. But I am ran into a different error when compiling the model when it says "CPU Plugin: Input image format dynamic is not supported yet..." Please see attached screenshot for the error message.

[mvafin]

@kolabearafk This happens because some of model inputs have unspecified (dynamic) data type. To solve this problem you need to specify data type of the input using something like this:

import openvino as ov

model = ov.convert_model(<...>)  # your ov model
params = model.get_parameters()
params[0].set_element_type(ov.Type.f32)

[kolabearafk]

import openvino as ov

model = ov.convert_model(<...>) # your ov model
params = model.get_parameters()
params[0].set_element_type(ov.Type.f32)

That doesn't seem to work for some reason and gives the same "CPU Plugin: Input image format dynamic is not supported yet..." error.

[mvafin]

import openvino as ov
model = ov.convert_model(<...>) # your ov model
params = model.get_parameters()
params[0].set_element_type(ov.Type.f32)

That doesn't seem to work for some reason and gives the same "CPU Plugin: Input image format dynamic is not supported yet..." error.

Could you do print(ov_model) where ov_model is the object you getting from convert_model and paste it here? The model most likely has more then 1 input.

[andrei-kochin]

Closing due to no activity