How can OnnxRuntime load multi onnx files? #10121
Replies: 2 comments 1 reply
-
One InferenceSession per model if you want to load multiple onnx files. e.g. session1 = ort.InferenceSession('model1.onnx')
session2 = ort.InferenceSession('model2.onnx') However your issue seems to be the model is invalid. There is an Einsum node with inputs of different types. The ONNX spec requires that all inputs are the same type. https://github.com/onnx/onnx/blob/main/docs/Operators.md#einsum Note that the way the constraints work is that it can be any type from the type constraints list, but only one type for a given node. i.e. That would be an issue with the converter that created the model. |
Beta Was this translation helpful? Give feedback.
-
I had the same problem, but it wasn't a problem with loading multiple files. Instead, it was due to the arguments of torch.einsum having different types (float and int64). At this line https://github.com/huggingface/transformers/blob/master/src/transformers/models/gptj/modeling_gptj.py#L55, change:
to
Then re-run the onnx export. |
Beta Was this translation helpful? Give feedback.
-
Env Settings
Question
For example, the following files are onnx files for GPT-J-6B.
When onnxruntime loads the
model.onnx
, error occurs as follows:How can I load such multi onnx files by using onnxruntime?
This questions are relevant to the following issue:
huggingface/transformers#14836 (comment)
Beta Was this translation helpful? Give feedback.
All reactions