What is the ordering of outputs for IOBinding.GetOutputValues (c++)?

[admayber]

We want to run inference with multiple dynamically-shaped outputs using the C++ api with IOBinding, but the documentation is not clear on how to retrieve the outputs. If we've bound multiple dynamically shaped outputs using the BindOutput with MemoryInfo signature on IOBinding, how will the outputs be ordered when we retrieve them using IOBinding.GetOutputValues()?

As a concrete example, take this snippet from the repo (snippet link):

  // Now replace binding of Y with an on device binding instead of pre-allocated memory.
  // This is when we can not allocate an OrtValue due to unknown dimensions
  {
    Ort::MemoryInfo info_cpu_dev("Cpu", OrtAllocatorType::OrtArenaAllocator, 0, OrtMemTypeDefault);
    binding.BindOutput("Y", info_cpu_dev);
    session.Run(Ort::RunOptions(), binding);
  }

  // Check the output value allocated based on the device binding.
  {
    std::vector<Ort::Value> output_values = binding.GetOutputValues();

In this situation, we are binding one output and it is retrieved using GetOutputValues(), which returns a vector<Ort::Value> with a single element.

My question is, what if we were to bind multiple outputs? How would the results be ordered in the return value of GetOutputValue()? For example:

  Ort::IoBinding binding(session);
  binding.BindInput("X", bound_x);
  binding.BindOutput("Y1", info_cpu_dev);
  binding.BindOutput("Y2", info_cpu_dev);

  session.Run(Ort::RunOptions(), binding);

  // two-element vector, how do the elements correspond to Y1 and Y2?
  std::vector<Ort::Value> output_values = binding.GetOutputValues();

[hariharans29]

Tagging @pranavsharma / @natke to help link any docs on this subject here.

By dynamic shaped outputs, do you know the shape of the outputs apriori or do you not know it? ORT should be able to handle both scenarios. This C++ shared lib test is what you will need, and it has sample code for both cases (where shape is known apriori and when shape is not known apriori)-

onnxruntime/onnxruntime/test/shared_lib/test_inference.cc

Line 1046 in 2f9b559

TEST(CApiTest, io_binding_cuda) {

.

Regarding multiplicity of the outputs, you would just use as many BindOutput() calls as the number of outputs you will be binding.

[admayber]

Thanks @hariharans29 - that doesn't quite address my question unfortunately, yes I am wondering about the scenario where we don't know the outputs a priori but then also adding the complication that there is more than one output of unknown size to be bound.

I've updated my question, I think looking through the examples more closely I may have actually found what I was looking for. Please let me know if the answer I've posted separately seems correct.

[admayber]

I realized that I had been overlooking something in the API that seems to be the answer - I'm assuming that binding.GetOutputNames() will return the names in the same order as the output vectors:

onnxruntime/onnxruntime/test/shared_lib/test_inference.cc

Line 1014 in 2f9b559

std::vector<std::string> output_names = binding.GetOutputNames();

[hariharans29]

Yes, that seems to be right. I checked the implementation. As you can see the ordering of output_names_ (which is returned by the internal method GetOutputNames() and ultimately by the C++ API GetOutputNames()) is consistent with outputs_ (which is returned by the internal method GetOutputs() and ultimately by the C++ API GetOutputValues()).

Anyway, tagging the @yuslepukhin who exposed the C++ interface for IOBinding if he wants to add any more info