Performance penalties for using Language bindings?

[welcomemandeep]

I want to know what are the pragmatic penalties we would have to bear if we are not using the native onnxruntime and rather using Python or Java language bindings. Is any benchmarking data available for the same? Is the penalty in a few micro-seconds or greater? Are there other factors to consider to build an inference service with high throughput and low latency?

[Craigacp]

For Java, if you write your inputs to a direct java.nio.Buffer then there is a zero copy path through into the native ONNX Runtime library. If you use a non-direct java.nio.Buffer as an input then it does a copy when building the OnnxTensor, and if you pass in a Java multidimensional array then it does a pile of slow pointer chasing to copy the data into native memory. It will do a copy for the outputs back into the Java heap using either the java.nio.Buffer path or the multidimensional array path, though the former is faster. Other than that it should be pretty similar to using it natively as the Java code is a thin wrapper around the C API. I've not written any benchmarks to compare the different language bindings though.

[welcomemandeep]

Thanks, @Craigacp. That certainly helps. I was also interested to learn more about the Python bindings and usage in a certain fashion and their implication on performance.

[Craigacp]

I'll let someone from the ORT comment about python, I'm not that familiar with their internals as I maintain the Java bindings.

[hariharans29]

Apart from "language-induced" perf lags, there shouldn't be a whole lot of penalty for using the Python bindings. However, we don't have any benchmarking data available at this time.