Parallelism of ONNX session objects with multiple threads

[carbotaniuman]

If I want to run multiple ONNX sessions of the same model with various objects, I can either create one session object and have multiple threads call it simultaneously, or create one session object per thread and have each thread with it's own. Asides from providers like DirectML that do not have parallelism, are there are benefits to one or the other?

[pranavsharma]

You should create one session and use it in multiple threads. ORT is designed for this. Creating multiple sessions for each thread is a huge waste of resources and we strongly recommend against it.

[Renekton]

I think you are right, the onnxruntime session is thread-safe, but I would like to know what the use cases of share allocator are. In the example code of unit testing, two sessions were created using the same resources.

[pranavsharma]

Each session uses its own arena based allocator instance which can lead to excessive memory usage. If you've multiple sessions (where each session corresponds to a different model) and want to share an allocator between them, you can use the CreateAndRegisterAllocator API. Obviously, this has its own disadvantages (for e.g. more lock contention between threads of different sessions to acquire memory), so please experiment before using in production.

[Renekton]

Thank you for your response.
May I ask, when using a shared allocator for sessions based on different models, which part of the memory area is being shared?
I haven't read the source code of Onnxruntime in detail yet.
Can the memory area of the model resources be shared, or only the part of memory area for allocation request parameters?

[pranavsharma]

What is it that you're trying to accomplish? What's your use case?

[Renekton]

I'm trying to use a shared allocator between sessions based on different models to reduce memory usage, but it seems that memory has not been optimized significantly.
Here is associated issue link : #16692 ，thanks.