Why does memory still increase when using a share allocator?

[Renekton]

Hello, I run the code with share allocator, and notice that the memory will increase. May I ask which part of the memory is shared by the share allocator?

Test Examle Code ：
`
//TEST

const OrtApi &api = Ort::GetApi();
Ort::Env env(ORT_LOGGING_LEVEL_WARNING, "Default");

Ort::AllocatorWithDefaultOptions allocator;
const OrtMemoryInfo *mem_info = allocator.GetInfo();

OrtArenaCfg* arena_cfg = nullptr;
(api.CreateArenaCfg(0, -1, -1, -1, &arena_cfg) == nullptr);
std::unique_ptr<OrtArenaCfg, decltype(api.ReleaseArenaCfg)> rel_arena_cfg(arena_cfg, api.ReleaseArenaCfg);

env.CreateAndRegisterAllocator(mem_info, arena_cfg);

Ort::SessionOptions session_options;
session_options.SetIntraOpNumThreads(1);
session_options.SetInterOpNumThreads(1);
session_options.AddConfigEntry(kOrtSessionOptionsConfigUseEnvAllocators, "1");

Ort::Session session_shared_0(env, "model-optimized-quantized.ort", session_options);
//Ort::Session session_shared_1(env, "model-optimized-quantized.ort", session_options);
//Ort::Session session_shared_3(env, "model-optimized-quantized.ort", session_options);`

The command top shows the memory :

1. one session, 18.2m
   VmHWM: 21536 kB
   VmRSS: 18624 kB
   RssAnon: 12688 kB
   RssFile: 5936 kB
   RssShmem: 0 kB

2. two session, 29.3m
   VmHWM: 29976 kB
   VmRSS: 29976 kB
   RssAnon: 24040 kB
   RssFile: 5936 kB
   RssShmem: 0 kB

3. three session, 36.5m
   VmHWM: 37368 kB
   VmRSS: 37368 kB
   RssAnon: 31428 kB
   RssFile: 5940 kB
   RssShmem: 0 kB

[xadupre]

You can find more information here: https://onnxruntime.ai/docs/get-started/with-c.html. onnxruntime allocates memory and manages it on its own. It can be three independant buffers or one buffer used by three sessions. In that case, the memory peak usage is lower but sessions do not share pointers. The shared allocator usually reduces memory fragmentation. In your case, if it is the same model, the session supports multithreading, the same session should work for inference from three different threads.

[Renekton]

Thank you for your response.

1. I know that the onnxruntime sesssion is thread-safe. The example is used to describe how the share allocator is used and to determine if the memory for the same model can be shared.
2. The test results indicate that even for the same model, memory corresponding to the model cannot be shared through a shared allocator during session initialization.
3. Is it true that sharing the memory of the same model can only be achieved by using the "CreateSessionWithPrepackedWeightsContainer" API or by directly using the same session within multiple threads?
4. The shared allocator usually reduces memory fragmentation. Is it true that the inputs which created by api "Ort::Value::CreateTensor(memory_info, ...)" should be shared in the memory which manged by share allocator?

[xadupre]

onnxruntime does optimize a model after it is loaded. Two sessions could end up with different optimized models even if they are loading the same one. Memory is not shared unless CreateSessionWithPrepackedWeightsContainer is used. Most of the memory consumption is taken by the model coefficients. For the last point, that should be true even if the allocation is not done by the same allocator. onnxruntime tries to avoid every unnecessary copy.

[Renekton]

1. Is the model optimization you mentioned referring to graph optimization? If I disable graph optimization by setting GraphOptimizationLevel to ORT_DISABLE_ALL, will the final model be the same?