Usage of new virtual heap allocator in SOF/Zephyr architecture

[dabekjakub]

Currently in sof/zephyr combo all allocations are made on one heap.

All in \sof\zephyr\lib\alloc.c

With introduction of virtual memory heaps we can start transitioning to using those heaps and benefits they bring.

My idea of implementing this changes is:

Use mem zone attributes as basis for choosing the heap

enum mem_zone {
	SOF_MEM_ZONE_SYS = 0,		/**< System zone */
	SOF_MEM_ZONE_SYS_RUNTIME,	/**< System-runtime zone */
	SOF_MEM_ZONE_RUNTIME,		/**< Runtime zone */
	SOF_MEM_ZONE_BUFFER,		/**< Buffer zone */
	SOF_MEM_ZONE_RUNTIME_SHARED,	/**< Runtime shared zone */
	SOF_MEM_ZONE_SYS_SHARED,	/**< System shared zone */
};

I want to start by:

• directing SOF_MEM_ZONE_RUNTIME_SHARED allocations to shared heap and checking if most of the runtime allocs shouldn't go there too.
• creating a core heap on core startup and create appropriate functions that would direct to core heap on alloc - currently I do not see many uses of old rzalloc_core_sys() and I do not know if there is any current arch that needs redirecting to core virtual heaps - comments welcome
• redirecting buffer zone allocations to core heaps ?

If anyone can provide insight please free to do so.
@mmaka1 @marcinszkudlinski Please advice in spare time if my starting points are flawed or should I go with this plan :)

@kv2019i added: Some relevant links to understand context
https://thesofproject.github.io/latest/architectures/firmware/sof-zephyr/rtos_layer/memory_management/heap_sharing.html
https://thesofproject.github.io/latest/architectures/firmware/sof-zephyr/rtos_layer/memory_management/index.html

[kv2019i]

Thanks for the write-up. I wonder if we need to extend the alloc.h interface to expose the new semantics to applications. Not sure if the current zone semantics are enough to expose the new capabilities. I.e. RUNTIME_SHARED is currently used to allocate buffers that are used across the cores. On platforms with cached/uncached aliases, this API has returned uncached pointers.

But if e.g. per-core heaps are to be created, I don't see other options than to extend alloc.h with some way to applications to indicate that it wants to do an allocation that does not need to be visible to other cores. To get benefits, a large amount of changes would be need to be made to application code to mark places where allocations can be per core, and where they are not. As @lyakh pointed out, we have a lot of generic code that doesn't know whether it is run tied to a single core or not, so this is not necessarily easy interface to use.

Also how would you allocate the physical RAM to virtual memory heaps? One big issue we have inow s that we don't know the split between different zones (and we don't want to make a fixed partitioning). How do you plan to tackle this, how is RAM partitioned between system heap and the 1...N virtual heaps?

[dabekjakub]

When it comes to uncached ptrs it is a case of creating appropriate functions/extending existing api. Problem is to know when it is unsafe to use cashed when switching to a new api. Then greed maybe first step should be extending current enum or using different mechanism if there will be a proposition for it.
I have a mpp memory manager in the back of my mind as a thing that in future would use this api separately for per core operations (directly calling heap allocations). And this api was created with that in mind since those layers are indicated in documentation and I assume it will be direction in future development.
Partitioning heaps is a separate topic in itself. The idea is that physical memory is mapped as needed. So if we want to limit the heap we need to limit the virtual region that it is attached to. Or do it via creating proper heap config. In current implementation if You provide heap config it won't try to allocate more than configured space.
When creating heap You actually chose between span allocations or fixed sizes allocations. So You can have a continuous allocation heap or fixed block sizes allocations.
So there are ways of limiting heaps sizes but it was designed with overbooking of virtual memory as a feature.

[lyakh]

So far the only really new functionality, provided by this proposal, that I can see, is physical-to-virtual address mapping. But your current PR is also doing a whole lot more. Why not just add a memory mapping layer on top of the current heap implementation?
As for per-core - what is the purpose of that? Other cores don't need to see that memory or they shouldn't see it? If just "don't need" - that's fine, they just don't use it. And I don't think we can currently control memory access at the core level?

[lyakh]

Maybe we can collect features that are missing in our current heap implementation that we want to address:

1. Allocation failures because of fragmentation. It can happen that after some time we're unable to allocate large memory buffers because of fragmentation. We could solve this using virtual address mapping
2. Memory banks staying powered on also because of fragmentation. It can happen that we have multiple memory banks powered on only because some small parts of them are in use for heap allocations. Also can be solved by virtual remapping.
3. Per-core allocations. Not sure really why we need these. To bulk-free them when a core goes down? If only to say, that that memory can be cached, then we can just allocate it as such already now.

Any other points?