Skip to content

Commit

Permalink
Make co-locales respect CHPL_RT_MAX_HEAP_SIZE (#24967)
Browse files Browse the repository at this point in the history
This PR fixes the way that co-locales determine their fixed heap size
when using COMM=ofi. Previously, co-locales would divy up 85% of the
memory. Now, co-locales divy up the memory based upon the max heap size
set in `CHPL_RT_MAX_HEAP_SIZE`. In other words, `CHPL_RT_MAX_HEAP_SIZE`
represents the maximum heap per node, and the co-locales just divide it
evenly. If `CHPL_RT_MAX_HEAP_SIZE` is not set, then the previous default
of 85% is used.

Tested using COMM=ofi with no co-locales, 2 co-locales, and 4
co-locales, with/without `CHPL_RT_MAX_HEAP_SIZE` set

[Reviewed by @jhh67]
  • Loading branch information
jabraham17 authored May 2, 2024
2 parents be41ede + 0a35160 commit 8f54fe1
Showing 1 changed file with 13 additions and 15 deletions.
28 changes: 13 additions & 15 deletions runtime/src/comm/ofi/comm-ofi.c
Original file line number Diff line number Diff line change
Expand Up @@ -3540,26 +3540,24 @@ void init_fixedHeap(void) {
}
if (createHeap) {


//
// Don't use more than 85% of the total memory for heaps.
// Either use MAX_HEAP_SIZE, or 85% of total memory
//
uint64_t total_memory = chpl_sys_physicalMemoryBytes();
uint64_t max_heap_memory = (size_t) (0.85 * total_memory);

int num_locales_on_node = chpl_get_num_locales_on_node();
size_t max_heap_per_locale = (size_t) (max_heap_memory / num_locales_on_node);

uint64_t max_heap_memory;
if (envMaxHeapSize > 0) {
// use user-specified envMaxHeapSize
max_heap_memory = envMaxHeapSize;
} else {
uint64_t total_memory = chpl_sys_physicalMemoryBytes();
max_heap_memory = (size_t) (0.85 * total_memory);
}

//
// If the maximum heap size is not specified or it's greater than the maximum heap per
// locale, set it to the maximum heap per locale.
// Split up the heap among all the colocales
//
ssize_t size = envMaxHeapSize;
CHK_TRUE(size != 0);
if ((size < 0) || (size > max_heap_per_locale)) {
size = max_heap_per_locale;
}
int num_locales_on_node = chpl_get_num_locales_on_node();
size_t max_heap_per_locale = (size_t) (max_heap_memory / num_locales_on_node);
ssize_t size = max_heap_per_locale;

//
// Check for hugepages. On certain systems you really ought to use
Expand Down

0 comments on commit 8f54fe1

Please sign in to comment.