diff --git a/doc/rst/platforms/libfabric.rst b/doc/rst/platforms/libfabric.rst index 43ba89befe53..60f077138a51 100644 --- a/doc/rst/platforms/libfabric.rst +++ b/doc/rst/platforms/libfabric.rst @@ -217,6 +217,12 @@ Alternatively, you can prevent creation of a fixed heap entirely by setting ``CHPL_RT_MAX_HEAP_SIZE=0``. This may cause the selection of a different provider than the highest-performing one, however. +``CHPL_RT_MAX_HEAP_SIZE`` sets the heap size per compute node. If the +application is run with multiple locales per node (co-locales), the heap size +will be divided equally among the co-locales. For example, setting +``CHPL_RT_MAX_HEAP_SIZE=100G`` and running with 4 co-locales (i.e., ``-nl Nx4``) will +result in a 25 GiB heap per co-locale. + .. note:: In the future we hope to be able to reduce the user impact of memory registration and fixed heaps when using the ofi communication layer. @@ -250,6 +256,17 @@ size ``CHPL_RT_MAX_HEAP_SIZE`` even if there is insufficient physical memory, which may lead to out-of-memory errors during program execution. + +Transparent Hugepages +_____________________ + +With some providers, the use of transparent hugepages is required to register +more memory. This is default when using the EFA provider on AWS. To explicitly +enable transparent hugepages, set the ``CHPL_RT_COMM_OFI_THP_HINT`` environment +variable to true. This will enable transparent hugepages for the duration of +the program. Note that this requires the kernel to have transparent hugepages +enabled and enough free hugepages to be available. + .. _mpirun4ofi-launcher: The mpirun4ofi Launcher