Add more documentation on OFI heap environment variables (#24982)

Adds more documentation to `doc/rst/platforms/libfabric.rst` about
`CHPL_RT_MAX_HEAP_SIZE` and `CHPL_RT_COMM_OFI_THP_HINT`

This is a follow-on to #24971
and #24967

Built and checked docs locally

[Reviewed by @jhh67]

doc/rst/platforms/libfabric.rst

@@ -217,6 +217,12 @@ Alternatively, you can prevent creation of a fixed heap entirely by
 setting ``CHPL_RT_MAX_HEAP_SIZE=0``.  This may cause the selection of a
 different provider than the highest-performing one, however.
 
+``CHPL_RT_MAX_HEAP_SIZE`` sets the heap size per compute node. If the
+application is run with multiple locales per node (co-locales), the heap size
+will be divided equally among the co-locales. For example, setting
+``CHPL_RT_MAX_HEAP_SIZE=100G`` and running with 4 co-locales (i.e., ``-nl Nx4``) will
+result in a 25 GiB heap per co-locale.
+
 .. note::
   In the future we hope to be able to reduce the user impact of memory
   registration and fixed heaps when using the ofi communication layer.
@@ -250,6 +256,17 @@ size ``CHPL_RT_MAX_HEAP_SIZE`` even if there is insufficient physical
 memory, which may lead to out-of-memory errors during program
 execution.
 
+
+Transparent Hugepages
+_____________________
+
+With some providers, the use of transparent hugepages is required to register
+more memory. This is default when using the EFA provider on AWS. To explicitly
+enable transparent hugepages, set the ``CHPL_RT_COMM_OFI_THP_HINT`` environment
+variable to true. This will enable transparent hugepages for the duration of
+the program. Note that this requires the kernel to have transparent hugepages
+enabled and enough free hugepages to be available.
+
 .. _mpirun4ofi-launcher:
 
 The mpirun4ofi Launcher