Skip to content

YAKL and SAMXX CIME and CMake Integration Notes

Jump to bottom Edit New page
Matt Norman edited this page Jul 1, 2021 · 7 revisions

Adding SAMXX to CIME and CAM configure integration

Here, we work with files in E3SM/components/eam/cime_config and E3SM/components/eam/bld.

To add samxx to CIME and the CAM configure, we added the following to cime_config/config_componenet.xml in the <entry id="CAM_CONFIG_OPTS"> and <values modifier='additive'> sections:

<value compset="_EAM%MMFXX">-crm samxx</value>

Then, in cime_config/config_componenet.xml, we added the following in the <description> section:

<desc compset="_EAM%MMFXX">E3SM-MMF (C++) with 64x1km CRM, RRTMGP radiation, 1-mom micro, prescribed aerosol</desc>

These tell CIME what to add to the CAM configure line based on having MMFXX in the compset string. Next, we define alises and compset strings in cime_config/config_compsets.xml with the following:

<compset>
  <alias>F-MMFXX-AQP1</alias>
  <lname>2000_EAM%MMFXX-AQUA_SLND_SICE_DOCN%AQP1_SROF_SGLC_SWAV</lname>
</compset>
<compset>
  <alias>F-MMFXX-RCEMIP</alias>
  <lname>2000_EAM%MMFXX-RCE_SLND_SICE_DOCN%AQPCONST_SROF_SGLC_SWAV</lname>
</compset>
<compset>
  <alias>F-MMFXX</alias>
  <lname>2000_EAM%MMFXX_ELM%SP_CICE%PRES_DOCN%DOM_SROF_SGLC_SWAV</lname>
</compset>

Finally, we work in the E3SM/components/eam/bld/configure script to specify what the samxx parameter passed to the configure script does to the CPP definitions and the source directories automatically GLOB'd into the EAM CMake scripts. First, we add a definition for the crm XML value and its valid parameters in E3SM/components/eam/bld/config_files/definition.xml:

<entry id="crm" valid_values="sam,samomp,samxx" value="sam">
MMF CRM model
</entry>

Next, we add documentation to the CAM configure command: -crm <model> CRM model [sam | samomp | samxx]. We then pass a CPP definition if -crm samxx is passed to the configure command with:

if (defined $opts{'use_MMF'}) {
  ...
  if    ($crm eq 'sam') {
      $cfg_cppdefs .= " -DMMF_SAM "
  } elsif ($crm eq 'samomp') {
      $cfg_cppdefs .= " -DMMF_SAMOMP"
  } elsif ($crm eq 'samxx') {
      $cfg_cppdefs .= " -DMMF_SAMXX "
  }    
  ...
}

Typically there are source directories to add to the CAM CMake GLOB commands to search for source files, but since samxx does it's own custom CMake build, this is skipped, i.e.:

  } elsif ($crm eq 'samxx') {
    #####################################################
    ## -crm samxx will add_subdirectory in CMake rather
    ##            than GLOBing.
    #####################################################
  }

SAMXX and YAKL CMake Integration

The E3SM CMake files are located in E3SM/components and E3SM/components/cmake.

Adding the USE_CUDA and CUDA_FLAGS CMake variables

The USE_CUDA variable needs to be visible at the highest level CMakeLists.txt, so it has to be specified through CIME using the Macros.cmake file in E3SM/cime_config/machines/config_compilers.xml. Similarly, the CUDA_FLAGS variable is compiler dependent and therefore will have to come from the same source through the same route into CMake.

To make these variables available to add to that file, however, they had to first be added to the XML schema in cime/config/xml_schemas/config_compilers_v2.xsd. It was added in the <xs:group name="compilerVars"> group as a "choice" as follows:

  <xs:element name="CUDA_FLAGS" type="flagsVar"/>
  <xs:element name="USE_CUDA" type="upperBoolean"/>

Then they are added to appropriate machine, compiler configurations in config_compilers.xml. An example is below for Summit with GNU + CUDA:

  <CUDA_FLAGS>
    <append> -O3 -arch sm_70 --use_fast_math </append>
  </CUDA_FLAGS>
  <USE_CUDA>TRUE</USE_CUDA>

Now, when CMake starts its top-level CMakeLists.txt for E3SM, it sees these things set in the local Macros.cmake. This still isn't quite enough, though because the main CMakeLists.txt only looks at the Macros.cmake file in a local setting inside a CMake function called set_compilers_e3sm. To make it visible to the overall project, you have to set it in parent scope inside that scope as follows:

  # USE_CUDA is set through Macros.cmake / config_compilers.xml
  # If it exists, then set parent's scope to true; otherwise to false
  if (USE_CUDA)
    set(USE_CUDA TRUE PARENT_SCOPE)
  else()
    set(USE_CUDA FALSE PARENT_SCOPE)
  endif()

Finally, we use this at the top level to turn on the CUDA language. CMake doesn't like doing this at a lower scope, so it needs to be done at the beginning of the project declaration:

project(E3SM C CXX Fortran)

if(USE_CUDA)
  enable_language(CUDA)
endif()

Creating USE_SAMXX and USE_YAKL CMake variables

Go to E3SM/components/cmake/common_setup.cmake, and add new variables there. Use existing code to get the feel for where to test for adding a given variable. CAM often hooks into CIME through the ${CAM_CONFIG_OPTS} variable that comes from the CAM configure command. USE_YAKL and USE_SAMXX are created with the code below in cmake/build_model.cmake

# Look for -crm samxx in the CAM_CONFIG_OPTS CIME variable
# If it's found, then enable USE_SAMXX
string(FIND "${CAM_CONFIG_OPTS}" "-crm samxx" HAS_SAMXX)
if (NOT HAS_SAMXX EQUAL -1)
  # The following is for the SAMXX code:
  set(USE_SAMXX TRUE)
endif()

# If samxx is being used, then YAKL must be used as well
set(USE_YAKL ${USE_SAMXX})

# If YAKL is being used, then we need to enable USE_CXX
if (${USE_YAKL})
  set(USE_CXX TRUE)
endif()

We then add the YAKL and SAMXX builds into the cmake/build_model.cmake file inside if (COMP_NAME STREQUAL "eam") with the following code:

  if (COMP_NAME STREQUAL "eam")
    ...
    # If YAKL is needed, then set YAKL CMake vars
    if (USE_YAKL)
      # YAKL_ARCH can be CUDA, HIP, SYCL, OPENMP45, or empty
      # USE_CUDA is set through Macros.cmake / config_compilers.xml
      if (USE_CUDA)
        set(YAKL_ARCH "CUDA")
        # CUDA_FLAGS is set through Macros.cmake / config_compilers.xml
        set(YAKL_CUDA_FLAGS "${CPPDEFS} ${CUDA_FLAGS}")
      else()
        # For CPU C++ compilers duplicate flags are fine, the last ones win typically
        set(YAKL_CXX_FLAGS "${CPPDEFS} ${CXXFLAGS}")
        set(YAKL_ARCH "")
      endif()
      message(STATUS "Building YAKL")
      # Build YAKL as a static library
      # YAKL_HOME is YAKL's source directlry
      set(YAKL_HOME ${CMAKE_CURRENT_SOURCE_DIR}/../../../externals/YAKL)
      # YAKL_BIN is where we're placing the YAKL library
      set(YAKL_BIN  ${CMAKE_CURRENT_BINARY_DIR}/yakl)
      # Build the YAKL static library
      add_subdirectory(${YAKL_HOME} ${YAKL_BIN})
      # Add the YAKL bin directory, mainly due to a Fortran module if it's needed
      include_directories(${YAKL_BIN})
    endif()

    # if samxx is needed, build samxx as a static library
    if (USE_SAMXX)
      message(STATUS "Building SAMXX")
      # SAMXX_HOME is where the samxx source code lives
      set(SAMXX_HOME ${CMAKE_CURRENT_SOURCE_DIR}/../../eam/src/physics/crm/samxx)
      # SAMXX_BIN is where the samxx library will live
      set(SAMXX_BIN  ${CMAKE_CURRENT_BINARY_DIR}/samxx)
      # Build the static samxx library
      add_subdirectory(${SAMXX_HOME} ${SAMXX_BIN})
      # Add samxx F90 files to the main E3SM build
      set(SOURCES ${SOURCES} cmake/atm/../../eam/src/physics/crm/samxx/cpp_interface_mod.F90
                             cmake/atm/../../eam/src/physics/crm/samxx/params.F90
                             cmake/atm/../../eam/src/physics/crm/crm_ecpp_output_module.F90 )
    endif()

  endif()

From here, each component (YAKL and SAMXX) can fend for itself with its own CMakeLists.txt, which will build a library. Finally, we link the library into the EAM component's library with the following code near the end of cmake/build_model.cmake:

    if (COMP_NAME STREQUAL "eam")
      if (USE_YAKL)
        target_link_libraries(${TARGET_NAME} PRIVATE yakl)
      endif()
      if (USE_SAMXX)
        target_link_libraries(${TARGET_NAME} PRIVATE samxx)
      endif()
    endif()

The samxx CMake subdirectory has the following CMakeLists.txt:

###############################################################################
## This assumes you have already run add_subdirectory(${YAKL_HOME} ${YAKL_BIN})
## It also assumes you've enabled CXX and C as languages (and CUDA if used)
###############################################################################

file(GLOB F90_SRC   cpp_interface_mod.F90 fft.F90 params.F90 fftpack5.F90 fftpack5_1d.F90)
file(GLOB CXX_SRC   *.cpp)
file(GLOB SAMXX_SRC ${CXX_SRC} ${F90_SRC})

# samxx will be a static library
add_library(samxx STATIC ${SAMXX_SRC})
if ("${YAKL_ARCH}" STREQUAL "CUDA")
  # samxx will be CUDA-linked with device symbols resolved at library creation
  set_target_properties(samxx PROPERTIES LINKER_LANGUAGE CUDA CUDA_SEPARABLE_COMPILATION OFF CUDA_RESOLVE_DEVICE_SYMBOLS ON)
endif()
# samxx needs to link with the yakl library
target_link_libraries(samxx yakl)
target_compile_features(samxx PUBLIC cxx_std_14)

# Set fortran compiler flags
set_source_files_properties(${F90_SRC} PROPERTIES COMPILE_FLAGS "${CPPDEFS} ${FFLAGS}")

# Set YAKL compiler flags
include(${YAKL_HOME}/process_cxx_source_files.cmake)
# Have to quote the list of files in order to pass it into a macro
process_cxx_source_files("${CXX_SRC}")
message(STATUS "YAKL Flags: ${YAKL_COMPILER_FLAGS}")

# Include YAKL source and library directories
include_directories(${YAKL_BIN})
Add a custom footer
Add a custom sidebar
Clone this wiki locally