ALCF Vesta

Instructions for 2016

My .soft file looks like this:

@default
+cmake
+bgqtoolchain-gcc472
+mpiwrapper-gcc

Note that the +bgqtoolchain-gcc472 line does not appear to set the toolchain, so I also placed the line:

soft add +bgqtoolchain-gcc472

in my .bash_profile

Building Trilinos

The Trilinos configure script used for the tutorials this year is committed to the Albany repository at:

doc/do-cmake-trilinos-vesta

Note that this build uses LAPACK 3.6.1. This version has deprecated the calls dggsvd and sggsvd. Please edit the Trilinos file packages/epetra/src/Epetra_LAPACK_wrappers.h and change the lines:

#define DGGSVD_F77  F77_BLAS_MANGLE(dggsvd3,DGGSVD3)
#define SGGSVD_F77  F77_BLAS_MANGLE(sggsvd3,SGGSVD3)

Building Albany

The Albany configure script used for the tutorials this year is committed to the Albany repository at:

doc/do-cmake-albany-vesta

Running the Albany regression test suite

Note that the above Albany configure script also sets up ctest to run the regression suite on BG/Q, employing the script /projects/FASTMath/ATPESC-2016/installs/albany/bin/run_Albany to dispatch each parallel test to the compute nodes while ctest itself runs on the frontend node. From the build directory on the frontend node, typing
```
  qsub -A ATPESC16_Instructors -n 4 -t 60 --mode script /projects/FASTMath/ATPESC-2016/installs/albany/bin/reserve.sh
  qstat
```

Get the blockid info from qstat, and now run ctest

  export MPIEXEC_BLOCKID=VST-02400-13511-32
  ctest

will run the regression suite in this manner. Note also that all the usual ctest options apply to this command.

Instructions for 2015

Add this to your .soft file:

+mpiwrapper-bgclang
+cmake-3.0.2

Configure Trilinos as follows. (The block starting with Trilinos_ENABLE_Zoltan is for another user of this Trilinos build and can be omitted in a build meant only for Albany.)

# July 2015.
#   Need cmake:
#     => Use this line in your .soft file:
#          +cmake-3.0.2
#   C++11 requirement in Trilinos, especially in Kokkos:
#     => Use bgclang.
#     => Use this line in your .soft file:
#          +mpiwrapper-bgclang
#   However, (i) bgclang has no Fortran equivalent, and (ii) XL's
# binaries appear not to be compatible with bgclang's:
#     => Use the CMAKE_Fortran_COMPILER flag to point to gcc's Fortran compiler.

# Install next to the build directory.
TRILINSTALLDIR=../install-atpesc

# These TPLs are already built for us.
NETCDFDIR=/projects/FASTMath/install/netcdf/4.3.0/linux-rhel_6-ppc64-gcc-4.4.6
HDF5DIR=/projects/FASTMath/install/hdf5/1.8.11/linux-rhel_6-ppc64-gcc-4.4.6
ZLIBDIR=/projects/FASTMath/install/zlib/1.2.8/linux-rhel_6-ppc64-gcc-4.4.6
BLASDIR=/projects/FASTMath/install/blas/19Apr11/linux-rhel_6-ppc64-gcc-4.4.6
LAPACKDIR=/projects/FASTMath/install/lapack/3.4.2/linux-rhel_6-ppc64-gcc-4.4.6
BOOSTDIR=/projects/FASTMath/install/boost/1.54.0/linux-rhel_6-ppc64-gcc-4.4.6
# These are not.
MYTPLDIR=/projects/FASTMath/ATPESC-2015/install/Trilinos/tpl

# Make Trilinos choose the .a rather than .so files.
BOOST_LIBS="$BOOSTDIR/lib/libboost_program_options.a;$BOOSTDIR/lib/libboost_system.a"

# netcdf and hdf5 ordering are wrong for static linking. Fix with the
# usual hack of appending these as extra link flags. List libz.a
# specifically so libz.so is not used.
EXTRA_LINK_FLAGS="${HDF5DIR}/lib/libhdf5_hl.a;${HDF5DIR}/lib/libhdf5.a;${ZLIBDIR}/lib/libz.a"

# Turn on for a 64-bit local ordinal.
HAVE_LL=OFF

rm -f CMakeCache.txt;

cmake \
    -D Trilinos_DISABLE_ENABLED_FORWARD_DEP_PACKAGES=ON \
    -D CMAKE_INSTALL_PREFIX:PATH=${TRILINSTALLDIR} \
    -D CMAKE_BUILD_TYPE:STRING="RELEASE" \
    -D TPL_ENABLE_MPI:BOOL=ON \
    -D CMAKE_VERBOSE_MAKEFILE:BOOL=OFF \
    -D BUILD_SHARED_LIBS:BOOL=OFF \
    -D Trilinos_ENABLE_SECONDARY_TESTED_CODE:BOOL=OFF \
    \
    -D Trilinos_ENABLE_ALL_PACKAGES:BOOL=OFF \
    -D Trilinos_WARNINGS_AS_ERRORS_FLAGS:STRING="" \
    -D Trilinos_ENABLE_Teuchos:BOOL=ON \
    -D Trilinos_ENABLE_Shards:BOOL=ON \
    -D Trilinos_ENABLE_Sacado:BOOL=ON \
    -D Trilinos_ENABLE_Epetra:BOOL=ON \
    -D Trilinos_ENABLE_EpetraExt:BOOL=ON \
    -D Trilinos_ENABLE_Ifpack:BOOL=ON \
    -D Trilinos_ENABLE_AztecOO:BOOL=ON \
    -D Trilinos_ENABLE_Amesos:BOOL=ON \
    -D Trilinos_ENABLE_Anasazi:BOOL=ON \
    -D Trilinos_ENABLE_Belos:BOOL=ON \
    -D Trilinos_ENABLE_ML:BOOL=ON \
    -D Trilinos_ENABLE_Phalanx:BOOL=ON \
    -D Trilinos_ENABLE_Intrepid:BOOL=ON \
    -D Trilinos_ENABLE_NOX:BOOL=ON \
    -D Trilinos_ENABLE_Stratimikos:BOOL=ON \
    -D Trilinos_ENABLE_Thyra:BOOL=ON \
    -D Trilinos_ENABLE_Rythmos:BOOL=ON \
    -D Trilinos_ENABLE_MOOCHO:BOOL=ON \
    -D Trilinos_ENABLE_Stokhos:BOOL=ON \
    -D Trilinos_ENABLE_Piro:BOOL=ON \
    -D Trilinos_ENABLE_Teko:BOOL=ON \
    -D Trilinos_ENABLE_Moertel:BOOL=ON \
    \
    -D Trilinos_ENABLE_STKIO:BOOL=ON \
    -D Trilinos_ENABLE_STKMesh:BOOL=ON \
    -D TPL_ENABLE_Boost:BOOL=ON \
    -D Boost_INCLUDE_DIRS:FILEPATH="$BOOSTDIR/include" \
    -D TPL_Boost_LIBRARIES=${BOOST_LIBS} \
    -D TPL_ENABLE_BoostLib:BOOL=ON \
    -D BoostLib_INCLUDE_DIRS:FILEPATH="$BOOSTDIR/include" \
    -D TPL_BoostLib_LIBRARIES=${BOOST_LIBS} \
    \
    -D Trilinos_ENABLE_SEACAS:BOOL=ON \
    -D TPL_ENABLE_X11:BOOL=OFF \
    -D TPL_ENABLE_Matio:BOOL=OFF \
    -D Trilinos_ENABLE_SEACASIoss:BOOL=ON \
    -D Trilinos_ENABLE_SEACASExodus:BOOL=ON \
    \
    -D TPL_ENABLE_Netcdf:BOOL=ON \
    -D Netcdf_INCLUDE_DIRS:PATH="$NETCDFDIR/include" \
    -D TPL_Netcdf_LIBRARIES="$NETCDFDIR/lib/libnetcdf.a" \
    -D TPL_ENABLE_HDF5:BOOL=ON \
    -D TPL_HDF5_LIBRARIES="$HDF5DIR/lib/libhdf5_hl.a;$HDF5DIR/lib/libhdf5.a;$ZLIBDIR/lib/libz.a" \
    -D HDF5_INCLUDE_DIRS:PATH="$HDF5DIR/include" \
    \
    -D Trilinos_ENABLE_Tpetra:BOOL=ON \
    -D Trilinos_ENABLE_Kokkos:BOOL=ON \
    -D Trilinos_ENABLE_Ifpack2:BOOL=ON \
    -D Trilinos_ENABLE_Amesos2:BOOL=ON \
    -D Trilinos_ENABLE_Zoltan2:BOOL=ON \
    -D Trilinos_ENABLE_MueLu:BOOL=ON \
    -D Amesos2_ENABLE_KLU2:BOOL=ON \
    -D Amesos2_ENABLE_Basker:BOOL=ON \
    \
    -D Trilinos_ENABLE_SCOREC:BOOL=ON \
    -D SCOREC_DISABLE_STRONG_WARNINGS:BOOL=ON \
    -D Trilinos_ENABLE_EXPORT_MAKEFILES:BOOL=OFF \
    -D Trilinos_ASSERT_MISSING_PACKAGES:BOOL=OFF \
    -D MDS_ID_TYPE:STRING="int" \
    \
    -D TPL_ENABLE_ParMETIS:STRING=ON \
    -D ParMETIS_INCLUDE_DIRS:PATH="${MYTPLDIR}/include" \
    -D ParMETIS_LIBRARY_DIRS:PATH="${MYTPLDIR}/lib" \
    -D TPL_ENABLE_METIS:STRING=ON \
    -D METIS_INCLUDE_DIRS:PATH="${MYTPLDIR}/include" \
    -D METIS_LIBRARY_DIRS:PATH="${MYTPLDIR}/lib" \
    -D Zoltan_ENABLE_ULLONG_IDS:BOOL=ON \
    -D Teuchos_ENABLE_LONG_LONG_INT:BOOL=$HAVE_LL \
    \
    -D Trilinos_ENABLE_EXPLICIT_INSTANTIATION:BOOL=ON \
    -D Tpetra_INST_FLOAT=OFF \
    -D Tpetra_INST_INT_INT=ON \
    -D Tpetra_INST_DOUBLE=ON \
    -D Tpetra_INST_COMPLEX_FLOAT=OFF \
    -D Tpetra_INST_COMPLEX_DOUBLE=OFF \
    -D Tpetra_INST_INT_LONG=OFF \
    -D Tpetra_INST_INT_UNSIGNED=OFF \
    -D Tpetra_INST_INT_LONG_LONG=$HAVE_LL \
    \
    -D Trilinos_ENABLE_Kokkos:BOOL=ON \
    -D Trilinos_ENABLE_KokkosCore:BOOL=ON \
    -D Phalanx_KOKKOS_DEVICE_TYPE:STRING="SERIAL" \
    -D Phalanx_INDEX_SIZE_TYPE:STRING="INT" \
    -D Phalanx_SHOW_DEPRECATED_WARNINGS:BOOL=OFF \
    -D Kokkos_ENABLE_Serial:BOOL=ON \
    -D Kokkos_ENABLE_OpenMP:BOOL=OFF \
    -D Kokkos_ENABLE_Pthread:BOOL=OFF \
    \
    -D Trilinos_ENABLE_Zoltan:BOOL=ON \
    -D Zoltan_ENABLE_TESTS:BOOL=ON \
    -D Zoltan_ENABLE_EXAMPLES:BOOL=ON \
    -D Zoltan_ENABLE_F90INTERFACE:BOOL=ON \
    -D Zoltan_ENABLE_ParMETIS:BOOL=ON \
    -D Zoltan_ENABLE_Scotch:BOOL=ON \
    -D Scotch_LIBRARY_DIRS:FILEPATH="${MYTPLDIR}/lib" \
    -D Scotch_INCLUDE_DIRS:FILEPATH="${MYTPLDIR}/include" \
    \
    -D CMAKE_CXX_FLAGS:STRING="-Wno-deprecated-declarations -Wno-sign-compare" \
    -D TPL_BLAS_LIBRARIES="$BLASDIR/lib/libblas.a" \
    -D TPL_LAPACK_LIBRARIES="$LAPACKDIR/lib/liblapack.a;$LAPACKDIR/lib/libtmglib.a" \
    -D CMAKE_Fortran_COMPILER="/soft/compilers/wrappers/gcc/mpif90" \
    -D Trilinos_EXTRA_LINK_FLAGS:STRING=${EXTRA_LINK_FLAGS} \
    -D F77_FUNC:STRING="(name,NAME) name" \
    -D F77_FUNC_:STRING="(name,NAME) name" \
    \
    ../

Instructions for 2014

What follows are instructions for the 2014 ATPESC build.

Set up your build environment

Please place the following lines in your ~/.soft file:

	@default
	+hpctoolkit
	+cmake
	+mpiwrapper-gcc

Install the required Third Party Libraries (TPLs) for Trilinos

These install directions assume that a base install directory of /home/ghansen/FASTMath/src/ is used

Building ZLib

Version 1.2.8

  cd /home/ghansen/FASTMath/src/
  wget http://zlib.net/zlib-1.2.8.tar.gz
  tar -xvf zlib-1.2.8.tar.gz
  cd zlib-1.2.8
  CC=mpicc CXX=mpicxx CFLAGS=-O3 CXXFLAGS=-O3 ./configure --64 --static --prefix=/home/ghansen/FASTMath
  make install

Building HDF5

Version 1.8.13

  cd /home/ghansen/FASTMath/src/
  wget http://www.hdfgroup.org/ftp/HDF5/current/src/hdf5-1.8.13.tar.gz
  tar -xvf hdf5-1.8.13.tar.gz
  cd hdf5-1.8.13
  ./configure CC=mpicc FC=mpif90 CXX=mpicxx CXXFLAGS="-fPIC -O3" CFLAGS="-fPIC -O3" FCFLAGS="-fPIC -O3" \
  --enable-parallel --enable-shared=no --with-zlib=/home/ghansen/FASTMath --prefix=/home/ghansen/FASTMath
  make -j 8
  make install

Building Netcdf

Version of the day from the git repo. Note that hdf5-1.8.13 made some changes that are not reflected in the latest release, so we are using the votd for now.
```
  cd /home/ghansen/FASTMath/src/
  git clone https://github.com/Unidata/netcdf-c.git
  cd netcdf-c
```

Edit the file include/netcdf.h. Make the following edits near line 228:

  #define NC_MAX_DIMS    65536    /* max dimensions per file */
  #define NC_MAX_ATTRS    8192    
  #define NC_MAX_VARS   524288    /* max variables per file */
  #define NC_MAX_NAME   256 
  #define NC_MAX_VAR_DIMS      8     /* max per variable dimensions */

Now, configure using CMake and build:

  mkdir build
  cd build
  cmake \
  -DCMAKE_C_COMPILER=mpicc \
  -DCMAKE_C_FLAGS="-I/home/ghansen/FASTMath/include -O3" \
  -DCMAKE_EXE_LINKER_FLAGS="-L/home/ghansen/FASTMath/lib -O3" \
  -DCMAKE_INSTALL_PREFIX=/home/ghansen/FASTMath \
  -DENABLE_DAP=OFF -DBUILD_SHARED_LIBS=OFF -DENABLE_FSYNC=OFF -DENABLE_CDMREMOTE=OFF \
  -DENABLE_DOXYGEN=OFF -DENABLE_NETCDF_4=ON -DCMAKE_BUILD_TYPE=RELEASE -DENABLE_EXAMPLES=OFF \
  -DENABLE_TESTS=OFF -DCMAKE_SKIP_INSTALL_RPATH=ON \
  -DZLIB_INCLUDE_DIRS=/home/ghansen/FASTMath/include \
  -DZLIB_LIBRARY=/home/ghansen/FASTMath/lib/libz.a \
  -DBUILD_UTILITIES=OFF \
  ..

  make -j 8 
  make install

Building boost

Version 1.55.0

  cd /home/ghansen/FASTMath/src/
  wget http://sourceforge.net/projects/boost/files/boost/1.55.0/boost_1_55_0.tar.gz/download
  tar -xvf boost_1_55_0.tar.gz
  cd boost_1_55_0

Build boost with the commands

  echo "using gcc : 4.4.7 : /bgsys/drivers/ppcfloor/gnu-linux/bin/powerpc64-bgq-linux-g++ ;" \
  >> ./tools/build/v2/user-config.jam
  echo "using mpi : /bgsys/drivers/ppcfloor/comm/bin/gcc/mpicxx ;" >> ./tools/build/v2/user-config.jam
  ./bootstrap.sh \
  --with-libraries=signals,regex,filesystem,system,mpi,serialization,thread,program_options,exception \
  --prefix=/home/ghansen/FASTMath
  ./b2 -j 8
  ./b2 -j 8 install

Building ParMetis

Version 4.0.3

  cd /home/ghansen/FASTMath/src/
  wget http://glaros.dtc.umn.edu/gkhome/fetch/sw/parmetis/parmetis-4.0.3.tar.gz
  tar -xvf parmetis-4.0.3.tar.gz
  cd parmetis-4.0.3

Edit metis.h to use an IDXTYPEWIDTH of 64

  cd metis/include
  vi metis.h
  #define IDXTYPEWIDTH 64

Run CMake to build parmetis

  cd /home/ghansen/FASTMath/src/parmetis-4.0.3
  cd build
  export metis=/home/ghansen/FASTMath/src/parmetis-4.0.3/metis

  cmake \
  -DCMAKE_INSTALL_PREFIX=/home/ghansen/FASTMath \
  -DMETIS_PATH=$metis \
  -DGKLIB_PATH=$metis/GKlib \
  -DCMAKE_C_COMPILER=mpicc \
  -DCMAKE_CXX_COMPILER=mpicxx \
  ..

  make -j 8
  make install

  cp libmetis/libmetis.a /home/ghansen/FASTMath/lib
  cp $metis/include/metis.h /home/ghansen/FASTMath/include

SuperLU

Version 4.3

  cd /home/ghansen/FASTMath/src
  wget http://crd-legacy.lbl.gov/~xiaoye/SuperLU/superlu_4.3.tar.gz
  tar -xvf superlu_4.3.tar.gz
  cd superlu_4.3

  mkdir /home/ghansen/FASTMath/SuperLU_4.3
  mkdir /home/ghansen/FASTMath/SuperLU_4.3/include
  mkdir /home/ghansen/FASTMath/SuperLU_4.3/lib

Edit make.inc for your machine and environment

  PLAT = _linux
  SuperLUroot = /home/ghansen/FASTMath/SuperLU_4.3
  SUPERLULIB      = $(SuperLUroot)/lib/libsuperlu_4.3.a
  TMGLIB          = libtmglib.a
  BLASDEF     = -DUSE_VENDOR_BLAS
  BLASLIB     = /gpfs/vesta-fs0/projects/FASTMath/ATPESC-2014/install/blas/19Apr11/linux-rhel_6-ppc64-gcc-4.4.6/lib/blas.a
  LIBS        = $(SUPERLULIB) $(BLASLIB)
  ARCH         = ar
  ARCHFLAGS    = cr
  RANLIB       = ranlib
  CC           = mpicc
  CFLAGS       = -DPRNTlevel=0 -O3
  NOOPTS       =
  FORTRAN      = mpif90
  FFLAGS       = -O3
  LOADER       = $(CC)
  LOADOPTS     =
  CDEFS        = -DNoChange
  MATLAB       =

Build SuperLU

  make -j 4
  cd SRC
  cp *.h /home/ghansen/FASTMath/SuperLU_4.3/include
  ar d home/ghansen/FASTMath/SuperLU_4.3/lib/libsuperlu_4.3.a dlamch.o slamch.o

The last command deletes the dlamch and slamch objects, which already exist in the LAPACK library on Vesta

Build Trilinos

Download Trilinos

  cd /home/ghansen/FASTMath/src
  git clone https://software.sandia.gov/trilinos/repositories/publicTrilinos

Download SCOREC

  cd /home/ghansen/FASTMath/src/publicTrilinos
  git clone https://github.com/SCOREC/core.git SCOREC

Configure the Trilinos build

  mkdir build
  cd build
  ./do-configure

Example do-configure script executed above

  #!/bin/sh
  #
  export TRILINOS_HOME=/home/ghansen/FASTMath/src/publicTrilinos
  BUILD_DIR=`pwd`
  INSTALL_DIR=/home/ghansen/FASTMath/Trilinos/MPI_REL
  BOOST_DIR=/home/ghansen/FASTMath
  MPI_BASE_DIR=/bgsys/drivers/V1R2M1/ppc64/comm
  NETCDF=/home/ghansen/FASTMath
  HDFDIR=/home/ghansen/FASTMath
  PARMETISDIR=/home/ghansen/FASTMath
  BLAS=/gpfs/vesta-fs0/projects/FASTMath/ATPESC-2014/install/blas/19Apr11/linux-rhel_6-ppc64-gcc-4.4.6
  LAPACK=/gpfs/vesta-fs0/projects/FASTMath/ATPESC-2014/install/lapack/3.5.0/linux-rhel_6-ppc64-gcc-4.4.6
  
  export BOOST_ROOT=$BOOST_DIR
  EXTRA_ARGS=$@
  
  cmake \
  -Wno-dev \
  -D Trilinos_EXTRA_REPOSITORIES:STRING=SCOREC \
  -D Trilinos_ENABLE_SCOREC:BOOL=ON \
  -D SCOREC_DISABLE_STRONG_WARNINGS:BOOL=ON \
  \
  -D Trilinos_CONFIGURE_OPTIONS_FILE:FILEPATH=$TRILINOS_HOME/sampleScripts/AlbanySettings.cmake \
  -D CMAKE_BUILD_TYPE:STRING=NONE \
  -D TPL_FIND_SHARED_LIBS:BOOL=OFF \
  \
  -D CMAKE_C_FLAGS:STRING="-O3 -ffast-math -funroll-loops -DCODE_MP -DDISABLE_MPIIO -DNO_FILE_LOCK -DTEMPLATE_FRIENDS_NOT_SUPPORTED -DMPICH_SKIP_MPICXX -DNDEBUG -DHAVE_CONFIG_H" \
  -D CMAKE_CXX_FLAGS:STRING="-O3 -ffast-math -funroll-loops -DCODE_MP -DDISABLE_MPIIO -DNO_FILE_LOCK -DTEMPLATE_FRIENDS_NOT_SUPPORTED -DMPICH_SKIP_MPICXX -DNDEBUG -DHAVE_CONFIG_H" \
  -D CMAKE_Fortran_FLAGS:STRING="-O3 -ffast-math -funroll-loops" \
  \
  -D TPL_ENABLE_MPI:BOOL=ON \
  -D MPI_BASE_DIR:PATH=$MPI_BASE_DIR \
  -D Trilinos_EXTRA_LINK_FLAGS:STRING="-ldl" \
  \
  -D SEACAS_ENABLE_SEACASSVDI:BOOL=OFF \
  -D Trilinos_ENABLE_SEACASFastq:BOOL=OFF \
  -D Trilinos_ENABLE_SEACASBlot:BOOL=OFF \
  -D Trilinos_ENABLE_SEACASPLT:BOOL=OFF \
  -D TPL_ENABLE_X11:BOOL=OFF \
  \
  -D Boost_INCLUDE_DIRS:PATH=$BOOST_DIR/include \
  -D Boost_LIBRARY_DIRS:PATH=$BOOST_DIR/lib \
  -D BoostLib_INCLUDE_DIRS:PATH=$BOOST_DIR/include \
  -D BoostLib_LIBRARY_DIRS:PATH=$BOOST_DIR/lib \
  -D BoostAlbLib_INCLUDE_DIRS:PATH=$BOOST_DIR/include \
  -D BoostAlbLib_LIBRARY_DIRS:PATH=$BOOST_DIR/lib \
  \
  -D TPL_ENABLE_Netcdf:BOOL=ON \
  -D Netcdf_INCLUDE_DIRS:PATH="${NETCDF}/include" \
  -D Netcdf_LIBRARY_DIRS:PATH="${NETCDF}/lib" \
  -D TPL_Netcdf_LIBRARIES:STRING="/home/ghansen/FASTMath/lib/libnetcdf.a;/home/ghansen/FASTMath/lib/libhdf5_hl.a;/home/ghansen/FASTMath/lib/libhdf5.a;/home/ghansen/FASTMath/lib/libz.a" \
  \
  -D TPL_ENABLE_HDF5:BOOL=ON \
  -D HDF5_INCLUDE_DIRS:PATH="${HDFDIR}/include" \
  -D HDF5_LIBRARY_DIRS:PATH="${HDFDIR}/lib" \
  \
  -D TPL_ENABLE_Zlib:STRING=ON \
  -D Zlib_INCLUDE_DIRS:PATH="${HDFDIR}/include" \
  -D Zlib_LIBRARY_DIRS:PATH="${HDFDIR}/lib" \
  \
  -D TPL_ENABLE_ParMETIS:STRING=ON \
  -D ParMETIS_INCLUDE_DIRS:PATH="${PARMETISDIR}/include" \
  -D ParMETIS_LIBRARY_DIRS:PATH="${PARMETISDIR}/lib" \
  \
  -D TPL_ENABLE_SuperLU:STRING=ON \
  -D SuperLU_INCLUDE_DIRS:PATH="/home/ghansen/FASTMath/SuperLU_4.3/include" \
  -D SuperLU_LIBRARY_DIRS:PATH="/home/ghansen/FASTMath/SuperLU_4.3/lib" \
  \
  -D TPL_ENABLE_BLAS:STRING=ON \
  -D TPL_BLAS_LIBRARIES:FILEPATH="$BLAS/lib/blas.a" \
  -D TPL_ENABLE_LAPACK:STRING=ON \
  -D LAPACK_LIBRARY_DIRS:FILEPATH="$LAPACK/lib" \
  \
  -D CMAKE_INSTALL_PREFIX:PATH=${INSTALL_DIR} \
  \
  $EXTRA_ARGS \
  ${TRILINOS_HOME}

Build Trilinos
```
  make -j 8
  make install
```

Build Albany

Clone the repo

  cd /home/ghansen/FASTMath/src
  git clone https://github.com/SNLComputation/Albany.git
  cd Albany
  git checkout -b tpetra origin/tpetra

Configure the Albany build

  mkdir build
  cd build
  ./do-configure

CMake configure script ./do-configure:

  #!/bin/sh
  #
  export ALB_ENABLE_SCOREC=ON
  export ALB_ENABLE_LCM=ON
  export ALB_ENABLE_MOR=OFF
  export ALB_ENABLE_SPECULATIVE=OFF
  export ALB_ENABLE_LAME=OFF
  export ALB_ENABLE_SG_MP=OFF
  export ALB_ENABLE_INSTALL=ON
  export TRILINOS_INSTALL_DIR=/home/ghansen/FASTMath/Trilinos/MPI_REL
  export ALBANY_INSTALL_DIR=/home/ghansen/FASTMath/Albany

  cmake \
  -D ALBANY_TRILINOS_DIR:FILEPATH="$TRILINOS_INSTALL_DIR" \
  -D ENABLE_LCM:BOOL=${ALB_ENABLE_LCM} \
  -D ENABLE_LCM_SPECULATIVE:BOOL=${ALB_ENABLE_SPECULATIVE} \
  -D ENABLE_SCOREC:BOOL=${ALB_ENABLE_SCOREC} \
  -D ENABLE_MOR:BOOL=${ALB_ENABLE_MOR} \
  -D ENABLE_SG_MP:BOOL=${ALB_ENABLE_SG_MP} \
  -D ENABLE_INSTALL:BOOL=${ALB_ENABLE_INSTALL} \
  -D CMAKE_INSTALL_PREFIX:FILEPATH=${ALBANY_INSTALL_DIR} \
  -D ALBANY_MPI_OPTIONS:BOOL=ON \
  -D ALBANY_MPI_EXEC:STRING="/home/ghansen/FASTMath/scripts/run_Albany" \
  -D ALBANY_MPI_EXEC_NUMPROCS_FLAG:STRING="-n" \
  -D ALBANY_MPI_EXEC_MAX_NUMPROCS:STRING="4" \
  -D ALBANY_MPI_LEADING_OPTIONS:STRING="" \
  \
  /home/ghansen/FASTMath/src/Albany

Build Albany
```
  make -j 8
  make install
```

Running the Albany regression test suite

Note that the above Albany configure script also sets up ctest to run the regression suite on BG/Q, employing the script /home/ghansen/FASTMath/scripts/run_Albany to dispatch each parallel test to the compute nodes while ctest itself runs on the frontend node. From the build directory on the frontend node, typing
```
  qsub -A FASTMath -n 4 -t 60 --mode script /home/ghansen/FASTMath/scripts/reserve.sh
  qstat
```

Get the blockid info from qstat, and now run ctest

  export MPIEXEC_BLOCKID=VST-20220-31331-32
  ctest

will run the regression suite in this manner. Note also that all the usual ctest options apply to this command.

Running a typical Albany analysis

Using Albany to perform an analysis is very similar to using any typical finite element analysis code. Two preparation steps are required: a) define the problem spatial domain and nature by meshing the parts and assemblies that participate in the analysis, and b) define the materials, properties, boundary and initial conditions and specify numerical solver parameters within the Albany input (driver) file.

Given the mesh, one decomposes the mesh into a set of subdomains for parallel execution, with one subdomain per processing element (or MPI rank).

One then constructs an xml input file for Albany that specifies how to perform the simulation on the supplied decomposed mesh files.

Reserve a run allocation, this example is for 128 processors for 30 minutes

  qsub -A FASTMath -n 128 -t 30 --mode script /home/ghansen/FASTMath/scripts/reserve.sh
  qstat

Get the blockid info from qstat and set the environment variable
```
  export MPIEXEC_BLOCKID=VST-20220-31331-32
```

Use the script /home/ghansen/FASTMath/scripts/run_Albany to dispatch the parallel job to the compute nodes.

  export PATH=/home/ghansen/FASTMath/scripts:/home/ghansen/FASTMath/bin
  run_Albany -np 128 AlbanyT input.xml

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ALCF Vesta

Instructions for 2016

Building Trilinos

Building Albany

Running the Albany regression test suite

Instructions for 2015

Instructions for 2014

Set up your build environment

Install the required Third Party Libraries (TPLs) for Trilinos

Building ZLib

Building HDF5

Building Netcdf

Building boost

Building ParMetis

SuperLU

Build Trilinos

Build Albany

Running the Albany regression test suite

Running a typical Albany analysis

Docker Image

SPACK Package

ToDo List

Albany FAQs

Capabilities

Sub-Projects

Compiling

Nightly Tests

Code HowTo

User Group Meetings

PAALS Tutorials

Wiki HowTo

Visualization Utilities

Good Commits

Python and Albany

Clone this wiki locally