Building on Amos with bgclang

Install bgclang on the RPI BGQ

Get the bgclang rpms

From the website http://www.mcs.anl.gov/~hfinkel/bgclang/ locally download the following rpms:

bgclang-binutils-r284961-stable-1-5.ppc64.rpm
bgclang-compiler-rt-r284961-stable-1-5.ppc64.rpm
bgclang-gdb-r284961-stable-1-5.ppc64.rpm
bgclang-libcxx-r284961-stable-1-5.ppc64.rpm
bgclang-mpich3-r284961-stable-1-5.ppc64.rpm
bgclang-r284961-stable-1-5.ppc64.rpm
bgclang-stage1-3.4-1.ppc64.rpm
bgclang-stage1-libcxx-3.4-2.ppc64.rpm
bgclang-stage2-3.5.1-1.ppc64.rpm
bgclang-stage2-libcxx-3.5.1-1.ppc64.rpm
bgclang-stage3-libcxx-r266865-1.ppc64.rpm
bgclang-stage3-r266865-1.ppc64.rpm
toolchain-fixup-4.7.2-4.ppc64.rpm
vpkg-bin-sh-1-1.ppc64.rpm

scp the rpms to the BGQ

scp *.*.rpm PGESgnzw:lp01.cci.rpi.edu:~scratch/bgq/bgclang/rpm
scp the rpms to the BGQ

Install the bgclang packages

First specify the directories:

export prefix=/gpfs/u/home/PGES/PGESgnzw/scratch/bgq/bgclang
export dbdir=/gpfs/u/home/PGES/PGESgnzw/scratch/bgq/bgclang/rpmdb

Install the vpkg package:

rpm -Uhv --dbpath $dbdir --prefix $prefix \
vpkg-bin-sh-1-1.ppc64.rpm

Install the stage1 packages:

rpm -Uhv --dbpath $dbdir --prefix $prefix \
bgclang-stage1-libcxx-3.4-2.ppc64.rpm \
bgclang-stage1-3.4-1.ppc64.rpm

Install the stage2 packages:

rpm -Uhv --dbpath $dbdir --prefix $prefix \
bgclang-stage2-3.5.1-1.ppc64.rpm \
bgclang-stage2-libcxx-3.5.1-1.ppc64.rpm

Install the stage3 packages:

rpm -Uhv --dbpath $dbdir --prefix $prefix \
bgclang-stage3-r266865-1.ppc64.rpm \
bgclang-stage3-libcxx-r266865-1.ppc64.rpm

Install the toolchain fixup package:

rpm -Uhv --dbpath $dbdir --prefix $prefix \
toolchain-fixup-4.7.2-4.ppc64.rpm

Install the bin utils

rpm -Uhv --dbpath $dbdir --prefix $prefix \
bgclang-binutils-r284961-stable-1-5.ppc64.rpm

Install gdb:

rpm -Uhv --dbpath $dbdir --prefix $prefix \
bgclang-gdb-r284961-stable-1-5.ppc64.rpm

Install bgclang:

rpm -Uhv --dbpath $dbdir --prefix $prefix \
bgclang-r284961-stable-1-5.ppc64.rpm

Install compiler rt:

rpm -Uhv --dbpath $dbdir --prefix $prefix \
bgclang-compiler-rt-r284961-stable-1-5.ppc64.rpm

Install lib cxx:

rpm -Uhv --dbpath $dbdir --prefix $prefix \
bgclang-libcxx-r284961-stable-1-5.ppc64.rpm

Install mpich:

rpm -Uhv --dbpath $dbdir --prefix $prefix \
bgclang-mpich3-r284961-stable-1-5.ppc64.rpm

Third party library installation for Trilinos

Setup the environment

export PATH=/gpfs/u/home/PGES/PGESgnzw/scratch/bgq/bgclang/bin:$PATH
export PATH=/gpfs/u/home/PGES/PGESgnzw/scratch/bgq/bgclang/mpi/bgclang/bin:$PATH
export tpls=/gpfs/u/home/PGES/PGESgnzw/scratch/bgq/tpls/install

Install zlib

Download zlib from:
https://zlib.net/
scp it to the BGQ.

scp zlib-1.2.8.tar.gz [email protected]:~/scratch/bgq/tpls

On the BGQ, install

cd ~/scratch/bgq/tpls
tar -xzvf zlib-1.2.8.tar.gz
cd zlib-1.2.8
CC=mpiclang CFLAGS="-O3 -fPIC" ./configure --64 --prefix=${tpls} --static

Install boost

Download boost (v 1.65.1) from:
http://www.boost.org/users/download/
scp it to the BGQ

scp boost_1_65_1.tar.gz [email protected]:~/scratch/bgq/tpls

On the BGQ, install

cd ~/scratch/bgq/tpls
tar -xzvf boost_1_65_1.tar.gz
cd boost_1_65_1
echo "using clang : /gpfs/u/home/PGES/PGESgnzw/scratch/bgq/bgclang/bin/bgclang ;" >> ~/user-config.jam
echo "using mpi : /gpfs/u/home/PGES/PGESgnzw/scratch/bgq/bgclang/mpi/bgclang/bin/mpiclang++11 ;" >> ~/user-config.jam
./bootstrap.sh --with-toolset=clang --with-libraries=signals,regex,filesystem,system,mpi,serialization,thread,program_options,exception --prefix=${tpls}
./b2 link=static
./b2 install
cd $tpls/lib
rm *.so *.so.*

Copy fortran math libraries

Download the files found here (for-bgq.zip)

scp the zip file to the BGQ

scp for-bgq.zip [email protected]:~/scratch/bgq/tpls

On the BGQ, unzip the files and move the libraries into the install directory

cd ~/scratch/bgq/tpls
unzip for-bgq.zip
cd for-bgq
mv liblapack-gnu472-O3-unroll-nofastmath.a ../install/lib
mv librefblas-gnu472-O3-unroll-nofastmath.a ../install/lib

Install Trilinos

Clone Trilinos

cd ~/scratch/bgq
mkdir trilinos
cd trilinos
git clone https://github.com/trilinos/Trilinos.git src
mkdir build

Build Trilinos

do-config-trilinos-lite.sh

Here is the Trilinos CMake configuration I used. For different installs, the first few lines of this file will need to be modified to point to the correct location.

#!/bin/bash
# Modify these paths for your system.
TOOLS=/gpfs/u/home/PGES/PGESgnzw/scratch/bgq/tpls/install
MPIDIR=/gpf/u/home/PGES/PGESgnzw/scratch/bgq/bgclang/mpi/bgclang/bin
TRIL_INSTALL_DIR=/gpfs/u/home/PGES/PGESgnzw/scratch/bgq/trilinos/install
LAPACKDIR=$TOOLS
BOOSTDIR=$TOOLS
ZLIBDIR=$TOOLS
PARMETISDIR=$TOOLS

cmake \
\
 -D CMAKE_INSTALL_PREFIX=$TRIL_INSTALL_DIR \
 -D CMAKE_BUILD_TYPE=NONE \
 -D CMAKE_C_COMPILER=mpiclang \
 -D CMAKE_CXX_COMPILER=mpiclang++11 \
 -D CMAKE_C_FLAGS="-O2 -g" \
 -D CMAKE_CXX_FLAGS="-O2 -g -Wno-deprecated-declarations -Wno-sign-compare" \
 -D CMAKE_VERBOSE_MAKEFILE=OFF \
 -D BUILD_SHARED_LIBS=OFF \
 -D CMAKE_INSTALL_RPATH_USE_LINK_PATH=True \
\
 -D Trilinos_WARNINGS_AS_ERRORS_FLAGS="" \
 -D Trilinos_ENABLE_Fortran=OFF \
 -D Trilinos_EXTRA_LINK_FLAGS="-ldl -lgfortran" \
 -D Trilinos_ENABLE_ALL_PACKAGES=OFF \
 -D Trilinos_ENABLE_EXPLICIT_INSTANTIATION=ON \
 -D Trilinos_ENABLE_SECONDARY_TESTED_CODE=ON \
 -D Trilinos_DISABLE_ENABLED_FORWARD_DEP_PACKAGES=ON \
\
 -D Trilinos_ENABLE_Shards=ON \
 -D Trilinos_ENABLE_Sacado=ON \
 -D Trilinos_ENABLE_Epetra=ON \
 -D Trilinos_ENABLE_EpetraExt=ON \
 -D Trilinos_ENABLE_Ifpack=ON \
 -D Trilinos_ENABLE_AztecOO=ON \
 -D Trilinos_ENABLE_Amesos=ON \
 -D Trilinos_ENABLE_Anasazi=ON \
 -D Trilinos_ENABLE_Belos=ON \
 -D Trilinos_ENABLE_ML=ON \
 -D Trilinos_ENABLE_MiniTensor=ON \
 -D Trilinos_ENABLE_NOX=ON \
 -D Trilinos_ENABLE_Stratimikos=ON \
 -D Trilinos_ENABLE_Thyra=ON \
 -D Trilinos_ENABLE_Rythmos=ON \
 -D Trilinos_ENABLE_Piro=ON \
 -D Trilinos_ENABLE_Teko=ON \
 -D Trilinos_ENABLE_Ifpack2=ON \
 -D Trilinos_ENABLE_MueLu=ON \
 -D Trilinos_ENABLE_Intrepid2=ON \
 -D Trilinos_ENABLE_Intrepid=ON \
\
 -D Trilinos_ENABLE_Tpetra=ON \
 -D Tpetra_INST_INT_LONG_LONG=ON \
 -D Tpetra_INST_INT_LONG=OFF \
 -D Tpetra_INST_INT_INT=OFF \
 -D Tpetra_INST_DOUBLE=ON \
 -D Tpetra_INST_FLOAT=OFF \
 -D Tpetra_INST_COMPLEX_FLOAT=OFF \
 -D Tpetra_INST_COMPLEX_DOUBLE=OFF \
 -D Tpetra_INST_INT_UNSIGNED=OFF \
 -D Tpetra_INST_INT_UNSIGNED_LONG=OFF \
\
 -D Trilinos_ENABLE_Zoltan2=ON \
 -D Zoltan_ENABLE_ULONG_IDS=ON \
 -D ZOLTAN_BUILD_ZFDRIVE=OFF \
\
 -D Trilinos_ENABLE_Teuchos=ON \
 -D Teuchos_ENABLE_COMPLEX=OFF \
 -D Teuchos_ENABLE_LONG_LONG_INT=ON \
\
 -D Trilinos_ENABLE_Phalanx=ON \
 -D Phalanx_INDEX_SIZE_TYPE="KOKKOS" \
\
 -D Trilinos_ENABLE_Kokkos=ON \
 -D Kokkos_ENABLE_Serial=ON \
 -D Kokkos_ENABLE_OpenMP=OFF \
 -D Kokkos_ENABLE_Pthread=OFF \
\
 -D Trilinos_ENABLE_Amesos2=ON \
 -D Amesos2_ENABLE_KLU2=ON \
 -D Amesos2_ENABLE_SuperLUDist=OFF \
 -D Amesos2_ENABLE_Epetra=OFF \
\
 -D Trilinos_ENABLE_STKIO=OFF \
 -D Trilinos_ENABLE_STKMesh=OFF \
 -D Trilinos_ENABLE_Stokhos=OFF \
 -D Trilinos_ENABLE_SEACAS=OFF \
 -D Trilinos_ENABLE_SEACASIoss=OFF \
 -D Trilinos_ENABLE_SEACASExodus=OFF \
 -D Trilinos_ENABLE_SEACASFastq=OFF \
 -D Trilinos_ENABLE_SEACASBlot=OFF \
 -D Trilinos_ENABLE_SEACASPLT=OFF \
\
 -D TPL_ENABLE_BLAS=ON \
 -D TPL_BLAS_LIBRARIES="$LAPACKDIR/lib/librefblas-gnu472-O3-unroll-nofastmath.a" \
\
 -D TPL_ENABLE_LAPACK=ON \
 -D TPL_LAPACK_LIBRARIES="$LAPACKDIR/lib/liblapack-gnu472-O3-unroll-nofastmath.a" \
\
 -D TPL_ENABLE_Zlib=ON \
 -D Zlib_INCLUDE_DIRS="$ZLIBDIR/include" \
 -D Zlib_LIBRARY_DIRS="$ZLIBDIR/lib" \
\
 -D TPL_ENABLE_ParMETIS=ON \
 -D ParMETIS_INCLUDE_DIRS="$PARMETISDIR/include" \
 -D ParMETIS_LIBRARY_DIRS="$PARMETISDIR/lib" \
\
 -D TPL_ENABLE_METIS=ON \
 -D METIS_INCLUDE_DIRS="$PARMETISDIR/include" \
 -D METIS_LIBRARY_DIRS="$PARMETISDIR/lib" \
\
 -D TPL_ENABLE_Boost=ON \
 -D Boost_INCLUDE_DIRS="$BOOSTDIR/include" \
 -D Boost_LIBRARY_DIRS="$BOOSTDIR/lib" \
 -D TPL_ENABLE_BoostLib=ON \
 -D BoostLib_INCLUDE_DIRS="$BOOSTDIR/include" \
 -D BoostLib_LIBRARY_DIRS="$BOOSTDIR/lib" \
\
 -D TPL_ENABLE_MPI=ON \
 -D MPI_BASE_DIR=$MPIDIR \
\
 -D TPL_ENABLE_Netcdf=OFF \
 -D TPL_ENABLE_HDF5=OFF \
 -D TPL_ENABLE_Matio=OFF \
 -D TPL_ENABLE_X11=OFF \
\
../src \
2>&1 | tee config_log

Install Trilinos

On the BGQ, install

cd ~/scratch/bgq/trilinos/build
chmod +x do-config-trilinos.sh
./do-config-trilinos.sh
make -j 4
make install

Install SCOREC/core

clone core

cd ~/scratch/bgq
git clone https://github.com/SCOREC/core.git
cd core
mkdir build

do-config-scorec.sh

All appropriate file paths will need to be modified

cmake .. \
-DCMAKE_C_COMPILER:FILEPATH=mpiclang \
-DCMAKE_CXX_COMPILER:FILEPATH=mpiclang++11 \
-DCMAKE_INSTALL_PREFIX:PATH=/gpfs/u/home/PGES/PGESgnzw/scratch/bgq/core/install \
-DBUILD_SHARED_LIBS:BOOL=OFF \
-DENABLE_ZOLTAN:BOOL=ON \
-DENABLE_STK:BOOL=ON \
-DENABLE_STK_MESH:BOOL=OFF \
-DTrilinos_PREFIX:PATH=/gpfs/u/home/PGES/PGESgnzw/scratch/bgq/trilinos/install \
-DZOLTAN_PREFIX:PATH=/gpfs/u/home/PGES/PGESgnzw/scratch/bgq/trilinos/install \
-DPARMETIS_PREFIX:PATH=/gpfs/u/home/PGES/PGESgnzw/scratch/bgq/tpls/install \
-DCMAKE_SKIP_RPATH:BOOL=ON \
2>&1 | tee config_log

Install SCOREC/core

On the BGQ

cd ~/scratch/bgq/core/build
chmod +x do-config-scorec.sh
./do-config-scorec.sh
make -j 4
make install

Install Albany

Clone Albany

cd ~/scratch/bgq
git clone https://github.com/gahansen/Albany albany
cd albany
mkdir build

do-config-albany.sh

Here is my Albany CMake configuration script. The first two CMake option will need to be modified to point to the correct location.

#!/bin/sh

cmake \
 -D ALBANY_TRILINOS_DIR:FILEPATH=/gpfs/u/home/PGES/PGESgnzw/scratch/bgq/trilinos/install \
 -D ALBANY_SCOREC_DIR:FILEPATH=/gpfs/u/home/PGES/PGESgnzw/scratch/bgq/core/install \
 -D ENABLE_LCM:BOOL=ON \
 -D ENABLE_CTM:BOOL=OFF \
 -D ENABLE_AMP:BOOL=OFF \
 -D ENABLE_ANISO:BOOL=OFF \
 -D ENABLE_OMEGA_H:BOOL=OFF \
 -D ENABLE_QCAD:BOOL=OFF \
 -D ENABLE_LCM_SPECULATIVE:BOOL=OFF \
 -D ENABLE_LAME:BOOL=OFF \
 -D CMAKE_VERBOSE_MAKEFILE:BOOL=OFF \
 -D ENABLE_DEBUGGING:BOOL=OFF \
 -D ENABLE_CHECK_FPE:BOOL=OFF \
 -D ENABLE_HYDRIDE:BOOL=OFF \
 -D ENABLE_SCOREC:BOOL=ON \
 -D ENABLE_FELIX:BOOL=OFF \
 -D ENABLE_MOR:BOOL=OFF \
 -D ENABLE_ALBANY_CI:BOOL=OFF \
 -D ENABLE_ASCR:BOOL=OFF \
 -D ENABLE_64BIT_INT:BOOL=ON \
 -D ENABLE_ALBANY_EPETRA_EXE:BOOL=OFF \
 .. \
 2>&1 | tee config_log

Install Albany

cd ~/scratch/bgq/albany/build
chmod +x do-config-albany.sh
./do-config-albany.sh
make -j 4
make install