Building FleCSI with stand alone Legion Gasnet

Get MPICH (currently using version 3.2 stable) http://www.mpich.org

wget http://www.mpich.org/static/downloads/3.2/mpich-3.2.tar.gz

Enable thread multiple (GASNet will call MPI multi-threaded)

./configure --prefix=/home/gshipman/local --enable-threads=multiple

Get GASNet (currently using version 1.26.4)

https://gasnet.lbl.gov

wget https://gasnet.lbl.gov/GASNet-1.26.4.tar.gz

Configure GASNet

• make sure to select an MPI interop compatible conduit such as mpi or ibv conduit.

• make sure to compile with fPIC (position independent code) as we will be linking the static libs generated from gasnet into a legion shared library

  ./configure --disable-udp --prefix= --enable-mpi-compat --enable-mpi --enable-par CC="gcc -g -fPIC" CXX="g++ -g -fPIC" MPI_CC="mpicc -g -fPIC"

Build and install GASNet:

make -j 4 VERBOSE=1 2>&1 | tee make.out && make install 

Clone Legion:

git clone https://github.com/StanfordLegion/legion.git

Checkout master:

git checkout master 

Make a build directory, say /build

mkdir build && cd build 

Run cmake (or ccmake if you prefer), the following params are useful, make sure the conduit and the gasnet path match what you configured in GASNet

cmake .. -DBUILD_SHARED_LIBS=ON -DCMAKE_INSTALL_PREFIX=<your desired install directory>  -DCMAKE_PREFIX_PATH=<gasnet install directory> -DGASNet_CONDUIT=mpi  -DLegion_USE_GASNet=ON

Now run make

make -j 4 VERBOSE=1 2>&1 | tee make.out  && make install 

You should now have liblegion.so and librealm.so in your install dir

ls -ltr <your desired install directory>

-rwxr-xr-x. 1 name name 21919984 Sep 20 11:19 librealm.so.1

-rwxr-xr-x. 1 name name 67707976 Sep 20 11:20 liblegion.so.1

Clone Third Party (to get "cereal") git clone --recursive https://github.com/losalamos/flecsi-third-party.git

Make a build dir in the top level Third Party

mkdir build && cd build

ccmake Third Party, Enable CEREAL, build

ccmake ..

Clone FleCSI

git clone --recursive https://github.com/gshipman/flecsi.git

Make a build dir in the top level FleCSI src dir

mkdir build && cd build 

Cmake FleCSI (or ccmake if you prefer)

cmake ..  -DENABLE_MPI=ON -DMPI_LIBRARY=<path to mpi install dir> -DENABLE_UNIT_TESTS=ON -DFLECSI_RUNTIME_MODEL=mpilegion -DCereal_INCLUDE_DIR=<path to cereal> -DLegion_ROOT=<path to legion install dir> 

make -j 4 VERBOSE=1 2>&1 | tee make.out 

If you receive a failure in compilation indicating that AM symbols are missing, ensure that libam.a was installed in your gasnet install path. We've seen an issue where this library is not picked up and then the Legion CMAKE build happily builds and links without it. If this is the case, you will need to rerun cmake and then rebuild Legion and install. Then rerun make here and the symbols should be picked up properly in the new librealm.so.

You can now test your installation of FleCSI with Legion. If using openmpi go to test/execution (from your Legion build dir). Run: mpirun -N 2 ./task This should pass and you should see this only once as output: mpilegion_runtime_driver started And you should also see this output: Attempting to connect 2 processors with 1 points per processor

If the driver task reports twice than you are running two distinct Legion runtimes as GASNet was not properly configured and built into the librealm.so library and hence the Legion processes cannot communicate with one another.

Enabling GASNet debug info (helpful in verifying Legion+GASNet is working with FleCSI) In your .rc (say ~/.bashrc file) export the following

export GASNET_VERBOSEENV=1
export GASNET_DEBUG_VERBOSE=1

When you run a FleCSI+Legion application you should see output such as below (note the ENV parameter output). If this is not present you may have an issue with your Legion+GASNet build

gshipman@vm1:~/local/src/flecsi-execution/build/test/execution$ mpirun -np 2 ./sprint 
[==========] Running 1 test from 1 test case.
[----------] Global test environment set-up.
[----------] 1 test from task
[ RUN      ] task.execute
[==========] Running 1 test from 1 test case.
[----------] Global test environment set-up.
[----------] 1 test from task
[ RUN      ] task.execute
ENV parameter: GASNET_FREEZE = NO                               (default)
ENV parameter: GASNET_NETWORKDEPTH = 4                          (default)
ENV parameter: GASNET_DISABLE_ARGDECODE = NO                    (default)
ENV parameter: GASNET_BACKTRACE = NO                            (default)
ENV parameter: GASNET_TMPDIR = *not set*                        (default)
ENV parameter: TMPDIR = *not set*                               (default)
ENV parameter: GASNET_BACKTRACE_TYPE = GSTACK,PSTACK,GDB,EXECINFO   (default)
ENV parameter: GASNET_FREEZE_ON_ERROR = NO                      (default)
ENV parameter: GASNET_FREEZE_SIGNAL = *not set*                 (default)