Building FleCSI with stand alone Legion Gasnet

This assumes you have an MPI configured for your system as GASNet and FleCSI will require it for this configuration. Try open-mpi http://www.open-mpi.org

Get GASNet (currently using version 1.26.4)

https://gasnet.lbl.gov

wget https://gasnet.lbl.gov/GASNet-1.26.4.tar.gz

Configure GASNet

• make sure to select an MPI interop compatible conduit such as mpi or ibv conduit.

• make sure to compile with fPIC (position independent code) as we will be linking the static libs generated from gasnet into a legion shared library

  ./configure --disable-udp --prefix= --enable-mpi-compat --enable-mpi --enable-par CC="gcc -g -fPIC" CXX="g++ -g -fPIC" MPI_CC="mpicc -g -fPIC"

Build and install GASNet:

make -j 4 VERBOSE=1 2>&1 | tee make.out && make install 

Clone Legion:

git clone https://github.com/StanfordLegion/legion.git

Checkout master:

git checkout master 

Make a build directory, say /build

mkdir build && cd build 

Run cmake (or ccmake if you prefer), the following params are useful, make sure the conduit and the gasnet path match what you configured in GASNet

cmake .. -DBUILD_SHARED_LIBS=ON -DCMAKE_INSTALL_PREFIX=<your desired install directory>  -DCMAKE_PREFIX_PATH=<gasnet install directory> -DGASNet_CONDUIT=mpi  -DLegion_USE_GASNet=ON

Now run make

make -j 4 VERBOSE=1 2>&1 | tee make.out  && make install 

You should now have liblegion.so and librealm.so in your install dir

ls -ltr <your desired install directory>

-rwxr-xr-x. 1 name name 21919984 Sep 20 11:19 librealm.so.1

-rwxr-xr-x. 1 name name 67707976 Sep 20 11:20 liblegion.so.1

Clone FleCSI

git clone --recursive https://github.com/gshipman/flecsi.git

Make a build dir in the top level FleCSI src dir

mkdir build && cd build 

Cmake FleCSI (or ccmake if you prefer)

cmake .. -DCMAKE_PREFIX_PATH=<path to legion install dir> -DENABLE_MPI=ON -DMPI_LIBRARY=<path to mpi lib>/libmpi.so -DENABLE_UNIT_TESTS=ON -DFLECSI_RUNTIME_MODEL=mpilegion  


make -j 4 VERBOSE=1 2>&1 | tee make.out 

If you receive a failure in compilation indicating that AM symbols are missing, ensure that libam.a was installed in your gasnet install path. We've seen an issue where this library is not picked up and then the Legion CMAKE build happily builds and links without it. If this is the case, you will need to rerun cmake and then rebuild Legion and install. Then rerun make here and the symbols should be picked up properly in the new librealm.so.

You can now test your installation of FleCSI with Legion. If using openmpi go to test/execution (from your Legion build dir). Run: mpirun -N 2 ./task This should pass and you should see this only once as output: mpilegion_runtime_driver started And you should also see this output: Attempting to connect 2 processors with 1 points per processor

If the driver task reports twice than you are running two distinct Legion runtimes as GASNet was not properly configured and built into the librealm.so library and hence the Legion processes cannot communicate with one another.