Tool to collect and report model factors (aka. fundamental performance factors) for hybrid MPI + OpenMP applications on-the-fly.
The C/C++ compiler needs to support OMPT (which is provided by icc/clang and others, but not gcc)
mkdir BUILD
cd BUILD
cmake ../
make -j8
To use clang with IntelMPI/OpenMPI/MPICH, export the following env variables first:
export MPICH_CC=clang; export MPICH_CXX=clang++;
export OMPI_CC=clang; export OMPI_CXX=clang++;
mkdir BUILD
cd BUILD
cmake ../
make -j8
The MPICH compiler wrapper on Ubuntu bricks MPI detection in cmake, work-around as following:
export MPICH_CC=clang; export MPICH_CXX=clang++;
mkdir BUILD
cd BUILD
CC=mpicc.mpich CXX=mpicxx.mpich LDFLAGS=-flto=auto cmake ../
make -j8
export OMPI_CC=clang; export OMPI_CXX=clang++;
mkdir BUILD
cd BUILD
CC=mpicc.openmpi CXX=mpicxx.openmpi cmake ../
make -j8
The Makefile accepts several configuration variables to be overwritten:
- MPI defaults to
openmpi
. This will be used as infix in most generated files - MPICC defaults to
mpicc.$(MPI)
. In combination with the MPI variable this allows easy switching of MPI on Ubuntu - MPICXX defaults to
mpicxx.$(MPI)
. In combination with the MPI variable this allows easy switching of MPI on Ubuntu - MPIRUN defaults to
mpirun.$(MPI)
. In combination with the MPI variable this allows easy switching of MPI on Ubuntu
run-test
- runs 4 tests loading the shared library tool implementationrun-test-static
- runs 4 tests that have the tool implementation statically built into the test
Depending on the system and how libomp.so is built, LD_PRELOAD and OMP_TOOL_LIBRARIES might both be necessary. Assuming a cmake build as described above, an application with OTF-CPT is executed like:
$(MPIRUN) -np 2 env OMP_NUM_THREADS=4 LD_PRELOAD=./BUILD/libOTFCPT.so OMP_TOOL_LIBRARIES=./BUILD/libOTFCPT.so ./app
At the moment, the tool supports selective instrumentation with a single pair of start/stop markers:
MPI_Pcontrol(1); // start
// region of interest
MPI_Pcontrol(0); // stop
or alternatively for OpenMP applications:
omp_control_tool(omp_control_tool_start, 0, NULL); // start
// region of interest
omp_control_tool(omp_control_tool_stop, 0, NULL); // stop
In both cases the runtime option stopped=1
should be used, see below.
The behavior of OTF-CPT can be changed with different runtime options. All
runtime options are exported as a space separated string assigned to
OTFCPT_OPTIONS
.
E.g.:
export OTFCPT_OPTIONS="verbose=1 start_stopped=1"
Flag Name | Default value | Description |
---|---|---|
stopped | 0 | Delay the start of measurement until a start marker is encountered. |
data_path | stdout | Write metric data to "<data_path>-<#procs>x<#threads>.txt". Special values are "stdout" and "stderr". Overwrites the file without checking. |
log_path | stdout | Write logging output to "<log_path>.<pid>". Special values are "stdout" and "stderr". Only relevant with verbose=1 |
verbose | 0 | Print additional statistics. |
enable | 1 | Use OTF-CPT during execution. |
- completion-wrappers.cpp - Request completion calls
- man-wrappers.cpp - Manually modified MPI wrappers with special semantics regarding handles
- gen-nb-wrappers.cpp - Generated non-blocking communication calls
- gen-wrappers.cpp - Generated MPI wrappers with simple IN or OUT semantics regarding handles
- tracking.cpp - implementation of some class functions defined in handle-data.h and tracking.h
- critical-core.cpp - OTF-CPT core functions
- ompt-critical.cpp - OMPT specific code for OTF-CPT
- mpi-critical.cpp - MPI specific code for OTF-CPT
- gen-nb-wrappers.w
- gen-wrappers.w
external/wrap/wrap.py -s -n gen-wrappers.w -o gen-wrappers.cpp
- Joachim Protze, Fabian Orland, Kingshuk Haldar, Thore Koritzius, Christian Terboven: On-the-Fly Calculation of Model Factors for Multi-paradigm Applications. Euro-Par 2022
- Joachim Jenke, Michael Knobloch, Marc-André Hermanns, Simon Schwitanski: A Shim Layer for Transparently Adding Meta Data to MPI Handles. EuroMPI 2023