New feature: re-architecture

[TysonRayJones]

contributor: Tyson Jones
release: v4

The entire QuEST backend has been re-architectured to be more modular, contain less code-duplication, contain less boilerplate, improve the defensive design, and enable a variety of new facilities. This involves multiple changes

• The entire backend is now compiled in C++11, but the API remains C99 or C++11 agnostic

  • developer benefits

    • C++ enables faster backend development by leveraging features like function and operator overloading, use of vectors, optional arguments, exceptions, namespaces, templates, objects, standard C++ libraries, etc.
    • All backend deployments (CPU, GPU, distributed) are now written in a consistent language (C++), whereas the old QuEST uses C++11 for the GPU backend and C99 elsewhere.
    • File suffix gives a clue as to whether user compiler will parse the file; .cpp and .hpp files are only ever parsed by the backend compilation (by C++ and hardware-accelerated variants), while .h files are parsed by both user and backend compilers.

  • build benefits

    • Compiling is simpler and easier. All QuEST source files are individually compiled in C++, and only the user's C code needs C compilation, before final linking. It is harder to erroneously use an incompatible compiler for a particular backend file.

  • user benefits

    • The API can include C++-only function overloads to give C++ users a cleaner interface.

• The QuEST file structure has been more modularised, via a compromise of linux kernel convention and this proposal. Code is semantically grouped into small files. E.g. qureg.h declares only the Qureg API, environment.h declares only the QuESTEnv functions, communication.cpp handles all MPI communication, etc. Internally-public funtions in these files are prefixed with qureg_, env_, comm_, etc (though these may be refactored to C++ namespaces: qureg:: etc). Code containing compiler-specific pragmas (always in guards; see next item) are isolated in separate folders (gpu/, omp/, mpi/).

  • developer benefits

    • More modularised code structure, and inter-dependencies are much clearer.
    • It is easier to find the source file of a called function, and browser functions therein.

  • build benefits

    • Since each file is independently compiled (before a final link), it's easier to find the source of compilation errors.
    • Specialised compilers (e.g. nvcc, mpicc, or OpenMP-compatible ones) need only compile a single source file. I.e. nvcc compiles gpu.cpp, mpicc compiles communication.cpp, and the openmp flags are given only to the compiler of cpu.cpp. This means non-special source files (all others than aforementioned) are safely compiled with non-special compilers, reducing chance of file-specific compatibility issues. For example, nvcc never compiles cpu.cp nor witnesses the openmp directives therein.

  • security benefits

    • As a result of the previous build benefit, we eliminate the risk of undefined behaviour of a specialised compiler processing another specialised source.

• The backend-specific source files (gpu.cpp et al, cpu.cpp and communication.cpp) can always be compiled by generic (non-accelerated) compilers, which simply disables their backend facilities using pre-compiler guards.

  • build benefits

    • No matter the user's target hardware, all source files are always compiled. There is ergo less (or no) control flow / file switching in the build scripts.
    • Using the wrong compiler for an intended hardware backend is safely detected and an illustrative QuEST pre-compiler error thrown, rather than a generic confused compiler error.

  • user benefits

    • It is trivial for a user to manually compile QuEST (foregoing CMake or GNUmake) against their own code; they can use the same compiler/flags for all source files. We demonstrate this with a compile.sh script supporting all deployment modes 😆

• Source code separates data communication and the subsequent data processing logic, as per the distributed manuscript

  • developer benefits

    • More modularised code, less code duplication.
    • Easier to integrate external libraries in distributed settings - simply invoke them in the post-communication data processing logic.
    • Permutations of deployment settings (local GPU vs distributed GPU) do not require bespoke code.

  • build benefits

    • Multiple deployment modes are enabled by merely separately compiling their source files with their separate specialised compilers. There is no need to use CUDA-aware MPI compilers, or vice versa, to enable multi-GPU.

  • feature benefits

    • All data-processing backends (e.g. serial, multithreaded CPU or GPU accelerated) can be distributed, and hybridised!
    • Both tightly-coupled and loosely-coupled multi-GPU simulation is ergo possible, simple and elegant to implement.

• Communication code run-time checks for CUDA-aware (or ROCm-aware) MPI (and compatible GPUs), otherwise falls back to routing GPU memory through CPU before communication.

  • developer benefits

    • Efficient tightly-coupled multi-GPU simulation is achieved for free by invoking MPI communication directly between GPU memories (when supported by GPUs and MPI).

  • security benefits

    • Building with non-GPU-MPI hybrid compilers don't cause any build or runtime bugs or errors, nor disable communication or GPU acceleration.

  • user benefits

    • Users do not need to source a GPU-aware MPI compiler in order to simultaneously use both facilities.
    • Users can accelerate their multi-GPU simulation by merely changing their MPI compiler to a GPU-aware one (if GPUs are compatible) with no other setting change.

[TysonRayJones]

Implemented since deb68f9