This library provides a high-level API for writing and reading scientific data according to openPMD. It allows one to build logical file structures which drive scientific I/O libraries such as HDF5 and ADIOS through a common, intuitive interface. Where supported, openPMD-api implements both serial and MPI parallel I/O capabilities.
#include <openPMD/openPMD.hpp>
#include <iostream>
// ...
auto s = openPMD::Series("samples/git-sample/data%T.h5", openPMD::AccessType::READ_ONLY);
for( auto const& i : s.iterations ) {
std::cout << "Iteration: " << i.first << "\n";
for( auto const& m : i.second.meshes ) {
std::cout << " Mesh '" << m.first << "' attributes:\n";
for( auto const& val : m.second.attributes() )
std::cout << " " << val << '\n';
}
for( auto const& p : i.second.particles ) {
std::cout << " Particle species '" << p.first << "' attributes:\n";
for( auto const& val : p.second.attributes() )
std::cout << " " << val << '\n';
}
}
import openpmd_api
# ...
series = openpmd_api.Series("samples/git-sample/data%T.h5", openPMD.Access_Type.read_only)
for k_i, i in series.iterations.items():
print("Iteration: {0}".format(k_i))
for k_m, m in i.meshes.items():
print(" Mesh '{0}' attributes:".format(k_m))
for a in m.attributes:
print(" {0}".format(a))
for k_p, p in i.particles.items():
print(" Particle species '{0}' attributes:".format(k_p))
for a in p.attributes:
print(" {0}".format(a))Curious? Our manual shows full read & write examples, both serial and MPI-parallel!
Required:
- CMake 3.11.0+
- C++11 capable compiler, e.g. g++ 4.8+, clang 3.9+, VS 2015+
Shipped internally in share/openPMD/thirdParty/:
- MPark.Variant 1.4.0+ (BSL-1.0)
- Catch2 2.6.1+ (BSL-1.0)
- pybind11 2.3.0+ (new BSD)
- NLohmann-JSON 3.7.0+ (MIT)
I/O backends:
while those can be built either with or without:
- MPI 2.1+, e.g. OpenMPI 1.6.5+ or MPICH2
Optional language bindings:
- Python:
- Python 3.5 - 3.8
- pybind 2.3.0+
- numpy 1.15+
- mpi4py 2.1+
Choose one of the install methods below to get started:
# optional: +python +adios1 +adios2 -hdf5 -mpi
spack install openpmd-api
spack load -r openpmd-api
# optional: OpenMPI support =*=mpi_openmpi*
# optional: MPICH support =*=mpi_mpich*
conda install -c conda-forge openpmd-api
Behind the scenes, this install method compiles from source against the found installations of HDF5, ADIOS1, ADIOS2, and/or MPI (in system paths, from other package managers, or loaded via a module system, ...). The current status for this install method is experimental. Please feel free to report how this works for you.
# we need pip 19 or newer
# optional: --user
python3 -m pip install -U pip
# optional: --user
python3 -m pip install openpmd-apior with MPI support:
# optional: --user
python3 -m pip install -U pip
# optional: --user
openPMD_USE_MPI=ON python3 -m pip install openpmd-api --no-binary openpmd-api
openPMD can then be installed using CMake:
git clone https://github.com/openPMD/openPMD-api.git
mkdir openPMD-api-build
cd openPMD-api-build
# optional: for full tests, with unzip
../openPMD-api/.travis/download_samples.sh
# for own install prefix append:
# -DCMAKE_INSTALL_PREFIX=$HOME/somepath
# for options append:
# -DopenPMD_USE_...=...
# e.g. for python support add:
# -DopenPMD_USE_PYTHON=ON -DPYTHON_EXECUTABLE=$(which python)
cmake ../openPMD-api
cmake --build .
# optional
ctest
# sudo might be required for system paths
cmake --build . --target installThe following options can be added to the cmake call to control features.
CMake controls options with prefixed -D, e.g. -DopenPMD_USE_MPI=OFF:
| CMake Option | Values | Description |
|---|---|---|
openPMD_USE_MPI |
AUTO/ON/OFF | Parallel, Multi-Node I/O for clusters |
openPMD_USE_HDF5 |
AUTO/ON/OFF | HDF5 backend (.h5 files) |
openPMD_USE_ADIOS1 |
AUTO/ON/OFF | ADIOS1 backend (.bp files up to version BP3) |
openPMD_USE_ADIOS2 |
AUTO/ON/OFF | ADIOS2 backend (.bp files in BP3, BP4 or higher) |
openPMD_USE_PYTHON |
AUTO/ON/OFF | Enable Python bindings |
openPMD_USE_INVASIVE_TESTS |
ON/OFF | Enable unit tests that modify source code 1 |
openPMD_USE_VERIFY |
ON/OFF | Enable internal VERIFY (assert) macro independent of build type 2 |
PYTHON_EXECUTABLE |
(first found) | Path to Python executable |
1 e.g. changes C++ visibility keywords, breaks MSVC 2 this includes most pre-/post-condition checks, disabling without specific cause is highly discouraged
Additionally, the following libraries are shipped internally. The following options allow to switch to external installs:
| CMake Option | Values | Library | Version |
|---|---|---|---|
openPMD_USE_INTERNAL_VARIANT |
ON/OFF | MPark.Variant | 1.4.0+ |
openPMD_USE_INTERNAL_CATCH |
ON/OFF | Catch2 | 2.6.1+ |
openPMD_USE_INTERNAL_PYBIND11 |
ON/OFF | pybind11 | 2.3.0+ |
openPMD_USE_INTERNAL_JSON |
ON/OFF | NLohmann-JSON | 3.7.0+ |
By default, this will build as a shared library (libopenPMD.[so|dylib|dll]) and installs also its headers.
In order to build a static library, append -DBUILD_SHARED_LIBS=OFF to the cmake command.
You can only build a static or a shared library at a time.
By default, the Release version is built.
In order to build with debug symbols, pass -DCMAKE_BUILD_TYPE=Debug to your cmake command.
By default, tests and examples are built.
In order to skip building those, pass -DBUILD_TESTING=OFF or -DBUILD_EXAMPLES to your cmake command.
The install will contain header files and libraries in the path set with -DCMAKE_INSTALL_PREFIX.
If your project is using CMake for its build, one can conveniently use our provided openPMDConfig.cmake package which is installed alongside the library.
First set the following environment hint if openPMD-api was not installed in a system path:
# optional: only needed if installed outside of system paths
export CMAKE_PREFIX_PATH=$HOME/somepath:$CMAKE_PREFIX_PATHUse the following lines in your project's CMakeLists.txt:
# supports: COMPONENTS MPI NOMPI HDF5 ADIOS1 ADIOS2
find_package(openPMD 0.9.0 CONFIG)
if(openPMD_FOUND)
target_link_libraries(YourTarget PRIVATE openPMD::openPMD)
endif()Alternatively, add the openPMD-api repository source directly to your project and use it via:
add_subdirectory("path/to/source/of/openPMD-api")
target_link_libraries(YourTarget PRIVATE openPMD::openPMD)If your (Linux/OSX) project is build by calling the compiler directly or uses a manually written Makefile, consider using our openPMD.pc helper file for pkg-config which are installed alongside the library.
First set the following environment hint if openPMD-api was not installed in a system path:
# optional: only needed if installed outside of system paths
export PKG_CONFIG_PATH=$HOME/somepath/lib/pkgconfig:$PKG_CONFIG_PATHAdditional linker and compiler flags for your project are available via:
# switch to check if openPMD-api was build as static library
# (via BUILD_SHARED_LIBS=OFF) or as shared library (default)
if [ "$(pkg-config --variable=static openPMD)" == "true" ]
then
pkg-config --libs --static openPMD
# -L/usr/local/lib -L/usr/lib/x86_64-linux-gnu/openmpi/lib -lopenPMD -pthread /usr/lib/libmpi.so -pthread /usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi_cxx.so /usr/lib/libmpi.so /usr/lib/x86_64-linux-gnu/hdf5/openmpi/libhdf5.so /usr/lib/x86_64-linux-gnu/libsz.so /usr/lib/x86_64-linux-gnu/libz.so /usr/lib/x86_64-linux-gnu/libdl.so /usr/lib/x86_64-linux-gnu/libm.so -pthread /usr/lib/libmpi.so -pthread /usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi_cxx.so /usr/lib/libmpi.so
else
pkg-config --libs openPMD
# -L${HOME}/somepath/lib -lopenPMD
fi
pkg-config --cflags openPMD
# -I${HOME}/somepath/includeopenPMD-api is developed by many people. It was initially started by the Computational Radiation Physics Group at HZDR as successor to libSplash, generalizing the successful HDF5 & ADIOS1 implementations in PIConGPU. The following people and institutions contributed to openPMD-api:
- Axel Huebl (HZDR, now LBNL): project lead, releases, documentation, automated CI/CD, Python bindings, installation & packaging, prior reference implementations
- Fabian Koller (HZDR): initial library design and implementation with HDF5 & ADIOS1 backend
- Franz Poeschel (HZDR): added JSON & ADIOS2 backend, data staging/streaming
- Carsten Fortmann-Grote (EU XFEL GmbH, now MPI-EvolBio): inital Python unit tests
openPMD-api stands on the shoulders of giants and we are grateful for the following projects included as direct dependencies:
- ADIOS1 and ADIOS2 by PI S. Klasky (ORNL), team, collaborators and contributors
- Catch2 by Phil Nash, Martin Hořeňovský and contributors
- HDF5 by the HDF group and community
- json by Niels Lohmann and contributors
- pybind11 by Wenzel Jakob (EPFL) and contributors
- all contributors to the evolution of modern C++ and early library preview developers, e.g. Michael Park (Facebook)
- the CMake build system and contributors
- packaging support by the conda-forge, PyPI and Spack communities, among others
- the openPMD-standard by Axel Huebl (HZDR, now LBNL) and contributors