-
Notifications
You must be signed in to change notification settings - Fork 117
Parallel MODFLOW User Guide
This guide describes the parallel capabilities in MODFLOW. Because these capabilities have recently been developed, this will be a living document for now.
The parallelization of MODFLOW is based on decomposition of the simulation domain and distributed memory computing with the message passing interface (MPI). The parallel run mode can be used to speed up a simulation run over a large spatial domain. The size and topology of the simulated domain largely determine the efficiency of the speedup. A first general rule is that the ratio between the number of nodes (unknowns) in the domain and the targeted number of CPU cores should not go below 25000 nodes per core. Another application is for cases where the model does not fit into the memory available to a single processor. This is typical for the simulation of a continental-scale area.
Parallel MODFLOW has been designed as a generic component that is targeted to be used for all hydrologic models in the MODFLOW 6 simulator. It is (near) feature complete: every package and formulation in the serial simulation is available in a parallel setup. The few exceptions to this rule will be documented here. The currently supported process models are GWF, GWT, and GWE.
All packages and features supported, except for
- the GNC package on GWF-GWF exchanges that connects two models that are being run in parallel. A more convenient alternative is to set the XT3D option to improve the flow calculation at the interface.
- ...
All packages and features supported, except for
- MVT the solute transport mover across models
All packages and features supported, except for
- MVE the energy transport mover across models
The parallel run mode is available as part of a separate executable that extends the serial program. It can be downloaded or built from sources depending on the target OS and the available software environment. There are two major external dependencies to the program that need to be available on the system: the MPI library for communication between the processes and the PETSc library (https://petsc.org/release/) for the parallel iterative solution.
A distribution for Windows (win64par.zip) is made available as part of the nightly build. It can be downloaded from this page. For users of FloPy, it can be installed with the get-modflow utility as well with the command
get-modflow /path/to/bin --repo modflow6-nightly-build --ostag win64par
Alternatively, if your system administrator allows it, you can install WSL on your Windows machine and follow the instructions for Linux below.
For Linux and MacOS, you can build parallel MODFLOW yourself with the following recipe. It needs to have a git client and conda (or an alternative such as micromamba) installed.
First clone the GitHub repository for MODFLOW 6 to get the program source files
git clone https://github.com/MODFLOW-USGS/modflow6.git
Next, you will create a conda environment, called mf6xtd. This mf6xtd environment will have all of the software needed to compile serial and parallel versions of MODFLOW 6, and the Python packages needed to pre- and post-process MODFLOW models.
Create a an environment file that has the following contents and call it mf6xtd_environment.yml
name: mf6xtd
channels:
- conda-forge
dependencies:
- pkg-config
- openmpi
- gcc
- gfortran
- petsc
- meson>=1.1.0
- ninja
With this file, the mf6xtd environment will be created by executing the following command
conda env create -f mf6xtd_environment.yml
To build the parallel version of MODFLOW, activate the environment
conda activate mf6xtd
and run the following commands from a terminal
cd modflow6
meson setup builddir -Ddebug=false -Dparallel=true --prefix=$(pwd) --libdir=bin
meson install -C builddir
meson test --verbose --no-rebuild -C builddir
If everything is working properly, then the last command should show that the tests completed ok and without errors.
The installation from sources in described in more detail in the PARALLEL.md file.
The following command runs MODFLOW in parallel on 8 cores
mpiexec -np 8 mf6 -p
There have been three classes on parallel MODFLOW to date. The presentation material and the exercises are a good starting point for running simulations with MODFLOW in parallel.
-
May '24: 2-Day MODFLOW 6 and FloPy class at the 2024 MODFLOW and More Conference in Princeton, NJ
-
Nov '23: 1-Day parallel MODFLOW 6 class in Delft, Netherlands
Basic examples scripted in FloPy, mostly consisting of synthetic models, are part of the autotest suite of MODFLOW 6 and can be a useful reference. The parallel test cases follow the file pattern test_par_*.py and can be found in the folder autotest
...
...