Merlin workflow example with heat conduction problem

Motivation

The overall ROM workflow involves several stages: offline FOM sample generation, POD/DMD training, projection-ROM operator building, and online ROM prediction. These stages in general cannot be executed automatically by one click:

• some manual pre/post-processing are required at each stage;
• even with the shell scripts that automatize such pre/post-processing, it is necessary for large-scale simulations to be submitted to a job-queueing system (such as slurm/moab) and wait for the queue and the actual job to be complete.

merlin makes the workflow seamless and automatic, even combined with HPC job submission system. Main advantages of using merlin are:

• Can overview/orchestrate the entire workflow with one yaml file
• HPC job submission can be automatic: the jobs will be submitted, queued, executed automatically as soon as the prerequisite jobs are complete.
• Each execution case is containerized: an execution of the workflow stores in a separate directory all the command scripts, the results, and the error messages. This helps debugging of the workflow. Also, it prevents unintended data corruption from the previous runs.

Prerequisites

Installation

The installation procedure for merlin is well documented here.

Server configuration

merlin can be used locally without any job submission, and in such case no server is required to be configured. In order to support job submission in parallel, server is required to be configured for merlin. Detailed procedures are documented in the merlin documentation.

For LC machines, dedicated IT servers can be created and configured. For the detailed instructions, see LLNL LC confluence page.