Use ECFlow as the workflow engine

[backeb]

Short description of the issue

In executing workflows, interTwin's WP6 is investigating interfaces to workflow composition and engines for workflow execution.

The Common Workflow Language (CWL) has been chosen as the common interface for workflow composition.

Here we executed a CWL workflow from ECFlow to explore interoperability.

It has been communicated to us that using ECFlow on DestinE is only really required when running large (global) models on EuroHPC. For limited area models like we use in DT Flood Early Warning and DT Flood Climate Impacts, we can rely on VMs in the cloud to run the models which also gives us more flexibility in terms of the workflow engine we are using.

Hence this issue is considered to be beyond the scope of the work being undertaken here for interTwin.

[backeb]

notes from meeting with Thomas

• Confirmed the ECFlow is mainly used in EuroHPC
• ECFlow will be used in operational setup DestinE so could be useful for monitoring and alerting
• We can launch the CWL workflow from ECFlow
• Converting CWL steps to ECFlow steps is not an ongoing activity, but could be a requirement for operational implementation.
• ECFlow as a Service model on DestinE still needs to be figured out. But you can request the ECFlow server to be deployed to monitor your DT and linking that to the operational set up it can provide notifications when models have issues.
• In the future we might have a CWL workflow which you can launch from ECFlow, then we can benefit from ECFlow monitoring and alerting - this way we benefit from the operational setup.
• EUMETSAT (DEDL) is deploying openEO but not the infrastructure orchestration part, but not integration with OpenStack to spin-up VMs

action

• once complete CWL workflow runs, work on launching it from ECFlow, and vice versa.

[kinow]

Using ecFlow to monitor CWL workflows sounds really interesting! Short notice, but we are having the CWL Conference 2024 in a few days. I think there is not much time to submit a talk on this, but registrations for the hybrid Online/Amsterdam conference are still open - https://www.commonwl.org/conferences/2024/ 👍