Machine Learning Workflow Orchestration

[BerndDoser]

We are looking for a suitable workflow framework to construct our machine learning pipeline.

Data download -> Data preprocessing -> Training -> Evaluation -> Inferenz -> Deployment

This discussion aims to outline the specific requirements for us and the astronomy community.

• Easy to install (Kubernetes is not standard in HPC community)
• Flexible pipeline definition
• Integration of containers / Apptainers
• Integration of HPC clusters (SLURM)

Flyte

Flyte docu
Spherinator branch to evaluate Flyte

+ Sandbox for single node usage
+ 4.7k GitHub stars
+ Flexible workflow definition by using python function decorators @task and @workflow
- Kubernetes needed for production
- SLURM plugin requested but not under active development

StreamFlow

StreamFlow Docu
Spherinator branch to evaluate StreamFlow

+ Developed by Alpha UNITO (SPACE partner)
+ SLURM plugin will be released soon (WP3 meeting Doriana Medic)
+ Using Common workflow language
+ CWL implementation in production supporting v1.2
- 42 GitHub stars
- very early stage of development

[BerndDoser]

Phone discussion with @andife:
Before defining the requirements, it would be helpful to define the use cases. Three possible use cases were identified:

1. Definition, execution, and monitoring of workflows only for version control and inhouse usage
2. Automated workflow execution through a web interface for external use
3. Distribution of the workflow infrastructure for external collaborators to deploy on their environment

Open question: Should the ML training be part of the workflow or only the inference?