source workflows

[oliver-sanders]

At present the UI can only see installed workflows (either running or stopped).

• We should add the ability to install workflows from source (e.g. to re-run a workflow).
• We should add the ability to reinstall workflows from source (i.e. in order to do a reload).

So how should source workflows fit into the UI?

Here is my proposal:

• List all un-installed workflows in GScan.
• (Uninstalled workflow list from the configured Cylc source dir(s) - to be done by the UIS via cylc scan).
• Add "is installed" as a field to the flow filtering options.
• Display only installed flows by default.
• Users could exploit the existing search functionality to find workflows.

This would create a landing page for the source workflow:

• The landing page should default to a file browser as the default view.
• This page can be used for editing workflow files (flow.cylc, rose-suite.conf, etc).
• Source directories should have a special icon to distinguish them from workflow dirs (not implemented in sketch below).
• This would provide a smooth transition from editing the source, to (re)installing the flows.
• We could plug in basic version control operations later on.
• When we have version control information we can mark workflows which were installed from a previous revision differently until re-installed.

• This provides a single, simple interface to Cylc/Rose with the bare minimum of clicks from source -> running.
• Given the search and filter functionality already implemented in GScan this should be suitable to heavy usage too.
• Future enhancements would add to the complexity of GScan, but, would also yield benefits to beyond source dirs.

The only big design barrier to overcome is that this restricts us to use the same name in the source and run dies (which I am actually ok with and have argued for in the past). If we can’t go with this then we will need a bit of UI magic to make this problem go away...

(assigning myself until the question label is removed or this issue closed)

[hjoliver]

I like the proposal, can't see any obvious problems with it. 👍

[oliver-sanders]

Linking through to part (2) of cylc/cylc-flow#3400 which is about making users aware when running workflows run directories no longer reflect the latest state of the source directory (i.e. the user has made change to the source dir since the flow was last (re)installed). Already on my radar, suggest using passive notification for these sorts of things (e.g. reinstallation/reload, removal of old run dirs, uninstallation).

[oliver-sanders]

Here's an updated proposal which is more compatible with the new Cylc support for multiple configurable source dirs.

Overview:

• We now allow sites/users to configure multiple source dirs.
• cylc install <name> will trawl through these like a search path looking for <name>.
• The default is ~/cylc-src.

Modified Proposal:

• Add a drop-down box for selecting source/run dir.
• We only list uninstalled source workflows from the relevant source dir.
• Mark source dirs with a special icon to allow mingling source and run dirs together for convenience.

Ideas?

[hjoliver]

Nice proposal 👍

Ideas?

With search and filter, and especially for those with a smaller number of workflows, it might be good to show all configured source dirs and the run dir, at once? This could be a user preference, if gscan performance is a problem for those with a large number of workflows.

[oliver-sanders]

We could just bung them all into the main GScan view and just filter out uninstalled workflows (and by extension their sources) by default.

We can scan for both source and run workflows and can separate these scan intervals and make them configurable. I think we might want something like PT1M for run dirs and manual refresh for src dirs for our site.

[oliver-sanders]

Notes from Dave:

• Consider toggling between src/run rather than using a dropdown selector.
• Need to handle src flows that are installed using different flow names.
• Installing from src in the cylc-run directory should result in an installation with the same flow name (irrespective of source name).
• Cylc tutorials best installed by rsync'ing into ~/cylc-src/tutorial.

[oliver-sanders]

Here's an amended design with Dave's suggestions.

• There is now a toggle switch rather than a dropdown.
  • Perhaps less obvious to users but also subtler design.
• Configured source directories are now the top level of the hierarchy.
• Custom flow names marked when in "src mode"
  • If workflows are installed using a custom flow name (e.g. cylc install --flow-name=foo/bar)
  • Then we need to add foo/bar to the workflow name when viewed against the src workflow.
  • See foo/bar/run1 in the sketch.

[oliver-sanders]

A couple of loose ends I've not jotted down anywhere so dumping them here for the time being:

• If a workflow has only one run we could auto-collapse it to a single line to take up less space. It could then be expanded if the user wants to toggle between the src/run dirs.
• We can display the number of jobs in each square (future work).
• GScan window may need to be slightly wider or user-adjustable.

[oliver-sanders]

issue covering the switching of workspaces and its effect on data, subs and tabs: #662

[hjoliver]

Looks good!

[oliver-sanders]

Source workflows arriving in the schema with: 512