template vars: simpler syntax for defining a list of strings

[oliver-sanders]

Template Variables

Currently we would define a list of strings like this:

-s 'MY_LIST=["foo", "bar", "baz"]'

This approach is highly flexible and supports other types e.g:

-s MIXED_LIST=["foo",42,True]
-s INT_SET={2,5,9}
-s DICT='{"foo": 1, "bar": 2, "baz": 3}'

Rose Stem

The rose stem system has two options called --tasks and --groups which confusingly do the same thing. They set a Jinja2 variable called RUN_NAMES to a list of strings defined in comma separated format.

E.G. This:

--tasks=foo,bar,baz

Is actually equivalent to this:

-s RUN_NAMES='["foo", "bar", "baz"]'

So, the --tasks and --groups options are actually nothing to do with rose stem at all, they are really just a shorthand for specifying template variables to Cylc. This is a bit obstructive as these arguments are only supported by the rose stem command itself, which only runs cylc install.

Future Rose Stem Interface?

It would make things a lot if we could pull the --tasks option out of rose stem and have it specified directly to Cylc. However, this option is a bit lengthy to write out in full.

We could make this easier by specifying a shortcut for defining lists of comma separated strings.

+s RUN_NAMES=foo,bar,baz

This might just be enough to allow us to deprecate the --tasks and --groups options (leaving them in as a wrapper of course), opening up this functionality to other Cylc commands e.g. cylc validate, cylc graph and cylc vr.

[hjoliver]

(Sounds sensible to me)

[wxtim]

That feels like a nice solution to the Rose Stem issue.

[oliver-sanders]

Ok, I think we have agreement here (removing the question label), this seems to be the least nasty way of getting around our rose-stem issues.

To close this issue we need only support the +s syntax. The rose-stem side of things will be solved by cylc/cylc-rose#205.

Proposed behaviour:

#!Jinja2

{{ FOO }}

$ cylc view -p -s 'FOO=["a", "b", "c"]'
['a', 'b', 'c']
$ cylc view -p +s FOO=a,b,c
['a', 'b', 'c']
$ cylc view -p +s FOO=a,"b,b",c
['a', 'b,b', 'c']
$ cylc view -p +s FOO=a, b, c
bash: b,: command not found

Note the quotes in the third example can be solved using the logic implemented in this PR - #5587

[wxtim]

Modification to proposal - It's not possible to use +s without subclassing 3 methods (100+ lines of code) from optparse's Option object. As such another option should be picked.

Front runners are

-L:

• We're defining a list. "L" for list makes sense.
• It's somewhat memorable because it makes sense.

-z:

• It's like a backward "s" as used for template variables.
• It's unlikely to conflict with other variables.
• It's memorable because it's wierd and arbitrary.
• It's actually a string list not plain "List"

[wxtim]

Possible additional examples

$ cylc view -z FOO=a,b -z FOO=c,d
['a', 'b', 'c', 'd']

$ cylc view -s 'FOO="a"' -z FOO=b,c
Error: Don't do this.

Are these reasonable @oliver-sanders ?