meta packages

[oliver-sanders]

closes: #76, #27, #75, cylc/cylc-flow#4158

Open proposal to discuss the packaging-end-game for Cylc 8.

Doesn't really need a full-blown proposal-document, there isn't any fine print to haggle over.

cylc => cylc-flow

Early in the Cylc 8 project we renamed cylc => cylc-flow, some of the motivation:

• We were considering having a stripped-down cylc-flow package which just had client/message functionality for use on job hosts.
• We wanted the cylc namespace for the meta-package.
• We considered separating the library and application layers of Cylc.
• Cylc Flow sounded cool.

In retrospect this might not have been the best idea:

• The client/message only package never materialised.
• All of the cylc-* projects have cylc-flow as a dependency making cylc-flow the "core" component.
  • Note: The cylc command is provided by cylc-flow other cylc projects register their subcommands via entry points.
• Separating the library and application would be a lot of work for no gain (especially given the previous point).
• We are less convinced by the role of the meta package than we were.

The meta-package

The intention of the meta package was to provide a one-click install for a fully-functional Cylc stack.

# i.e. this:
$ conda install cylc

# does the same as this:
$ conda install cylc-flow cylc-uiserver cylc-rose metomi-rose

Advantages:

1. The versions of each component get "pinned" in the meta-package ensuring environment consistency.

   • However, we don't actually need a meta-package to enforce this consistency (1).
   • For example cylc-rose has both metomi-rose and cylc-flow as dependencies.
   • This means that if you install cylc-rose your package manager (pip, conda) will ensure the two are compatible.

2. Ease of installation.

   • Seemed like a good idea at the time, however, environments may have lots of variables.
   • What bits of the system do they want installed in which places?
   • What if they want to set the version of Python? Wouldn't we want to fix that in the meta-package?
   • What if they want (or don't want) optional Cylc dependencies (e.g. Pandas used for cylc-report-timings)?
   • What if they want to install the UI Server with the optional "hub" dependency.
   • What if they want to install the Hub without the "configurable-http-proxy".
   • Handling the matrix of "Cylc packages", "Cylc package installation options", "Cylc package dependency installation options" gets really, really messy.

Disadvantages:

3. We can't "deselect" packages to install (with conda, pypi is different).

   • With Conda you can have multiple outputs, however, these aren't like optional-dependencies in PyPi.
   • They are more like top-level projects. E.G. both cPython and PyPy produce the python output.
   • So at the moment if you conda install cylc you get the whole lot whether you wanted it or not.
   • To get around this we would need special meta outputs e.g. cylc-meta-flow, cylc-meta-uiserver, etc.

4. Differing versions between cylc and cylc-flow.

   • We would expect the meta-package version to start out as the cylc-flow version?
   • If we want to make a maintenance release for cylc-uiserver we would need to bump the meta-package maintenance version?
   • So the meta-package and cylc-flow versions can mismatch.
   • Unless we mandate that there is a cylc-flow maintenance version for each meta maintenance release?
   • Otherwise cylc version shows the version of cylc-flow not the meta package which is confusing if you installed via the meta package.
   • Where does the meta-package version get shown?

And so the meta-package has begun to look like a [complex] solution in need of a problem.

We would probably be better off adding a few example Conda recipes in cylc-doc and letting users decide how they want to install the stack.

The Future

Consistency

Currently all Cylc projects have a dependency on Cylc Flow.

We can give this dependency a very narrow version range (version "pinning") in order to ensure consistency between the different components.

Terminology: <major>.<minor>.<maintenance>

Packages

There should be one-- and preferably only one --obvious way to do it.

If the meta-package serves little function we should probably cut our losses and get rid of it.

Doing so would leave us with three problems:

• The cylc Conda package.
  • Currently provides cylc-flow, cylc-rose, cylc-uiserver, metomi-rose.
• The cylc PyPi package.
  • Currently provides cylc-flow 8.0a0 and nothing else.
• The slightly-confusing cylc-flow project name.
  • Just seems a bit odd considering that it is the core component of the system.

Options:

1. We could delete the cylc packages (makes us vulnerable to typo-squatting)
2. We could make them dead-end packages (i.e. if you try to install them you get an error telling you not to install them).
   • I think you can do this on PyPi.
   • I have asked the conda-forge people, so far they have said that typosquatting protection comes from the review process...
   • ... This is fair, however, would this happen in practice?
3. We could retire the cylc-flow packages and use these instead.

Related: Should we claim the namespace cycl (and possibly clyc?) since it is such a common typo on pypi/conda-forge.

I think:

• If we were to rename cylc-flow back to cylc (lots of work, no gain) then (3).
• Otherwise (2)

Proposal

Ok, enough background, the actual proposal is quite simple:

1. Consistency

   Make Cylc projects depend on minor releases of cylc-flow.

   Examples:

   • cylc-uiserver==1.1.0 might depend on cylc-flow=8.0.* NOT cylc-flow>=8.0.0.
   • cylc-rose==1.1.1 might depend on cylc-flow>=8.0.1,<8.1 NOT cylc-flow>=8.0.0.

   This means all [core] Cylc packages would have to be released for every minor release of cylc-flow.

   I think this is probably the sort of direction we were heading in anyway. We could potentially relax this later on (to the major version) if interfaces settle down.

2. Namespace

   Go for option (2), we don't need a special meta-package, try to make the existing cylc packages "dead-ends".

3. WIP

   Close:

   • proposal: meta package cylc-flow#4158
   • proposal: meta package cylc-doc#234

Thoughts @cylc/core?

[dpmatthews]

Makes sense to me

[hjoliver]

Agreed 👍

[oliver-sanders]

No one screamed, actioning.

[kinow]

Regarding the old cylc metapackage, I think we will have to:

• mark all releases of cylc-feedstock as broken via pull requests to conda-forge/admin-requests: https://conda-forge.org/docs/maintainer/updating_pkgs.html#maint-fix-broken-packages
• mark the cylc-feedstock repository as archived - https://conda-forge.org/docs/maintainer/updating_pkgs.html#archiving-feedstocks

[hjoliver]

Thanks for investigating that @kinow

[oliver-sanders]

Progress:

Version Pinning (pin to cylc-flow minor version):

• cylc-uiserver
• cylc-rose

Conda Forge Outputs:

• cylc-flow - cylc-flow v8.0b3 conda-forge/cylc-flow-feedstock#29
• cylc-uiserver
• metomi-rose

Misc:

• cylc-doc (installation instructions)
• Close "cylc" meta-package PyPi
• Close "cylc" meta-package Conda-Forge
• Suggest hanging on to the cylc-ui Conda Forge package for the moment as we don't yet have a plan for deploying cylc-ui versions [on shorter timescales than UI releases]. Although dropping tarballs in from GH releases is probably the easiest solution.