Version numbers for documentation deprecation removal in the collection world

This is about version numbers for version_added='xxx' (module, option, return value documentation), deprecation.removed_in='xxx' (module docs), removed_in_version='xxx' (argspec), deprecated_aliases.version='xxx' (argspec) and module.deprecate(version='xxx') (module/module_utils code).

version_added='xxx' for module and return value documentations can only refer to the collection version of the collection the module is part of, since these must be specified in the module code itself.

All other versions can be specified in the module itself, in doc_fragments or module_utils of the same collection, in doc_fragments or module_utlis of other collections, or in doc_fragments or module_utils of ansible-base. And there the fun begins... :)

Problems of cross-collection doc_fragments and module_utils

If a doc_fragment is extended (or changed), this affects every module using that doc_fragment. If a user is in another collection, the module in another collection suddenly gets new (or changed) options without a change in the collection's version.

If options are added to the argspec by code from module_utils in other collections, modifying that code might modify/add/change/remove options in modules from other collections that use this module_utils. Most importantly, this can change modules in other collections without their collection version changing: this is especially dramatic if changes are not backwards compatible, i.e. the meaning of options changes, or options are deprecated and finally removed.

Also, it is important to note that by just updating a doc_fragment, or by just updating module_utils, this will definitely break users from other collections! When a new option is added to a doc_fragment, there must be code in module_utils which a) is used by all users of that doc_fragment and b) adds support for this new option. If this is not the case, the doc_fragment should not be updated.

Cross-collection usage of doc_fragments and module_utils should be made only very carefully, if at all!

Determining origin of version numbers

1. version_added for module options can be specified directly in the module, or in doc_fragments. When being specified in the module or in doc_fragments from the same collection, the version refers to the current collection's version. Otherwise they refer to versions of another collection, or to versions of ansible-base.
2. removed_in_version and deprecated_aliases.version can be specified directly in the module, or in module_utils. Since the argument spec can be create by arbitrarily complex code spread over arbitrarily many module_utils, it is essentially impossible to track where different parts of it come from.

How to implement tracking

For version_added, the fragment loader / combiner can be adjusted to keep track of where a version_added came from. This is work, but feasible and requires no changes to content (i.e. module_utils, doc_fragments, modules both in ansible-doc or collections).

For removed_in_version or deprecated_aliases.version, we could use some tagging system, i.e. if defined in ansible-base they would use ansible-base:2.12 instead of simply 2.12; and if defined in collection ansible.posix they would use ansible.posix:3.0.0 instead of simply 3.0.0. Leaving them away should only be allowed for content from the current collection. Since it is impossible to detect this, one could enforce that removed_in_version and depreated_aliases.version always need a tag.

Forward-compatible uses of cross-collection doc fragments and module_utils

Example 1: files doc_fragment of ansible-base

The files doc_fragment of ansible-base comes with the option add_file_common_args of AnsibleModule.__init__(). The doc fragment should be used, and only be used, if add_file_common_args=True. This makes it possible to add new options without breaking modules in other collections. (Even if the module already used that option name for something else, the way the system works nothing breaks since the option is kept as it was - it only affects modules which did not have this option before.)

Version numbers for version_added for new options will be ansible-base versions.

Deprecation and/or removal of options could be done, but should be avoided for the above reasons: this can randomly break modules in other collections which might depend on these options in some way. (The only good side is that this is ansible-base, and won't "randomly" update as other collections might do.)

Example 2: urls doc_fragment of ansible-base

The url doc_fragment defines some options, some of which are used by lib/ansible/module_utils/urls.py automatically, and some others have to be passed directly. Also, all these options need to be declared in the argspec manually by ever module using the url doc_fragment!

This makes it impossible to add new options without breaking random modules (sanity checks will fail because there are options documented which aren't in argspec), or to deprecate / remove existing options.

Generalization

doc_fragments such as files are the way to go if you want to have something that works and can evolve in collections. Deprecation/removal is still problematic, but at least new options can be added. Special care should be taken that new options added will not overwrite options of the same name in modules! Also, it is important to note that just because extending such fragments with module_utils is possible, it is not always a good idea: when these options are used by modules in other ways (like the files options are used by code outside module.set_file_attributes_if_different), it can have unintended consequences or introduce bugs.

doc_fragments such as url should never be touched again. If they ever should be adjusted/extended, it is better to create a new doc_fragment (like url2), and this time accompany it with module_utils code to handle the argspec (so there isn't need for yet another doc_fragment url3 later). Also, the code in module_utils should not rely on the values of the required options to be passed explicitly, as code in other collections will not magically start doing that.