Understanding how the Ansible docs build

• Understanding the Ansible Docs Build
  • Sphinx - The workhorse of it all
  • make webdocs - What does it do?
  • make webdocs - How does it do it?
    • Makefiles
      • Docs Makefile
  • docs_build.py and the fun it has!
  • But where's the collection info from?
  • Hacking more collections into the build or - how to stress test build memory
  • Turning off breadcrumbs in docs build
• Jenkins builds - where the magic happens
  • Staging a PR with the Jenkins builds
• Much Ado about Redirects
• Preparing for a release
  • Updating intersphinx links

Table of contents generated with markdown-toc

Understanding the Ansible Docs Build

There are many components to creating the Ansible documentation. This article attempts to start documenting some of these parts and pieces. It's a work in progress for sure, so don't take this all as proven fact just yet :-)

Sphinx - The workhorse of it all

Ansible uses Sphinx to convert our rst files to html. Fairly straightforward, but the main file it uses to configure it all is conf.py. We won't repeat the Sphinx conf.py documentation, but we do make use of a few items that we describe later in order to create two docsites from the ansible/ansible Github repository.

make webdocs - What does it do?

Whether you are building locally, or triggering the internal to Red Hat build script that publishes to docs.ansible.com, the most common command you use is make webdocs.

Prior to Ansible 2.10, make webdocs was the main command to make all the documentation. In 2.10 and later, there are two commands:

• make webdocs ANSIBLE_VERSION=[2.10,3,4] - this chooses between building all the docs (for 2.10) or building the Ansible package docs for 3 or later. The main difference is 3 or later does not include ansible-core roadmap or porting guides. If you don't select a version, it chooses the latest available from ansible-build-data.
• make coredocs - Excludes network and scenario guides, and any Ansible package roadmap/porting guides.

make webdocs - How does it do it?

This level of detail is mostly for the curious or anyone who has to change what and how these commands work. Feel free to ignore.

Not ignoring? Okay, let's take a walk through Makefiles and some python code.

Makefiles

In docs/docsite/ you'll see two makefiles:

• Makefile - sets up what make webdocs and make coredocs do at the highest level.
• Makefile.sphinx - sets up what Sphinx (the tool that converts our rst files to html) does.

There is also a section in the main Ansible Makefile make webdocs etc that allows you to run the make commands from the root directory instead of from docs/docsite. If you add some new variant on making the docs, consider if you should also add that variant to the main Ansible Makefile as well. Otherwise your new variant only works from `docs/docsite/

Docs Makefile

There's quite a lot in this Makefile, but let's start with the basics. Starting in Ansible 2.10, we need to create two separate docsites effectively from the same set of .rst files. To do this, we use a few Sphinx and Makefile tricks:

• Use the sphinx -c <path> option for a separate conf.py file to control the docsites and some other parameters.
• Use exclude_pattern[] to exclude network, galaxy, and scenario guides and the individual porting guides and roadmap files from the appropriate sites.
• Use symlinks in the Makefile to link index.rst from either ansible_index file for ansible or core_index.rst for core.

docs_build.py and the fun it has!

running notes from a session today... will need to make these more coherent later

docs-build from the Makefile gets saved into a variable PLUGIN_FORMATTER=../../hacking/build-ansible.py docs-build

The build-ansible.py script is a bit non-standard. The structure is that it's a lite wrapper script which sets up the location that the script can find its python files : https://github.com/ansible/ansible/blob/devel/hacking/build-ansible.py#L40-L41 : hacking/build-ansible.py:40-41 sys.path.insert(0, ansible_lib_path()) sys.path.insert(0, build_lib_path())

And then it loads the parts of the script that do actual work as plugins. So instead having a literal import docs_build somewhere, the loading of the code is done programmatically in a loop: https://github.com/ansible/ansible/blob/devel/hacking/build-ansible.py#L64
hacking/build-ansible.py:64 subcommands = load('build_ansible.command_plugins', subclasses=commands.Command)

Then the plugin we want (docs_build) is selected in this for loop: https://github.com/ansible/ansible/blob/devel/hacking/build-ansible.py#L82 hacking/build-ansible.py:82 for subcommand in subcommands:

And we execute the plugin's main() function here: https://github.com/ansible/ansible/blob/devel/hacking/build-ansible.py#L92 (edited) hacking/build-ansible.py:92 retval = command.main(args)

Explaining load('build_ansible.command_plugins', subclasses=commands.Command)

Okay, so load() comes from the plugin library we're using (straight.plugin). It takes a python package as its first argument and looks inside of that location for any python files . The second argument, subclasses=commands.Command, tells load () that valid plugins are commands.Command classes within the files its examining. So in our case, we added the hacking/build_library directory to sys.path (the directories that python checks for python libraries) on Line 41. So load finds the python package we specified in https://github.com/ansible/ansible/tree/devel/hacking/build_library/build_ansible/command_plugins

Each of those files should have a class inside of them that inherits from build_ansible.commands.Command (example: https://github.com/ansible/ansible/blob/devel/hacking/build_library/build_ansible/command_plugins/docs_build.py#L123 ) (edited) hacking/build_library/build_ansible/command_plugins/docs_build.py:123 class CollectionPluginDocs(Command):

load() finds that class in each of the files, and returns a list of all of them.

So make coredocs builds ansible-core docs. make webdocs ANSIBLE_VERSION=[2.10,3,4] builds ansible package docs. The latter even works in devel because the package builder is selecting what the version of the collection docs are.

We will Autodetect the latest collection *.deps file and use that.

(future ) Implement building the docs for the latest version of collections on galaxy?

But where's the collection info from?

Good question! So the plugin documentation is pulled in from collections using the antsibull tool. It checks for the collection build data for the version of Ansible we're building and pulls in the plugin docs for the version of each collection in that data file.

Hacking more collections into the build or - how to stress test build memory

Since we've had some problems with the Jenkins builds failing due to memory problems (see Jenkins general info in the next section) - we now have a way to dummy up an Ansible package build with additional collections to stress test things before a release.

To do this:

1. Checkout ansible-build-data
   • git clone https://github.com/ansible-community/ansible-build-data
2. For testing the devel tree, modify the latest ansible.in file (currently ansible-build-data/5/ansible.in)
   • Example, this adds the community.dns collection to the collection docs that will be built:
   • echo 'community.dns ' >> ansible-build-data/5/ansible.in
3. For testing the stable tree, you will have to modify the latest ansible-X.Y.Z.deps file:
   • echo 'community.dns: 1.0.1' >> ansible-build-data/4/ansible-4.2.0.deps
4. Checkout the ansible-core code:
   • git clone https://github.com/ansible/ansible
5. cd ansible
6. Tell the docs build to use your local copy of ansible-build-data when building the ansible package devel docs:
   • Note: the ansible-build-data path can be absolute. If relative as in the example, it is relative to the ansible/docs/docsite directory
   • make webdocs EXTRA_PLUGIN_FORMATTER_ARGS='--ansible-build-data=../../../ansible-build-data'

Turning off breadcrumbs

The collection-level breadcrumbs can be a memory hog when building docs. The Jenkins builds (below) have a toggle to turn them off as needed. To turn of breadcrumb generation on your local docs build:

1. echo "breadcrumbs = false" > ~/.antsibull.cfg
2. Build the docs (e.g. make webdocs).

Jenkins builds - where the magic happens

The Jenkins builds internal to Red Hat are what finally publishes all these documents. There are:

• Nightly builds -for ansible-core and Ansible, based on devel.
• Manual builds - mostly for tests, but also to republish docs for a given minor release of ansible-core or Ansible.

When a new release happens, We need to update the Ansible jenkins build to recognize the new release number, and update redirects (see below).

Staging a PR with the Jenkins builds

While this is a task that only a Red Hat person can do, here's some notes for said Red Hat person to remember ;-)

• Fill in build parameters for the github fork (ansible or the PR owner), and branch (devel, stable-2.11, PR branch name, etc).

Much Ado about Redirects

Starting with Ansible 2.10, we have server side redirects in place to allow the version switcher to switch between a module in 2.9 (say https://docs.ansible.com/ansible/2.9/modules/docker_image_module.html) to the same module now in 2.10 (https://docs.ansible.com/ansible/2.10/collections/community/general/docker_image_module.html). A significant (aka thousands) batch of redirects were generated for 2.9 and 2.10 to allow for the version switcher (and user bookmarks) to go to the correct new location for modules migrated to collections.

• For future releases, we need to create a release based folder at https://github.com/ansible/docsite/tree/master/ansible (private repo) and copy the 2.10 redirects into that new folder (for example 3/).

Preparing for a release

Since we now have two releases - Ansible and ansible-core, we also have two checklists for how to prepare and publish a new docset for each. The most recent examples are:

• Ansible 6 github issue
• ansible-core 2.13 issue

These checklists change over time so search for a similar issue for releases that come after these two.

Updating intersphinx links

We use the intersphinx links extension to provide easy cross-references. We use this so we can link to python and jinja2 modules on other docsites, with a simple format like python:foo_module instead of a hard-coded url. We also use these so local builds of the docs will succeed despite the absence of external resources like the Python docs or older versions of our docs. See this pr for other details.

After each release, we need to update those links as follows from within a local copy of the devel branch:

1. add intersphinx_mapping for the new release to each of the conf.py files except 2.10 ( eg docs/docsite/sphinx_conf/all_conf.py adds 'ansible_5': ('https://docs.ansible.com/ansible/5/', (None, '../ansible_5.inv')),)
2. source hacking/env_setup
3. hacking/build-ansible.py update-intersphinx-cache -o docs/docsite -c docs/docsite/sphinx_conf/all_conf.py
4. merge and backport to the appropriate stable branch associated with the release.

(note - if we ever split the docsite up, we may need to reintroduce a patch that was removed here - https://github.com/ansible/ansible/pull/70826)