Add howto for applying a cowboy on qastaging

.custom_wordlist.txt

@@ -196,6 +196,8 @@ hba
 hg
 hirsute's
 hostnames
+hotfix
+hotfixes
 howto
 HSTS
 html
@@ -417,6 +419,7 @@ realizes
 ReleaseCycles
 repl
 repo
+RequestLogging
 RESTful
 resultset
 ResultSets

custom_conf.py

@@ -142,6 +142,7 @@
     r'https://github\.com/canonical/fetch-operator*',  # private
     'https://git.launchpad.net/charm-launchpad-buildd-image-modifier/tree/files/scripts/setup-ppa-buildd',  # private
     'https://git.zx2c4.com/cgit/', # unfortunately very flaky
+    'https://wiki.canonical.com/InformationInfrastructure/OSA/RequestLogging/LP/Cowboys',  # private
     ]
 
 # Pages on which to ignore anchors

how-to/applying-a-cowboy-to-qastaging.rst

@@ -0,0 +1,80 @@
+==============================
+Applying a cowboy to qastaging
+==============================
+
+.. note::
+
+    A cowboy is a hotfix applied to an instance, avoiding a proper deployment
+    by applying code changes directly on an instance.
+
+.. caution::
+
+    A cowboy is a valuable tool to either apply urgent hotfixes or test things
+    quickly e.g. in a staging environment.
+
+    As it avoids a proper deployment, it also comes with a couple of negative
+    side effects:
+
+    - a regular deployment overwrites the cowboy
+
+    - the state of the instance is not clearly defined in the current code
+      and/or deployment configuration
+
+    - and probably some more reasons
+
+    All in all a cowboy should be applied only in rare and extraordinary
+    circumstances.
+
+Before you apply a cowboy, please have a quick conversation with your team
+where the reason for the cowboy is thoroughly discussed.
+
+Applying the cowboy
+===================
+
+#. SSH into bastion ``ssh launchpad-bastion-ps5`` and switch to the
+   ``stb-vbuilder-qastaging`` user by running
+   ``sudo -iu stg-vbuilder-qastaging``.
+
+#. Run ``source ~/.mojorc.qastaging``.
+
+#. Run ``juju config launchpad-appserver build_label`` and take a note of the
+   currently applied git commit id.
+
+#. Checkout that commit id locally.
+
+#. Apply the planned changes to your local checkout and create a local patch
+   file via e.g. ``git diff > <description>.patch``.
+
+#. Log into the appropriate unit where you want to apply the patch to.
+
+    - The following steps describe the process of applying a patch to the
+      application server.
+
+    - Get a list of units you need to apply the patches to. For the application
+      server run ``juju status launchpad-appserver``.
+      As we run more than one instance, we need to apply the next steps for
+      each of the units.
+
+    - ssh into one of the units via ``juju ssh launchpad-appserver/2``
+
+    - Change into the directory with the source code via
+      ``cd /srv/launchpad/code``.
+
+    - Apply the patch via ``git apply <description>.patch``.
+
+    - Restart the system via ``sudo systemctl restart launchpad``.
+
+      .. note::
+
+          The restart command occasionally times out. Just run it again until
+          it works.
+
+    - As already mentioned, repeat the previous steps for all units to make
+      sure all units have your patch applied.
+
+
+Related information
+===================
+
+Applying cowboys to production need to be performed by IS, see
+https://wiki.canonical.com/InformationInfrastructure/OSA/RequestLogging/LP/Cowboys.

how-to/operating-launchpad.rst

@@ -15,4 +15,4 @@ Operating Launchpad
    transfer-project-ownership
    create-bot-account
    porting-builders-to-newer-ubuntu-versions
-
+   applying-a-cowboy-to-qastaging