Changes made to reflect jules-shared developments (#138)

Co-authored-by: James Bruten <[email protected]>

source/Reviewers/howtocommit.rst

@@ -110,7 +110,7 @@ If there are conflicts in versions.py then see the details in the macro section
 
 2. Macros (if required)
 -----------------------
-**If** the ticket includes meta-data changes, upgrade macro changes or a new rose-stem app
+**If** the ticket includes metadata changes, upgrade macro changes or a new rose-stem app
 then you will need to upgrade the test-suite.
 
 .. dropdown:: versions.py
@@ -139,14 +139,24 @@ then you will need to upgrade the test-suite.
 
             .. code-block:: RST
 
-                ~frum/bin/update_all.py --path=/path/to/working/copy/of/trunk --um=vnXX.Y_tZZZZ
+                ~frum/bin/update_all.py --path=/path/to/working/copy/of/trunk --um=vnXX.Y_tZZZZ [--jules-path=/path/to/working/copy/of/jules/trunk]
 
-            where `--um=vnXX.Y_tZZZZ` is the `AFTER_TAG` of the latest upgrade macro.
+            where `-\-um=vnXX.Y_tZZZZ` is the `AFTER_TAG` of the latest
+	    upgrade macro.
 
             If there is a macro for fcm_make or createbc then check that the makes `version*_*.py` has the
-            correct BEFORE and AFTER tags and append `--makeum=vnXX.Y_tZZZZ` and/or `--createbc=vnXX.Y_tZZZZ`
+            correct BEFORE and AFTER tags and append `-\-makeum=vnXX.Y_tZZZZ` and/or `-\-createbc=vnXX.Y_tZZZZ`
             to the above command.
 
+	    .. warning::
+	       Please ensure that Cylc7 is used with `update_all.py` @vn13.5.
+
+	    .. note::
+	       The `-\-jules-path` option is only required if there
+	       are linked `jules-shared
+	       <https://code.metoffice.gov.uk/trac/jules/browser/main/trunk/rose-meta/jules-shared>`_
+	       metadata changes.
+
         .. tab-item:: JULES
 
             .. code-block:: RST
@@ -180,6 +190,14 @@ then you will need to upgrade the test-suite.
 
             rose macro --validate -M path/to/working_copy/rose-meta
 
+    .. note::
+       For UM tickets, if there are linked `jules-shared
+       <https://code.metoffice.gov.uk/trac/jules/browser/main/trunk/rose-meta/jules-shared>`_
+       metadata changes these will need to be added to the metadata
+       path. Please see the :ref:`rose config-edit
+       example<metadata_changes>`.
+
+
 .. dropdown:: Temporary Logical?
 
     If a new temporary logical has been added, or an old one retired, then

source/WorkingPractices/TestSuites/jules.rst

@@ -16,6 +16,8 @@ The output is checked for correctness both by comparing the output to a set of
 stored :ref:`KGO files <kgo>`.
 
 .. note::
+
+=======
     If there are JULES changes to shared science code or metadata then these
     changes will need to be tested :ref:`with the UM<um_testing>` and
     :ref:`with LFRic Apps<lfric_apps_test>`. If you have access to LFRic, the
@@ -24,6 +26,12 @@ stored :ref:`KGO files <kgo>`.
     your Met Office contact.
 
     See :ref:`multirepo` for details on how to carry out this testing.
+
+    .. important::
+      For **jules-shared** changes, when LFRic testing, the changes
+      need to be manually synced to the LFRic location. When UM
+      testing, this is not required as **jules-shared** is imported
+      from the JULES branch.
 
 Below is a (by no means comprehensive) set of groups that you may wish to use on
 Met Office systems.

source/WorkingPractices/metadata_guidance.rst

@@ -12,20 +12,57 @@ with a consistent set of information, such as follows:
 
 .. code-block::
 
-    [namelist:<namelist>=<variable>]
-    description=<what is the variable for?>
-    help=<detailed information about the variable>
-    sort-key=<XY>
-    etc
+   [namelist:<namelist>=<variable>]
+   description=<what is the variable for?>
+   help=<detailed information about the variable>
+   sort-key=<XY>
+   url=<http://jules-lsm.github.io/latest/namelists/<namelist>.nml.html#<NAMELIST>::<item>>
+   etc
 
-then each input is presented as:
+where JULES namelist items should only contain the ``url=`` field and
+not the ``help=`` field regardless of the repository they reside
+in. Each input is presented in the GUI as:
 
 .. code-block::
 
     variable name
     short description of the variable name
 
-Help text is displayed when one clicks the variable name.
+Either the help text is displayed or web help opened when one clicks
+the variable name. If both fields exist the Web help is opened
+preferentially. If a ``url=`` is present the ``help=`` field should be
+removed to avoid duplication.
+
+.. _shared-namelists:
+
+Shared namelists
+----------------
+
+Shared namelist items, or items which exist in more than one
+repository, should have identical metadata regardless of the
+repository where they reside (e.g. JULES items in the UM
+metadata). The only caveats are where a ``trigger`` or a ``fail-if``
+reference a namelist item from the parent model (e.g. UM).
+
+Shared JULES metadata is in the process of being migrated to
+`rose-meta/jules-shared
+<https://code.metoffice.gov.uk/trac/jules/browser/main/trunk/rose-meta/jules-shared>`_,
+which resides in the JULES repository. The sub-directories are
+imported by **rose-meta/um-atmos** and **rose-meta/jules-standalone**
+and is manually synced with a copy in LFRic. Please see `Sharing JULES
+metadata <https://code.metoffice.gov.uk/trac/jules/wiki/SharingJULESmetadata>`_
+for more details including what should be in `jules-shared
+<https://code.metoffice.gov.uk/trac/jules/wiki/SharingJULESmetadata#Whatsinjules-shared>`_
+and in `jules-standalone, jules-lfric or um-atmos
+<https://code.metoffice.gov.uk/trac/jules/wiki/SharingJULESmetadata#Whatsinjules-standalonejules-lfricorum-atmos>`_. When
+developing shared JULES metadata, you will need :ref:`linked tickets
+<multirepo>`. The metadata migration is currently dictated by LFRic
+porting of science, although the ultimate aim is to have a single
+source of truth.
+
+Please see the :ref:`rose config-edit example<metadata_changes>` for
+an illustration of how to pick up **jules-shared** changes from a
+JULES working copy.
 
 ..
  We need to check if this is all still the case with cylc 8.
@@ -55,7 +92,8 @@ see inputs that need to be set rather than everything.
 .. tip::
   The GUI provides an option to un-hide triggered variables if one wants to see them all.
 
-Please set ``compulsory=true`` for items and use triggers for when it is not required.The settings
+Please set ``compulsory=true`` for items and use triggers for when it
+is not required. The settings
 of all variables will then be present, in all apps to aid configuration management. When a variable
 is triggered off, it will be commented out in the apps e.g. ``!!variable``.
 
@@ -73,18 +111,64 @@ is triggered off, it will be commented out in the apps e.g. ``!!variable``.
 
   It is not possible to trigger a variable based on ``OR`` logic.
 
-Viewing meta-data changes as you go along
+.. _metadata_changes:
+
+Viewing metadata changes as you go along
 -----------------------------------------
 
-One can easily review their meta-data changes with the rose config editor, opening up an example app file. For example
+One can easily review their metadata changes with the rose config editor, opening up an example app file. For example:
 
 .. code-block::
 
-  cd <path of working copy of branch>/rose-stem/app/um_n48_eg
-  rose config-edit -M <path of working copy of branch/rose-meta/>
+   cd <path of working copy of branch>/rose-stem/app/um_n48_eg
+   rose config-edit -M <path of working copy of branch/rose-meta/>
+
+If making **jules-shared** changes, when reviewing these changes from a
+different parent repository, you will first need to set the
+`ROSE_META_PATH` system variable:
 
-then once the app opens click on the LHS appname to being up the app meta panel. Update this to HEAD rather than the version number and apply.
+.. code-block::
+
+   export ROSE_META_PATH=<path of working copy of JULES branch/rose-meta/>
+
+or add the path instead as a colon separated list:
+
+.. code-block::
 
-Please note that if you have used an upgrade macro on the app then the meta line at the top of the app file will have changed (i.e. meta=um-atmos/vn11.0_t46). Since no meta-data exists at this version rose edit will produce an error saying that it cannot find it, instead it will use the meta-data in um-atmos/HEAD. Please click OK and continue.
+   rose config-edit -M <path of working copy of branch/rose-meta/>:<path of working copy of JULES branch/rose-meta/>
 
-Your updates should now appear. 
+then once the app opens click on the LHS appname to display the app
+meta panel. Update this to HEAD rather than the version number and
+apply.
+
+Please note that if you have used an upgrade macro on the app then the
+meta line at the top of the app file will have changed
+(e.g. meta=um-atmos/vn11.0_t46). Since no metadata exists at this
+version rose edit will produce an error saying that it cannot find it,
+instead it will use the metadata in e.g. um-atmos/HEAD. Please click
+OK and continue.
+
+Your updates should now appear.
+
+Ensuring metadata changes are valid
+-----------------------------------
+
+Developments to the metadata can be checked for errors by running
+`rose metadata-check <https://metomi.github.io/rose/doc/html/api/command-reference.html#rose-metadata-check>`_
+
+.. code-block::
+
+   rose metadata-check -C /path/to/rose-meta/<config>/HEAD
+
+where the ``-C`` option can be omitted if inside the directory containing the metadata file.
+
+.. note::
+   If there are **jules-shared** changes then these need to be
+   added to the metadata path even in the JULES repository. As the
+   metadata checker does not have the ``-M`` option, this has to be
+   done using the `ROSE_META_PATH` environnment variable as in the
+   :ref:`previous example<metadata_changes>`.
+
+   If the metadata checker returns "not a configuration metadata
+   directory" then this may indicate that the wrong path has been
+   set.

source/WorkingPractices/rose_stem.rst

@@ -8,7 +8,8 @@ by the rose stem suite.
 
 .. warning::
   If you find that you need to update all the apps in the rose stem suite
-  to get your change to work then you should use an upgrade macro. See :ref:`inputs`.
+  to get your change to work then you should use an upgrade macro. See
+  :ref:`inputs`.
 
 .. tip::
   Familiarise yourself with the `Rose documentation
@@ -47,6 +48,12 @@ already in the code.
         app directory, or selecting Metadata -> Check fail-if, warn-if from the
         drop-down menu of the rose edit gui.
 
+	.. important::
+	   If you are working in a UM branch and have **jules-shared**
+	   changes, the JULES metadata path will also need
+	   adding. Please see the shared metadata
+	   :ref:`guidance<shared-namelists>`.
+
 The next steps are site and rose-stem specific, but fall broadly into two
 categories - those using a jinja2 templated design which populate runtime and graph
 sections for you, such as the UM METO suite, and those which are manually configured,

source/WorkingPractices/testing.rst

@@ -74,26 +74,45 @@ change then this macro will be run by your code reviewer as part of the commit
 process. In order to prove that the upgrade macro will be successful, and to
 make those new model inputs available to your tests, you should create a test
 branch as described above and run the upgrade macro with one of the following
-commands
+commands, noting that ``--jules-path`` is only required if you have
+**jules-shared** metadata changes. Please see the shared metadata
+:ref:`guidance<shared-namelists>`.
 
-    +-------+----------------------------------------------------------------------------------------+
-    | UM    | $UMDIR/bin/update_all.py --path=/path/to/working/copy/of/test/branch --um=vnX.X_tXXXX  |
-    +-------+----------------------------------------------------------------------------------------+
-    | JULES | ./bin/upgrade_jules_test_apps vnX.X_tXXXX                                              |
-    +-------+----------------------------------------------------------------------------------------+
+    +-------+------------------------------------------------------------------------------------------------------------------------------------------------+
+    | UM    | ``$UMDIR/bin/update_all.py --path=/path/to/working/copy/of/test/branch --um=vnX.X_tXXXX [--jules-path=/path/to/jules/working/copy/of/branch]`` |
+    +-------+------------------------------------------------------------------------------------------------------------------------------------------------+
+    | JULES | ``./bin/upgrade_jules_test_apps vnX.X_tXXXX``                                                                                                  |
+    +-------+------------------------------------------------------------------------------------------------------------------------------------------------+
 
 ..
-    | LFRic | *Macro capabilities are currently in development*                                      |
-    +-------+----------------------------------------------------------------------------------------+
+    | LFRic | *Macro capabilities are currently in development*                                                                                              |
+    +-------+------------------------------------------------------------------------------------------------------------------------------------------------+
 
 .. todo: add an LFRic upgrade macro command... and make these more consistent?
 
+.. warning::
+   Please ensure that Cylc7 is used with `update_all.py` @vn13.5.
+
 .. Note::
-    The update_all.py script suppresses warnings produced by upgrade macros.
-    You can test these separately by upgrading a single app. A single app can be
-    upgraded for testing using ``rose app-upgrade -M /path/to/rose-meta -C /path/
-    to rose-stem/app/<app_name> -a <trunk_metadata_version>`` inside the app's
-    directory.
+   The update_all.py script suppresses warnings produced by upgrade macros.
+   You can test these separately by upgrading a single app. A single app can be
+   upgraded for testing using:
+
+   .. code-block::
+
+      rose app-upgrade -M /path/to/rose-meta -C /path/to/rose-stem/app/<app_name> -a <trunk_metadata_version>
+
+   where the ``-C`` option can be omitted if inside the app's
+   directory.
+
+   .. Important::
+      If there are **jules-shared** metadata changes these will need to
+      be added to the metadata path. Please see the :ref:`rose
+      config-edit example<metadata_changes>`.
+
+      Please refer to `rose app-upgrade
+      <https://metomi.github.io/rose/doc/html/api/command-reference.html#rose-app-upgrade>`_
+      command reference for more details.
 
 .. _traclog:
 
@@ -106,4 +125,4 @@ included when passing a ticket for review.
 
 .. code-block::
 
-    ~/cylc-run/<suite_name>/trac.log
+    ~/cylc-run/<suite_name>/trac.log