From 0aed071416ac164ff12a23fbd4128c4faa45b024 Mon Sep 17 00:00:00 2001 From: James Bruten <109733895+james-bruten-mo@users.noreply.github.com> Date: Tue, 16 Apr 2024 09:25:58 +0100 Subject: [PATCH] Move multi repo testing (#168) --- .../TestSuites/multi-repo_testing.rst | 106 ++++++++++++++++++ source/WorkingPractices/multi_repository.rst | 100 +---------------- source/WorkingPractices/testing.rst | 6 +- 3 files changed, 110 insertions(+), 102 deletions(-) create mode 100644 source/WorkingPractices/TestSuites/multi-repo_testing.rst diff --git a/source/WorkingPractices/TestSuites/multi-repo_testing.rst b/source/WorkingPractices/TestSuites/multi-repo_testing.rst new file mode 100644 index 00000000..e9531157 --- /dev/null +++ b/source/WorkingPractices/TestSuites/multi-repo_testing.rst @@ -0,0 +1,106 @@ +.. _multi-repo_testing: + +Multi-Repository Testing +======================== + +Multi-repository changes are expected to pass the regression tests for all the +repositories involved. To carry out the tests involved in a linked ticket it can +be helpful to refer to the :ref:`semi-concentric circles figure `; layering the testing +from the inside out as needed. Further details of how testing in each +repository is handled can be found :ref:`here `. Compatible +code revisions are needed for testing across repositories as described above. + +Testing changes in JULES, LFRic Core, UKCA, or any other child repositories is +as simple as running the standalone test procedures for these codebases. + +.. important:: + + When specifying an alternative source in the ``dependencies.sh`` file the revision for the source **must** be updated. + + * If setting the source as an fcm URL, the mirror needs to be used and the revision can either be blank (for latest commit) or any valid revision for that branch. + * If setting the source as a Working Copy, the hostname needs to be provided (as Hostname:Path) and the revision must be blank. + +Testing the UM with other repositories +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +To test the UM, any changes to JULES, UKCA, Socrates, CASIM etc will also need +to be included. This is done by adding another source to the rose stem command +line. + + 1. :ref:`Checkout` a UM working copy + + - this may be your branch from a linked ticket, or a clean trunk copy + at either the last release or a suitable head of trunk revision. + + 2. Run rose stem, including a source code path to every branch involved. As a minimum + run ``developer`` group and all groups that cover the repositories being tested. + + .. code-block:: + + rose stem --group=developer,jules,ukca --source=. --source=/path/to/jules/changes --source=/path/to/ukca/changes + +The source paths involved can either be to local working copies or links to the +fcm source control e.g. ``fcm:jules.xm_br/dev/user/branch_name``. As many source +paths as needed can be added to the list. + +Testing LFRic Apps with other repositories +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +LFRic Apps testing needs to encompass all of the other repositories affected. +Paths to the other codebases involved should be added to +``dependencies.sh`` under each of the ``*_sources`` variables. Again +these paths can either be to local changes or those in the repository. + + 1. :ref:`Checkout` an LFRic Apps working copy + + - this may be your branch from a linked ticket, or a clean trunk copy + at either the last release or a suitable head of trunk revision. + + 2. Update dependencies.sh to point to all other code changes, e.g. + + .. code-block:: RST + + lfric_core_rev= + lfric_core_sources=fcm:lfric.xm_br/path/to/branch + + um_rev= + um_sources=vldXXX:/path/to/um/working/copy + + 3a. Run the lfric_atm developer test-suite + + - suitable for testing changes in other repositories that do not + include any LFRic Apps changes + + .. code-block:: + + export CYLC_VERSION=8 + rose stem --group=lfric_atm_developer + cylc play + cylc gui + + 3b. Run the full developer test-suite + + - suitable for testing LFRic Apps changes with other repositories, or expanding + testing if lfric_atm tests have shown errors. + + .. code-block:: + + export CYLC_VERSION=8 + rose stem --group=developer + cylc play + cylc gui + + More details on LFRic Apps testing are found :ref:`here`. + +.. note:: + If any of the testing shows up failures then there are two possible ways to + proceed: + + 1. The changes made should be re-written to avoid breaking the dependant + repositories + + 2. The changes made directly affect the interface between repositories and + therefore a change is also needed to the parent repository to adapt to that change. + + If you're uncertain which route to take then the Code Owners involved will + hopefully be able to advise. diff --git a/source/WorkingPractices/multi_repository.rst b/source/WorkingPractices/multi_repository.rst index bf2270fd..8492d9b0 100644 --- a/source/WorkingPractices/multi_repository.rst +++ b/source/WorkingPractices/multi_repository.rst @@ -52,101 +52,7 @@ Do: .. _multirepo_testing: -Testing Changes Together ------------------------- -Multi-repository changes are expected to pass the regression tests for all the -repositories involved. To carry out the tests involved in a linked ticket it can -be helpful to refer to the semi-concentric circles above; layering the testing -from the inside out as needed. Further details of how testing in each -repository is handled can be found :ref:`here `. Compatible -code revisions are needed for testing across repositories as described above. - -Testing changes in JULES, LFRic Core, UKCA, or any other child repositories is -as simple as running the standalone test procedures for these codebases. - -.. important:: - - When specifying an alternative source in the ``dependencies.sh`` file the revision for the source **must** be updated. - - * If setting the source as an fcm URL, the mirror needs to be used and the revision can either be blank (for latest commit) or any valid revision for that branch. - * If setting the source as a Working Copy, the hostname needs to be provided (as Hostname:Path) and the revision must be blank. - -Testing the UM with other repositories -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -To test the UM, any changes to JULES, UKCA, Socrates, CASIM etc will also need -to be included. This is done by adding another source to the rose stem command -line. - - 1. :ref:`Checkout` a UM working copy - - - this may be your branch from a linked ticket, or a clean trunk copy - at either the last release or a suitable head of trunk revision. - - 2. Run rose stem, including a source code path to every branch involved. As a minimum - run ``developer`` group and all groups that cover the repositories being tested. - - .. code-block:: - - rose stem --group=developer,jules,ukca --source=. --source=/path/to/jules/changes --source=/path/to/ukca/changes - -The source paths involved can either be to local working copies or links to the -fcm source control e.g. ``fcm:jules.xm_br/dev/user/branch_name``. As many source -paths as needed can be added to the list. - -Testing LFRic Apps with other repositories -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -LFRic Apps testing needs to encompass all of the other repositories affected. -Paths to the other codebases involved should be added to -``dependencies.sh`` under each of the ``*_sources`` variables. Again -these paths can either be to local changes or those in the repository. - - 1. :ref:`Checkout` an LFRic Apps working copy - - - this may be your branch from a linked ticket, or a clean trunk copy - at either the last release or a suitable head of trunk revision. - - 2. Update dependencies.sh to point to all other code changes, e.g. - - .. code-block:: RST - - um_sources=vldXXX:/path/to/um/working/copy - jules_source=vldXXX:/path/to/jules/working/copy - - 3a. Run the lfric_atm developer test-suite - - - suitable for testing changes in other repositories that do not - include any LFRic Apps changes - - .. code-block:: - - export CYLC_VERSION=8 - rose stem --group=lfric_atm_developer - cylc play - - 3b. Run the full developer test-suite - - - suitable for testing LFRic Apps changes with other repositories, or expanding - testing if lfric_atm tests have shown errors. - - .. code-block:: - - export CYLC_VERSION=8 - rose stem --group=developer - cylc play - - More details on LFRic Apps testing are found :ref:`here`. - -.. note:: - If any of the testing shows up failures then there are two possible ways to - proceed: - - 1. The changes made should be re-written to avoid breaking the dependant - repositories - - 2. The changes made directly affect the interface between repositories and - therefore a change is also needed to the parent repository to adapt to that change. +Multi-Repo Testing +------------------ - If you're uncertain which route to take then the Code Owners involved will - hopefully be able to advise. +Please see :ref:`this page ` for details on how to test multiple repositories together. diff --git a/source/WorkingPractices/testing.rst b/source/WorkingPractices/testing.rst index 11fe16cc..92db35b5 100644 --- a/source/WorkingPractices/testing.rst +++ b/source/WorkingPractices/testing.rst @@ -33,16 +33,12 @@ Bespoke: TestSuites/lfric_core TestSuites/jules TestSuites/ukca + TestSuites/multi-repo_testing .. todo: TestSuites/casim TestSuites/shumlib -.. important:: - - Changes that affect multiple repositories will need testing together as - detailed :ref:`here `. - Test branches & Upgrade Macros ------------------------------ There are a few cases where testing your change will require you to make changes