-
Notifications
You must be signed in to change notification settings - Fork 188
Testing Procedures
Contents
- 1 Purpose of this page
-
2 Information that should be included in each component's documentation
- 2.1 Brief summary of the testing, merging and tagging process
- 2.2 How to checkout the code for testing
- 2.3 Test commands to run
- 2.4 Baseline location(s)
- 2.5 How to determine expected failures
- 2.6 Does full testing need to be run on each set of changes?
- 2.7 Does a branch need to be up-to-date with the latest main branch before final testing?
- 2.8 Rough estimate of test turnaround time and core-hour cost
- 2.9 Any other important notes about the testing process
- 2.10 Where to find information on testing procedures for the component's externals
- 3 Components
This page provides links to documentation describing the testing procedures for each component / repository that makes up CESM (or in some cases this page may provide this documentation directly). The audience is CESM software engineers who are familiar with CESM / CIME system testing in general, but who need to run testing on a component that they do not regularly work with.
Many components have different testing procedures that are run depending on what has changed. This page is not meant to be exhaustive, but rather is meant to capture the most common testing procedures – e.g., procedures that should cover about 90% of cases.
Components can organize their documentation in whatever way they wish. However, they should aim to include at least the following information. Ideally, the documentation's organization will allow readers to quickly find these pieces of information. The goal is to make it relatively easy for CESM software engineers to run the testing for a different component when the need arises; needing to spend an hour reading through documentation defeats that goal.
Give a very brief overview of the testing, merging and tagging process for this component. Some example questions to cover are:
- Is it expected that an issue is opened for every planned change?
- Is there a ChangeLog entry for each merge to the main branch?
Does this component support a standalone checkout for testing, or does it need to be checked out in the context of CESM or some other "umbrella" repository? If the latter, describe how to determine a reasonable version of the umbrella repository to use for testing – and in particular, how to determine what versions will have baselines available for system tests of the component.
Give details of the exact test commands to run to achieve full system testing, and on what machine(s). This does not need to exhaustively cover all possible testing scenarios. Rather, it is meant to cover the typical cases – say, the common 90% of scenarios. The testing commands should be detailed enough that someone can essentially copy and paste the commands into a terminal (replacing some placeholder text, e.g., for the baseline tag name to be used for comparison). Different components have different procedures for how to kick off their test suites – for example, whether a separate create_test invocation is made for each compiler in the test suite – so please be very explicit in these instructions.
If there is anything non-standard in how to check the test results, document that as well.
For each test machine, give the location on disk in which baselines can be found for the above testing.
Describe how expected failures can be determined for any given tag.
Does the full testing described above need to be run on each set of changes? Or is it common to combine multiple changes / PRs together and have a component SE run full testing on the batch of changes? If the latter, is there some minimum expected / recommended testing on the individual changes?
State whether a feature branch needs to merge in the latest version of the main branch before running final testing. If so, there will typically be some process for tag ordering, so briefly describe that process.
Give a very rough estimate of test turnaround time and core-hour cost, to help answer the question: if I run this test suite a number of times, am I going to burn through my allocation?
Add anything else that is important but not covered above. Include any references to additional relevant documentation.
For components that have their own externals (e.g., CTSM's FATES, or CAM's CARMA, CLUBB, PUMAS, etc.): provide links to any documentation on procedures for changing those externals.