Suggestions/ideas for updated and/or new WE2E tests

[mkavulich]

Description

This Issue is just a list of updates/changes that are needed to the existing WE2E tests: mainly to address currently untested features. We should keep this issue updated as new items are discovered, and note as others are resolved.

• The vast majority of tests still use old, staged nemsio input data. We should start migrating some (but not all!) of those tests to more up-to-date data. This will have added benefits of saving disk space by using much smaller netCDF or grib2 data files, and shorter data transfer times for tests that retrieve data from HPSS
• There are currently no tests for 8- or 9-hour forecasts; since most of the workflow scripts are still in bash, they can be prone to legacy "octal format" errors, which are only revealed by values "08" and "09".
• There are no tests for "off-hour" initial and end times; i.e. forecast start and/or end times that do not correspond to the traditional "main" times of 00z, 06z, 12z, and 18z
• There are dozens of SPP settings that are currently un-tested https://github.com/ufs-community/ufs-srweather-app/blob/develop/ush/config_defaults.yaml#L2282
• The Cheyenne coverage tests will need to be re-distributed to other platforms, since Cheyenne is being decommissioned at the end of 2023