Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[develop] Consolidate verification tasks using retrieve_data.py #864

Merged
merged 24 commits into from
Aug 15, 2023

Commits on Aug 14, 2023

  1. Configuration menu
    Copy the full SHA
    9070063 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    faf5f72 View commit details
    Browse the repository at this point in the history
  3. Convert exregional_get_obs_ccpa.sh to use retrieve_data.py...this ver…

    …sion may have bugs, I am committing so I can test on another platform since my Jet "fairshare" is apparently off-the-charts low
    mkavulich committed Aug 14, 2023
    Configuration menu
    Copy the full SHA
    1a5da5d View commit details
    Browse the repository at this point in the history
  4. Configuration menu
    Copy the full SHA
    b383c3f View commit details
    Browse the repository at this point in the history
  5. Fix bug in retrieve_data.py: when the glob expression for files in th…

    …e tarball contains a full path, we need to lstrip the leading "/" before globbing
    mkavulich committed Aug 14, 2023
    Configuration menu
    Copy the full SHA
    93a6ca4 View commit details
    Browse the repository at this point in the history
  6. Consolidate all "get_obs" tasks for vx into single J-job/exscript.

    All tasks now working for simple test case! Next up is to try all WE2E
    vx tests
    mkavulich committed Aug 14, 2023
    Configuration menu
    Copy the full SHA
    77fcc4c View commit details
    Browse the repository at this point in the history
  7. Now that retrieve_data.py checks for the existence of the output dire…

    …ctory, we need to create it prior to calling the script
    mkavulich committed Aug 14, 2023
    Configuration menu
    Copy the full SHA
    5230d7a View commit details
    Browse the repository at this point in the history
  8. Complete transition to new obs pulling script. Currently works to pull

    obs for tests MET_verification, MET_verification_only_vx, and
    MET_ensemble_verification (modified to pull rather than use staged data)
    
     - Remove old j-jobs and exscripts
     - Cleanup and update mrms_pull_topofhour.py to be more pythonic
     - Move to obs dir prior to pulling data to avoid potential conflicts
    from temporarily extracted files from HPSS
     - Move many comments from config_defaults.yaml to more appropriate
    location in new data pulling/staging script; add more details about data
    quirks
    mkavulich committed Aug 14, 2023
    Configuration menu
    Copy the full SHA
    58ef36f View commit details
    Browse the repository at this point in the history
  9. Get rid of unneccessary directory structure requirement for obs. All …

    …verification tests now pass when retrieving from HPSS!
    mkavulich committed Aug 14, 2023
    Configuration menu
    Copy the full SHA
    b199379 View commit details
    Browse the repository at this point in the history
  10. Configuration menu
    Copy the full SHA
    906846b View commit details
    Browse the repository at this point in the history
  11. Updates to exregional_get_verif_obs.sh:

     - Remove usage of "*_vrfy" functions due to issues mentioned in Issue
       861 (ufs-community#861)
     - Remove extra "mkdir" statement where it shouldn't be for MRMS
     - Use correct variable (vhh_noZero) for checking valid hour for NDAS
    mkavulich committed Aug 14, 2023
    Configuration menu
    Copy the full SHA
    e998ded View commit details
    Browse the repository at this point in the history
  12. Updates to verification tests:

     - Remove stanzas eliminating the "task_get_obs*" tasks; these should
       now be run every time (if using staged data, they will check that the
       data exists)
     - Modify tasks RRFS_CONUS_25km_ics_FV3GFS_lbcs_RAP_suite_RAP
       (MET_verification) and MET_ensemble_verification_only_vx_time_lag to
       retrieve obs data from HPSS
     - Fix MET_ensemble_verification_only_vx_time_lag test by correcting
       variable NET to NET_default (addresses issue 851)
     - Add new verification task custom_ESGgrid_Eastern_NA_snow_8km that
       will have winter weather verification tasks (for subsequent PR)
    mkavulich committed Aug 14, 2023
    Configuration menu
    Copy the full SHA
    d4cb266 View commit details
    Browse the repository at this point in the history
  13. More testing changes/updates

     - Don't retrieve obs data from HPSS for
       grid_RRFS_CONUS_25km_ics_FV3GFS_lbcs_RAP_suite_RAP; too many
       platforms use this test
     - Rename custom_ESGgrid_Eastern_NA_snow_8km to
       custom_ESGgrid_Great_Lakes_snow_8km; add to comprehensive and Jet
       coverage test
     - Add MET_ensemble_verification_only_vx to comprehensive for Cheyenne
       and Orion, as well as to coverage suite via Hera/GNU
     - Move MET_verification_only_vx coverage from Hera/GNU to Cheyenne/GNU
     - Add new mysterious "2020_CAD" test to all comprehensive lists, not
       just Orion (what's up with that?), sort alphabetically
    mkavulich committed Aug 14, 2023
    Configuration menu
    Copy the full SHA
    b666879 View commit details
    Browse the repository at this point in the history
  14. Configuration menu
    Copy the full SHA
    2882610 View commit details
    Browse the repository at this point in the history
  15. Configuration menu
    Copy the full SHA
    9cfbcd1 View commit details
    Browse the repository at this point in the history
  16. Configuration menu
    Copy the full SHA
    712207e View commit details
    Browse the repository at this point in the history
  17. Configuration menu
    Copy the full SHA
    48fb9d3 View commit details
    Browse the repository at this point in the history
  18. Remove immediate quit when specified output path does not exist; this…

    … didn't account for templating behavior
    mkavulich committed Aug 14, 2023
    Configuration menu
    Copy the full SHA
    2dbdf94 View commit details
    Browse the repository at this point in the history
  19. Configuration menu
    Copy the full SHA
    ef161a2 View commit details
    Browse the repository at this point in the history
  20. Roll new snowfall/NOHRSC verification into new data retrieval system

     - Remove stand-alone j-jobs and exscripts for NOHRSC
     - Add NOHRSC_obs stanza to parm/data_locations.yml
     - Update documentation and comments to include NOHRSC obs
     - Add "ASNOW" (accumulated snow) verification to
       custom_ESGgrid_Great_Lakes_snow_8km
     -
    mkavulich committed Aug 14, 2023
    Configuration menu
    Copy the full SHA
    0fde1cb View commit details
    Browse the repository at this point in the history
  21. Cheyenne can't automatically get AWS data, removing 2020_CAD test fro…

    …m comprehensive.cheyenne
    mkavulich committed Aug 14, 2023
    Configuration menu
    Copy the full SHA
    b39decf View commit details
    Browse the repository at this point in the history
  22. Address Gerard's comments

    mkavulich committed Aug 14, 2023
    Configuration menu
    Copy the full SHA
    e35d28d View commit details
    Browse the repository at this point in the history
  23. Move machine file definitions of *_OBS_DIRs to TEST_*_OBS_DIR for WE2…

    …E testing purposes, similar to TEST_ALT_EXTRN_MDL_SYSBASEDIR_ICS
    mkavulich committed Aug 14, 2023
    Configuration menu
    Copy the full SHA
    624084d View commit details
    Browse the repository at this point in the history

Commits on Aug 15, 2023

  1. Configuration menu
    Copy the full SHA
    6b97050 View commit details
    Browse the repository at this point in the history