Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

GFSv16.3.20 - OBSPROCv1.3 and updated USAF snow filename #2913

Open
KateFriedman-NOAA opened this issue Sep 12, 2024 · 15 comments
Open

GFSv16.3.20 - OBSPROCv1.3 and updated USAF snow filename #2913

KateFriedman-NOAA opened this issue Sep 12, 2024 · 15 comments
Assignees
Labels
production update Processing update in production

Comments

@KateFriedman-NOAA
Copy link
Member

KateFriedman-NOAA commented Sep 12, 2024

Description

The snow data file provided by the USAF has changed names and the new name will go into affect with obsproc.v1.3. The GFS will need to update to use the new obsproc.v1.3 and the new snow filename. Additionally, the GSI will require a change in the GSI code (read_prepbufr.f) for the saildrone data.

The prepobs package is also updating to v1.2.0 with obsproc/v1.3.0. This version is internal to the obsproc package but we'll want to update both obsproc and prepobs versions in develop for GFSv17. Will handle that separately.

FYI @ilianagenkova @GeorgeGayno-NOAA @ADCollard

Target version

v16.3.20 (need confirmation)

Expected workflow changes

Update obsproc_ver=v1.3.

Updated GSI tag with saildrone change.

The updated UFS_UTILS release tag is being worked under: Issue #973

The test branch is here: release/ops-gfsv16.3.1

Will need to coordinate to pass in the global afwa file from the JJOB to ./scripts/exemcsfc_global_sfc_prep.sh

@KateFriedman-NOAA KateFriedman-NOAA added the production update Processing update in production label Sep 12, 2024
@KateFriedman-NOAA KateFriedman-NOAA changed the title GFSv16.?.? - updated USAF snowfile GFSv16.?.? - Update to obsproc/v1.3 and updated USAF snowfile Sep 12, 2024
@KateFriedman-NOAA KateFriedman-NOAA changed the title GFSv16.?.? - Update to obsproc/v1.3 and updated USAF snowfile GFSv16.?.? - Update to obsproc.v1.3 and updated USAF snowfile Sep 12, 2024
@KateFriedman-NOAA
Copy link
Member Author

Release branch for this created: https://github.com/NOAA-EMC/global-workflow/tree/release/gfs.v16.3_obsproc_snow

Will rename release branch to version once the version is known.

@GeorgeGayno-NOAA
Copy link
Contributor

GeorgeGayno-NOAA commented Sep 13, 2024

The new snow file will need to be exported from here.
https://github.com/NOAA-EMC/global-workflow/blob/release/gfs.v16.3.20/jobs/JGLOBAL_ATMOS_EMCSFC_SFC_PREP#L67

I have been using AFWA_GLOBAL_FILE. The AFWA_NH_FILE and AFWA_SH_FILE variables can be removed.

How can we test this?

@KateFriedman-NOAA
Copy link
Member Author

How can we test this?

@ilianagenkova Do you have a para COM for obsproc.v1.3 that we could point to for testing this?

@ilianagenkova
Copy link
Contributor

On Dogwood: /lfs/h2/emc/stmp/iliana.genkova/CRON/R13/com/obsproc/v1.3/g*/*/atmos/usaf

-rw-r--r-- 1 iliana.genkova emc 42315832 Sep 12 05:40 gdas.20240912/00/atmos/gdas.t00z.snow.usaf.grib2
-rw-r--r-- 1 iliana.genkova emc 42315832 Sep 12 11:40 gdas.20240912/06/atmos/gdas.t06z.snow.usaf.grib2
-rw-r--r-- 1 iliana.genkova emc 42315832 Sep 12 17:40 gdas.20240912/12/atmos/gdas.t12z.snow.usaf.grib2
-rw-r--r-- 1 iliana.genkova emc 42315832 Sep 12 23:40 gdas.20240912/18/atmos/gdas.t18z.snow.usaf.grib2

@KateFriedman-NOAA
Copy link
Member Author

Branch renamed and recut based on release/gfs.v16.3.19 state: https://github.com/NOAA-EMC/global-workflow/tree/release/gfs.v16.3.20

@KateFriedman-NOAA KateFriedman-NOAA changed the title GFSv16.?.? - Update to obsproc.v1.3 and updated USAF snowfile GFSv16.3.20 - Update to obsproc.v1.3 and updated USAF snowfile Sep 23, 2024
@KateFriedman-NOAA KateFriedman-NOAA changed the title GFSv16.3.20 - Update to obsproc.v1.3 and updated USAF snowfile GFSv16.3.20 - OBSPROCv1.3 and updated USAF snow filename Sep 23, 2024
@KateFriedman-NOAA
Copy link
Member Author

KateFriedman-NOAA commented Sep 23, 2024

On Dogwood: /lfs/h2/emc/stmp/iliana.genkova/CRON/R13/com/obsproc/v1.3/g*/*/atmos/usaf

@ilianagenkova Is this available on Cactus as well?

KateFriedman-NOAA added a commit to KateFriedman-NOAA/global-workflow that referenced this issue Sep 23, 2024
KateFriedman-NOAA added a commit to KateFriedman-NOAA/global-workflow that referenced this issue Sep 23, 2024
Replace AFWA_NH_FILE and AFWA_SH_FILE with AFWA_GLOBAL_FILE

Refs NOAA-EMC#2913
@KateFriedman-NOAA
Copy link
Member Author

@ilianagenkova Do you have a copy of the new obsproc release/obsproc.v1.3.0 branch on Cactus that I can point to for testing the GFS? Or shall I build my own copy for testing?

KateFriedman-NOAA added a commit to KateFriedman-NOAA/global-workflow that referenced this issue Sep 23, 2024
Also include mention of related updates for AFWA snow filename
and GSI code update.

Refs NOAA-EMC#2913
@KateFriedman-NOAA
Copy link
Member Author

/lfs/h2/emc/stmp/iliana.genkova/CRON/R13/com/obsproc/v1.3/g*/*/atmos/usaf

Nevermind, I see it now, thanks!

@ilianagenkova Do you have a copy of the new obsproc release/obsproc.v1.3.0 branch on Cactus that I can point to for testing the GFS? Or shall I build my own copy for testing?

I've gone ahead and installed copies of both the obsproc and prepobs release branches for this upgrade.

I will use the provided COM and installs to test the GFS on Cactus.

@KateFriedman-NOAA
Copy link
Member Author

@ilianagenkova I ran the GFSv16 workflow with the new obsproc/v1.3, prepobs/v1.2, USAF snow file from your COM, and UFS_UTILS release branch with snow filename updates. I ran the emcsfc_sfc_prep job that produces the GFS snogrb files and which uses the new USAF snow filename. I then ran the prep job to create a prepbufr file using the generated snogrb files, new obsproc, and new prepobs. @GeorgeGayno-NOAA has checked the emcsfc_sfc_prep job and confirmed it ran successfully.

Please take a look at the following log and output on WCOSS2-Cactus and let me know if the prep job ran as expected. Thanks!

output: /lfs/h2/emc/ptmp/kate.friedman/comrot/testobsproc13/gdas.20240924/00/atmos/
log: /lfs/h2/emc/ptmp/kate.friedman/comrot/testobsproc13/logs/2024092400/gdasprep.log

@ilianagenkova
Copy link
Contributor

ilianagenkova commented Sep 24, 2024

@KateFriedman-NOAA , thanks for testing the processing end-to-end. The prep job output looks good!

[clogin06 /lfs/h2/emc/ptmp/ashley.stanfield/CRON]$ ll /lfs/h2/emc/ptmp/kate.friedman/comrot/testobsproc13/gdas.20240924/00/atmos/*prepbufr*
-rw-r----- 1 kate.friedman rstprod 115100896 Sep 24 18:01 /lfs/h2/emc/ptmp/kate.friedman/comrot/testobsproc13/gdas.20240924/00/atmos/gdas.t00z.prepbufr
-rw-r----- 1 kate.friedman rstprod  21224440 Sep 24 18:01 /lfs/h2/emc/ptmp/kate.friedman/comrot/testobsproc13/gdas.20240924/00/atmos/gdas.t00z.prepbufr.acft_profiles
lrwxrwxrwx 1 kate.friedman emc           102 Sep 24 17:51 lrwxrwxrwx 1 kate.friedman emc           106 Sep 24 17:51 
[clogin06 /lfs/h2/emc/stmp/iliana.genkova]$ ll  CRON/R13/com/obsproc/v1.3//gdas.20240924/00/atmos/*prepbufr*                                                                                                                                                                            
-rw-r----- 1 iliana.genkova rstprod 115100896 Sep 24 06:26 CRON/R13/com/obsproc/v1.3//gdas.20240924/00/atmos/gdas.t00z.prepbufr
-rw-r----- 1 iliana.genkova rstprod  21224440 Sep 24 06:26 CRON/R13/com/obsproc/v1.3//gdas.20240924/00/atmos/gdas.t00z.prepbufr.acft_profiles

@KateFriedman-NOAA
Copy link
Member Author

@ilianagenkova Excellent! I see I even reproduced your prepbufr file:

kate.friedman@clogin01:/lfs/h2/emc/ptmp/kate.friedman/comrot/testobsproc13> cmp gdas.20240924/00/atmos/gdas.t00z.prepbufr /lfs/h2/emc/stmp/iliana.genkova/CRON/R13/com/obsproc/v1.3//gdas.20240924/00/atmos/gdas.t00z.prepbufr
kate.friedman@clogin01:/lfs/h2/emc/ptmp/kate.friedman/comrot/testobsproc13>

Thanks for checking and confirming!

@KateFriedman-NOAA
Copy link
Member Author

@GeorgeGayno-NOAA I think we're ready for a UFS_UTILS tag to include in the release branch for this update.

@GeorgeGayno-NOAA
Copy link
Contributor

@GeorgeGayno-NOAA I think we're ready for a UFS_UTILS tag to include in the release branch for this update.

See: https://github.com/ufs-community/UFS_UTILS/releases/tag/ops-gfsv16.3.20

KateFriedman-NOAA added a commit to KateFriedman-NOAA/global-workflow that referenced this issue Sep 25, 2024
KateFriedman-NOAA added a commit to KateFriedman-NOAA/global-workflow that referenced this issue Sep 25, 2024
KateFriedman-NOAA added a commit to KateFriedman-NOAA/global-workflow that referenced this issue Sep 25, 2024
Add it back if decision is made to update for saildrone.

Refs NOAA-EMC#2913
KateFriedman-NOAA added a commit to KateFriedman-NOAA/global-workflow that referenced this issue Sep 26, 2024
Add further information to describe change to AFWA global snow file

Refs NOAA-EMC#2913
KateFriedman-NOAA added a commit that referenced this issue Sep 26, 2024
Updates for the new `obsproc/v1.3` and AFWA global snow filename.
A new UFS_UTILS tag (`ops-gfsv16.3.20`) is used for the snow filename changes.

The prepobs package is also updated to `v1.2.0` but is internal to the upstream obsproc
package so only the `prepobs_run_ver` variables are updated for dev mode usage.

Additionally, workflow updates are made to allow the EMCSFC_SFC_PREP (sfcprep)
job to be run as part of the rocoto mesh in dev/emc mode. A new switch `DO_SFCPREP`
is added to `config.base.emc.dyn` and is set to "NO" by default. Setting it to "YES" 
will add the `sfcprep` job to the rocoto mesh ahead of the `prep` job and will generate
the snogrb files that are usually generated in ops and used via the GDA.

Refs #2913
@KateFriedman-NOAA
Copy link
Member Author

Once the ARFC in issue #2951 goes in I will sync those updates into the release branch for this.

Only remaining question is whether a GSI update to handle saildrone data in the new obsproc/v1.3 should be included in this or possibly held for the upcoming GFSv16.4 DA upgrade. @RussTreadon-NOAA @ilianagenkova do we know what the status of that change is?

@KateFriedman-NOAA
Copy link
Member Author

Synced dev/gfs.v16 into the release/gfs.v16.3.20 branch and reverted the commit to remove WAFS (this branch was originally cut from the WAFS removal branch).

KateFriedman-NOAA added a commit to KateFriedman-NOAA/global-workflow that referenced this issue Oct 2, 2024
KateFriedman-NOAA added a commit to KateFriedman-NOAA/global-workflow that referenced this issue Oct 2, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
production update Processing update in production
Projects
None yet
Development

No branches or pull requests

3 participants