DEPHY conversion #496

grantfirl · 2024-07-24T20:49:58Z

Add dephy_converter.py script that takes one argument (-n name_of_case) where name_of_case is found in scm/etc/case_config and data file found in scm/data/processed_case_input and produces a DEPHY-formatted case namelist and data file
This script successfully converted all previous cases except for fv3_model_point_noah that was outdated.
Files in scm/etc/case_config are updated
Files in scm/data/processed_case_input were updated in v7.0.0-beta release asset
Fixed small bugs in run_scm.py, scm_forcing.F90, and scm_input.F90

…th non-conforming COMBLE case DEPHY file

…rcingT to be true and surfaceforcinglsm to be true - ingests LSM data and a forcing surface T

grantfirl · 2024-08-01T19:42:19Z

#481 and #463 have been combined into here.

grantfirl · 2024-08-01T19:42:47Z

I have updated DEPHY files for all data in data/processed_case_input ready to be made into a release asset.

…ts (with new DEPHY-format processed_case_input files)

grantfirl · 2024-08-01T20:10:33Z

contrib/get_all_static_data.sh

@@ -21,7 +21,7 @@ for file in "${data_files[@]}"; do
    mkdir -p $BASEDIR/scm/data/$file
    cd $BASEDIR/scm/data/$file
    echo "Retrieving $file"
-    wget https://github.com/NCAR/ccpp-scm/releases/download/v6.0.0/${file}.tar.gz
+    wget https://github.com/NCAR/ccpp-scm/releases/download/v7.0.0-beta/${file}.tar.gz


@mkavulich As soon as there is a v7.0.0-beta release on GitHub, this script is ready to go and CI should use the new files.

grantfirl · 2024-08-01T21:59:44Z

scm/etc/scripts/MOSAiC_AMPS_forcing_file_generator.py

+#height = ffc.get_height_from_pres(T_abs[:,0],levels,z_sfc)
+
+#the following variables are not in this forcing file, but are included in other cases
+soil_depth = np.zeros(4,dtype=float)


@hertneky It's not showing as changed because this file is new, but I added this (needed by the dephy_converter.py script when lsm_ics = T)

grantfirl · 2024-08-01T22:01:04Z

scm/etc/scripts/MOSAiC_AMPS_forcing_file_generator.py

+snwdph   = 2.e-4
+tg3      = 271.35
+zorl     = 15.0
+alvsf    = 0.06


@hertneky Updated these (alvsf, alnsf, alvwf, alnwf, facsf, facwf). The previous values were from the GABLS3 case, but those are specific to that location. These are the values from the UFS fix files for this lat/lon. It doesn't seem to change the results though.

grantfirl · 2024-08-01T22:01:27Z

scm/etc/scripts/MOSAiC_AMPS_forcing_file_generator.py

+canopy   = 0.0
+vegtyp   = 0
+soiltyp  = 0
+scolor   = 1


@hertneky Added scolor needed by lsm_ics.

grantfirl · 2024-08-01T22:01:44Z

scm/etc/scripts/MOSAiC_AMPS_forcing_file_generator.py

+shdmax   = 0.0
+slopetyp = 1
+snoalb   = 0.0
+sncovr   = 1.0


@hertneky Added sncovr needed by lsm_ics.

grantfirl · 2024-08-01T22:02:25Z

scm/etc/scripts/MOSAiC_AMPS_forcing_file_generator.py

+writefile_zorl_var.units = 'cm'
+writefile_zorl_var.description = 'composite surface roughness length'
+
+writefile_zorll_var = writefile_scalar_grp.createVariable('zorll', 'f4')


@hertneky Writing out zorll, zorlw, zorli as identical to zorl -- needed by lsm_ics.

grantfirl · 2024-08-01T22:03:21Z

scm/etc/scripts/MOSAiC_SS_forcing_file_generator.py

@@ -0,0 +1,590 @@
+#!/usr/bin/env python


@hertneky Everything that I added to the MOSAiC_AMPS_forcing... file, I repeated here.

grantfirl · 2024-08-01T22:07:44Z

scm/src/scm_input.F90

-  call NetCDF_read_var(ncid, "pa", .True., input_pres)
-  call NetCDF_read_var(ncid, "zh", .True., input_height)
+
+  if (lev_in_altitude) then


@bluefinweiwei This code change is to accommodate the COMBLE case that doesn't supply everything that a DEPHY file should have. One has a choice of what to put in the 'lev' data in the NetCDF file (according to the DEPHY file specification). This code can now handle whether the data is altitude in m or pressure levels in Pa. With this change, the other changes that you had to the scm_input.F90 file are taken care of.

grantfirl · 2024-08-01T22:10:47Z

scm/src/scm_input.F90

-  else if (trim(input_surfaceForcingLSM) == 'lsm') then
+  end if
+
+  if (trim(input_surfaceForcingLSM) == 'lsm') then


Note to self: This is needed to not have LSM ICs be mutually exclusive to surfaceForcingT == 'ts'. For the MOSAiC cases, we have a forcing surface T and LSM IC data for the sea ice. This allows us to read in LSM IC data and force the surface temperature, like the case wants. This does not break the GABLS3 cases (tested).

grantfirl · 2024-08-01T22:12:31Z

scm/src/scm_input.F90

@@ -2259,8 +2318,8 @@ subroutine get_case_init_DEPHY(scm_state, scm_input)
    !set all individual w forcing controls to .true. until finer control is available from the input file
    scm_state%force_sub_for_T = .true.
    scm_state%force_sub_for_qv = .true.
-    scm_state%force_sub_for_u = .true.
-    scm_state%force_sub_for_v = .true.
+    !scm_state%force_sub_for_u = .true.


This can potentially be uncommented. Subsidence forcing typically isn't applied to horizontal momentum and the DEPHY standard doesn't have a control for which variables to apply w or omega forcing to. This is simply an assertion on my part.

grantfirl · 2024-08-01T22:13:00Z

test/rt_test_cases.py

@@ -27,6 +27,17 @@
            {"case": "LASSO_2016051812",      "suite": "SCM_GFS_v16"},                                                                                    \
            {"case": "LASSO_2016051812",      "suite": "SCM_WoFS_v0"},                                                                                    \
            {"case": "LASSO_2016051812",      "suite": "SCM_HRRR_gf"},                                                                                    \
+            {"case": "COMBLE",                "suite": "SCM_GFS_v17_p8_ugwpv1"},                                                                          \


@ligiabernardet @dustinswales Added COMBLE/MOSAiC to RTs

…shorter and will make list maintenance less of a hassle if changes are needed

- Fix calls to logging.critical() with no argument

mkavulich

@grantfirl I have opened two PRs to your branch with proposed changes (only one of which should be merged): one of which is my "preferred" changes and the other I'd consider "essential" changes. Note that I haven't run any tests yet (since I'm not sure where to get that data) but the changes should still produce bit-for-bit identical output.

grantfirl#11 These are my preferred changes. I understand that it is invasive, but it eliminates so much code redundancy that doesn't seem necessary. Also some fixes to problems mentioned below.

grantfirl#12 These are my "essential" changes. It converts the huge multi-line regular string with arguments at the end to a multi-line f-string, which is much easier to maintain.

Both of the above PRs include two bug fixes:

a fix to a bug which may require that data gets re-converted: w_d was assigned the value of w_0.
Several calls to logging.critical() that included no message string as argument, which is required.

I also included "debug" as a new argument to get debug output rather than requiring manual editing of code.

scm/etc/scripts/dephy_converter.py

Essential Updates for PR 496

Co-authored-by: Michael Kavulich <[email protected]>

mkavulich · 2024-08-12T16:10:13Z

scm/etc/scripts/dephy_converter.py

        logging.basicConfig(format='%(levelname)s: %(message)s', level=logging.DEBUG)
+    else:


Whoops, good catch!

mkavulich

Thanks for making those changes, just a few more minor suggestions that are not required for merging.

scm/etc/scripts/scm_analysis.py

scm/src/scm_input.F90

scm/etc/scripts/scm_read_obs.py

dustinswales

@grantfirl These changes look good to me. Thanks for this, it was a big effort to convert all of the cases to DEPHY.
I had a super picky comment, but nothing to hold this up.

scm/etc/case_config/COMBLE.nml

scm/etc/scripts/MOSAiC_AMPS_forcing_file_generator.py

Co-authored-by: Michael Kavulich <[email protected]>

grantfirl · 2024-08-29T21:54:15Z

COMBLE case plots:
profiles_mean_T.pdf

profiles_mean_qv.pdf

profiles_mean_rad_cloud_fraction.pdf
time_series_shf.pdf
time_series_lhf.pdf
profiles_mean_multi_T_tendencies.pdf

profiles_mean_multi_q_tendencies.pdf
contour_qc.pdf

grantfirl · 2024-08-29T21:56:56Z

MOSAiC-AMPS plots:
profiles_mean_T.pdf
profiles_mean_qv.pdf
profiles_mean_rad_cloud_fraction.pdf
time_series_shf.pdf
profiles_mean_multi_T_tendencies.pdf
profiles_mean_multi_q_tendencies.pdf
contour_qc.pdf

Tracy and others added 10 commits April 4, 2024 18:27

Update hera module for rocky8, remove shell setup scripts

2c81c84

update spack versio

9da57a8

add files for 2 MOSAiC cases from Amy Solomon

936b893

Merge branch 'NCAR:main' into add_MOSAic_case

ce1c106

Merge branch 'main' into dephy_conversion

1aed401

add dephy_converter.py

dd673f2

Merge branch 'main' into dephy_conversion

636450a

working version (at least for bomex)

19aea49

only add vertical advective forcing if forc_w or forc_wap are not on

b25af21

add surface variables and updated case_config namelists

cc937f5

grantfirl requested review from dustinswales and mkavulich as code owners July 24, 2024 20:49

grantfirl added 4 commits July 30, 2024 10:53

Merge branch 'main' into dephy_conversion

298b59b

add COMBLE case configuration and changes to scm_input.F90 to work wi…

3ba2954

…th non-conforming COMBLE case DEPHY file

Merge branch 'add_MOSAic_case' into dephy_conversion

48d180e

update and convert MOSAiC cases to DEPHY format; allow both surfacefo…

7eae638

…rcingT to be true and surfaceforcinglsm to be true - ingests LSM data and a forcing surface T

This was referenced Aug 1, 2024

Add COMBLE case to CCPP SCM #463

Closed

Add MOSAiC case to CCPP-SCM #481

Closed

grantfirl added 2 commits August 1, 2024 16:04

add new cases to rt_test_cases.py

6e449c5

update get_all_static_data.sh script to grab v7.0.0-beta release asse…

684448d

…ts (with new DEPHY-format processed_case_input files)

grantfirl commented Aug 1, 2024

View reviewed changes

grantfirl mentioned this pull request Aug 8, 2024

NML changes for HRRR_gf and GFS_v17_p8_ugwpv1 #500

Closed

mkavulich added 2 commits August 8, 2024 20:01

Convert enormous regular string to multi-line f-string. This is much …

6fe54a5

…shorter and will make list maintenance less of a hassle if changes are needed

- Convert logging level to command-line argument

18883a7

- Fix calls to logging.critical() with no argument

mkavulich requested changes Aug 9, 2024

View reviewed changes

grantfirl and others added 6 commits August 9, 2024 19:05

Merge pull request #12 from mkavulich/dephy_conversion

7bbe0ce

Essential Updates for PR 496

fix w_0/w_d bug caught by @mkavulich

c6801d2

Update scm/etc/scripts/dephy_converter.py

1d5c77f

Co-authored-by: Michael Kavulich <[email protected]>

Update scm/etc/scripts/dephy_converter.py

f40427b

Co-authored-by: Michael Kavulich <[email protected]>

set default logging level to info

9a4c526

Merge branch 'main' into dephy_conversion

11ad77b

grantfirl requested a review from mkavulich August 10, 2024 00:04

fix debug logging logic

c5dc7b0

mkavulich reviewed Aug 12, 2024

View reviewed changes

mkavulich approved these changes Aug 12, 2024

View reviewed changes

scm/etc/scripts/scm_analysis.py Outdated Show resolved Hide resolved

scm/src/scm_input.F90 Outdated Show resolved Hide resolved

scm/src/scm_input.F90 Show resolved Hide resolved

scm/src/scm_input.F90 Outdated Show resolved Hide resolved

scm/etc/scripts/scm_read_obs.py Show resolved Hide resolved

dustinswales approved these changes Aug 12, 2024

View reviewed changes

scm/etc/case_config/COMBLE.nml Outdated Show resolved Hide resolved

scm/etc/scripts/MOSAiC_AMPS_forcing_file_generator.py Show resolved Hide resolved

grantfirl and others added 6 commits August 13, 2024 11:52

Update scm/etc/scripts/scm_analysis.py

4b4cfd3

Co-authored-by: Michael Kavulich <[email protected]>

Update scm/src/scm_input.F90

15bea31

Co-authored-by: Michael Kavulich <[email protected]>

Update scm/src/scm_input.F90

9a842b9

Co-authored-by: Michael Kavulich <[email protected]>

Update scm/src/scm_input.F90

dbce3ff

Co-authored-by: Michael Kavulich <[email protected]>

fix indenting in scm_read_obs.py"

4e12394

fix COMBLE case namelist format

4c9559c

grantfirl mentioned this pull request Aug 13, 2024

Namelist updates and runtime fixes #502

Merged

grantfirl merged commit 7ede232 into NCAR:main Aug 13, 2024
19 of 24 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

DEPHY conversion #496

DEPHY conversion #496

grantfirl commented Jul 24, 2024 •

edited

Loading

grantfirl commented Aug 1, 2024

grantfirl commented Aug 1, 2024

grantfirl Aug 1, 2024

grantfirl Aug 1, 2024

grantfirl Aug 1, 2024 •

edited

Loading

grantfirl Aug 1, 2024

grantfirl Aug 1, 2024

grantfirl Aug 1, 2024

grantfirl Aug 1, 2024

grantfirl Aug 1, 2024 •

edited

Loading

grantfirl Aug 1, 2024

grantfirl Aug 1, 2024

grantfirl Aug 1, 2024

mkavulich left a comment

mkavulich Aug 12, 2024

mkavulich left a comment

dustinswales left a comment

grantfirl commented Aug 29, 2024

grantfirl commented Aug 29, 2024

		logging.basicConfig(format='%(levelname)s: %(message)s', level=logging.DEBUG)
		else:

DEPHY conversion #496

DEPHY conversion #496

Conversation

grantfirl commented Jul 24, 2024 • edited Loading

grantfirl commented Aug 1, 2024

grantfirl commented Aug 1, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

grantfirl Aug 1, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

grantfirl Aug 1, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mkavulich left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mkavulich left a comment

Choose a reason for hiding this comment

dustinswales left a comment

Choose a reason for hiding this comment

grantfirl commented Aug 29, 2024

grantfirl commented Aug 29, 2024

grantfirl commented Jul 24, 2024 •

edited

Loading

grantfirl Aug 1, 2024 •

edited

Loading

grantfirl Aug 1, 2024 •

edited

Loading