Skip to content

Commit

Permalink
bug fix: disable concurrency in GFS_phys_time_vary_init NetCDF calls…
Browse files Browse the repository at this point in the history 
… AND #2040 and #2043 (#2041)

General clean-up, and CCPP concurrency bug fixes

1. Remove nfhout, nfhmax_hf, nfhout_hf and nsout from fv3atm and the regression tests. ( Remove nfhout, nfhmax_hf,  nfhout_hf etc configuration parameters NOAA-EMC/fv3atm#731 )
2. Add comments to smc pert and fix bug in stc pert ( Fix bugs in soil temperature perturbation #2042 )
3. Disable concurrency in NetCDF calls within CCPP GFS_phys_time_vary_init subroutine to avoid crashes
  • Loading branch information
@SamuelTrahanNOAA
SamuelTrahanNOAA authored Dec 19, 2023
1 parent 10635ef commit 1d90c04
Show file tree
Hide file tree
Showing 59 changed files with 5,347 additions and 5,775 deletions.
2 changes: 1 addition & 1 deletion FV3
Submodule FV3 updated 6 files
+2 −2 atmos_model.F90
+1 −1 ccpp/physics
+3 −70 fv3_cap.F90
+1 −1 io/module_fv3_io_def.F90
+2 −3 io/module_wrt_grid_comp.F90
+1 −1 module_fv3_config.F90
20 changes: 0 additions & 20 deletions doc/UsersGuide/source/InputsOutputs.rst
View file
Original file line number Diff line number Diff line change
Expand Up @@ -1404,10 +1404,6 @@ shows the following parameters that can be set in ``model_configure`` at run-tim
- atmosphere time step in second
- integer
- 1800 (for C96)
* - output_1st_tstep_rst
- output first time step history file after restart
- logical
- .false.
* - restart_interval
- frequency to output restart file or forecast hours to write out restart file
- integer
Expand Down Expand Up @@ -1452,22 +1448,6 @@ shows the following parameters that can be set in ``model_configure`` at run-tim
- j-dimension for output grid
- integer
- 190
* - nfhout
- history file output frequency
- integer
- 3
* - nfhmax_hf
- forecast length of high history file
- integer
- 0 (0:no high frequency output)
* - nfhout_hf
- high history file output frequency
- integer
- 1
* - nsout
- output frequency of number of time step
- integer
- -1 (negative: turn off the option, 1: output history file at every time step)
* - output_fh
- history file output forecast hours or history file output frequency if the second elelment is -1
- real
Expand Down
2 changes: 1 addition & 1 deletion stochastic_physics
Submodule stochastic_physics updated 1 files
+3 −1 lndp_apply_perts.F90
14 changes: 1 addition & 13 deletions tests/default_vars.sh
View file
Original file line number Diff line number Diff line change
Expand Up @@ -533,9 +533,6 @@ export NPZ=64
export NPZP=65
export NSTF_NAME=2,1,1,0,5
export OUTPUT_FH="12 -1"
export NFHOUT=12
export NFHMAX_HF=12
export NFHOUT_HF=6
export FHZERO=6
export FNALBC="'global_snowfree_albedo.bosu.t126.384.190.rg.grb'"
export FNVETC="'global_vegtype.igbp.t126.384.190.rg.grb'"
Expand Down Expand Up @@ -841,7 +838,6 @@ export FIELD_TABLE=field_table_thompson_noaero_tke_GOCART
export DIAG_TABLE=diag_table_p8_template
export DIAG_TABLE_ADDITIONAL=''

export NSOUT=-1
export FHZERO=6
export DT_INNER=${DT_ATMOS}

Expand Down Expand Up @@ -992,7 +988,6 @@ export SHOUR=00
export FHMAX=24
export DT_ATMOS=900
export FHROT=0
export NFHOUT=6

# required but unused
export WARM_START=.false.
Expand Down Expand Up @@ -1117,9 +1112,6 @@ export OUTPUT_GRID="'regional_latlon'"
export OUTPUT_FILE="'netcdf'"
export IDEFLATE=0
export QUANTIZE_NSD=0
export NFHOUT=3
export NFHMAX_HF=-1
export NFHOUT_HF=3
export CEN_LON=-62.0
export CEN_LAT=25.0
export LON1=-114.5
Expand Down Expand Up @@ -1310,11 +1302,7 @@ export NY_6=''
export DX_6=''
export DY_6=''

export NFHOUT=3
export NFHMAX_HF=-1
export NFHOUT_HF=3
export NSOUT=-1
export OUTPUT_FH=-1
export OUTPUT_FH='3 -1'
}
export_hrrr() {
export_fv3
Expand Down
48 changes: 24 additions & 24 deletions tests/logs/OpnReqTests_control_p8_hera.log
View file
Original file line number Diff line number Diff line change
@@ -1,9 +1,9 @@
Mon Dec 4 14:30:51 UTC 2023
Tue Dec 19 00:58:42 UTC 2023
Start Operation Requirement Test


baseline dir = /scratch1/NCEPDEV/stmp4/Zachary.Shrader/FV3_OPNREQ_TEST/OPNREQ_TEST/control_p8_bit_base_gnu
working dir = /scratch1/NCEPDEV/stmp2/Zachary.Shrader/FV3_OPNREQ_TEST/opnReqTest_45674/control_p8_gnu_bit_base
working dir = /scratch1/NCEPDEV/stmp2/Zachary.Shrader/FV3_OPNREQ_TEST/opnReqTest_181599/control_p8_gnu_bit_base
Checking test bit_base control_p8_gnu results ....
Moving baseline bit_base control_p8_gnu files ....
Moving sfcf000.nc .........OK
Expand Down Expand Up @@ -51,14 +51,14 @@ Moving baseline bit_base control_p8_gnu files ....
Moving RESTART/20210323.060000.sfc_data.tile5.nc .........OK
Moving RESTART/20210323.060000.sfc_data.tile6.nc .........OK

0: The total amount of wall time = 281.662628
0: The maximum resident set size (KB) = 1302480
0: The total amount of wall time = 288.506269
0: The maximum resident set size (KB) = 1301052

Test bit_base control_p8_gnu PASS


baseline dir = /scratch1/NCEPDEV/stmp4/Zachary.Shrader/FV3_OPNREQ_TEST/OPNREQ_TEST/control_p8_dbg_base_gnu
working dir = /scratch1/NCEPDEV/stmp2/Zachary.Shrader/FV3_OPNREQ_TEST/opnReqTest_45674/control_p8_gnu_dbg_base
working dir = /scratch1/NCEPDEV/stmp2/Zachary.Shrader/FV3_OPNREQ_TEST/opnReqTest_181599/control_p8_gnu_dbg_base
Checking test dbg_base control_p8_gnu results ....
Moving baseline dbg_base control_p8_gnu files ....
Moving sfcf000.nc .........OK
Expand Down Expand Up @@ -106,14 +106,14 @@ Moving baseline dbg_base control_p8_gnu files ....
Moving RESTART/20210323.060000.sfc_data.tile5.nc .........OK
Moving RESTART/20210323.060000.sfc_data.tile6.nc .........OK

0: The total amount of wall time = 909.259942
0: The maximum resident set size (KB) = 1289656
0: The total amount of wall time = 915.151021
0: The maximum resident set size (KB) = 1292240

Test dbg_base control_p8_gnu PASS


baseline dir = /scratch1/NCEPDEV/stmp4/Zachary.Shrader/FV3_OPNREQ_TEST/OPNREQ_TEST/control_p8_std_base_gnu
working dir = /scratch1/NCEPDEV/stmp2/Zachary.Shrader/FV3_OPNREQ_TEST/opnReqTest_45674/control_p8_gnu_dcp
working dir = /scratch1/NCEPDEV/stmp2/Zachary.Shrader/FV3_OPNREQ_TEST/opnReqTest_181599/control_p8_gnu_dcp
Checking test dcp control_p8_gnu results ....
Comparing sfcf000.nc .........OK
Comparing sfcf021.nc .........OK
Expand Down Expand Up @@ -160,14 +160,14 @@ Checking test dcp control_p8_gnu results ....
Comparing RESTART/20210323.060000.sfc_data.tile5.nc .........OK
Comparing RESTART/20210323.060000.sfc_data.tile6.nc .........OK

0: The total amount of wall time = 249.181311
0: The maximum resident set size (KB) = 1281316
0: The total amount of wall time = 263.405073
0: The maximum resident set size (KB) = 1281032

Test dcp control_p8_gnu PASS


baseline dir = /scratch1/NCEPDEV/stmp4/Zachary.Shrader/FV3_OPNREQ_TEST/OPNREQ_TEST/control_p8_std_base_gnu
working dir = /scratch1/NCEPDEV/stmp2/Zachary.Shrader/FV3_OPNREQ_TEST/opnReqTest_45674/control_p8_gnu_mpi
working dir = /scratch1/NCEPDEV/stmp2/Zachary.Shrader/FV3_OPNREQ_TEST/opnReqTest_181599/control_p8_gnu_mpi
Checking test mpi control_p8_gnu results ....
Comparing sfcf000.nc .........OK
Comparing sfcf021.nc .........OK
Expand Down Expand Up @@ -214,14 +214,14 @@ Checking test mpi control_p8_gnu results ....
Comparing RESTART/20210323.060000.sfc_data.tile5.nc .........OK
Comparing RESTART/20210323.060000.sfc_data.tile6.nc .........OK

0: The total amount of wall time = 250.654118
0: The maximum resident set size (KB) = 1279188
0: The total amount of wall time = 255.080358
0: The maximum resident set size (KB) = 1282552

Test mpi control_p8_gnu PASS


baseline dir = /scratch1/NCEPDEV/stmp4/Zachary.Shrader/FV3_OPNREQ_TEST/OPNREQ_TEST/control_p8_std_base_gnu
working dir = /scratch1/NCEPDEV/stmp2/Zachary.Shrader/FV3_OPNREQ_TEST/opnReqTest_45674/control_p8_gnu_rst
working dir = /scratch1/NCEPDEV/stmp2/Zachary.Shrader/FV3_OPNREQ_TEST/opnReqTest_181599/control_p8_gnu_rst
Checking test rst control_p8_gnu results ....
Comparing sfcf000.nc .........OK
Comparing sfcf021.nc .........OK
Expand Down Expand Up @@ -268,14 +268,14 @@ Checking test rst control_p8_gnu results ....
Comparing RESTART/20210323.060000.sfc_data.tile5.nc .........OK
Comparing RESTART/20210323.060000.sfc_data.tile6.nc .........OK

0: The total amount of wall time = 252.596116
0: The maximum resident set size (KB) = 1279660
0: The total amount of wall time = 249.762324
0: The maximum resident set size (KB) = 1280660

Test rst control_p8_gnu PASS


baseline dir = /scratch1/NCEPDEV/stmp4/Zachary.Shrader/FV3_OPNREQ_TEST/OPNREQ_TEST/control_p8_std_base_gnu
working dir = /scratch1/NCEPDEV/stmp2/Zachary.Shrader/FV3_OPNREQ_TEST/opnReqTest_45674/control_p8_gnu_std_base
working dir = /scratch1/NCEPDEV/stmp2/Zachary.Shrader/FV3_OPNREQ_TEST/opnReqTest_181599/control_p8_gnu_std_base
Checking test std_base control_p8_gnu results ....
Moving baseline std_base control_p8_gnu files ....
Moving sfcf000.nc .........OK
Expand Down Expand Up @@ -323,14 +323,14 @@ Moving baseline std_base control_p8_gnu files ....
Moving RESTART/20210323.060000.sfc_data.tile5.nc .........OK
Moving RESTART/20210323.060000.sfc_data.tile6.nc .........OK

0: The total amount of wall time = 250.001003
0: The maximum resident set size (KB) = 1280812
0: The total amount of wall time = 251.940839
0: The maximum resident set size (KB) = 1279716

Test std_base control_p8_gnu PASS


baseline dir = /scratch1/NCEPDEV/stmp4/Zachary.Shrader/FV3_OPNREQ_TEST/OPNREQ_TEST/control_p8_std_base_gnu
working dir = /scratch1/NCEPDEV/stmp2/Zachary.Shrader/FV3_OPNREQ_TEST/opnReqTest_45674/control_p8_gnu_thr
working dir = /scratch1/NCEPDEV/stmp2/Zachary.Shrader/FV3_OPNREQ_TEST/opnReqTest_181599/control_p8_gnu_thr
Checking test thr control_p8_gnu results ....
Comparing sfcf000.nc .........OK
Comparing sfcf021.nc .........OK
Expand Down Expand Up @@ -377,11 +377,11 @@ Checking test thr control_p8_gnu results ....
Comparing RESTART/20210323.060000.sfc_data.tile5.nc .........OK
Comparing RESTART/20210323.060000.sfc_data.tile6.nc .........OK

0: The total amount of wall time = 250.574652
0: The maximum resident set size (KB) = 1283668
0: The total amount of wall time = 256.498818
0: The maximum resident set size (KB) = 1280192

Test thr control_p8_gnu PASS

OPERATION REQUIREMENT TEST WAS SUCCESSFUL
Mon Dec 4 15:39:13 UTC 2023
Elapsed time: 01h:08m:22s. Have a nice day!
Tue Dec 19 02:24:57 UTC 2023
Elapsed time: 01h:26m:16s. Have a nice day!
24 changes: 12 additions & 12 deletions tests/logs/OpnReqTests_cpld_control_nowave_noaero_p8_hera.log
View file
Original file line number Diff line number Diff line change
@@ -1,9 +1,9 @@
Mon Dec 4 13:38:49 UTC 2023
Tue Dec 19 03:23:26 UTC 2023
Start Operation Requirement Test


baseline dir = /scratch1/NCEPDEV/stmp4/Zachary.Shrader/FV3_OPNREQ_TEST/OPNREQ_TEST/cpld_control_c96_noaero_p8_dbg_base_gnu
working dir = /scratch1/NCEPDEV/stmp2/Zachary.Shrader/FV3_OPNREQ_TEST/opnReqTest_221447/cpld_control_nowave_noaero_p8_gnu_dbg_base
working dir = /scratch1/NCEPDEV/stmp2/Zachary.Shrader/FV3_OPNREQ_TEST/opnReqTest_129523/cpld_control_nowave_noaero_p8_gnu_dbg_base
Checking test dbg_base cpld_control_nowave_noaero_p8_gnu results ....
Moving baseline dbg_base cpld_control_nowave_noaero_p8_gnu files ....
Moving sfcf021.tile1.nc .........OK
Expand Down Expand Up @@ -66,14 +66,14 @@ Moving baseline dbg_base cpld_control_nowave_noaero_p8_gnu files ....
Moving RESTART/iced.2021-03-23-21600.nc .........OK
Moving RESTART/ufs.cpld.cpl.r.2021-03-23-21600.nc .........OK

0: The total amount of wall time = 1255.630972
0: The maximum resident set size (KB) = 1411232
0: The total amount of wall time = 1244.541002
0: The maximum resident set size (KB) = 1405012

Test dbg_base cpld_control_nowave_noaero_p8_gnu PASS


baseline dir = /scratch1/NCEPDEV/stmp4/Zachary.Shrader/FV3_OPNREQ_TEST/OPNREQ_TEST/cpld_control_c96_noaero_p8_std_base_gnu
working dir = /scratch1/NCEPDEV/stmp2/Zachary.Shrader/FV3_OPNREQ_TEST/opnReqTest_221447/cpld_control_nowave_noaero_p8_gnu_rst
working dir = /scratch1/NCEPDEV/stmp2/Zachary.Shrader/FV3_OPNREQ_TEST/opnReqTest_129523/cpld_control_nowave_noaero_p8_gnu_rst
Checking test rst cpld_control_nowave_noaero_p8_gnu results ....
Comparing sfcf021.tile1.nc .........OK
Comparing sfcf021.tile2.nc .........OK
Expand Down Expand Up @@ -135,14 +135,14 @@ Checking test rst cpld_control_nowave_noaero_p8_gnu results ....
Comparing RESTART/iced.2021-03-23-21600.nc .........OK
Comparing RESTART/ufs.cpld.cpl.r.2021-03-23-21600.nc .........OK

0: The total amount of wall time = 388.631231
0: The maximum resident set size (KB) = 1406476
0: The total amount of wall time = 393.044208
0: The maximum resident set size (KB) = 1403064

Test rst cpld_control_nowave_noaero_p8_gnu PASS


baseline dir = /scratch1/NCEPDEV/stmp4/Zachary.Shrader/FV3_OPNREQ_TEST/OPNREQ_TEST/cpld_control_c96_noaero_p8_std_base_gnu
working dir = /scratch1/NCEPDEV/stmp2/Zachary.Shrader/FV3_OPNREQ_TEST/opnReqTest_221447/cpld_control_nowave_noaero_p8_gnu_std_base
working dir = /scratch1/NCEPDEV/stmp2/Zachary.Shrader/FV3_OPNREQ_TEST/opnReqTest_129523/cpld_control_nowave_noaero_p8_gnu_std_base
Checking test std_base cpld_control_nowave_noaero_p8_gnu results ....
Moving baseline std_base cpld_control_nowave_noaero_p8_gnu files ....
Moving sfcf021.tile1.nc .........OK
Expand Down Expand Up @@ -205,11 +205,11 @@ Moving baseline std_base cpld_control_nowave_noaero_p8_gnu files ....
Moving RESTART/iced.2021-03-23-21600.nc .........OK
Moving RESTART/ufs.cpld.cpl.r.2021-03-23-21600.nc .........OK

0: The total amount of wall time = 387.397993
0: The maximum resident set size (KB) = 1400340
0: The total amount of wall time = 386.989243
0: The maximum resident set size (KB) = 1402912

Test std_base cpld_control_nowave_noaero_p8_gnu PASS

OPERATION REQUIREMENT TEST WAS SUCCESSFUL
Mon Dec 4 14:28:11 UTC 2023
Elapsed time: 00h:49m:22s. Have a nice day!
Tue Dec 19 04:14:00 UTC 2023
Elapsed time: 00h:50m:34s. Have a nice day!
24 changes: 12 additions & 12 deletions tests/logs/OpnReqTests_regional_control_hera.log
View file
Original file line number Diff line number Diff line change
@@ -1,9 +1,9 @@
Mon Dec 4 13:01:50 UTC 2023
Tue Dec 19 02:30:18 UTC 2023
Start Operation Requirement Test


baseline dir = /scratch1/NCEPDEV/stmp4/Zachary.Shrader/FV3_OPNREQ_TEST/OPNREQ_TEST/regional_control_std_base_gnu
working dir = /scratch1/NCEPDEV/stmp2/Zachary.Shrader/FV3_OPNREQ_TEST/opnReqTest_20393/regional_control_gnu_dcp
working dir = /scratch1/NCEPDEV/stmp2/Zachary.Shrader/FV3_OPNREQ_TEST/opnReqTest_21952/regional_control_gnu_dcp
Checking test dcp regional_control_gnu results ....
Comparing dynf000.nc .........OK
Comparing dynf006.nc .........OK
Expand All @@ -14,14 +14,14 @@ Checking test dcp regional_control_gnu results ....
Comparing NATLEV.GrbF00 .........OK
Comparing NATLEV.GrbF06 .........OK

0: The total amount of wall time = 527.808815
0: The maximum resident set size (KB) = 587160
0: The total amount of wall time = 572.207367
0: The maximum resident set size (KB) = 589584

Test dcp regional_control_gnu PASS


baseline dir = /scratch1/NCEPDEV/stmp4/Zachary.Shrader/FV3_OPNREQ_TEST/OPNREQ_TEST/regional_control_std_base_gnu
working dir = /scratch1/NCEPDEV/stmp2/Zachary.Shrader/FV3_OPNREQ_TEST/opnReqTest_20393/regional_control_gnu_std_base
working dir = /scratch1/NCEPDEV/stmp2/Zachary.Shrader/FV3_OPNREQ_TEST/opnReqTest_21952/regional_control_gnu_std_base
Checking test std_base regional_control_gnu results ....
Moving baseline std_base regional_control_gnu files ....
Moving dynf000.nc .........OK
Expand All @@ -33,14 +33,14 @@ Moving baseline std_base regional_control_gnu files ....
Moving NATLEV.GrbF00 .........OK
Moving NATLEV.GrbF06 .........OK

0: The total amount of wall time = 519.287964
0: The maximum resident set size (KB) = 588100
0: The total amount of wall time = 515.401441
0: The maximum resident set size (KB) = 587380

Test std_base regional_control_gnu PASS


baseline dir = /scratch1/NCEPDEV/stmp4/Zachary.Shrader/FV3_OPNREQ_TEST/OPNREQ_TEST/regional_control_std_base_gnu
working dir = /scratch1/NCEPDEV/stmp2/Zachary.Shrader/FV3_OPNREQ_TEST/opnReqTest_20393/regional_control_gnu_thr
working dir = /scratch1/NCEPDEV/stmp2/Zachary.Shrader/FV3_OPNREQ_TEST/opnReqTest_21952/regional_control_gnu_thr
Checking test thr regional_control_gnu results ....
Comparing dynf000.nc .........OK
Comparing dynf006.nc .........OK
Expand All @@ -51,11 +51,11 @@ Checking test thr regional_control_gnu results ....
Comparing NATLEV.GrbF00 .........OK
Comparing NATLEV.GrbF06 .........OK

0: The total amount of wall time = 508.172814
0: The maximum resident set size (KB) = 585988
0: The total amount of wall time = 522.360834
0: The maximum resident set size (KB) = 585460

Test thr regional_control_gnu PASS

OPERATION REQUIREMENT TEST WAS SUCCESSFUL
Mon Dec 4 13:37:44 UTC 2023
Elapsed time: 00h:35m:55s. Have a nice day!
Tue Dec 19 03:21:43 UTC 2023
Elapsed time: 00h:51m:26s. Have a nice day!
Loading

0 comments on commit 1d90c04

Please sign in to comment.