Issue with datm_cdeps_mx025_cfsr

[mjhossen]

I am using a ufs weather model (datm-mom6-cice6) in my DA system built with GDASApp and soca-science workflow. I plot MOM.res.nc produced by regression test - datm_cdeps_mx025_cfsr that uses data in 20111001. The plot (attached) looks suspicious to me. I am trying to figure out where the problem is.
Can you please make your comments. Do you think the model output is right?

[jiandewang]

@mjhossen how long did you run the model ? You may also plot the cross-section at equator (one from your starting IC, one from your results). Also only plot from surface to 300m will make it easy for people to look at them.

[DeniseWorthen]

@mjhossen Are you wondering about, for example, the sharp black "spike" which show up at around 40N? Are you sure that isn't just reflecting the bottom bathy? MOM restarts fill all the values deeper than the last valid depth with the value of the last valid depth.

[jiandewang]

"MOM restarts fill all the values deeper than the last valid depth with the value of the last valid depth."
I think they are filled with zero

[DeniseWorthen]

See the following from the RT MOM restart at 0E.

[mjhossen]

I have started my experiment with initial restart file at 2019120100 (plotted).
After first cycle the vertical temp profile looks very different:

@DeniseWorthen I am worried about mixed depth (about 200m).
I also compare the model output with argo data. They look very different.

[jiandewang]

by comparing the two plots at 30S, apparently something is wrong (most likely in DA) after 6hr run. Will you be able to turn off DA in your run ?

[mjhossen]

Yes, I can. Do you mean to run only forcast for another 6 hrs?

[mjhossen]

Oops! I should mention that I do not do any DA yet. This is Free run without assimilation.

[jiandewang]

yes, in this way we want to make sure there is no big difference between 6hr run and its IC

[jiandewang]

let me repeat this run using UWM rt.sh, will get back to you later

[mjhossen]

Thank you!

[jiandewang]

@mjhossen I ran this job using latest UWM, but don't see big difference between IC and 12hr restart. You may check them on HERA
/scratch1/NCEPDEV/stmp2/Jiande.Wang/FV3_RT/rt_3432713/datm_cdeps_mx025_cfsr_intel

[mjhossen]

Thanks. I will have a look.

[mjhossen]

@jiandewang I plot I.C and 12hr restart from /scratch1/NCEPDEV/stmp2/Jiande.Wang/FV3_RT/rt_3432713/datm_cdeps_mx025_cfsr_intel. Looks different! It could be a bug in my plotting script. What do you think? Did you plot them? Can you please share your plotting script?

[jiandewang]

So the first plot is from IC while the 2nd one is from 12hr restart file, both of them are from my run, right, let me use my own way to do the plotting and get back to you.

[mjhossen]

Yes.

[jiandewang]

this is from 6-12hr averaged ocean output

it is close to your IC plot. I don't have a reliable way to plot 12hr restart file at this moment.

[mjhossen]

Looks great! So, just to make sure the plot is RESTART/20111001.120000.MOM.res.nc. Am I right?

[jiandewang]

no, it is /scratch1/NCEPDEV/stmp2/Jiande.Wang/FV3_RT/rt_3432713/datm_cdeps_mx025_cfsr_intel/ocn_2011_10_01_09.nc
this is 6-12hr averaged results, 20111001.120000.MOM.res.nc is the final snapshot at 12hr but I don't have a reliable way to plot that.

[mjhossen]

@jiandewang I also got the similar plot for ocn_2011_10_01_09.nc. It confirms that my plotting script is working correctly. So the problem could be MOM.res.nc. Is there any way to fix this issue-MOM6 restart file?

[jiandewang]

we need to confirm whether thers is problem in restart file or not, this may take some time as the file fomat/metadata make it hard to do a clean plot (unlike the ocn*nc file that is straightforward for us to plot).

[mjhossen]

Sounds good! Thank you.

[jiandewang]

@mjhossen sorry I forgot to mention that this is density layer model so in the restart file
Temp(Time, Layer, lath, lonh)
Layer = 1, 3, 5, 7, 9, 11, 13, 15.005, 17.015, 19.03, 21.055, 23.095, 25.16,
27.255, 29.385, 31.565, 33.81, 36.135, 38.56, 41.105, 43.795, 46.655,
49.715, 53.015, 56.6, 60.515, 64.805, 69.525, 74.74, 80.515, 86.92,
94.04, 101.96, 110.77, 120.575, 131.485, 143.615, 157.095, 172.06,
188.655, 207.035, 227.365, 249.82, 274.585, 301.86, 331.855, 364.795,
400.915, 440.46, 483.69, 530.885, 582.335, 638.345, 699.24, 765.355,
837.04, 914.66, 998.605, 1089.28, 1187.105, 1292.525, 1406, 1528.01,
1659.05, 1799.64, 1950.325, 2111.665, 2284.245, 2468.675, 2665.585,
2875.63, 3099.485, 3337.85, 3591.46, 5111.085
here the value of Layer doesn't really mean when you set z=1 then all grid points in the 1st layer is at 1m.
the z dimension in variable is misleading (as this is not a z level model). The reason your plot looks reasonable in IC restart is because this file is from ocean DA (years ago's stuff) that used z level in its vertical configuration.
when you use ocn*nc file the plot looks good this is because we interpolate the 75 density layer value to 40 z level.

[mjhossen]

FYI, I have some results from old ufs model (MOM.res.nc). The MOM6 restart file looks good

However, MOM6 restart file with new ufs model is the problem. I believe there is something wrong with MOM6 restart file.

[mjhossen]

Also, please see the layout from old and new ufs restart netcdf file.

[jiandewang]

FYI, I have some results from old ufs model (MOM.res.nc). The MOM6 restart file looks good However, MOM6 restart file with new ufs model is the problem. I believe there is something wrong with MOM6 restart file.

this one must come from z level configured run

[jiandewang]

so in short, there is no reliable way to plot vertical cross section temperature in the 12hr restart file as each grid is at different depth

[DeniseWorthen]

@mjhossen Where is your run directory?

[mjhossen]

Here it is: /work/noaa/aoml-osse/jhossen/OSEwGdasapp/FR_hofx

[DeniseWorthen]

I can't see into your aoml-osse directory. Do you have any publicly accessible stmp space?

[jiandewang]

@mjhossen
I have two sample IC restart files here

(2) /scratch1/NCEPDEV/climate/role.ufscpara/IC/HR3marine/2020073100/ocean/MOM.res.nc
(1) /scratch1/NCEPDEV/climate/role.ufscpara/IC//CPC3Dvar/2016070100/ocn/025/MOM.res.nc

here left is from (1), right is from (2)

you can see left panel looks reasonable but right panle looks bad
but (1) is from CPC ocean DA which used z-level in model configuration while (2) is from replay IC which used density level, although both of them z=75.

for your run and my run, IC is from (1). Once model started to run, the saved restart files will be in the format of (2) as we use density level in UFS. There is no simple way to plot cross section in z for restart files in the style of (2).

[mjhossen]

So, you are saying that MOM restart output is good; no issue with them at all.
If this is true, I agree with that as my obs_out plot provides very good matching with the observations.

[jiandewang]

Yes. In the future when you compare model results with OBS always use ocn*nc file.

[mjhossen]

Thank you so much for your time and explanation.

[gspetro-NOAA]

Closing this issue since it seems to be resolved. Please feel free to post a new issue if follow-on questions come up, or request to reopen this issue.