Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merge CICE-Consortium/Icepack main to E3SM-Project/Icepack main #35

Open
wants to merge 25 commits into
base: main
Choose a base branch
from

Conversation

apcraig
Copy link
Collaborator

@apcraig apcraig commented Apr 12, 2024

This is a simple PR created directly from the repositories.

apcraig and others added 16 commits October 5, 2023 17:42
Co-authored-by: David Bailey <[email protected]>
Co-authored-by: Elizabeth Hunke <[email protected]>
* Add .readthedocs.yaml

Copy CICE's readthedocs configuration as of
CICE-Consortium/CICE@06282a53 (Update version to 6.4.2 (#864),
2023-09-08).

* doc: mention code variables relating to 'calc_dragio'

At the end of the "Ocean" section, we mention that 'dragio' can be
computed from 'iceruf_ocn' and 'thickness_ocn_layer1', but do not
mention these code variables, nor the 'calc_dragio' setting used to
activate that computation. Mention the code variables next to their
mathematical symbol, for more clarity.
…ters (#465)

All parameters of icepack_parameters::icepack_query_parameters are
intent(out), so it does not make sense to recompute constants at the end
of that subroutine since no constants are changed. This superfluous call
dates back to the introduction of icepack_query_parameters (then called
icepack_query_constants) in 7646f2a (Migrate icepack_constants out of
the columnphysics/constants directory..., 2017-10-04).

Remove that call.
* Remove older "_old" tfrz_options, added for backwards compatibility

Modify tfrz_option logic in icepack_sea_freezing_temperature to trap unsupported values

* Update Icepack set_nml settings, remove _old from tfrz_option
* Clarify documentation associated with namelist inputs

Alphabetize cpl_frazil namelist documentation

* Update documentation
* Update 5-band dEdd to support test data

Add an interpolation method in icepack_shortwave.F90 for rsnw

Refactor portions of icepack_shortwave to improve robustness

Add snicar and snicartest test cases

* Update the 5-band snicar shortwave rsnw_snicar_tab interpolation

This changes answers but QC tests pass.  The new implementation checks
whether the rsnw_snicar_tab is an array of integer values with deltas
of value 1 (and sets the rsnw_snicar_chk variable).  If so, it uses a
shortcut to find the rsnw_snicar_tab bounds for interpolation, otherwise
it calls the shortwave_search routine.  In testing, the shortcut did not
change performance much, but that could change in the future.

* Clean up debug output

* Refactor the logic to control rsnw table interpolation in the shortwave.
Set a character string, rsnw_datatype, at initialization.
…ation (#476)

* correct ks units, clean up comments, big fix for BL99 enthalpy calculation

* make it pretty
Update pull request template to ask for additional information about the PR. This information can and will be used during the squash merge process to produce more useful commit logs.

Update Macros.conda_macos to remove ffpe-trap=invalid from debug flags.
This flag causes nf90_create to fail with the latest version of
the conda environment. This happens when debug flags are on
and a test with netcdf is turned on. I believe this is a compiler error.
* Add gnuplot to the conda environment

This fix allows the timeseries.csh script to be used for Icepack standalone runs.
@eclare108213
Copy link
Collaborator

eclare108213 commented Apr 12, 2024

I am testing this using the E3SM-Polar-Developer script.

@eclare108213
Copy link
Collaborator

Running QC test on the following directories:
/lcrc/group/e3sm/ac.eclare/E3SM-Polar/D12.qcbaseline.emc.qcbaseline20240413.master.E3SM-Project.anvil/run.k000/
/lcrc/group/e3sm/ac.eclare/E3SM-Polar/D12.qcbaseline.emc.update_icepack_20240412.master.E3SM-Project.anvil/run.k000
Number of files: 61
2 Stage Test Passed
Quadratic Skill Test Passed for Northern Hemisphere
Quadratic Skill Test Passed for Southern Hemisphere

@eclare108213
Copy link
Collaborator

Baseline E3SM:

--- Global Energy and Mass Conservation for Month 03, Year 0001:

    relativeEnergyError:  	+1.78312e-03
    relativeMassError:  	+1.18320e-06
    relativeSaltError:  	-9.78900e-17
    relativeCarbonError:  	+0.00000e+00

--- Average Global Values for Month 03, Year 0001:

			North 			South
    1980-2022 Extent:	+1.530286800000000e+07 	+4.128757000000000e+06
    totalIceExtent	+1.629369209297351e+07 	+1.705213445957203e+07
    totalIceVolume	+3.636707252614450e+04 	+2.499332039374441e+04
    totalSnowVolume	+3.268645596762610e+03 	+7.460772897206572e+02
    totalKineticEnergy	+2.233036288084481e+14 	+3.397156886195364e+14
    averageAlbedo	+3.595186295033303e-01 	+1.763111566892719e-01

			Minimum 		Maximum
    iceAreaCell 	+0.000000000000000e+00 	+9.999938011169434e-01
    iceVolumeCell 	+0.000000000000000e+00 	+4.600533485412598e+00
    icePressure 	+0.000000000000000e+00 	+1.692133593750000e+05
    uVelocityGeo 	-4.285480380058289e-01 	+5.492014884948730e-01
    vVelocityGeo 	-4.407684504985809e-01 	+4.059779047966003e-01

E3SM with updated Icepack

--- Global Energy and Mass Conservation for Month 03, Year 0001:

    relativeEnergyError:  	+1.78396e-03
    relativeMassError:  	+1.17827e-06
    relativeSaltError:  	-2.80610e-16
    relativeCarbonError:  	+0.00000e+00

--- Average Global Values for Month 03, Year 0001:

			North 			South
    1980-2022 Extent:	+1.530286800000000e+07 	+4.128757000000000e+06
    totalIceExtent	+1.629368454672141e+07 	+1.705176894139234e+07
    totalIceVolume	+3.636728686167497e+04 	+2.499222023248838e+04
    totalSnowVolume	+3.268699112949644e+03 	+7.451200732825436e+02
    totalKineticEnergy	+2.233177964238433e+14 	+3.397029549194099e+14
    averageAlbedo	+3.595267401230673e-01 	+1.762149195049135e-01

			Minimum 		Maximum
    iceAreaCell 	+0.000000000000000e+00 	+9.999940395355225e-01
    iceVolumeCell 	+0.000000000000000e+00 	+4.600161075592041e+00
    icePressure 	+0.000000000000000e+00 	+1.691377031250000e+05
    uVelocityGeo 	-4.285481274127960e-01 	+5.492023229598999e-01
    vVelocityGeo 	-4.407695531845093e-01 	+4.059588909149170e-01

@eclare108213
Copy link
Collaborator

Here are the stats comparing the 5-year QC runs.

$ ./E3SM-Polar-Developer.sh -s qcbaseline20240413 -k qcbaseline.nlk -e -d60 -a D12.qcbaseline.emc.update_icepack_20240412.master.E3SM-Project.anvil -v

--- D-CASE AT STANDARD RESOLUTION WITH JRA 1.5 FORCING
    2000_DATM%JRA-1p5_SLND_MPASSI_DOCN%SOM_DROF%JRA-1p5_SGLC_SWAV_TEST, 
    TL319_EC30to60E2r2, S layout, 32 combos allowed

--- Fork origin is different from the default: [email protected]:E3SM-Project/E3SM.git
    Sandbox origin is: [email protected]:E3SM-Project/E3SM

--- Skipping clone of code ---

--- Reading combinations from D12.qcbaseline.emc.qcbaseline20240413.master.E3SM-Project.anvil/qcbaseline.nlk

    k000 ->
    mpassi: 	config_AM_thicknesses_write_on_startup = .true.
    mpassi: 	config_AM_thicknesses_compute_on_startup = .true.
    mpassi: 	config_AM_thicknesses_enable = .true.

--- Skipping create newcase ---

--- Skipping build for k000

--- Skipping queue for k000

--- Starting analysis ---

    All submissions out to 0006-01-01-00000 -
    run.D12.qcbaseline.emc.qcbaseline20240413.master.E3SM-Project.anvil.666565 k000: Complete
    run.D12.qcbaseline.emc.qcbaseline20240413.master.E3SM-Project.anvil.666571 k000: Complete
    run.D12.qcbaseline.emc.qcbaseline20240413.master.E3SM-Project.anvil.666573 k000: Complete
    run.D12.qcbaseline.emc.qcbaseline20240413.master.E3SM-Project.anvil.666578 k000: Complete
    run.D12.qcbaseline.emc.qcbaseline20240413.master.E3SM-Project.anvil.666579 k000: Complete

    All submissions out to 0006-01-01-00000 -
    run.D12.qcbaseline.emc.update_icepack_20240412.master.E3SM-Project.anvil.666560 k000: Complete
    run.D12.qcbaseline.emc.update_icepack_20240412.master.E3SM-Project.anvil.666564 k000: Complete
    run.D12.qcbaseline.emc.update_icepack_20240412.master.E3SM-Project.anvil.666570 k000: Complete
    run.D12.qcbaseline.emc.update_icepack_20240412.master.E3SM-Project.anvil.666572 k000: Complete
    run.D12.qcbaseline.emc.update_icepack_20240412.master.E3SM-Project.anvil.666577 k000: Complete

--- Global Energy and Mass Conservation for Month 12, Year 0005:

    D12.qcbaseline.emc.qcbaseline20240413.master.E3SM-Project.anvil k000:

    relativeEnergyError:  	+4.59932e-03
    relativeMassError:  	+1.67645e-07
    relativeSaltError:  	-1.13700e-16
    relativeCarbonError:  	+0.00000e+00

    D12.qcbaseline.emc.update_icepack_20240412.master.E3SM-Project.anvil k000:

    relativeEnergyError:  	+4.60260e-03
    relativeMassError:  	+1.66784e-07
    relativeSaltError:  	-1.39900e-17
    relativeCarbonError:  	+0.00000e+00

--- Average Differences for Month 12, Year 0005:

    D12.qcbaseline.emc.qcbaseline20240413.master.E3SM-Project.anvil k000 - 
    D12.qcbaseline.emc.update_icepack_20240412.master.E3SM-Project.anvil k000:

			Minimum 	Maximum
    iceAreaCell 	-1.20475e-02 	+2.77486e-02
    iceVolumeCell 	-1.50131e-01 	+1.31364e-01
    icePressure 	-7.12052e+03 	+5.06067e+03
    uVelocityGeo 	-1.68541e-02 	+4.32362e-02
    vVelocityGeo 	-8.97751e-03 	+1.82618e-02

			North 		South
    totalIceExtent 	+3.39529e+02 	+6.57685e+03
    totalIceVolume 	+1.20362e+01 	+2.85916e+00
    totalSnowVolume 	+3.82790e+00 	-1.41116e+00
    totalKineticEnergy 	+5.55519e+11 	+1.73564e+10
    averageAlbedo 	+6.49974e-07 	+8.84199e-05

--- Case root: /lcrc/group/e3sm/ac.eclare/E3SM-Polar/
    D12.qcbaseline.emc.qcbaseline20240413.master.E3SM-Project.anvil

--- Code root: ~/E3SM-Polar/code/qcbaseline20240413

--- Provenance root: ~/E3SM-Polar/provenance/qcbaseline20240413/
    D12.qcbaseline.emc.qcbaseline20240413.master.E3SM-Project.anvil

--- Script saved: E3SM-Polar-Developer.sh.20240415-113612

Copy link

@proteanplanet proteanplanet left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We have worked further on the high resolution version of E3SM and determined that the model is also crashing with a data sea ice model, thereby eliminating anything in this PR as a potential cause of model instability. In consultation with the HR team and with @eclare108213, I would like to suggest we close this PR so as not to complicate @njeffery 's work that would automatically see this change brought into E3SM as part of the BGC tag.

@eclare108213
Copy link
Collaborator

The contents of this PR will likely NOT be brought in with @njeffery's BGC mods. I think it's best to keep them separate, since they are both somewhat large collections of changes and their intent is different. The argument for not merging this PR yet is so that it's easier to compare Nicole's code / simulations to existing fully coupled (water cycle) runs. If the control runs will not be BFB due to other changes/bug fixes, then I we could go ahead with this PR. Then @njeffery would need to rebase and retest -- that's extra work that we'd prefer to avoid. Are non-BFB changes going into master now?

@apcraig
Copy link
Collaborator Author

apcraig commented Apr 16, 2024

Also, there really isn't any risk/cost to merge this. If the plan is to keep the E3SM-Project/Icepack main synced up with the Consortium/Icepack main, this PR is just doing that. We could probably create a github action that would do it automatically once a week and it would have no impact on any other development or plans. Merging this PR has no impact on the bgc branch or whether bgc is updated to the current main. That is a totally independent step. It also has no impact on the Icepack hash that E3SM points to. That hash can be on any branch. At some points, it could be a hash from main, but never has to be. All that assumes the plan is for the E3SM-Project/Icepack main to match the Consortium/Icepack main exactly always and for that update to happen periodically.

@eclare108213
Copy link
Collaborator

eclare108213 commented Apr 16, 2024

That's true, E3SM will not point to this new hash. But I think this branch (main) is where we had planned to force-push changes from the Consortium in order to keep the history clean, so I don't think we would actually merge this PR anyhow. Mostly I don't want to confuse the order in which things happen, relative to the BGC merge. If we're going to make these changes in E3SM first, then let's do it following the established process, which I think means force-pushing to E3SM Icepack main and then opening another PR into E3SM itself, linking the new Icepack hash, and moving the documentation above into that PR. But that's harder for Nicole to deal with, so I'm waiting. It's good to know that my tests above succeeded cleanly. I do not know whether Rob et al will want us to update the Icepack link in E3SM separately for the resync and for the BGC, or if we could PR them together - my guess is that they'll need to be done separately.

@apcraig
Copy link
Collaborator Author

apcraig commented Apr 16, 2024

I'm not sure I understand the plan. The current E3SM Icepack main is up to date with Consortium from a few months ago. I think we forced pushed then, so the history is identical, just E3SM is a bit behind. This PR just does this push from Consortium main to E3SM main, we don't need to force push, just don't squash merge and the mains will remain synced up with identical history. That is taken care of with this PR. As I said, we could probably automate this merge weekly if we wanted, don't need to force push as long as nobody pushed onto E3SM main when they shouldn't.

For bgc, we may want or have to fix the history, but that would NOT have to happen from the head of main and what's on main wouldn't matter. We would find the appropriate hash to branch from on main (or elsewhere), create a branch from it, then merge the bgc changes onto it. When bgc is ready, the hash on that branch would be used in E3SM and then could be merged to Consortium main. We cannot and should not require the head of main in E3SM to be somehow associated with some current version of Icepack that's useful in E3SM. There may be hashes on main that are particularly useful wrt E3SM development, we can know/discover what those are quite easily. But the head of main should not be part of the equation of that.

We should never be force pushing again in general. Only if someone commits something to main that shouldn't be there. We should probably clarify the process again, seems like there is some confusion.

@eclare108213
Copy link
Collaborator

Okay, we need to sort through this. It's is already on the agenda for the Icepack meeting tomorrow: https://acme-climate.atlassian.net/wiki/spaces/ICE/pages/4254990453/2024-04-17+Icepack+Merge+Meeting+notes

apcraig and others added 2 commits April 26, 2024 16:41
* add tutorial documentation

* add tutorial

* add tutorial

* add tutorial

* add tutorial

* add tutorial

* Minor clarifications and grammatical updates.

---------

Co-authored-by: Elizabeth Hunke <[email protected]>
Update Copyright to 2024 in code and LICENSE file
apcraig and others added 7 commits May 14, 2024 11:15
Add run_testoutput script to generate release plots for Icepack
Documentation is being update based on the Consortium workshop in 2024. The "adding a tracer" section has been updated as has the tutorial section related to adding a fluff tracer. An example set of code diffs for the tutorial is also includes as reference.
Port to NCAR's cluster, casper using the HTC nodes using standard cpus. This will support testing and development on another piece of hardware other than derecho and izumi and provides a good option for smaller test cases. Added intel, inteloneapi, and gnu compiler options.

The -check debug option is very sensitive in this version of Inteloneapi, so it has been turned off.
Add congel_freeze namelist and add 'one-step' option to the default 'two-step' option.

Namelist flag congel_freeze chooses which formulation to use. The original is ‘two-step’, since only the mushy boundary moves in the first step and the phase change happens in the next step. Plante et al. (‘one-step’) moves the boundary and performs the phase change in a single time step.  Maintain 'two-step' as default for now.
This is a significant update in the BGC including refactoring Icepack interfaces.

Deprecate skl BGC but leave code alone for now hoping we get help from the community to validate the latest code.

Add BGC parameters to icepack_parameters.F90

Update BGC physics, consistent with E3SM-Project/E3SM#6457.  Remove redundant arguments in BGC interfaces.

    icepack_aerosol.F90
        revised subroutine update_snow_bgc
    icepack_algae.F90
        revised subroutine zbio
        revised subroutine z_biogeochemistry
        revised subroutine algal_dyn
        add subroutine bgc_carbon_sum
    icepack_brine.F90
        revise subroutine prepare_hbrine
        revise subroutine update_hbrine
    icepack_mechred.F90
        add mbio calculation to subroutine ridge_shift
        add flux_bio calculation to subroutine ridge_ice
    icepack_therm_itd.F90
        update calculation of dvssl and dvint in subroutine lateral_melt
    icepack_zbgc.F90
        lots of stuff
    icepack_zbgc_shared.F90
        lots of stuff

Remove redundant arguments in non-BGC interfaces.

    icepack_atmo.F90
    icepack_fsd.F90
    icepack_isotope.F90
    icepack_itd.F90
    icepack_meltpond_topo.F90
    icepack_mushy_physics.F90
    icepack_snow.F90
    icepack_therm_bl99.F90
    icepack_therm_mushy.F90
    icepack_therm_shared.F90
    icepack_therm_vertical.F90
    icepack_tracers.F90
    icepack_wavefracspec.F90

Generalize merge_fluxes to make all arguments optional

Fix bug in subroutine snow_redist computation of hsn_new when nslyr=1

Update the Icepack driver consistent with Icepack interface changes

Update zbgc initialization in the Icepack driver, move bgc parameter intiialization to icepack_init_parameters, update icepack_init_zbgc call.

Update some calls to icepack_warnings_setabort to add file and line number.

Update warning package to improve OpenMP robustness. Fixes potential race conditions that show up when lots of output is produced.

Modified congel implementation in subroutine thickness_changes in icepack_therm_vertical.F90 to recover bit-for-bit results. New congel and new bgc implementation were bit-for-bit independently, but when combining the changes, the intel compiler (with -O2) introduces non bit-for-bit changes (roundoff).

Update bgc namelist defaults and settings in icepack_in

Update bgc tracer sizing in set_env files

Update testing, remove skl tests, add zaero tests.

---------

Co-authored-by: Elizabeth Hunke <[email protected]>
Co-authored-by: David Bailey <[email protected]>
Co-authored-by: Nicole Jeffery <[email protected]>

Update inteloneapi, cray, intel, nvhpc modules/compiler. Changes answers for inteloneapi and cray.

Switch default Derecho queue from main to develop to support quicker turn around and lower costs. Develop is a shared queue.

Error in the snow physics with the cray compiler needs further assessment, but suggest we merge this now as is. That error does not present itself in CICE with snow physics with the same cray compiler.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants