Adding Provenance to experiments with git

[jedwards4b]

The run database used in cmip6 experiments is probably not suitable for cmip7 and it has been proposed to replace it with a simple lookup table in google sheets. This would give us a lot of what we need to keep track of experiments but would not easily provide us provenance information on individual experiment cases. I propose that we add a new option to the cesm create_newcase command --create-git-repo (I'm not stuck on the name)

This command would cause a new git repo to be created under a predefined git domain, perhaps github.com/CESM/CMIP7. We already have a capability in cime that updates a CaseStatus log file with each change in a case, using this option to the create_newcase command would in addition to updating the CaseStatus file, add a new git commit. When files are added or removed from the case, such as files in SourceMods, they would also be added or removed from the git repository. I think that this functionality would be relatively easy to implement and would greatly improve our ability to track and reproduce an experiment.

[nanr]

I like this idea too! We could also add notes about the run (e.g., debugging, etc.) in this repo.

[cacraigucar]

@jedwards4b - I'm curious on why you are suggesting a new repo for each CMIP7 experiment as opposed to creating a new branch and/or issue for each one within a single github.com/CESM/CMIP7_RUNS repo? @cecilehannay has effectively been using https://github.com/NCAR/amwg_dev/issues to track her experimental runs. While it might not contain all the provenance you are proposing, it has been useful for her documentation purposes. This isn't a vote for one way or the other, but rather simply a question on why you are proposing this workflow.

[dlawrenncar]

I think I like the idea as well. Perhaps after you all have flushed the idea out a bit, we could have a quick demonstration of what it would look like at a CESM Project meeting to see how people like it.

[aidanheerdegen]

Sorry to drop in unannounced (and uninvited), but I have a bit of experience using git for capturing climate model run provenance that might be useful.

At ACCESS-NRI we're using payu to run our climate models. It is a python tool developed by @marshallward (GFDL) when he was working here in Australia. It's subsequently been supported by a couple of research centres (by myself), and now has been adopted by ACCESS-NRI.

payu has been utilising git for almost 10 years to capture provenance of experiments and share and distribute experiment configurations. It has evolved quite a bit over that time. Features now include

• git tracking of all text based configuration files
• Capture of executable, input and output provenance by tracking (with git) manifest files that record filesystem path and identifying md5 hashes
• Automatic git commit every time the model runs, as the restart manifest changes for every run
• Automated UUID Experiment ID generation and automated model metadata generation
• Support of git branches by wrapping git clone, branch and checkout commands

payu releases are documented on our forum

https://forum.access-hive.org.au/t/payu-a-workflow-manager-for-some-access-models/1098/5

and there is also a tutorial which gives a flavour for how it works

https://forum.access-hive.org.au/t/access-om2-payu-tutorial/1750

An (oldish) example of an experiment showing how the automated commits work is here

https://github.com/rmholmes/1deg_jra55_ryf/commits/C7_3D_MLT_rcpwarm/

And this is the repo where we release our supported versions of the Ocean Sea-Ice model:

https://github.com/ACCESS-NRI/access-om2-configs

We're actively working on new features to improve experiment provenance.

I'm not suggesting that you should adopt payu, but there is clear overlap with what you're doing, and it may be useful to see the features we've added and how we've approached using git, and experiment provenance in general.

[jedwards4b]

@aidanheerdegen Thank you so much, I'm sure we will find value in the experience.

[cecilehannay]

I am not sure whether it is helpful but here are the scripts I use to save the case directories on amwg_dev

/glade/u/home/hannay/Notes/amwg_dev/b_cases_github.csh
/glade/u/home/hannay/Notes/amwg_dev/f_cases_github.csh