Should manage_externals be replaced with a new tool

[gold2718]

Background

• Both CESM and its component models require a robust tool for managing external repositories
• manage_externals has served this role well but is slow and difficult to maintain or improve
• Now that all component models and submodels use git for their revision control, a multi-protocol tool such as manage_externals is no longer required

Requirements

This section should represent a CSEG and community consensus of the requirements that MUST or SHOULD be met by a CESM externals management tool / process. Note that manage_externals currently supports all of these requirements except those explicitly noted below.

1. A model MUST be able to control the revision of its sub-components.
2. The set of sub-component revisions (usually tags) MUST be easily retrieved before running the externals management tool. This information MUST be stored with a model revision. Note: in manage_externals, this requirement is met with the Externals[_xxx].cfg files.
3. An externals management tool MUST be able to support, suppress, or override the behavior of git submodules in any of its sub-components. support means that all git submodules are checked out as configured. suppress means that no git submodules are checked out. override means that git submodules are selectively checked out.
4. An externals management tool MUST be able to support sparse checkout.
5. An externals management tool MUST be able to correctly update a repository worktree or provide a clear reason with workaround instructions in the case that it cannot perform the update. Note: manage_externals does not currently meet this requirement.
6. An externals management tool MUST be able to handle situations where a developer is modifying code in one or more sub-components. In particular, it MUST NOT make desctructive changes to a sub-component clone.
7. An externals management tool SHOULD be able to support optional sub-components.
8. An externals management tool MUST be able to sucessfully install the externals of any checked-out tag, older or newer (with an obvious hard limit before the tool existed or when a bug in the tool existed). Note: manage_externals does not currently meet this requirement in all cases.
9. An externals management tool MUST be able to display the current status of a set of checked-out externals.

[gold2718]

Given the set of DRAFT PRs submitted by @jedwards4b, I thought it would be good to have a single place where we could discuss the pros and cons of this new approach and any possible required changes if it is to be accepted. I wrote down a set of requirements as a strawman. I hope the community weighs in and that these are updated as a consensus emerges.

[jedwards4b]

I hope you don't mind I edited your list so that I can refer to it by index.

Issue 1: I believe this is fully covered. Please provide a counter example if you disagree.

Issue 2 is a bit more difficult, git submodules use hashes and not tags, you can see the list of submodules in the toplevel .gitmodules file and you can see the list of hashes in github. One suggestion to solve this problem is to continue to use a file like Externals.cfg and add code to the install script to maintain the consistency of that file. See discussion here.

Issue 3 currently the install scripts can either install internal only or internal plus external submodules - extending this to be able to control individual submodules should be a straightforward change.

Issue 4 is fully covered by this change and in fact the proposed code does a better job of this than manage externals - in manage externals the full code is checked out and then pruned to be the requested sparse code. In the proposed change the pruning is done before the code is checked out and only the requested sparse checkout files are downloaded.

Issue 5 I think this is covered, although I find git messages to be fairly obscure sometimes.

Issue 6 I don't believe that there is a problem here, a counter example may help.

Issue 7 Not currently handled but as with Issue 3 I don't think that it would be a difficult extension.

[gold2718]

I think using a numbered list is a good idea although as this list changes, some of these notes might end up with stale numbering. Note that when I put together this initial list, I was not thinking of what might be deficient or not with either manage_externals or with your proposed approach. That is, these are general requirements, not issues. I filled in the manage_external deficiency later.
I think your analysis is good although I do not have enough experience with the new system to know if I agree with everything.

[gold2718]

Regarding requirement 2 (see tags before checkout):

One suggestion to solve this problem is to continue to use a file like Externals.cfg and add code to the install script to maintain the consistency of that file

I have been thinking that this is a good solution. We can use the same format but drop the protocol field. I have not yet thought through how requirement 3 will be met.

[Katetc]

Hi Jim, I think Adam relayed my concern correctly. For updating externals in cism_wrapper (for example), it's nice to be able to look up the current or previous externals lists for CTSM on GitHub. To use the the script would require the following extra steps: 1, checkout a clone of CTSM. 2. Checkout the tag I'm interested in. 3. Run the script. 4. Delete or clean up the clone. It is much more convenient to just look at CTSM on GitHub, choose the tag I need from the drop down and click on "Externals.cfg" in the browser. I can then see a list of externals and this gives me a good view of the tags that work together for a given CTSM release. This also applies to CAM and its externals and CESM with its externals.

[gold2718]

Right now, I can do something like:

git diff cam6_3_135 cam6_3_143 -- Externals.cfg

This is really useful to me and a lot more flexible than the "currently checked out commit".

[cacraigucar]

@jedwards4b - I concur with all of the above statements about needing to be able to identify the externals without checking out anything. When I update CAM's externals to match a CESM tag, I do look at the Externals.cfg file and I typically do not clone the CESM tag I am wanting to duplicate. I just look at the file in github.

Also, I would suggest that the Externals.cfg file remain part of the checkout process so we know it is correct and not just a file that may or may not have been updated with a tell-me script prior to a commit.

[gold2718]

I'm wondering if we aren't thinking about this the wrong way. Currently, in order to update Externals.cfg you have to do these steps (using a CAM update into CESM as an example):

1. Update CAM
2. Create new tag
3. Update Externals.cfg with new tag
4. Merge and create new CESM tag

Using the proposed (git submodules) system, this sequence might look something like this:

1. Update CAM
2. Create new tag
3. Run the ./update_externals Externals.cfg script (currently called tell-me with some output mods)
4. Merge and create new CESM tag

This is the same number of steps and might even be a bit less error-prone (since you have to run this from a testable checkout). Thoughts?

[cacraigucar]

@gold2718 - If your reply was to my updating externals, my workflow is:

1. Create branch for new tag
2. Update Externals.cfg to reflect externals for upcoming CAM tag
3. Run manage_externals to get the updated externals
4. Run regression tests
5. Merge and create new CAM tag

Steps 2 and 3 in the git submodules step can be replaced with the new script (see my request below)

You'll notice the update is the first step and the testing results are based on the actual externals in the file. That is what I want to make sure is linked. I am saying that a manual step to running ./update_externals Externals.cfg is error prone as if someone forgets that step, then the results are wrong. This already happens on the "plans page" at times, so I have learned to ignore it there.

I believe it is ESSENTIAL that Externals.cfg be used or auto-generated in the actual testing sequence so you can be guaranteed that the results are correct. This is a feature of the current manage_externals that I want to make sure is maintained. It can be as simple as ./update_externals Externals.cfg being run as part of the new install script.

[gold2718]

Regarding requirement 6 (developer modifying code):

I don't believe that there is a problem here, a counter example may help.

I think the biggest issue here is to define what behavior we want in all cases. Maybe it is as simple as having a --ignore-modified-externals or --warn-on-modified-externals switch to allow the tool to update everything except the modified repositories.

[gold2718]

I believe it is ESSENTIAL that Externals.cfg be used or auto-generated in the actual testing sequence so you can be guaranteed that the results are correct. This is a feature of the current manage_externals that I want to make sure is maintained. It can be as simple as ./update_externals Externals.cfg being run as part of the new install script.

Two thoughts on this:

1. I like the idea of the install script updating a (mandatory?) Externals[_xxx].cfg file.
2. Running ./update_externals --check Externals.cfg or something like that could also be part of the CAM test suite to either run the install or check to make sure the externals match Externals[_xxx].cfg (or error and quit).

[cacraigucar]

Whatever solution is adopted, it should be identical for all components and CESM, so I believe modifying CAM's testing is not a solution. We need to be able to trust the "Externals.cfg" contents and its existence should be mandatory in every repository. The name and format of the file could be changed with this script, but needs to be consistent across all repositories.

[jedwards4b]

I agree and hope to have something ready to discuss at the 1/9 cseg meeting.

[gold2718]

I believe modifying CAM's testing is not a solution

I guess I was not clear. My thought is that this would be an improvement in CAM's testing, even with manage_externals. It would enforce testing with the current Externals.cfg and Externals_CAM.cfg. Currently, you can run tests without running manage_externals first and obtain test results that do not reflect what is being committed. Adding a check to the test driver is not a replacement for having the externals files in the repo, it was meant as a improvement to the testing process.

[jedwards4b]

I have something I would like to get feedback on. I've only updated cam so far so if you would like to try and send me feedback please run:

git clone  [email protected]:jedwards4b/CAM.git -b remove_manage_externals cam_rme
cd cam_rme
./esm --help  (To see the options)
./esm (To checkout a full cam_development tree)

If you change tags in .gitmodules running ./esm --update will update the sandbox to the new tags.
./esm --status shows the status of the sandbox relative to the tags in .gitmodules

[gold2718]

Some of the externals do not have tags. Is that just a 'work in progress' thing?

[gold2718]

One potential issue I see is that CAM now has a single .gitmodules file that includes all the externals needed to run the model (e.g., CMEPS, CTSM).
What happens if this version were included as a submodule of CESM that also ran the new tool?
It is a little hard to tell since those (non CAM) externals seem to be dummies right now (.git directory but no files).

[jedwards4b]

If you ran the ./esm (formerly install) you would install the externals including clm which will illustrate how the tool manages toplevel vs component installs. Your comment about "(non CAM) externals seem to be dummies right now (.git directory but no files)." doesn't make any sense to me. Either they should be empty directories with no .git directory or they should be populated - can you try again with a fresh checkout and let me know if that's still the case? If it is can you tell me what version of git you are using? Thanks,

[jedwards4b]

Yes that externals do not have tags is a work in progress issue.

[gold2718]

If you ran the ./esm ...

I found the bug. It was not the git version, it was the PBKAC version. Fixed that and the external components (e.g., clm) checkout just fine.

I'm not sure what the mechanism is, however, running this, I get empty directories for clm externals (e.g., share, components/cdeps). I guess this is a bit harmless but also a bit weird. What is the mechanism that checks out fates but not share? It might be easier to see if another external, say ctsm, was also moved to the new system (currently, CAM has esm and CTSM has install.

Also, ./esm --update reports that the action if update but gives no output indicating what is updated. If I ask git, I get something like:

$git status
On branch remove_manage_externals
Your branch is up to date with 'origin/remove_manage_externals'.

Changes not staged for commit:
  (use "git add <file>..." to update what will be committed)
  (use "git restore <file>..." to discard changes in working directory)
	modified:   .gitmodules
	modified:   components/cmeps (new commits)

which is also weird since components/cmeps does not have new commits, it just has a new tag checked out.

Finally, (and correct me if I'm wrong), it seems that instead of a single place where external versions are defined, we now sort of have two. For instance, if I did update the .gitmodules file and commit just that (i.e., not run ./esm --update), then the branch would store two different versions for some externals.

To demonstrate this, if someone ran (as many do),

git clone --recurse-submodules [email protected]:jedwards4b/CAM.git -b remove_manage_externals cam_rme

then, they get the versions stored in the .git directory, not the versions stored in the .gitmodules file. Also, they will now get extra copies of externals such as share and cmeps since many components have them loaded.

This behavior is quite different from manage_externals since currently, there is only one place where externals are defined.

[jedwards4b]

Update: I've renamed the driver script git-fleximod and refactored a little.
If you put this directory in your path you can run it as:
git fleximod [options]

[gold2718]

After updating, ./esm --update crashes.

Also are users supposed to modify their PATH every time they clone a repo?

$ ./esm --update
Getting checkout tools
Cloning into 'bin'...
remote: Enumerating objects: 11, done.
remote: Counting objects: 100% (11/11), done.
remote: Compressing objects: 100% (9/9), done.
remote: Total 11 (delta 0), reused 7 (delta 0), pack-reused 0
Receiving objects: 100% (11/11), 8.61 KiB | 8.61 MiB/s, done.
root_dir is /home/goldy/scratch/cam_rme
Traceback (most recent call last):
  File "/home/goldy/scratch/cam_rme/./bin/git-fleximod", line 334, in <module>
    _main_func()
  File "/home/goldy/scratch/cam_rme/./bin/git-fleximod", line 324, in _main_func
    submodules_update(gitmodules, root_dir)
  File "/home/goldy/scratch/cam_rme/./bin/git-fleximod", line 264, in submodules_update
    if esmtag not in tags:
TypeError: 'in <string>' requires string as left operand, not NoneType
Exception ignored in: <function GitModules.__del__ at 0x769302af4ca0>
Traceback (most recent call last):
  File "/home/goldy/scratch/cam_rme/bin/modules/gitmodules.py", line 41, in __del__
  File "/home/goldy/scratch/cam_rme/bin/modules/gitmodules.py", line 38, in save
NameError: name 'open' is not defined

[jedwards4b]

The empty directories are a git submodules feature, I don't think I can do much about that.
The issue with --update should be fixed now.

Also are users supposed to modify their PATH every time they clone a repo?

I think maybe they get the git-fleximod repo independent of the model repo - what do you think?
We can add something in git-fleximod that tells you if you need to update for a particular model version.
But I'm not sure I see a reason they need to be distributed together.

[jedwards4b]

So I've changed the --status and --update options so that they are positional and added install.
So there are three basic commands

git fleximod status
git fleximod update
git fleximod install

or git-fleximod if you don't want to put it in the PATH.

[cacraigucar]

I was trying to give this a try, but when I git checkout remove_manage_externals, all I see is install instead of esm. When I run install, I get an error (but that is not unexpected since it is not the script I am supposed to be using)

[jedwards4b]

@cacraigucar It's been renamed to install again, so running ./install is correct. What error did you get? I just tried and it seems to be working.

[cacraigucar]

Here is the error message:

izumi$ ./install
Getting checkout tools
Cloning into 'bin'...
remote: Enumerating objects: 14, done.
remote: Counting objects: 100% (14/14), done.
remote: Compressing objects: 100% (12/12), done.
remote: Total 14 (delta 0), reused 7 (delta 0), pack-reused 0
Receiving objects: 100% (14/14), 11.71 KiB | 2.93 MiB/s, done.
File "./bin/git-fleximod", line 31
help=f"Subcommand of fleximod, choices are {choices}",
^
SyntaxError: invalid syntax

[jedwards4b]

I think maybe you are using an older python version? Is it choking on the f string format?

[cacraigucar]

That got me further, (I tried it twice) but it still bombs out:

izumi$ module load lang/python/3.11.5
module-info mode load in progress
Unloading python module: lang/python/3.11.5
izumi$ ./install
root_dir is /home/cacraig/cam_remove_manage_externals
Successfully checked out ali_arms
Recursively checking out submodules of cime /home/cacraig/cam_remove_manage_externals/cime
ERROR:root:Failed to checkout CIME/non_py/cprnc
Traceback (most recent call last):
File "/home/cacraig/cam_remove_manage_externals/./bin/git-fleximod", line 348, in
_main_func()
File "/home/cacraig/cam_remove_manage_externals/./bin/git-fleximod", line 340, in _main_func
submodules_install(gitmodules, root_dir, fxrequired)
File "/home/cacraig/cam_remove_manage_externals/./bin/git-fleximod", line 305, in submodules_install
submodule_checkout(root_dir, name, path)
File "/home/cacraig/cam_remove_manage_externals/./bin/git-fleximod", line 214, in submodule_checkout
submodules_install(gitmodules, repodir, ["I:T"])
File "/home/cacraig/cam_remove_manage_externals/./bin/git-fleximod", line 305, in submodules_install
submodule_checkout(root_dir, name, path)
File "/home/cacraig/cam_remove_manage_externals/./bin/git-fleximod", line 218, in submodule_checkout
utils.fatal_error(f"Failed to checkout {name}")
File "/home/cacraig/cam_remove_manage_externals/bin/modules/utils.py", line 130, in fatal_error
raise RuntimeError("{0}ERROR: {1}".format(os.linesep, message))
RuntimeError:
ERROR: Failed to checkout CIME/non_py/cprnc

[jedwards4b]

git-fleximod - maybe you should remove the your clone and start over and please just test cam, I'm still working on an issue in cesm.

[jedwards4b]

Okay I think I've fully fixed the ssh issue now.

[jedwards4b]

And I've fixed CESM, so at this point CESM, CAM and CLM should all work correctly. Thank you for letting me know if you run into any issues.

[cacraigucar]

@jedwards4b - I think this is probably an important point to understand and might impact our ability to track updates to the gitmodules. Why did git pull say everything was up-to-date (and the git log is still saying your last edit was Jan 7)? It did work to blow away the checkout and reclone it.

[jedwards4b]

Because git-fleximod is not connected to the cam or cesm git repo. We can discuss connecting it using git subtree (it can't be a submodule) but my preference would be to keep it separate and document that it isn't connected and needs to be treated independently.

[jedwards4b]

git-fleximod is now available on pypi: https://pypi.org/project/fleximod/
And on conda-forge: https://github.com/conda-forge/fleximod-feedstock

[gold2718]

I still do not understand the advantage of requiring installation of a new tool over just including this fairly compact tool with the code. Can you please add an explanation here?
Also, can you address how users would work if the underlying data format or content (currently stored in .gitmodules file and the .git directory) changes?

[jedwards4b]

I'm totally open to providing this tool with the model distribution - I just wanted to open a discussion about keeping it separate. I think that most of our users are sophisticated enough to download and install another prerequisite - but maybe I'm wrong.

So if the format of the .gitmodules file were to change as long as it remains a config file that should not be a problem, I intentionally chose variable names that are not likely to be repeated by another tool and variables not used by this tool can be safely ignored. If the .gitmodules or .git directory format were to change then git-fleximod would need to change as well - but I would expect plenty of lead time were that to happen.

[gold2718]

I would expect plenty of lead time were that to happen.

Can you ensure that this would always be backwards compatible (so new versions could check out old tags)? Any chance it could also recognize an old version that uses manage_externals and call that instead?

[mattldawson]

Following up on @mwaxmonsky 's suggestion in the CSEG meeting, has CMake been explored as a possible solution? There is built-in functionality like find_package that might be able to handle these requirements and avoid the need to build and maintain a custom tool. @mwaxmonsky @K20shores - what do you think?

[K20shores]

• @gold2718 yes, find_package, as well as fetch_content, both support version matching. fetch_content also resolves circular dependencies when multiple packages are requesting the same dependency.
• create_newcase could also be replaced entirely with cmake. Cmake supports running any script on the system, and has built in support to generate files for whatever purpose while allowing the use of any defined cmake file to be placed into the generated file
• Cmake configures packages to be built and also builds them. You are right that create_newcase could be reimplemented in cmake, but it doesn't have to be; cmake could be used a little or as much as makes sense to serve our user community
• If by modifiied externals, you mean local version of code checked out with modifications; those changes are used when code is compiled. You can also specify explicit tags or commit hashes if we use fetch_content for software components, at which case changing the tag will trigger cmake to checkout the correct git thing and the recompile and link the code
• Cmake uses git for fetch_content. find_package searches the filesystem to look for installed packages. fetch_content can be configured to first check the filesystem for existing packages of the required version and on a failure pull from git

The biggest downside is that using cmake is easiest if all of our downstream packages support cmake. However, cmake can be setup to support any build system, which a functionality called ExternalProject. Cmake can cover any use case we have, including replacing manage_externals with a similarly named thing. ExternalProject should be our last preferred option and we should instead favor adding enough cmake glue into downstream projects to enable fetch_content since fetch_content is minimally invasive and makes including things easy to use. Here's an example in our micm project which includes google test and a json parser easily into our build

[gold2718]

@K20shores, I am not arguing about the power of CMake -- I have been using CMake since mid-2010 and am a fan.
The issue here is that create_newcase is the heart of the CIME case-control system (CCS). It is shared between several institutions (NCAR, DOE, CMCC in Italy, and NorESM in Scandinavia). If using CMake involves changing create_newcase (which I think it must because the system does not even know which components it needs loaded until after it processes the compset), this will make it much harder to maintain the CIME CCS since the other institutions have no interest in this change.

[K20shores]

@gold2718 I don't know enough about create_newcase to answer that question.

[mwaxmonsky]

Apologies for the delay here! Meant to respond sooner but got caught up in trying out different CMake approaches.

What we'd be recommending is using a system of CMakeLists.txt (or similar) in the respective repos under NCAR/ESCOMP and FetchContent commands to place the dependencies in the appropriate place and not for building at the current time.

This has the cost of having non-build related CMake files in the repos for now but has the benefit of using CMake to manage the dependencies and does not require modifying .gitmodules with custom attributes. I have a local proof of concept that pulls in a git repo (Project A) with another dependent repo (Project B) that has it's own submodules (to represent a project outside of the organization) and calling CMake from the top level project pulls down the dependencies and places them in the desired place in the source tree.

There is other functionality in manage_externals that I am not familiar with that I would need to dig further into but the main use case that I have used it is for getting new dependencies which CMake should be capable of handling with a little debugging to verify.

I would need to look into the current workflow for having locally modified externals. My gut instinct is that CMake would not modify it by default but the current way that manage_externals handles them should be reproducible.

[jedwards4b]

@mwaxmonsky, @K20shores First thank you for voicing your ideas and even taking the time to work out a prototype. I have no doubt that this could be done with cmake, and as you point out the entire case control system could perhaps be replaced given enough time and resources to apply to the problem.

But we don't have unlimited time and resources. My proposal is to replace the manage externals tool with something supporting the functionality that we need now while reducing the support burden on the team. git-fleximod has less than 20% the code of manage-externals. The work to develop this tool is already done, its functionality is proven and it is ready to be put into place. I believe that the question at this time is not what other ways could we do this, but should we replace the manage externals with git-fleximod now. If at some point in the future we find the resources and this tool appears inadequate to the job then we can certainly consider replacing it at that time.

[gold2718]

Why call this tool a git tool?

git help fleximod does not work ...

This was a helpful answer, however, I do not feel like you addressed the initial question. Further down, you say:

It is a more flexible extension to the capabilities of git submodule

but I still do not see the advantage of tying it to the git infrastructure. Everything would work the same with a simple script name (like install and esm). What am I missing?

[gold2718]

So you need to use the full git-fleximod name - is that a problem?

and

What do you want to call it?

• checkout_externals
• checkout_model_sandbox

Something that reflects its purpose.

It is a more flexible extension to the capabilities of git submodule

I'm not sure it is an extension since it does not support actions such as add or set-branch. That is another reason I would prefer a name aligned with its specific purpose.

[gold2718]

I do not understand the specific wording used in status messages. For instance, submodule ccs_config not checked out, aligned at tag ccs_config_cesm0.0.88. What is aligned exactly? If it is not checked out, how can it be aligned with anything? Can you use more straightforward wording (e.g., 'should be' instead of 'aligned at')?

. . . Constructive suggestions for better wording are welcome.

But I did include a suggestion. What am I missing here?

[gold2718]

I didn't use annotated tags - perhaps this is a reason to do so and it should be included in the
workflow instructions - however I think that in the current context you are just looking for reasons to be critical

Again, that was not the intent, please let me know why you came to that conclusion.

I was confused by the mismatch and wanted to make sure it was just the specific tags that were used in the example and not a "feature" of the tool. Thanks for clarifying.

[jedwards4b]

I have updated CESM, CAM and CTSM branches with a new version of git-fleximod. Please give it another try. @gold2718

[jedwards4b]

Updated to add support for fxhash. My feeling is that I should add this to the github workflow testing and fail if a PR is going to a protected branch with an fxhash reference.

[jedwards4b]

I updated CESM, CAM and CTSM branches with a new build of git-fleximod.

[fischer-ncar]

I just tried using git-fleximod with the checkout of the latest cesm remove_manage_externals branch. In CESMROOT I'm getting a src directory with the directory structure for cam physics and dynamics.

[fischer@izumi CESM_RME]$ ls -R src
src:
dynamics physics
src/dynamics:
mpas
src/dynamics/mpas:
dycore
src/dynamics/mpas/dycore:
src/physics:
clubb cosp2
src/physics/clubb:
src/physics/cosp2:
src
src/physics/cosp2/src:

[jedwards4b]

Thank you Chris, I'll look into it. It is checking out the code correctly but seems to also be creating some empty directories in the wrong place.

[fvitt]

I am seeing these empty dirs:

> find * -empty -type d

components/cism/ccs_config
components/cism/cime
components/cism/components/cdeps
components/cism/components/clm
components/cism/components/cmeps
components/cism/components/cpl7
components/cism/components/mosart
components/cism/libraries/mct
components/cism/libraries/parallelio
components/cism/share
components/clm/ccs_config
components/clm/cime
components/clm/components/cdeps
components/clm/components/cism
components/clm/components/cmeps
components/clm/components/cpl7
components/clm/components/mizuRoute
components/clm/components/mosart
components/clm/components/rtm
components/clm/doc/doc-builder
components/clm/libraries/mct
components/clm/libraries/parallelio
components/clm/share

This is with the CAM branch

[jedwards4b]

Fixed, please try again.

[jedwards4b]

@fvitt these are expected. They are a feature of git submodule. If I were to remove them locally then other git commands would not work correctly - always showing your source as locally modified.

[peverwhee]

Hi @jedwards4b

manage_externals used to return more information when a checkout failed due to a lack of disk space:

ERROR:root:Failed with output:
fatal: fsync error on '/home/courtneyp/Projects/CAM-test1/components/cice/.git/objects/pack/tmp_pack_DSL9Vz': Disk quota exceeded
fatal: fetch-pack: invalid index-pack output

fleximod just says it fails but not why:

root - ERROR - Failed to checkout cice6
Traceback (most recent call last):
File "/home/courtneyp/Projects/CAM_fleximod/bin/git-fleximod", line 8, in
sys.exit(main())
^^^^^^
File "/home/courtneyp/Projects/CAM_fleximod/.lib/git-fleximod/git_fleximod/git_fleximod.py", line 429, in main
submodules_checkout(gitmodules, root_dir, fxrequired, force)
File "/home/courtneyp/Projects/CAM_fleximod/.lib/git-fleximod/git_fleximod/git_fleximod.py", line 365, in submodules_checkout
single_submodule_checkout(
File "/home/courtneyp/Projects/CAM_fleximod/.lib/git-fleximod/git_fleximod/git_fleximod.py", line 191, in single_submodule_checkout
utils.fatal_error(f"Failed to checkout {name}")
File "/home/courtneyp/Projects/CAM_fleximod/.lib/git-fleximod/git_fleximod/utils.py", line 130, in fatal_error
raise RuntimeError("{0}ERROR: {1}".format(os.linesep, message))
RuntimeError:
ERROR: Failed to checkout cice6

Would it be possible to pass additional information through to the error handler on the reason for the failure?

Thanks!

[jedwards4b]

Yes, I can add better error trapping.

[cacraigucar]

Here is a workflow I do routinely, but doesn't work as expected. I will clone a tag, edit my .gitmodules file to reflect what I want and then bin/git-fleximod checkout to get exactly the setup I want. I don't get the updated external, but rather the one that was originally in the clone.

Details:
git clone (this contains a reference in .gitmodules to carma4_05)
edit .gitmodules with the following

---

[submodule "carma"]
    path = src/physics/carma/base
    url = https://github.com/ESCOMP/CARMA_base.git
    fxurl = https://github.com/ESCOMP/CARMA_base.git
    fxtag = carma4_01
    fxrequired = I:T

---

bin/gitfleximod checkout (get carma4_05 and not the carma4_01 which I specified and was listed in the git-fleximod output when it ran.

[jedwards4b]

The command to match externals to .gitmodules is 'update', 'checkout' is intended to initially create the repository matching the submodule hashes (as opposed to the fxtag's)

[cacraigucar]

Repeating the above listed workflow, but substituting the final command with update (instead of checkout):

bin/git-fleximod update

I do not get anything in the src/physics/carma/base directory (it is empty), and all of the other externals are not populated either.

Neither checkout nor update work by themselves to get the requested carma tag. The only way I've been able to get this to work is to do two commands, checkout followed by update. If this is truly the case (and not a bug), then perhaps "checkout" can perform the current "checkout" followed by "update" automatically for the user.

[cacraigucar]

In discussions with other users about the need to differentiate and use the correct argument of either checkout or update, I ruminated that perhaps we should always do checkout followed by update. The errors from using the wrong argument can lead to unexpected results based on what has been done with the .gitmodules and the code base (for example, deleting a directory and not doing a checkout before doing an update). Unless you go in and check your checkouts, you may not have what you expect if you use the wrong argument.

@mattldawson pointed out the flag --recurse-submodules might solve this problem. Could this be explored as way for git-fleximod to perhaps combine checkout and update into a single checkout command (and remove the need for update)?

[jedwards4b]

It's not clear to me exactly what behavior you expect - can you be more precise? If you are getting a tag from the official repository the github workflow design means that you never have to run update (gitmodules and fxtags are always consistant). The update command is intended for developers who want to get a combination of tags not currently available - does that make sense?

[cacraigucar]

There are a couple of scenarios:

A developer wants to modify the externals they are using from a clone of the official repository. In manage_externals, they would edit the External file(s) and run manage_externals. Now they need to edit the file, and run git-fleximod twice, first with checkout and then update.

A developer wants to "start over" with an external they are using (maybe they are removing debug statements they put in there temporarily). In the past, they would rm -rf directory, and then run manage_externals to get the unmodified code. Now they need to do checkout and then remember to do udpate if they had modified the .gitmodules file for that particular external.

I'm pointing out that this new tool requires a bit more thought by the user whereas before manage_externals did everything for the user with a single command. If we could eliminate the differentiation between checkout and update, and have a single command do both, I think that would be ideal.

[jedwards4b]

I think that I can modify update so that if a component is not already checked out it will be checked out before updating. I think that would resolve both these scenarios - I'm not sure that you would want to fully eliminate the difference, but I'll take some time to think about why before I respond to that.

[cacraigucar]

If you do modify update to checkout a component that is not there, then I would propose that update might make checkout no longer necessary. I am making this statement from a user point of view and not with any knowledge of what is going on internally with git-fleximod

[cacraigucar]

@jedwards4b - As we get closer to release of this tool, could the documentation perhaps be augmented? Here are some suggestions of what might need to be included:

• The -h option is a good start, but having a little more description on the options would be useful (what do update, status and test actually do?).
• For the uninitiated, when they find manage_externals missing, where do they find the replacement (something to direct them to bin/git-fleximod)?
• Where do they find the replacement for Externals.cfg (.gitmodules is a hidden file)?
• What are the additional fields in .gitmodules and what do they do, when do we need to add them, etc?

[jedwards4b]

Besides the -h option the README.md contains this information, https://github.com/jedwards4b/git-fleximod?tab=readme-ov-file#git-fleximod
I'm not sure how the individual components want to present this information and I have refrained from modifying their documentation.

[jedwards4b]

New versions of several component models are available:

https://github.com/ESCOMP/CESM.git  rme/cesm2_3_alpha17c
https://github.com/jedwards4b/CAM   zz_rme/cam6_3_150.01    
https://github.com/jedwards4b/CESM_CICE  rme/cesm_cice6_5_0_7 
https://github.com/jedwards4b/CTSM  rme_tags/CESM3_dev.n04_ctsm5.1.dev166.01  

[cacraigucar]

@jedwards4b - I would propose for CAM - that we have in addition to bin/git-fleximod, a soft link in the bin directory back to your ./lib/git-fleximod/README.md since this directory is hidden from the users.

Also, a reminder that README.md needs to be updated with the current implementation (I'm sure it's on your radar).

I'm also contemplating having a text file in CAM's home directory Where_is_manage_externals.txt which gives the directions on using bin/git-fleximod and that Externals.cfg/Externals_CAM.cfg have been replaced with .gitmodules (again since this file is hidden). Most users who have downloaded CAM or CESM before don't think of looking at the README.md file, because they "know" what it says. I don't know if other components and CESM might want to do something similar, in which case CAM is willing to adopt a uniform look/feel for presenting this information to users. Of course, this is all my personal opinion and is subject to change when others weigh in.

[jedwards4b]

CESM3_0_beta01 is out and manage_externals has been replaced.