Manifest Reproduce true fix #475
Conversation
Previous behaviour would only use the exe path and restarts already set in the Manifest. So they would pick up when the exe/restart that were currently pointing toin the manifest would change, but not the configured one.
… when files are added to manifest - When reproduce is True, set scanInputs to True so new/changed inputs are discovered - When reproduce is True, only create work symlinks when filepaths are added in manifest during model.setup - Check for files that exist in the manifest, but haven't been added to work directory, or no longer exist
|
Hello @jo-basevi! Thanks for updating this PR. We checked the lines you've touched for PEP 8 issues, and found:
Comment last updated at 2024-08-22 01:38:38 UTC |
|
One goal of the original behaviour was to be able to re-run an experiment from any commit, by using the restart manifest to populate the restart file symlinks as they were at that commit. This would bypass the It is an open question as to the utility of this functionality. As far as I know no-one has used it, and with the typical workflow of using ephemeral One use-case is perturbation experiments branching from the same point in a control experiment. Each perturbation experiment could be a |
Fix some typos Co-authored-by: Aidan Heerdegen <[email protected]>
- Rename existing_filepaths to previous_filepaths - Add description that previous_filepaths keep track of filepaths from previous experiment run - Move reproduce manifest exists check to load()
|
I've updated the manifests to keep track of the pre-existing manifest and to compare directly with the created manifests to try simplify some of the logic. A key difference is that previous manifests are used as source of truth for |
|
I'm doing some tests on performance, which is one of the key criteria for the design and re-use of manifest hashes etc. The I'll report back what I find. |
|
Timing looks similar after the first run of Typical timings for for 708 files. If the input manifest is deleted, forcing recalculation, the timing for this PR is Current The times vary (I tried a couple of times) and likely it is very sensitive to disk usage and cache status. Conclusion: this change does not degrade performance. |
The binhash values may be updated, and maybe the fullpaths if they have changed? But that should only happen if all MD5 hashes match. |
That's great! Thanks for running those tests :) How did you run those timing tests? |
You can pass command line arguments to that command to output all sorts of useful info, above is just the default. There is also a bash builtin |
Ok so I've had a go changing the manifests reproduce True logic. Currently (on the main branch), it create symlinks to the work directory from the manifests, and only raise errors if the files that existing on the manifest are modified.
In the first commit, I changed logic in model setup to always add model exe path and prior restart path to the manifest. This picks up if exe path is changed in
config.yaml. If there's new restart files, in the prior restart path, it will also pick it up that there's been a change.It still had the logic that symlinks from files to the work directory are generated from the manifest initially when reproduce is True. This could run into weird issues when there's a file in the manifest, thats existing unchanged on the filesystem, but it's no longer added to manifest in
Model.setup. It'll pass the manifest checks because the hashes are unchanged and this file will still be linked into the work directory.In the second commit, I have changed it to when reproduce is True, all the symlinks are created from filepaths added in the
model.setup()when it callsmanifest.add_filepath(). I have used the pre-existingexisting_filepathsset on thePayuManifestclass to keep track of what files haven't been added to the work directory but are in the manifest. There is now a check at the end ofPayuManifest.fastcheckchecking if that set is empty when reproduce is True.The second commit, also has changes to the
input/scanInputslogic. I've changed it to havescaninputsTrue whenreproduceis True. This is so changes to inputs inconfig.yamlare picked up, or if new files are added to input directories.scaninputscan be set to False, whenreproduceis False. This is will still not check for changes inconfig.yamlas it'll create symlinks only from the input manifest. If a file in the manifest no longer exists, it will still be removed from the manifests before manifests are checked (only when reproduce isFalse).I'll appreciate any feedback on these changes as it is changing the behaviour of reproduce:True quite a bit, and I'm unsure of my assumptions..
Closes #456