Sync Path

[anton-seaice]

Should the sync path configure follow the style as short path? Where the experiment name is added to the path provided.

i.e. if:

sync: 
    enable: True
    path: /g/data/$PROJECT/$USER

should this create a folder

/g/data/$PROJECT/$USER/<<experiment_name+uuid>>

automatically to sync to?

This seems especially useful when using the payu branch from a point in another experiment and you don't want to override the existing results ?

Current behavior is it will create the output folders directly in the path directory

[jo-basevi]

Yeah that's a good idea. My only concern is that it would break existing usage of sync (if there is any), by syncing all output in a new subfolder in the path directory. But it would be good way to ensure uniqueness in remote archive folder name.

I thought I had previously added in logic in the sync code to check if there was pre-existing metadata.yaml file with a matching UUID in the sync path to ensure it was the same experiment, but after looking at the code, I had not sorry..

[anton-seaice]

I think its only an issue if users change payu version part way through an experiment? Or try to add to an existing experiment which was run on an older payu?

The worst case is that an extra folder is created - it wouldn't be too hard to tidy up manually ... ?

[jo-basevi]

Yeah true, it would not be difficult to tidy up manually, and would only be an issue if changing payu versions during running an experiment.

Just a couple more thoughts about implementation:

• Experiments that were started prior to UUID changes (payu versions < 1.1), still have an experiment name (the name used for archive directories) being the control directory name. For these experiments, should the default archive folder name that is being synced to, include an UUID to ensure uniqueness?
• There could be an experiment option for the sync config, to allow overriding the default folder name if it's needed.