Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Modify COSMIC flows to use reconstruct both locally and at NERSC #14

Open
dylanmcreynolds opened this issue Feb 23, 2024 · 0 comments
Open
Assignees

Comments

@dylanmcreynolds
Copy link
Contributor

dylanmcreynolds commented Feb 23, 2024

Talking with @davidalexandershapiro, he gave some interesting requirements for his flows:

Whenever a file is created, the file (in parrallel):

  • copied to NERSC
  • run through a low-res reconstruction on a beamline machine

When the file is finished copying to NERSC

  • A high-res reconstruction is performed

The low-res reconstruction lets the user quickly get some feedback on the success of the scan. The high-res will be used for actual research.

This introduces a few new concepts for his flows.

The main flow will invoke another flow, but not as a sub-flow. It will instead launch a new flow from a deployment so that it can get picked up from the worker on the local machine. I've seen examples of doing this in a sub-flow on a new deployment, which would be nice because you'd see failures in the main flow's status. But I can't find those examples right now. Anyway, we already have a utility method that schedules flows onto a different deployment in the future. In the simple case, we could reuse that code and schedule is for "now":

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants