Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

DM-38041: rewrite pre-exec-init logic to work without QGs and respect storage class differences #444

Merged
merged 16 commits into from
Sep 12, 2024
Merged

DM-38041: rewrite pre-exec-init logic to work without QGs and respect storage class differences #444

merged 16 commits into from
Sep 12, 2024

Commits on Sep 12, 2024

  1. Add method to test whether a connection is optional.

    @TallJimbo
    TallJimbo committed Sep 12, 2024
    Configuration menu
    Copy the full SHA
    0f99ac1 View commit details
    Browse the repository at this point in the history

  2. Move dataset type registration logic from PreExecInit to PipelineGraph.

    @TallJimbo
    TallJimbo committed Sep 12, 2024
    Configuration menu
    Copy the full SHA
    eba7aac View commit details
    Browse the repository at this point in the history

  3. Implement QG-free version of pre-exec-init's init-output writing. 

    This does not use TaskFactory (which is in ctrl_mpexec anyway) because
it takes a lot more care with storage class conversions and components,
and it saves all writes until after it's done all checks, in order to
reduce the chance that we leave a repository messy when we encounter
an error.
    @TallJimbo
    TallJimbo committed Sep 12, 2024
    Configuration menu
    Copy the full SHA
    80abceb View commit details
    Browse the repository at this point in the history

  4. Use task labels rather than TaskDefs as keys in QG in init maps.

    @TallJimbo
    TallJimbo committed Sep 12, 2024
    Configuration menu
    Copy the full SHA
    5ed0199 View commit details
    Browse the repository at this point in the history

  5. Add methods to get init refs from QG without TaskDef. 

    The old ones that take TaskDef will be deprecated on DM-40442.
    @TallJimbo
    TallJimbo committed Sep 12, 2024
    Configuration menu
    Copy the full SHA
    8cc5a38 View commit details
    Browse the repository at this point in the history

  6. Add convenience method to build QBB for pre-exec-init to QG. 

    This code was moved here from ctrl.mpexec.CmdLineFwk.preExecInitQBB
with minimal changes.
    @TallJimbo
    TallJimbo committed Sep 12, 2024
    Configuration menu
    Copy the full SHA
    13d6283 View commit details
    Browse the repository at this point in the history

  7. Rework QG-based init-output writing as a QG method. 

    This reimplements functionality previously in ctrl_mpexec's PreExecInit
by delegating to the new PipelineGraph.instantiate_tasks instead.

PreExecInit.saveInitOutputs now delegates to this method.
    @TallJimbo
    TallJimbo committed Sep 12, 2024
    Configuration menu
    Copy the full SHA
    ce13fc2 View commit details
    Browse the repository at this point in the history

  8. Implement QG-free version of pre-exec-init's config writing.

    @TallJimbo
    TallJimbo committed Sep 12, 2024
    Configuration menu
    Copy the full SHA
    db9c870 View commit details
    Browse the repository at this point in the history

  9. Rework QG-based config writing as a QG method.

    @TallJimbo
    TallJimbo committed Sep 12, 2024
    Configuration menu
    Copy the full SHA
    6e0feb7 View commit details
    Browse the repository at this point in the history

  10. Implement QG-free version of pre-exec-init's packages writing.

    @TallJimbo
    TallJimbo committed Sep 12, 2024
    Configuration menu
    Copy the full SHA
    ff8a123 View commit details
    Browse the repository at this point in the history

  11. Rework QG-based package versions writing as a QG method.

    @TallJimbo
    TallJimbo committed Sep 12, 2024
    Configuration menu
    Copy the full SHA
    ad90609 View commit details
    Browse the repository at this point in the history

  12. Add convenience methods for initializing output runs.

    @TallJimbo
    TallJimbo committed Sep 12, 2024
    Configuration menu
    Copy the full SHA
    87dcde6 View commit details
    Browse the repository at this point in the history

  13. Minor clarification in code comment.

    @TallJimbo
    TallJimbo committed Sep 12, 2024
    Configuration menu
    Copy the full SHA
    23e361d View commit details
    Browse the repository at this point in the history

  14. Fix docstring typo.

    @TallJimbo
    TallJimbo committed Sep 12, 2024
    Configuration menu
    Copy the full SHA
    ee18f67 View commit details
    Browse the repository at this point in the history

  15. Add tests for init_output_run and related methods.

    @TallJimbo
    TallJimbo committed Sep 12, 2024
    Configuration menu
    Copy the full SHA
    9f0f3b6 View commit details
    Browse the repository at this point in the history

  16. Add changelog entry.

    @TallJimbo
    TallJimbo committed Sep 12, 2024
    Configuration menu
    Copy the full SHA
    9a3e471 View commit details
    Browse the repository at this point in the history