Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

DM-41605: Command-line aggregator for pipetask report #437

Draft
wants to merge 22 commits into
base: main
Choose a base branch
from
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
22 commits
Select commit Hold shift + click to select a range
125d947
Add stub for quantum graph provenance
eigerx Nov 14, 2023
99497ab
Add new graphs to combined provenance graph
eigerx Nov 14, 2023
c83a242
Walk multiple quantum graphs, annotating the status of specific datas…
eigerx Nov 15, 2023
32d48ba
Fix successful/succeeded bug
eigerx Feb 3, 2024
fed8576
Start working on dataset summaries.
TallJimbo Feb 20, 2024
428db30
Formalize overall status with match statements
eigerx Mar 9, 2024
b0a60db
Summarize info from graph in a dictionary
eigerx Mar 12, 2024
23568d1
Add diagnostic messages
eigerx Mar 12, 2024
7e2e044
Output contents of QuantumProvenanceGraph in a summary dictionary
eigerx Mar 21, 2024
994e675
Pydantify Quantum and datasetType summaries
eigerx Apr 30, 2024
0ce2020
Resolve bugs with datasets
eigerx May 2, 2024
7e3fefd
Add documentation for DM-41711
eigerx May 8, 2024
201f5e2
Add to task and dataset summaries
eigerx May 17, 2024
e3ec5a6
Fix bug in which blocked quanta were not being saved
eigerx May 17, 2024
5fa6590
Raise RunTimeError if resolve_duplicates is used improperly
eigerx May 17, 2024
83da95f
Prevent published logs from unsuccessful quanta being marked as cursed
eigerx May 17, 2024
b5824d0
Add tests for QuantumProvenanceGraph
eigerx May 22, 2024
c59ae38
Add flag for what to do with failed log datasets
eigerx May 23, 2024
7ced2aa
Use Pydantic models for tests
eigerx Aug 3, 2024
51db95e
Add aggregate function for quantum provenance graph
eigerx Jul 18, 2024
fa37f9c
Add tests for QuantumProvenanceGraph.aggregate
eigerx Aug 14, 2024
96c40bd
Add documentation for DM-41605
eigerx Aug 16, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 7 additions & 0 deletions doc/changes/DM-41605.feature.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
Add functionality to aggregate multiple `QuantumProvenanceGraph.Summary`
objects into one `Summary` for a wholistic report.

While the `QuantumProvenanceGraph` was designed to resolve processing over
dataquery-identified groups, `QuantumProvenanceGraph.aggregate` is designed to
combine multiple group-level reports into one which totals the successes,
issues and failures over the same section of pipeline.
11 changes: 11 additions & 0 deletions doc/changes/DM-41711.feature.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
Create a QuantumProvenanceGraph, which details the status of every quantum
and dataset over multiple attempts at executing graphs, noting when quanta
have been recovered.

Step through all the quantum graphs associated with certain tasks or
processing steps. For each graph/attempt, the status of each quantum and
dataset is recorded in `QuantumProvenanceGraph.add_new_graph` and outcomes
of quanta over multiple runs are resolved in
`QuantumProvenanceGraph.resolve_duplicates`. At the end of this process,
we can combine all attempts into a summary. This serves to answer the
question "What happened to this data ID?" in a wholistic sense.
Loading
Loading