Skip to content
This repository has been archived by the owner on Sep 30, 2024. It is now read-only.

Partial data ingest #354

Open
esheehan-gsl opened this issue Jun 21, 2023 · 0 comments
Open

Partial data ingest #354

esheehan-gsl opened this issue Jun 21, 2023 · 0 comments
Assignees
Labels
bug Something isn't working

Comments

@esheehan-gsl
Copy link
Contributor

When we have an error in the save step of our ingest pipeline—as we did with #353—we can end up with partially ingested data. For example, the Parquet write happens after the Zarr is written. If an error happens there, we end up with the data written to a new Zarr group, but not available in the database (or in the Parquet files, of course). So the data is not presented in the application.

We should set up our error handling so that if any part of the ingest pipeline fails, we clear out all of the data and log a useful message.

@esheehan-gsl esheehan-gsl added the bug Something isn't working label Jun 21, 2023
@esheehan-gsl esheehan-gsl added this to the Cycle 2023.4 milestone Oct 3, 2023
@esheehan-gsl esheehan-gsl self-assigned this Nov 15, 2023
@esheehan-gsl esheehan-gsl removed this from the Cycle 2023.5 milestone Nov 15, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

1 participant