Support for alternate data formats #276
Closed
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Fixing logic where pip installs the updated packages (to avoid the entire image rebuild), so make sure deps are ignored (as this was the slow part).
Adding type GET_SERIALIZED_FORM to get the entire serialized state of a dataset.
Refactor to ensure it takes advantage of connection handling via its async context manager logic.
Add new function to get serialized dataset details to DatasetExternalClient.
Fix issue with not acquiring a session, and wrapping some things in try block to catch and log exceptions.
Fixing bug in response to LIST_FILES query, where reason text was not being assembled entirely correctly.
Adding optional offset and length params to abstract interface definition of get_data, and adding get_file_stat abstract method.
Adding support for querying about dataset items.
Updating to support indicating start and size of data for partial transfers.
Optimizing the reloading of datasets on object store manager startup.
Accounting for changes to get_data parameters and implementing get_file_stat.
Updating service to respond to GET_DATASET_ITEMS queries and requests for data with an offset start (i.e. partials).
Adding initial second view display and manage existing datasets, along with navigation functionality to toggle between "manage" and "create" views.
Sending serialized datasets to HTML template as list/array rather than dict/map.
Implementing layout and initial details-viewing behavior for dataset management GUI view.
this also includes some commented out code that *might* need to be added in the future to support data_requirement dissemination
…bjectLockConfiguration error
…en calibration jobs
Fixing bug in deserialization function where processed (i.e., by this type) optional kwargs were being combined into list of additional (i.e., potentially by subtypes) additional kwargs to be passed to __init__, with both lists being passed instead of just the combined.
Fixing setup for TestSimpleHydrofabricSubset that used the MappedGraphHydrofabric class after it was made abstract, replacing its use with its GeoJsonHydrofabric subtype.
Fixing argument handling for those args related to communicating with evaluation service, including the dest name for args, defaults, and use when creating main service handler class.
Fixing imports, class usage, and code style spacing for several things related to evaluation requests within main request-service handler class.
Adding DataFormat.can_format_fulfill class method to test if alternative data format is compatible, and updating logic of DataDomain.contains method to test and account for this type of compatibility.
Updating dmod.dataservice to depend on updated dmod.core (>=0.5.0).
Updating service manager find_dataset_for_requirement method to account for potentially compatible alternative dataset formats and examine any datasets having such formats if there are no matches with exactly the specified format.
This was referenced Mar 14, 2023
|
Closing; replaced by #299, which isolates the relevant changes. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Adding support for data formats to have satisfactory alternate formats. This enables, e.g., a requirement for a forcing dataset to be fulfilled by a dataset with a format that doesn't strictly match that specified by the requirement.
The branch for this commit is based on the branch in #240, making that a blocker for this. It should not be reviewed until #240 is complete.