Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Added fields to enhance mapping and filtering possibilities #12

Open
wants to merge 1 commit into
base: main
Choose a base branch
from

Conversation

boecklic
Copy link

@boecklic boecklic commented Oct 10, 2024

Besides it's original intention (satellite data), STAC and STAC API proved to be useful for many other data having a georeference of some kind as well. For many of them the core specification of STAC and STAC API is sufficient. In case of weather forecast data, there are limitations however due to the vast amount of data, the frequent update cycles and the nature of the data.

Weather forecast data is typically calculated or recalculated every X hours (e.g. 3h) and results conceptually in a large number of datapackages, each containing the result of a specific selection of parameters among the many degrees of freedom (spatial, temporal, others...). The fields offered by STAC core allow to distinguish spatial and temporal parameters, but not more. Mapping these > 100'000 datapackages to STAC would either mean few items with an extensive list of assets, or a large list of items with one or a few assets each, or combining several parameter combinations in one file and having large files (50GB +). Each of these solutions is not very user-friendly, since selecting a subset of the data is hard to impossible (E.g. the spatial extent is often the same for every datapackage (covering the whole weather model grid) so it's of limited use for e.g. filtering).

We propose to add additional fields to the forecast extension to enhance mapping and filtering possibilities for forecast data. The degrees of freedom the data contains are

  • 2D extent of the model grid
  • datetime when the data was calculated
  • datetime in the future to which the forecast corresponds
  • level (third spatial dimension, mostly just an index)
  • physical parameter (temperature, pressure etc.)
  • perturbation sample

Open questions:

  • We have implemented the query extension (https://github.com/stac-api-extensions/query, https://data.geo.admin.ch/api/stac/static/spec/v0.9/apitransactional.html#tag/STAC/operation/postSearchSTAC) so basically we are able to filter on these attributes as well.
  • forecast:param can be a string or a list or strings. Having just a string would mean each parameter has to be in a separate file, having a list of strings would allow to combine parameters in one file. Although a more practical scenario is probably rather to just have a few parameter groups, e.g. "surface" or "full atmosphere" parameter group which could be implemented with just a string. Having just strings and not lists is also easier for querying/filtering the data, so we propose to just have strings. The allowed values for forecast:param are intentionally unrestricted since these heavily depend on the forecast model configuration.
  • As for perturbation: users are often either interested in either the control run or all of the perturbation runs. For this it would be sufficient to have a forecast:mode attribute with values of either ctrl or perturb. If desired/necessary it could be paired with a forecast:sample_nr with the number of the perturbation run. Since we'll not currently use this it can also be omitted for now and added later if there's a usecase for it.
  • The forecast:level will also not be implemented/used in our usecase, but could be useful for others.

@boecklic boecklic marked this pull request as draft October 10, 2024 12:24
@boecklic boecklic changed the title PB-1060 added fields to inhance mapping and filtering possibilities PB-1060 added fields to enhance mapping and filtering possibilities Oct 10, 2024
@boecklic boecklic marked this pull request as ready for review October 16, 2024 19:23
@m-mohr
Copy link
Contributor

m-mohr commented Oct 16, 2024

Hey @zakjan, @cboettig, @aaronspring,
you were active when we defined the first version of this extension.
I'd love to hear your feedback as I don't really have an understanding of the details.

@cboettig
Copy link
Contributor

cc @rqthomas

@m-mohr m-mohr changed the title PB-1060 added fields to enhance mapping and filtering possibilities Added fields to enhance mapping and filtering possibilities Oct 17, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants