Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

FSS: test dask #505

Open
nikeethr opened this issue Jun 10, 2024 · 0 comments
Open

FSS: test dask #505

nikeethr opened this issue Jun 10, 2024 · 0 comments
Milestone

Comments

@nikeethr
Copy link
Collaborator

nikeethr commented Jun 10, 2024

https://docs.xarray.dev/en/stable/generated/xarray.apply_ufunc.html is used to aggregate the N-dimensional data. The current api for fss_2d and fss_2d_binary accept a similar argument dask # default='forbidden'.

However, this is not tested on dask arrays. Note that the underlying operations are performed on numpy arrays so only "forbidden" or "parallelize" are the only valid options.

Note

  • "forbidden" option will raise an error for dask arrays when it hits the ufunc. To circumvent this we should coerce to core xarray prior to any computations, if the user sets dask = "forbidden" but feeds in a dask array.
  • dask arrays can be coerced to core xarray or numpy using .compute() .load() or equivalent - so setting dask = "forbidden" shouldn't halt the computations, they just cannot utilize dask's scheduling and memory management.

Test cases:

  1. [benchmark] test against chunked large dask arrays > RAM. "forbidden" should most likely crash (though this is non-deterministic), while "parallelize" (dask) should hopefully still work.
  2. [benchmark] test speed of "forbidden" (core-numpy) v.s. "parallelize" (dask) options.
  3. [unit test] test that output arrays are equivilent for both options
  4. [unit test] "allowed" option should raise an error (ufunc doing the aggregation expects a numpy array)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants