Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

savecube does not take backend kwargs anymore #215

Closed
felixcremer opened this issue Jan 13, 2023 · 5 comments
Closed

savecube does not take backend kwargs anymore #215

felixcremer opened this issue Jan 13, 2023 · 5 comments

Comments

@felixcremer
Copy link
Member

Before #150 it was possible to give backend kwargs to example set the compression.
This seems to be not possible anymore.

We would need to find a good way to also provide documentation about the available kwargs for different backends.

@felixcremer
Copy link
Member Author

This is fixed by #267. We still don't have a good way to expose the backend documentation.

@Balinus
Copy link
Contributor

Balinus commented Jan 16, 2024

Hello!

Is there an example on how to set compression? My output gives a ~ 35-40GB dataset and I'd like to compress it as much as possible. Thanks!

@felixcremer
Copy link
Member Author

No unfortunately there isn't and we should add one to the docs.
It depends on the backend how you have to specify the compression.
When you are using NetCDF you can use the compress keyword to the savecube function where you can set the compression level between 0 and 9 as Integers. Thereby 0 is no compression and 9 is the highest compression which would then take the longest for writing the data.

For Zarr one can set a compressor keyword. The compressor can be set as a Zarr.BloscCompressor(clevel=n) where n is the compression level with the same meaning as in the NetCDF case.
See the docs here: https://juliaio.github.io/Zarr.jl/latest/reference/#Compressors

Maybe we could streamline this and have an YAXArrays internal compress keyword which would then be converted to the BloscCompressor with the correct clevel.

@Balinus
Copy link
Contributor

Balinus commented Jan 17, 2024

Thanks! I will try it and add it to the documentation

@Balinus
Copy link
Contributor

Balinus commented Sep 13, 2024

#438

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants