using the HDF5 file space strategy property

[itcarroll]

Is there any way to create a file that does not use the HDF5 default file space strategy? In the HDF5 library, you would use the set_file_space_strategy to adjust file creation properties passed to H5Fcreate.

[WardF]

There is not currently a pass-through to this function in the HDF5 API, but that might be in part because you are (as far as I know) the first, or one of the first, who has asked for it. At the very least, it's not a commonly requested feature. It's not out of the question (at first reading) that we might be able to add such a method, but it would take some consideration to make sure no unintended consequences are being overlooked.

[itcarroll]

Good to know. For your situational awareness, I asked due to indications from the HDF5 group that page aggregation can improve performance for reads from S3 when used with the page buffer. See the video on Cloud-optimized HDF5 Files.

[DennisHeimbigner]

I am pretty sure this is the first time I have seen this request.
As always we have a problem that there is an arbitrarily large number of such potential feature requests.
I think we need to start thinking about some mechanism for handling this in a general way.
One possibility is this: We define hook events in various places in the operation of the netcdf library: e.g. open, create, close, etc. We then call a user defined function at the end of each event telling it the event and selected implementation-specific info (e.g. HDF5 hids).

[edwardhartnett]

Bear in mind that curating the complexity of HDF5 is a key netCDF feature. Exposing all of HDF5's numerous settings is probably not helpful to the scientists. But this could be a good way to enable the use of HDF5 features without explicitly exposing them.

Perhaps this would be implemented similarly to the user-defined format, in which the user can pass in a struct of function pointers to hijack the entire netCDF back end. (Which is not accessible in Fortran, BTW). So a function that accepts a struct with function pointers would seem straightforward and only require one or two functions.

I think only advanced users would use this feature, but I can easily imagine the PIO library using this to good effect. The PIO library is a C core with Fortran wrappers, so Fortran users would benefit without having to use the feature directly.

[edwardhartnett]

Is this a feature only for S3 files? Or does it do something for other netCDF files as well? I read the docs but it wasn't clear to me what this would be used for - what is the benefit?

[DennisHeimbigner]

Nice use of the word "curate" :-)

[betolink]

This would be very useful indeed, the main benefit is that it will speed up access to files hosted in the cloud (AWS, Google, Azure) so not only S3. Related: nsidc/earthaccess#251

[edwardhartnett]

OK, well that sounds useful. But what I meant by s3 was actually zarr. In other words, this is a feature to speed up zarr access and will not affect any data in netCDF-4. Is that correct?

[betolink]

Correct, changing the file space strategy should not break or affect anything in the netCDF-4 or underlying HDF5 storage. Just to clarify, if data producers could access and modify these lower level settings in the HDF5 API they wouldn't need to reprocess the files later.

[edwardhartnett]

@DennisHeimbigner isn't there already some dot-file for zarr settings? Could this find it's way into that?

[ethanrd]

Sorry, now I'm confused. The Cloud-optimized HDF-5 is about the netCDF-4/HDF5 format not about Zarr. My understanding was that by using the HDF5 file space strategy it would affect the layout of the HDF5 files and make them more efficient to read when stored on S3 or other object stores.
...
Ah, I think I understand the point @betolink is making. (Am I getting this right?) The netCDF-C library doesn't need any changes to be able to read HDF5/nc4 files that are written with different space strategies as the current HDF5 library looks at an existing data file to decide which strategy was used.

[itcarroll]

I believe you are understanding it right, at the basic/defaults level. There are, however, additional options that a reader could implement to use a non-default page buffer size when reading a netCDF-4/HDF5 file that uses the page file space strategy. For instance, h5py allows a page_buf_size value that is passed to the hdf5 library.