HDF5 pre-built filter shared libraries..

[DennisHeimbigner]

I see that HDF5 is now providing pre-built filter plugin shared libraries. See here:

http://portal.hdfgroup.org/display/support/HDF5+Filter+Plugins

How if at all should we consider making use of these?

[czender]

A few thoughts:
First, at least two of the codecs (ZFP and BitShuffle) offer desirable characteristics not currently easily (i.e., without tedious build/installation procedures) available to netCDF users. To maximize dataset interoperability, it is important to use only codecs that have well-defined, stable, APIs so downstream users can uncompress the data for years/decades. Until quite recently (Aug. 1) the ZFP codec did not guarantee the stable API needed for back compatibility. They just released 1.0.0 so I think ZFP may be "ready for primetime. I'm not sure about the the stability of the BitShuffle codec.
Second, I think user feedback from about the burgeoning filter capabilities in the netCDF 4.9.X series should be heeded. That feedback rate will certainly increase once 4.9.1 is released since many package systems seem to be delaying their upgrades from 4.8.1 until 4.9.1 is available to fix the packaging issues posed by 4.9.0. Hence I think it's too early to make a fully informed decision on whether/how to use the HDF-provided plugins.

[edwardhartnett]

Users are already exploring some of the advanced capabilities offered by filters.

I was surprised (indeed: shocked) by the blithe way that users in our EGU session described their workflows, involving HDF5 filters that we have not yet touched upon in our discussions. (Frequently LZF.)

No concern for portability or interoperability were expressed by anyone! ;-)

I suggest that, as with zstandard, we move decisively to add support for filters that are well-maintained and provide an important feature for netCDF users, and can be supported in netcdf-Java. The HDF5 team have already done a lot of the work by selecting and supporting these filters: BZIP2, JPEG, LZF, BLOSC, MAFISC, LZ4, Bitshuffle, and ZFP.

By supporting a curated set of filters (and ideally we would support the full list of HDF5 pre-built filters), we can give users the full functionality available in the world of filters, and avoid a "Wild West" scenario, where many different filters, or versions of filters, become embedded in important Earth science workflows, causing downstream trouble for users (and us). Or where filters make data impossible to read with native Java code.

By well-supported, I mean the same level of support we give to zlib and other currently-supported filters, like the recently-added zstandard support. This generally consists of two new functions, for example nc_def_var_zstandard()/nc_inq_var_zstandard()) which call the filter code appropriately. These functions then can appear in the documentation and users can use them without really caring that they are filters instead of built-in features. This support can be provided in the netcdf-c/netcdf-fortran libraries, or in the CCR C/Fortran libraries.

This is a little extra work, but we can rejoice in the fact that we are leveraging the work of Dennis, the HDF5 team, the filter code writer, and (usually) the library used by the filter, to get advanced features into user hands with remarkably little effort. ;-)

LZF and BLOSC are next on the list for CCR, as far as I am concerned. What would be useful would be understanding how they will impact netcdf-java. LZF is a compression library, BLOSC is something else, and itself uses different compression libraries. But BLOSC is about HPC I/O performance - I'm not sure whether or how the resulting data files need to care about it. @czender knows more...

Our recent efforts have really improved the compression situation, and started to catch us up to current state of the art, but there is much more to be done...

[dopplershift]

I'm not going to weigh in on whether this is a good idea or not (though I appreciate that netCDF-java is listed among the considerations), but just want to request that we stabilize the last set of complexity added before embarking on increasing it further (e.g. #2428 #2429 #2484 conda-forge/libnetcdf-feedstock#125)

[edwardhartnett]

The complexity is already here - at the EGU we heard speakers from multiple large data producing organizations describe their use of different filters.

The question now is: how do we best manage that complexity?

The wild west scenario is not locked in - it will only come to pass if we take no action, or move too slowly. Users have urgent needs - they won't wait.

[DennisHeimbigner]

At this point I know of at least three independent sources of filter implementations:

1. netcdf-c/plugins
2. ccr
3. HDF5 pre-built filters

• (1) is not going away completely because it is the only place where Zarr filter support exists.
• (2) I do not know Charlie's and Ed's plans for this going forward.
• (3) is unlikely to go away

So I have no idea where this is all headed, but Ed's wild west scenario is looking likely for the foreseeable future.

[edwardhartnett]

CCR is the same as HDF5. I just took copies.

@DennisHeimbigner does the netcdf-c/plugins have a different filter implementation for any of the HDF5 filters? Or just duplicates (as in the case of bzip2)?

For the HDF5 filters BZIP2, JPEG, LZF, BLOSC, MAFISC, LZ4, Bitshuffle, and ZFP can we agree that HDF5 is the canonical source of filter implementations?

[DennisHeimbigner]

Yes, some of the wrappers differ, but AFAIK the underlying filter library (e.g. libzstd)
is the same.

For the HDF5 filters BZIP2, JPEG, LZF, BLOSC, MAFISC, LZ4, Bitshuffle, and ZFP can we agree that HDF5 is the canonical source of filter implementations?

That seems reasonable. I must confess that I am surprised that zstd is not in the HDF5 prebuilt
filters.

[DennisHeimbigner]

It occurs to me that if we rely on the HDF5 prebuilts, then I will need some
way to coordinate the install directory with the remaining filters in the plugins directory.

[edwardhartnett]

I do not think we need to rely on the HDF5 prebuilds, just that we can accept the source code that built them as the canonical version. In other words, I think we can accept HDF5 as the arbiter of filter implementations for these filters.

But pre-builts does not usually work in HPC settings. Either because the users want special compilers and flags, or because they have security or other rules about not running software that they don't have the source code for.

So most users of CCR will still build the filters from source.

[DennisHeimbigner]

I am constructing a PR to add support for programmatic plugin path manipulation.
Here is the relevant discussion. Please provide comments.

==================================================
Extend the netcdf API to support programmatic changes to the plugin search path

re: #2753

This is a draft PR that needs community feedback before it gets committed.

As suggested by Ed Hartnett, This PR extends the netcdf.h API to support programmatic control over the search path used to locate plugins.

Below, I provide a discussion about the decisions I made in designing the API. I am open to suggestions for alternate designs.

The API proposed in this PR looks like this (from netcdf-c/include/netcdf_filter.h).

1. Functions to read/write the plugin path sequence:

int nc_plugin_path_getall(size_t* npathsp, char** pathlist);
int nc_plugin_path_getith(size_t index, char** entryp);
int nc_plugin_path_load(const char* paths);
int nc_plugin_path_append(const char* path);
int nc_plugin_path_prepend(const char* path);
int nc_plugin_path_remove(const char* dir);

2. Functions to propagate the plugin path to dispatchers

int nc_plugin_path_sync(int formatx);

3. Initialize/finalize function

int nc_plugin_path_initialize(void);
int nc_plugin_path_finalize(void);

As a rule, the initialize and finalize functions do not need to be called because they are called as part of nc_initialize()/nc_finalize(). The sync function pushes the current global plugin list state down to one or more of the dispatchers that use plugin paths: libhdf5 and libnczarr currently.

==================================================

There were several constraints that I had to consider in designing the API.

1. It should be compatible with the HDF5 capabilities AND limitations.
2. It must be usable with multiple dispatchers: HDF5 and NCZarr currently.
3. It must be possible to set up the sequence of plugin paths before any other library actions (but note below).

We also have to consider this constraint as desirable but not completely necessary:

4. The plugin path sequence may be different for different dispatchers.

Ideally, it would be desirable to stick the API into the existing struct NC_Dispatch structure so every dispatcher could handle it as desired.

Unfortunately, this approach requires that the user already has a valid netcdf file descriptor -- an ncid -- so as to use the proper dispatch table. This ostensibly violates constraint 3 above.

But for the sake of argument, suppose we violate that constraint and require the user to open a file in order to manipulate the plugin path sequence. This would generally require the following set of netcdf API calls:

1. nc_open (or nc_create) a netcdf-c dataset
2. manipulate the plugin path sequence using the ncid from step 1.
3. do other netcdf-c API operations using the ncid from step 1.
4. nc_close the step 1 ncid.

It is not clear to me that any plugin path changes in step 2 would apply to the step 3 operations. It may be that they will only apply to actions after step 4.

Note also that this approach supports constraint 4 above.

I personally dislike this solution.

The alternative I have chosen keeps constraints 1-3 and partially violates constraint 4. This solution is embodied in the API described at the beginning of this document.

The basic idea is that the netcdf library keeps a global plugin path sequence that is distinct from that of any of the dispatchers. Before any files are opened, it is possible to set up an arbitary plugin path sequence using the proposed API.

The critical thing to note is the nc_plugin_path_sync() API function. This function pushes the current plugin path sequence down to either all dispatchers that use plugin paths, or to one specific dispatcher.

So the typical usage pattern is as follows:

1. Use the read/write functions to create the desired plugin path sequence.
2. Use the nc_plugin_path_sync(0) function to push the plugin path sequence to all the dispatchers that use plugin paths.
3. Continue with normal netcdf-c API operations.

If it is desired to set the plugin path sequence for a specific dispatcher, then change step 2 to call nc_plugin_path_sync(n) where n specifies the dispatcher to change. The legal values for n are: NC_FORMATX_NC_HDF5 or NC_FORMATX_NCZARR or 0 (zero). The zero case pushes the current plugin path sequence to all dispatchers that use plugin paths.

[Alexander-Barth]

The proposal sounds very useful. I am wondering if you have considered to make the formatx parameter (with 0 meaning all dispatchers as you proposed) as part of the path manipulation functions:

int nc_plugin_path_append(int formatx,const char* path);

nc_plugin_path_append would apply the changes immediately.

I guess that nc_plugin_path_load sets all paths in one call. If a user need to make a change to the path in an atomic way, then all call to nc_plugin_path_load (possibly preceded by nc_plugin_path_getall) would be enough.

But this is just a minor suggestion. I am quite certain that the proposed API will work for the julia community.

Can you maybe clarify if nc_plugin_path_sync clears all changes once applied?

nc_plugin_path_append("foo");
nc_plugin_path_sync(NC_FORMATX_NC_HDF5);
nc_plugin_path_append("bar");
nc_plugin_path_sync(NC_FORMATX_NCZARR); // path of NcZarr is just "bar" or "foo:bar" ?

[DennisHeimbigner]

Adding the formatx argument to all of the manipulation functions is a reasonable alternative.
It would have a minor drawbacks. Namely, currently each dispatch library only has to implement initialize, finalize, and sync functions. Your change would require implementing the whole API for each dispatch library, but that is not a significant problem. On the plus side, I would no longer need the sync function.

[DennisHeimbigner]

Also WRT:

Can you maybe clarify if nc_plugin_path_sync clears all changes once applied?

It does not clear all changes, so one would have to add a call to nc_plugin_path_load(NULL) to clear the plugin path.

[edwardhartnett]

I am happy to see this being addressed, thanks @DennisHeimbigner .

[DennisHeimbigner]

It has occurred to me that the simplest possible interface is one that has two operations:

1. read the whole current path list for a given dispatcher (specified by NC_FORMATX_XXXX)
2. replace the whole current path list for a given dispatcher.

This puts the burden on the client to construct the path list they want, but also gives them
the freedom to build it any way they want.

The only reason I can see for using a more complex interface is if the dispatcher need to figure
out what changed. So for example, using append might allow the dispatcher to focus only on the
specific new directory added to the path list in order to update is set of loaded plugins.
Not sure that e.g. HDF5 does anything like this.

[edwardhartnett]

I agree! Simpler is better.

[DennisHeimbigner]

Further reflection: the simple api might be good in a multithreaded environment.