Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

add support for /vsi datasets in GDAL plugin #44

Merged
merged 3 commits into from
Nov 27, 2023
Merged

Conversation

gillins
Copy link
Member

@gillins gillins commented Nov 24, 2023

@t-hackwood @petescarth this adds the mysterious hdf5vfl.h from GDAL (which weirdly I'm listed as one of the authors, must have been a long time ago). Conveniently this file was already under the same license as libkea.

Anyway, with this addition (plus a few tweaks) when building KEA as a GDAL plugin (ie against Ubuntu GDAL in a container) you now get access to /vsi filesystems. Which might be handy for those of us with large KEA files on S3....

@gillins gillins merged commit adbb33e into ubarsc:master Nov 27, 2023
2 checks passed
@gillins gillins deleted the vsi branch November 27, 2023 04:43
@gillins
Copy link
Member Author

gillins commented Dec 6, 2023

Interesting: https://www.youtube.com/watch?v=bDH59YTXpkc. Might have to look into Paged Aggregation.
@petebunting should we start tweaking some of the default chunk sizes etc to make the files run better on S3?

@petebunting
Copy link
Collaborator

@gillins, yeah, it looks like it would be a good idea to tweak the chunk sizes. That should also be for the RAT as well as the image data?

From a quick scan through the RFC for page buffering, it does not sound like it would be a negative thing when using KEA files locally to have paging enabled and would also help out people using them on a HPC system was well...

https://docs.hdfgroup.org/archive/support/HDF5/docNewFeatures/NewFeaturesPageBuffering.html

@gillins
Copy link
Member Author

gillins commented Dec 6, 2023

@petebunting yeah I guess RAT also. I didn't quite understand that bit about putting all the metadata in one chunk, but sounds like might be worth looking at.
I'm hoping to put out a 1.5.3 shortly and maybe we can look at this for 1.6.0?

@gillins
Copy link
Member Author

gillins commented Dec 21, 2023

@petebunting have bumped the version to 1.6.0 in #50 so we can fiddle with these low level things...

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants