Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Feature]: Support parallelization for export #112

Open
bendichter opened this issue Jul 31, 2023 · 0 comments
Open

[Feature]: Support parallelization for export #112

bendichter opened this issue Jul 31, 2023 · 0 comments
Assignees
@mavaylon1
Labels
category: enhancement improvements of code or code behavior priority: low alternative solution already working and/or relevant to only specific user(s) topic: storage issues related to storage schema and formats
Milestone
Future

Comments

@bendichter
Copy link
Contributor

bendichter commented Jul 31, 2023 •
edited
Loading

A nice-to-have feature would be to enhance ZarrIO.export (

hdmf-zarr/src/hdmf_zarr/backend.py

Line 231 in d47fc82

def export(self, **kwargs):
) to use parallel write when converting from HDF5 to Zarr. One way to achieve this would be to automatically wrap datasets in DataChunkIterator in ZarrIO.write_dataset. This is probably beyond this PR and can be addressed later in a separate issue. Just wanted to mention it.

Originally posted by @oruebel in #111 (comment)
@bendichter bendichter changed the title Support parallelization for export [Feature]: Support parallelization for export Jul 31, 2023
@mavaylon1 mavaylon1 added category: enhancement improvements of code or code behavior priority: low alternative solution already working and/or relevant to only specific user(s) topic: storage issues related to storage schema and formats labels Apr 16, 2024
@mavaylon1 mavaylon1 self-assigned this Apr 16, 2024
@mavaylon1 mavaylon1 added this to the Future milestone Apr 16, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@mavaylon1 mavaylon1

Labels
category: enhancement improvements of code or code behavior priority: low alternative solution already working and/or relevant to only specific user(s) topic: storage issues related to storage schema and formats
Projects
None yet
Milestone
Future
Development

No branches or pull requests
2 participants
@bendichter @mavaylon1