You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
For CSVs (and later when we support them for JSONs), if all that is being changed is compression, we can do a shortcut by compressing or decompressing the files directly instead of routing through a pyarrow dataset.
Doing this ends up being slightly tricky because reading lines and gzipping in python isn't particularly fast, so we should try and find something that doesn't do that (or we could rely on system utilities if they exist...)
The text was updated successfully, but these errors were encountered:
For CSVs (and later when we support them for JSONs), if all that is being changed is compression, we can do a shortcut by compressing or decompressing the files directly instead of routing through a pyarrow dataset.
Doing this ends up being slightly tricky because reading lines and gzipping in python isn't particularly fast, so we should try and find something that doesn't do that (or we could rely on system utilities if they exist...)
The text was updated successfully, but these errors were encountered: