-
Notifications
You must be signed in to change notification settings - Fork 13
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
DM-45386: Fix serialization of datetime64 to parquet via numpy and astropy. #1041
Conversation
Codecov ReportAll modified and coverable lines are covered by tests ✅
Additional details and impacted files@@ Coverage Diff @@
## main #1041 +/- ##
=======================================
Coverage 89.37% 89.37%
=======================================
Files 359 359
Lines 45622 45630 +8
Branches 9348 9349 +1
=======================================
+ Hits 40775 40783 +8
Misses 3521 3521
Partials 1326 1326 ☔ View full report in Codecov by Sentry. |
ad78a93
to
6bbd594
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This looks good to me. Just one suggestion:
This bug I found it first using the datetime[us]
so adding it to the test cases might be good.
tests/test_parquet.py
Outdated
@@ -138,6 +139,7 @@ def _makeSimpleNumpyTable(include_multidim=False, include_bigendian=False): | |||
("f", "i8"), | |||
("strcol", "U10"), | |||
("bytecol", "a10"), | |||
("dt", "datetime64[ns]"), |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Can you add to the test the us
as well? Is it relevant?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I will add that!
This conversion has to be done by hand, but once the correct arrow type is specified the rest of the code handles the datatype and round-trips.
The datetime column does not work correctly with the direct astropy serialization.
6bbd594
to
e246ec6
Compare
Checklist
doc/changes
configs/old_dimensions