Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add support for TIMESTAMP and other logical types #99

Open
craxal opened this issue Nov 4, 2023 · 0 comments
Open

Add support for TIMESTAMP and other logical types #99

craxal opened this issue Nov 4, 2023 · 0 comments
Assignees
@saraswatpuneet
Labels
enhancement New feature or request
Milestone
Q2/Q3 2024 Improv...

Comments

@craxal
Copy link

craxal commented Nov 4, 2023 •
edited by wilwade
Loading

According to the Parquet type definitions, ConvertedTypes (such as TIME_MILLIS, TIME_MICROS TIMESTAMP_MILLIS, and TIMESTAMP_MICROS) are deprecated in favor of LogicalTypes (such as TIME and TIMESTAMP). However, I don't see any support for these types (such as in SchemaDefinition, ParquetType, etc.). Please add support for these types.

Useful tool for inspecting output files: https://github.com/manojkarthick/pqrs
@wilwade wilwade added good first issue Good for newcomers enhancement New feature or request labels Nov 8, 2023
@craxal craxal mentioned this issue Nov 8, 2023
Closed
@craxal craxal mentioned this issue Jan 6, 2024
Closed
3 tasks
@craxal craxal changed the title Support for TIME and TIMESTAMP parameterized types Add support for logical types Mar 5, 2024
@craxal craxal changed the title Add support for logical types Add support for TIMESTAMP and other logical types Mar 7, 2024
@wilwade wilwade added help wanted Extra attention is needed and removed help wanted Extra attention is needed labels Jun 11, 2024
@wilwade wilwade added this to the Q2/Q3 2024 Improvements milestone Jun 11, 2024
@wilwade wilwade removed the good first issue Good for newcomers label Jun 11, 2024
@saraswatpuneet saraswatpuneet mentioned this issue Sep 11, 2024
Merged
saraswatpuneet added a commit that referenced this issue Sep 13, 2024
@saraswatpuneet
Logical type : TIME (#143)
6b7bea9 
# Problem

Part of #99 

![image](https://github.com/user-attachments/assets/3526d69a-b89b-4513-b02c-39ff03e91af3)

Support logical types in parquetjs starting with `TIME` support

Solution
========
Implementation following the parquet
[spec](https://github.com/apache/parquet-format/blob/master/LogicalTypes.md#deprecated-time-convertedtype)

## Change summary:

- Added a logical type to support parquet types
- Implemented type conversions for logical type time to primitive type
according to spec
- Implement field, schema and file tests to ensure everything works
together

## Steps to Verify:

1. npm run test
@saraswatpuneet saraswatpuneet mentioned this issue Sep 13, 2024
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@saraswatpuneet saraswatpuneet

Labels
enhancement New feature or request
Projects
None yet
Milestone
Q2/Q3 2024 Improvements
Development

No branches or pull requests
3 participants
@wilwade @craxal @saraswatpuneet