Decimal Support for Binary Precision

[wilwade]

Currently this library only supports DECIMAL reading and writing when the precision is <= 18

To annotate the Parquet Spec: https://github.com/apache/parquet-format/blob/master/LogicalTypes.md#decimal

DECIMAL can be used to annotate the following types:

• int32: for 1 <= precision <= 9
• int64: for 1 <= precision <= 18; precision < 10 will produce a
  warning
• fixed_len_byte_array: precision is limited by the array size. Length n
  can store <= floor(log_10(2^(8*n - 1) - 1)) base-10 digits
• binary: precision is not limited, but is required. The minimum number of
  bytes to store the unscaled value should be used.

Test Files:

• additional support for decimal type #81 (comment)
• See Decimal test files here: https://github.com/apache/parquet-testing/tree/master/data

Related Issues:

• Thanks to @nirmal82 in additional support for decimal type #81 bringing this up
• Current Decimal Read Support Added: Add ability to read decimal columns #79
• Current Decimal Write Support Added: Decimal Writer Support #90
• Byte array Support added: Add support to byte array decimal fields #97

[YECHUNAN]

I made a PR attempting to add rudimentary support for Decimal fields that are represented by byte arrays, which may have precision over 18.

[craxal]

I suspect that the earlier pull request has caused some regression issues related to DECIMAL values. Some folks are reporting the following error:

missing option: typeLength (required for FIXED_LEN_BYTE_ARRAY)

From what I can gather, this occurs even if there are no FIXED_LEN_BYTE_ARRAY backed DECIMAL values (only INT64 in one case).

[wilwade]

@craxal the fix from @JasonYeMSFT released in v1.6.1 (just this morning) should fix it.

[craxal]

@wilwade Ah, yes, I think it does. Just tested it myself. Sorry, I thought the pull request had already been released.

[craxal]

Is there any status update on this item? We're hoping we can start parsing fixed length array decimals in the near future.