[CORE] Fix incorrect precision of decimal literal

[jiangjiangtian]

For SQL as follows:

select (col0 / (col1 + 0.00000001)) from table;

In this case, col0 and col1 are 0.
The result may be NULL. The reason is that the result of Decimal(0.00000001).toString() is "1E-8", which will make the new precision be 4.
Therefore, we use toPlainString here to prevent scientific notation.

[jiangjiangtian]

@kecookier

[github-actions]

Run Gluten Clickhouse CI

[github-actions]

Run Gluten Clickhouse CI

[jiangjiangtian]

@rui-mo Can you review this PR?
I hava a question: why do we have this rescale in gluten? I can't find the same logic in Spark. I have read the comment and I still can't have a good understanding of it.
Thanks!

[rui-mo]

Hi @jiangjiangtian, this adjustment, as I recall, is for the situation where one performs an arithmetic operation between a decimal and a number. In this instance, the number is converted to decimal and its precision and scale acquired from Spark are (38, 18), which are inconsistent with the real values. E.g., in the case you mentioned, Decimal(0.00000001) should have a precision and scale of (8, 8) instead of (38, 18). In order to produce accurate results, we need additional logic to extract the accurate precision and scale that native computing requires.

Perhaps you could help confirm if it is the case for the example you provided. Thanks.

[rui-mo]

Thanks for the fix. Would you like to add the buggy case as a unit test?

[jiangjiangtian]

Hi @jiangjiangtian, this adjustment, as I recall, is for the situation where one performs an arithmetic operation between a decimal and a number. In this instance, the number is converted to decimal and its precision and scale acquired from Spark are (38, 18), which are inconsistent with the real values. E.g., in the case you mentioned, Decimal(0.00000001) should have a precision and scale of (8, 8) instead of (38, 18). In order to produce accurate results, we need additional logic to extract the accurate precision and scale that native computing requires.

Perhaps you could help confirm if it is the case for the example you provided. Thanks.

@rui-mo Thanks! It seems that Spark doesn't have the logic. I don't know why Spark doesn't need this logic.

In my case, the type of the literal 0.00000001 is Decimal(19, 8). After the adjustment, the type is still Decimal(19, 8). Because the string representation of the decimal contains . and the decimal is not a valid long number. Perhaps we should not return the original precision and scale in the end of the function. https://github.com/apache/incubator-gluten/blob/main/gluten-core/src/main/scala/org/apache/gluten/utils/DecimalArithmeticUtil.scala#L110

[github-actions]

Run Gluten Clickhouse CI

[jiangjiangtian]

Thanks for the fix. Would you like to add the buggy case as a unit test?

I add a unit test. Is there anything that I need to add or edit? Thanks.

[rui-mo]

Because the string representation of the decimal contains . and the decimal is not a valid long number.

Thanks for reminding me of this. I just remembered that this adjustment is typically for the arithmetic operation between a decimal and an integer/bigint. In your case, the literal is double, so to return the original precision and scale should be fine.

[rui-mo]

There is a test failure as below:

Fix wrong rescale *** FAILED ***
  org.apache.spark.sql.AnalysisException: unknown requires that the data to be inserted have the same number of columns as the target table: target table has 3 column(s) but the inserted data has 2 column(s), including 0 partition column(s) having constant value(s).

[jiangjiangtian]

Fixed.

[github-actions]

Run Gluten Clickhouse CI

[github-actions]

Run Gluten Clickhouse CI