-
Notifications
You must be signed in to change notification settings - Fork 116
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Support for Teradata's CLOB data type in row validations #1256
Comments
The TransUnicodeToUTF8 function requires a VARCHAR(32000) or CHAR data type. We can cast CLOB columns to VARCHAR, but there will be data loss since the maximum number of characters that can be stored in a VARCHAR is 64000, and the maximum for CLOB is 2097088000. The cast will truncate the full string in the CLOB column. |
Hi, CLOB is a variable length character data type in Teradata - which can hold strings longer 64000 characters. Being a character data type, CLOB can store both Latin1 and Unicode character sets. Strings in BigQuery (and other databases) are stored in UTF8 encoding. To compare CLOB columns in Teradata and BigQuery STRING, we will need to convert from Latin1 / Unicode character set to UTF8. The only function to convert Latin1 / Unicode to UTF8 in Teradata is Here are the options to compare CLOB columns in Teradata with BigQuery:
Sundar Mudupalli |
Decreasing priority, issue can be tackled in the future since there are workarounds available, including the ones mentioned by Neha and Sundar previously and the following ones:
|
Context: row validations
Source: Teradata
Target: BigQuery
Details:
Table in Teradata has a CLOB column which maps to a String column in BigQuery. When attempting row validations on this table, we are seeing the following error:
[Teradata Database] [Errpr 9881] Function 'TransUnicodeToUTF8' called with an invalid number or type of parameters
TransUnicodeToUTF8 cannot be applied to non-VARCHAR data types. Casting to VARCHAR normally happens for columns prior to this step, but this column is already a "string" field in ibis so the casting is not happening.
Adding support for the CLOB data type specifically would ensure the casting to VARCHAR occurs and this error is bypassed.
The text was updated successfully, but these errors were encountered: