Skip to content

Release 3.3.0
Compare
Choose a tag to compare
@Aryex Aryex released this 27 Jun 21:30
· 56 commits to main since this release
v3.3.0
9dfe322
This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
GPG key ID: 4AEE18F83AFDEB23
Expired
Learn about vigilant mode.

Overview

The connector can now read supported complex types like:

  • Rows type.
  • Complex arrays (any arrays that are not 1D arrays of primitives).

When reading complex types, all of the data will be exported as JSON files instead of Parquet files. The decision is done when complex type columns are detected, however the option json is available which will force the connector to always read using JSON exports.

For map type, the connector can only write to an external table but cannot read them. Vertica suggests using Array[Row(key, value)] instead for use in internal tables.

Requirements:
Reading complex data types require at least Vertica 11.1.1-0.

JSON Readers

For reading JSON files, we delegate to Spark's DataSource V2 to handle the reading. This includes partitioning the exported files and creating readers. On close, all exported files will be deleted using the same behaviours as that of reading Parquet files.

Assets 4
Loading