Moving data between Data Product versions

[murdo-moj]

User story

As a data consumer
I want as much historical data as possible in my data product as it gets versioned
So I can produce reports or analyse trends

Scenario

A data product maintainer has deleted a column in a existing table schema in a data product. This has created a new major version of the data product. This new data product will initially have no associated data.

Major versions of a data product have different tables so that breaking changes do not silently impact the consumers of that data product. Data consumers are expected to update their pipelines to consume the latest data product and update their processes accordingly.

Suggested solution

We want to copy over the data associated with the old table to the new table. If a column is added, new columns can be null in the new table. This could be achieved by select on the intersection of the two schemas in the old table, then insert into the new table.

Talking points

• Would you expect this behaviour when a new data product version is created?
• Should the data transfer be optional?
• There's potentially some overlap with using Iceberg here which would change some of the assumptions here, namely if we need separate tables to version data product tables.

[murdo-moj]

We agreed the suggested solution is what we'll do, on 24 Oct