Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Create a separate dataflow database migration task application #5956

Open
corneil opened this issue Oct 1, 2024 · 1 comment
Open

Create a separate dataflow database migration task application #5956

corneil opened this issue Oct 1, 2024 · 1 comment
Labels
area/db Belongs to database data-migration

Comments

@corneil
Copy link
Contributor

corneil commented Oct 1, 2024

Problem description:
When a release contains a large set of changes that may run longer than 1 minute Kubernetes may kill the application and relaunch.

Solution description:
The solution will be to create a task application that can apply the migrations and also output sql where possible for migrations that need to be applied. In the cases of migration which perform logic that cannot be output as sql only the output will stop and a message indicating the need to run the task afterwards.

We can also have the task list all migrations need by name and the use specify the target migration to apply and stop which will result in executing all the outstanding migrations including the target.

This task application can be packaged as a container that can be launched as a job before a deployment of spring cloud data flow and skipper.

We can include the migrations of both dataflow-server and skipper-server in the application or keep them as separate task applications.

@corneil corneil added area/db Belongs to database data-migration labels Oct 1, 2024
@cppwfs
Copy link
Contributor

cppwfs commented Oct 1, 2024

We also need add instructions in the migration guide that refer to potential of a longer than normal startup because of schema migration. This would include the use of the task or a set of migration scripts for the supported databases.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/db Belongs to database data-migration
Projects
None yet
Development

No branches or pull requests

2 participants