Deployment fails with External RDS database

[rengler-noodle]

Attempting to use external postgres instance on AWS RDS. have tried this with completely new database a couple times. on first start, the bootloader is able to create many tables and do some migrations but fails on 0.40.12.001.

postgres version is 11.19. Is that too old?

Seems there is a related discussion from the old board here: https://discuss.airbyte.io/t/upgrade-migration-failing/4667

airbyte-bootloader                | 2023-09-19 22:33:41 ERROR o.f.c.i.l.s.Slf4jLog(error):57 - Migration of schema "public" to version "0.40.12.001 - AddWebhookOperationColumns" failed! Changes successfully rolled back.
airbyte-bootloader                | 2023-09-19 22:33:41 ERROR i.a.b.Application(main):25 - Unable to bootstrap Airbyte environment.
airbyte-bootloader                | org.flywaydb.core.internal.command.DbMigrate$FlywayMigrateException: Migration failed !
airbyte-bootloader                | 	at org.flywaydb.core.internal.command.DbMigrate.doMigrateGroup(DbMigrate.java:378) ~[flyway-core-7.14.0.jar:?]
airbyte-bootloader                | 	at org.flywaydb.core.internal.command.DbMigrate.lambda$applyMigrations$1(DbMigrate.java:271) ~[flyway-core-7.14.0.jar:?]
airbyte-bootloader                | 	at org.flywaydb.core.internal.jdbc.TransactionalExecutionTemplate.execute(TransactionalExecutionTemplate.java:66) ~[flyway-core-7.14.0.jar:?]
airbyte-bootloader                | 	at org.flywaydb.core.internal.command.DbMigrate.applyMigrations(DbMigrate.java:270) ~[flyway-core-7.14.0.jar:?]
airbyte-bootloader                | 	at org.flywaydb.core.internal.command.DbMigrate.migrateGroup(DbMigrate.java:243) ~[flyway-core-7.14.0.jar:?]
airbyte-bootloader                | 	at org.flywaydb.core.internal.command.DbMigrate.lambda$migrateAll$0(DbMigrate.java:141) ~[flyway-core-7.14.0.jar:?]
airbyte-bootloader                | 	at org.flywaydb.core.internal.database.postgresql.PostgreSQLAdvisoryLockTemplate.execute(PostgreSQLAdvisoryLockTemplate.java:70) ~[flyway-core-7.14.0.jar:?]
airbyte-bootloader                | 	at org.flywaydb.core.internal.database.postgresql.PostgreSQLConnection.lock(PostgreSQLConnection.java:99) ~[flyway-core-7.14.0.jar:?]
airbyte-bootloader                | 	at org.flywaydb.core.internal.schemahistory.JdbcTableSchemaHistory.lock(JdbcTableSchemaHistory.java:141) ~[flyway-core-7.14.0.jar:?]
airbyte-bootloader                | 	at org.flywaydb.core.internal.command.DbMigrate.migrateAll(DbMigrate.java:141) ~[flyway-core-7.14.0.jar:?]
airbyte-bootloader                | 	at org.flywaydb.core.internal.command.DbMigrate.migrate(DbMigrate.java:101) ~[flyway-core-7.14.0.jar:?]
airbyte-bootloader                | 	at org.flywaydb.core.Flyway$1.execute(Flyway.java:217) ~[flyway-core-7.14.0.jar:?]
airbyte-bootloader                | 	at org.flywaydb.core.Flyway$1.execute(Flyway.java:168) ~[flyway-core-7.14.0.jar:?]
airbyte-bootloader                | 	at org.flywaydb.core.Flyway.execute(Flyway.java:584) ~[flyway-core-7.14.0.jar:?]
airbyte-bootloader                | 	at org.flywaydb.core.Flyway.migrate(Flyway.java:168) ~[flyway-core-7.14.0.jar:?]
airbyte-bootloader                | 	at io.airbyte.db.instance.FlywayDatabaseMigrator.migrate(FlywayDatabaseMigrator.java:45) ~[io.airbyte.airbyte-db-db-lib-0.50.19.jar:?]
airbyte-bootloader                | 	at io.airbyte.bootloader.Bootloader.runFlywayMigration(Bootloader.java:236) ~[io.airbyte-airbyte-bootloader-0.50.19.jar:?]
airbyte-bootloader                | 	at io.airbyte.bootloader.Bootloader.load(Bootloader.java:108) ~[io.airbyte-airbyte-bootloader-0.50.19.jar:?]
airbyte-bootloader                | 	at io.airbyte.bootloader.Application.main(Application.java:22) ~[io.airbyte-airbyte-bootloader-0.50.19.jar:?]
airbyte-bootloader                | Caused by: org.flywaydb.core.api.FlywayException: Migration failed !
airbyte-bootloader                | 	at org.flywaydb.core.internal.resolver.java.JavaMigrationExecutor.executeOnce(JavaMigrationExecutor.java:75) ~[flyway-core-7.14.0.jar:?]
airbyte-bootloader                | 	at org.flywaydb.core.internal.resolver.java.JavaMigrationExecutor.lambda$execute$0(JavaMigrationExecutor.java:54) ~[flyway-core-7.14.0.jar:?]
airbyte-bootloader                | 	at org.flywaydb.core.internal.database.DefaultExecutionStrategy.execute(DefaultExecutionStrategy.java:27) ~[flyway-core-7.14.0.jar:?]
airbyte-bootloader                | 	at org.flywaydb.core.internal.resolver.java.JavaMigrationExecutor.execute(JavaMigrationExecutor.java:53) ~[flyway-core-7.14.0.jar:?]
airbyte-bootloader                | 	at org.flywaydb.core.internal.command.DbMigrate.doMigrateGroup(DbMigrate.java:370) ~[flyway-core-7.14.0.jar:?]
airbyte-bootloader                | 	... 18 more
airbyte-bootloader                | Caused by: org.jooq.exception.DataAccessException: SQL [alter type "operator_type" add value 'webhook']; ERROR: ALTER TYPE ... ADD cannot run inside a transaction block
airbyte-bootloader                | 	at org.jooq_3.17.8.POSTGRES.debug(Unknown Source) ~[?:?]
airbyte-bootloader                | 	at org.jooq.impl.Tools.translate(Tools.java:3313) ~[jooq-3.17.8.jar:?]
airbyte-bootloader                | 	at org.jooq.impl.DefaultExecuteContext.sqlException(DefaultExecuteContext.java:678) ~[jooq-3.17.8.jar:?]
airbyte-bootloader                | 	at org.jooq.impl.AbstractQuery.execute(AbstractQuery.java:355) ~[jooq-3.17.8.jar:?]
airbyte-bootloader                | 	at io.airbyte.db.instance.configs.migrations.V0_40_12_001__AddWebhookOperationColumns.addWebhookOperationType(V0_40_12_001__AddWebhookOperationColumns.java:45) ~[io.airbyte.airbyte-db-db-lib-0.50.19.jar:?]
airbyte-bootloader                | 	at io.airbyte.db.instance.configs.migrations.V0_40_12_001__AddWebhookOperationColumns.migrate(V0_40_12_001__AddWebhookOperationColumns.java:32) ~[io.airbyte.airbyte-db-db-lib-0.50.19.jar:?]
airbyte-bootloader                | 	at org.flywaydb.core.internal.resolver.java.JavaMigrationExecutor.executeOnce(JavaMigrationExecutor.java:61) ~[flyway-core-7.14.0.jar:?]
airbyte-bootloader                | 	at org.flywaydb.core.internal.resolver.java.JavaMigrationExecutor.lambda$execute$0(JavaMigrationExecutor.java:54) ~[flyway-core-7.14.0.jar:?]
airbyte-bootloader                | 	at org.flywaydb.core.internal.database.DefaultExecutionStrategy.execute(DefaultExecutionStrategy.java:27) ~[flyway-core-7.14.0.jar:?]
airbyte-bootloader                | 	at org.flywaydb.core.internal.resolver.java.JavaMigrationExecutor.execute(JavaMigrationExecutor.java:53) ~[flyway-core-7.14.0.jar:?]
airbyte-bootloader                | 	at org.flywaydb.core.internal.command.DbMigrate.doMigrateGroup(DbMigrate.java:370) ~[flyway-core-7.14.0.jar:?]
airbyte-bootloader                | 	... 18 more
airbyte-bootloader                | Caused by: org.postgresql.util.PSQLException: ERROR: ALTER TYPE ... ADD cannot run inside a transaction block
airbyte-bootloader                | 	at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2676) ~[postgresql-42.5.4.jar:42.5.4]
airbyte-bootloader                | 	at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:2366) ~[postgresql-42.5.4.jar:42.5.4]
airbyte-bootloader                | 	at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:356) ~[postgresql-42.5.4.jar:42.5.4]
airbyte-bootloader                | 	at org.postgresql.jdbc.PgStatement.executeInternal(PgStatement.java:496) ~[postgresql-42.5.4.jar:42.5.4]
airbyte-bootloader                | 	at org.postgresql.jdbc.PgStatement.execute(PgStatement.java:413) ~[postgresql-42.5.4.jar:42.5.4]
airbyte-bootloader                | 	at org.postgresql.jdbc.PgStatement.executeWithFlags(PgStatement.java:333) ~[postgresql-42.5.4.jar:42.5.4]
airbyte-bootloader                | 	at org.postgresql.jdbc.PgStatement.executeCachedSql(PgStatement.java:319) ~[postgresql-42.5.4.jar:42.5.4]
airbyte-bootloader                | 	at org.postgresql.jdbc.PgStatement.executeWithFlags(PgStatement.java:295) ~[postgresql-42.5.4.jar:42.5.4]
airbyte-bootloader                | 	at org.postgresql.jdbc.PgStatement.execute(PgStatement.java:290) ~[postgresql-42.5.4.jar:42.5.4]
airbyte-bootloader                | 	at com.zaxxer.hikari.pool.ProxyStatement.execute(ProxyStatement.java:94) ~[HikariCP-5.0.1.jar:?]
airbyte-bootloader                | 	at com.zaxxer.hikari.pool.HikariProxyStatement.execute(HikariProxyStatement.java) ~[HikariCP-5.0.1.jar:?]
airbyte-bootloader                | 	at org.jooq.tools.jdbc.DefaultStatement.execute(DefaultStatement.java:102) ~[jooq-3.17.8.jar:?]
airbyte-bootloader                | 	at org.jooq.impl.SettingsEnabledPreparedStatement.execute(SettingsEnabledPreparedStatement.java:227) ~[jooq-3.17.8.jar:?]
airbyte-bootloader                | 	at org.jooq.impl.AbstractQuery.execute(AbstractQuery.java:420) ~[jooq-3.17.8.jar:?]
airbyte-bootloader                | 	at org.jooq.impl.AbstractQuery.execute(AbstractQuery.java:341) ~[jooq-3.17.8.jar:?]
airbyte-bootloader                | 	at io.airbyte.db.instance.configs.migrations.V0_40_12_001__AddWebhookOperationColumns.addWebhookOperationType(V0_40_12_001__AddWebhookOperationColumns.java:45) ~[io.airbyte.airbyte-db-db-lib-0.50.19.jar:?]
airbyte-bootloader                | 	at io.airbyte.db.instance.configs.migrations.V0_40_12_001__AddWebhookOperationColumns.migrate(V0_40_12_001__AddWebhookOperationColumns.java:32) ~[io.airbyte.airbyte-db-db-lib-0.50.19.jar:?]
airbyte-bootloader                | 	at org.flywaydb.core.internal.resolver.java.JavaMigrationExecutor.executeOnce(JavaMigrationExecutor.java:61) ~[flyway-core-7.14.0.jar:?]
airbyte-bootloader                | 	at org.flywaydb.core.internal.resolver.java.JavaMigrationExecutor.lambda$execute$0(JavaMigrationExecutor.java:54) ~[flyway-core-7.14.0.jar:?]
airbyte-bootloader                | 	at org.flywaydb.core.internal.database.DefaultExecutionStrategy.execute(DefaultExecutionStrategy.java:27) ~[flyway-core-7.14.0.jar:?]
airbyte-bootloader                | 	at org.flywaydb.core.internal.resolver.java.JavaMigrationExecutor.execute(JavaMigrationExecutor.java:53) ~[flyway-core-7.14.0.jar:?]
airbyte-bootloader                | 	at org.flywaydb.core.internal.command.DbMigrate.doMigrateGroup(DbMigrate.java:370) ~[flyway-core-7.14.0.jar:?]
airbyte-bootloader                | 	... 18 more
airbyte-bootloader                | 2023-09-19 22:33:41 INFO i.m.r.Micronaut(lambda$start$0):115 - Embedded Application shutting down
airbyte-bootloader exited with code 255
service "bootloader" didn't completed successfully: exit 255