Airbyte OSS via Helm on EKS: sync jobs are failing constantly #35301

alexremn · 2024-02-14T20:00:01Z

alexremn
Feb 14, 2024

Good day, colleagues!

We have Airbyte OSS deployed via Helm on EKS with Karpenter. Each try of sync is ending with failure due to timeout for jobs waiting - using carpenter to dynamically spin up nodes during the pod creation.

Here is the log example:

io.airbyte.workers.exception.WorkerException: Failed to create pod for check step
        at io.airbyte.workers.process.KubeProcessFactory.create(KubeProcessFactory.java:197) ~[io.airbyte-airbyte-commons-worker-0.50.48.jar:?]
        at io.airbyte.workers.process.AirbyteIntegrationLauncher.check(AirbyteIntegrationLauncher.java:148) ~[io.airbyte-airbyte-commons-worker-0.50.48.jar:?]
        at io.airbyte.workers.general.DefaultCheckConnectionWorker.run(DefaultCheckConnectionWorker.java:71) ~[io.airbyte-airbyte-commons-worker-0.50.48.jar:?]
        at io.airbyte.workers.general.DefaultCheckConnectionWorker.run(DefaultCheckConnectionWorker.java:44) ~[io.airbyte-airbyte-commons-worker-0.50.48.jar:?]
        at io.airbyte.workers.temporal.TemporalAttemptExecution.get(TemporalAttemptExecution.java:142) ~[io.airbyte-airbyte-workers-0.50.48.jar:?]
        at io.airbyte.workers.temporal.check.connection.CheckConnectionActivityImpl.lambda$runWithJobOutput$1(CheckConnectionActivityImpl.java:222) ~[io.airbyte-airbyte-workers-0.50.48.jar:?]
        at io.airbyte.commons.temporal.HeartbeatUtils.withBackgroundHeartbeat(HeartbeatUtils.java:57) ~[io.airbyte-airbyte-commons-temporal-core-0.50.48.jar:?]
        at io.airbyte.workers.temporal.check.connection.CheckConnectionActivityImpl.runWithJobOutput(CheckConnectionActivityImpl.java:207) ~[io.airbyte-airbyte-workers-0.50.48.jar:?]
        at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103) ~[?:?]
        at java.base/java.lang.reflect.Method.invoke(Method.java:580) ~[?:?]
        at io.temporal.internal.activity.RootActivityInboundCallsInterceptor$POJOActivityInboundCallsInterceptor.executeActivity(RootActivityInboundCallsInterceptor.java:64) ~[temporal-sdk-1.22.3.jar:?]
        at io.temporal.internal.activity.RootActivityInboundCallsInterceptor.execute(RootActivityInboundCallsInterceptor.java:43) ~[temporal-sdk-1.22.3.jar:?]
        at io.temporal.internal.activity.ActivityTaskExecutors$BaseActivityTaskExecutor.execute(ActivityTaskExecutors.java:107) ~[temporal-sdk-1.22.3.jar:?]
        at io.temporal.internal.activity.ActivityTaskHandlerImpl.handle(ActivityTaskHandlerImpl.java:124) ~[temporal-sdk-1.22.3.jar:?]
        at io.temporal.internal.worker.ActivityWorker$TaskHandlerImpl.handleActivity(ActivityWorker.java:278) ~[temporal-sdk-1.22.3.jar:?]
        at io.temporal.internal.worker.ActivityWorker$TaskHandlerImpl.handle(ActivityWorker.java:243) ~[temporal-sdk-1.22.3.jar:?]
        at io.temporal.internal.worker.ActivityWorker$TaskHandlerImpl.handle(ActivityWorker.java:216) ~[temporal-sdk-1.22.3.jar:?]
        at io.temporal.internal.worker.PollTaskExecutor.lambda$process$0(PollTaskExecutor.java:105) ~[temporal-sdk-1.22.3.jar:?]
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?]
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?]
        at java.base/java.lang.Thread.run(Thread.java:1583) [?:?]
Caused by: java.lang.RuntimeException: java.io.IOException: kubectl cp failed with exit code 1
        at io.airbyte.workers.process.KubePodProcess.copyFilesToKubeConfigVolume(KubePodProcess.java:356) ~[io.airbyte-airbyte-commons-worker-0.50.48.jar:?]
        at io.airbyte.workers.process.KubePodProcess.<init>(KubePodProcess.java:655) ~[io.airbyte-airbyte-commons-worker-0.50.48.jar:?]
        at io.airbyte.workers.process.KubeProcessFactory.create(KubeProcessFactory.java:193) ~[io.airbyte-airbyte-commons-worker-0.50.48.jar:?]
        ... 20 more
Caused by: java.io.IOException: kubectl cp failed with exit code 1
        at io.airbyte.workers.process.KubePodProcess.copyFilesToKubeConfigVolume(KubePodProcess.java:350) ~[io.airbyte-airbyte-commons-worker-0.50.48.jar:?]
        at io.airbyte.workers.process.KubePodProcess.<init>(KubePodProcess.java:655) ~[io.airbyte-airbyte-commons-worker-0.50.48.jar:?]
        at io.airbyte.workers.process.KubeProcessFactory.create(KubeProcessFactory.java:193) ~[io.airbyte-airbyte-commons-worker-0.50.48.jar:?]

Can you, please, assist with how to extend the waiting timeouts?

marcosmarxm · 2024-02-14T20:06:08Z

marcosmarxm
Feb 14, 2024
Maintainer

Can you confirm all pods are in Running state and doesn't have any problem with EBS/disk volume creation?
What version of Helm Chart are you using? Did you change anything or running without any change?

6 replies

marcosmarxm Feb 14, 2024
Maintainer

Definitely something wrong. Would be possible to try the default release? Just for sanity check :D

alexremn Feb 14, 2024
Author

@marcosmarxm cannot use it, as we need to have external db & s3, not inside eks.

Also tried with minio before - was the same.

Here is the full log of last start:

2024-02-14 21:26:31 platform > Retry State: RetryManager(completeFailureBackoffPolicy=BackoffPolicy(minInterval=PT10S, maxInterval=PT30M, base=3), partialFailureBackoffPolicy=null, successiveCompleteFailureLimit=5, totalCompleteFailureLimit=10, successivePartialFailureLimit=1000, totalPartialFailureLimit=10, successiveCompleteFailures=2, totalCompleteFailures=2, successivePartialFailures=0, totalPartialFailures=0)
2024-02-14 21:26:31 platform > Backing off for: 30 seconds.
2024-02-14 21:27:02 platform > Cloud storage job log path: /workspace/20/2/logs.log
2024-02-14 21:27:02 platform > Executing worker wrapper. Airbyte version: 0.50.48
2024-02-14 21:27:02 platform > Attempt 0 to save workflow id for cancellation
2024-02-14 21:27:02 platform > 
2024-02-14 21:27:02 platform > Using default value for environment variable SIDECAR_KUBE_CPU_LIMIT: '2.0'
2024-02-14 21:27:02 platform > Using default value for environment variable SOCAT_KUBE_CPU_LIMIT: '2.0'
2024-02-14 21:27:02 platform > ----- START CHECK -----
2024-02-14 21:27:02 platform > Using default value for environment variable SIDECAR_KUBE_CPU_REQUEST: '0.1'
2024-02-14 21:27:02 platform > 
2024-02-14 21:27:02 platform > Using default value for environment variable SOCAT_KUBE_CPU_REQUEST: '0.1'
2024-02-14 21:27:02 platform > Using default value for environment variable LAUNCHDARKLY_KEY: ''
2024-02-14 21:27:02 platform > Using default value for environment variable FEATURE_FLAG_CLIENT: ''
2024-02-14 21:27:02 platform > Using default value for environment variable OTEL_COLLECTOR_ENDPOINT: ''
2024-02-14 21:27:02 platform > Attempting to start pod = source-bing-ads-check-20-2-andtb for airbyte/source-bing-ads:2.1.4 with resources ConnectorResourceRequirements[main=io.airbyte.config.ResourceRequirements@1a3fafa[cpuRequest=2000m,cpuLimit=2000m,memoryRequest=4Gi,memoryLimit=4Gi,additionalProperties={}], heartbeat=io.airbyte.config.ResourceRequirements@5c7e9313[cpuRequest=0.1,cpuLimit=2.0,memoryRequest=25Mi,memoryLimit=50Mi,additionalProperties={}], stdErr=io.airbyte.config.ResourceRequirements@59047b02[cpuRequest=0.25,cpuLimit=2,memoryRequest=25Mi,memoryLimit=50Mi,additionalProperties={}], stdIn=io.airbyte.config.ResourceRequirements@64c53467[cpuRequest=0.1,cpuLimit=2.0,memoryRequest=25Mi,memoryLimit=50Mi,additionalProperties={}], stdOut=io.airbyte.config.ResourceRequirements@64c53467[cpuRequest=0.1,cpuLimit=2.0,memoryRequest=25Mi,memoryLimit=50Mi,additionalProperties={}]] and allowedHosts io.airbyte.config.AllowedHosts@7b03a6cc[hosts=[bingads.microsoft.com, login.microsoftonline.com, login.live.com, login.windows-ppe.net, ads.microsoft.com, api.ads.microsoft.com, clientcenter.api.bingads.microsoft.com, *.datadoghq.com, *.datadoghq.eu, *.sentry.io],additionalProperties={}]
2024-02-14 21:27:02 platform > source-bing-ads-check-20-2-andtb stdoutLocalPort = 9030
2024-02-14 21:27:02 platform > source-bing-ads-check-20-2-andtb stderrLocalPort = 9031
2024-02-14 21:27:02 platform > Creating stdout socket server...
2024-02-14 21:27:02 platform > Creating stderr socket server...
2024-02-14 21:27:02 platform > Creating pod source-bing-ads-check-20-2-andtb...
2024-02-14 21:27:02 platform > Waiting for init container to be ready before copying files...
2024-02-14 21:27:38 platform > Init container ready..
2024-02-14 21:27:38 platform > Copying files...
2024-02-14 21:27:38 platform > Uploading file: source_config.json
2024-02-14 21:27:38 platform > kubectl cp /tmp/85ca8db7-3d96-4772-a073-cac73add2ace/source_config.json airbyte/source-bing-ads-check-20-2-andtb:/config/source_config.json -c init --retries=3
2024-02-14 21:27:38 platform > Waiting for kubectl cp to complete
2024-02-14 21:27:38 platform > (pod: airbyte / source-bing-ads-check-20-2-andtb) - Destroying Kube process.
2024-02-14 21:27:38 platform > (pod: airbyte / source-bing-ads-check-20-2-andtb) - Closed all resources for pod
2024-02-14 21:27:38 platform > (pod: airbyte / source-bing-ads-check-20-2-andtb) - Destroyed Kube process.
2024-02-14 21:27:38 platform > Unexpected error while checking connection: 
io.airbyte.workers.exception.WorkerException: Failed to create pod for check step
	at io.airbyte.workers.process.KubeProcessFactory.create(KubeProcessFactory.java:197) ~[io.airbyte-airbyte-commons-worker-0.50.48.jar:?]
	at io.airbyte.workers.process.AirbyteIntegrationLauncher.check(AirbyteIntegrationLauncher.java:148) ~[io.airbyte-airbyte-commons-worker-0.50.48.jar:?]
	at io.airbyte.workers.general.DefaultCheckConnectionWorker.run(DefaultCheckConnectionWorker.java:71) ~[io.airbyte-airbyte-commons-worker-0.50.48.jar:?]
	at io.airbyte.workers.general.DefaultCheckConnectionWorker.run(DefaultCheckConnectionWorker.java:44) ~[io.airbyte-airbyte-commons-worker-0.50.48.jar:?]
	at io.airbyte.workers.temporal.TemporalAttemptExecution.get(TemporalAttemptExecution.java:142) ~[io.airbyte-airbyte-workers-0.50.48.jar:?]
	at io.airbyte.workers.temporal.check.connection.CheckConnectionActivityImpl.lambda$runWithJobOutput$1(CheckConnectionActivityImpl.java:222) ~[io.airbyte-airbyte-workers-0.50.48.jar:?]
	at io.airbyte.commons.temporal.HeartbeatUtils.withBackgroundHeartbeat(HeartbeatUtils.java:57) ~[io.airbyte-airbyte-commons-temporal-core-0.50.48.jar:?]
	at io.airbyte.workers.temporal.check.connection.CheckConnectionActivityImpl.runWithJobOutput(CheckConnectionActivityImpl.java:207) ~[io.airbyte-airbyte-workers-0.50.48.jar:?]
	at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103) ~[?:?]
	at java.base/java.lang.reflect.Method.invoke(Method.java:580) ~[?:?]
	at io.temporal.internal.activity.RootActivityInboundCallsInterceptor$POJOActivityInboundCallsInterceptor.executeActivity(RootActivityInboundCallsInterceptor.java:64) ~[temporal-sdk-1.22.3.jar:?]
	at io.temporal.internal.activity.RootActivityInboundCallsInterceptor.execute(RootActivityInboundCallsInterceptor.java:43) ~[temporal-sdk-1.22.3.jar:?]
	at io.temporal.internal.activity.ActivityTaskExecutors$BaseActivityTaskExecutor.execute(ActivityTaskExecutors.java:107) ~[temporal-sdk-1.22.3.jar:?]
	at io.temporal.internal.activity.ActivityTaskHandlerImpl.handle(ActivityTaskHandlerImpl.java:124) ~[temporal-sdk-1.22.3.jar:?]
	at io.temporal.internal.worker.ActivityWorker$TaskHandlerImpl.handleActivity(ActivityWorker.java:278) ~[temporal-sdk-1.22.3.jar:?]
	at io.temporal.internal.worker.ActivityWorker$TaskHandlerImpl.handle(ActivityWorker.java:243) ~[temporal-sdk-1.22.3.jar:?]
	at io.temporal.internal.worker.ActivityWorker$TaskHandlerImpl.handle(ActivityWorker.java:216) ~[temporal-sdk-1.22.3.jar:?]
	at io.temporal.internal.worker.PollTaskExecutor.lambda$process$0(PollTaskExecutor.java:105) ~[temporal-sdk-1.22.3.jar:?]
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?]
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?]
	at java.base/java.lang.Thread.run(Thread.java:1583) [?:?]
Caused by: java.lang.RuntimeException: java.io.IOException: kubectl cp failed with exit code 1
	at io.airbyte.workers.process.KubePodProcess.copyFilesToKubeConfigVolume(KubePodProcess.java:356) ~[io.airbyte-airbyte-commons-worker-0.50.48.jar:?]
	at io.airbyte.workers.process.KubePodProcess.<init>(KubePodProcess.java:655) ~[io.airbyte-airbyte-commons-worker-0.50.48.jar:?]
	at io.airbyte.workers.process.KubeProcessFactory.create(KubeProcessFactory.java:193) ~[io.airbyte-airbyte-commons-worker-0.50.48.jar:?]
	... 20 more
Caused by: java.io.IOException: kubectl cp failed with exit code 1
	at io.airbyte.workers.process.KubePodProcess.copyFilesToKubeConfigVolume(KubePodProcess.java:350) ~[io.airbyte-airbyte-commons-worker-0.50.48.jar:?]
	at io.airbyte.workers.process.KubePodProcess.<init>(KubePodProcess.java:655) ~[io.airbyte-airbyte-commons-worker-0.50.48.jar:?]
	at io.airbyte.workers.process.KubeProcessFactory.create(KubeProcessFactory.java:193) ~[io.airbyte-airbyte-commons-worker-0.50.48.jar:?]
	... 20 more
2024-02-14 21:27:38 platform > 
2024-02-14 21:27:38 platform > ----- END CHECK -----
2024-02-14 21:27:38 platform > 
2024-02-14 21:27:39 platform > Retry State: RetryManager(completeFailureBackoffPolicy=BackoffPolicy(minInterval=PT10S, maxInterval=PT30M, base=3), partialFailureBackoffPolicy=null, successiveCompleteFailureLimit=5, totalCompleteFailureLimit=10, successivePartialFailureLimit=1000, totalPartialFailureLimit=10, successiveCompleteFailures=3, totalCompleteFailures=3, successivePartialFailures=0, totalPartialFailures=0)
 Backoff before next attempt: 1 minute 30 seconds

marcosmarxm Feb 15, 2024
Maintainer

My question regards in validate your cluster is working properly. EKS you need to enable the EBS driver add-on to work with Airbyte Helm. Running the default value will make sure you have it in a good state.

marcosmarxm Feb 15, 2024
Maintainer

Second, can you make the changes in steps. Have the external database and after the external logs?

alexremn Feb 15, 2024
Author

@marcosmarxm good day! :)

It's enabled for sure, we're utilising it with our other services. Nevertheless, when Airbyte creates jobs, it is not creating volumes for it. Where can the configuration for creating volumes be found?

I don't actually sure that's the issue is in RDS nor in S3. When I tried to investigate yesterday, I found that's Airbyte code too early defines job pod as successfully loaded and tries to issue Kubectl cp to it with statically defined amount of retries (found here) - this happens due to we need to wait for karpenter to start node rolling up process, wait for node, wait for pod - it usually takes up to 2 minutes till the pod would be actually available, but Airbyte job controller don't wait for it.

Can we adjust somehow wait time for job pods or adjust number of retries?

alexremn · 2024-02-16T09:14:28Z

alexremn
Feb 16, 2024
Author

Good day, colleagues!

Any ideas?

1 reply

alexremn Feb 16, 2024
Author

Here is the last log:

2024-02-16 10:40:38 platform > Retry State: RetryManager(completeFailureBackoffPolicy=BackoffPolicy(minInterval=PT10S, maxInterval=PT30M, base=3), partialFailureBackoffPolicy=null, successiveCompleteFailureLimit=5, totalCompleteFailureLimit=10, successivePartialFailureLimit=1000, totalPartialFailureLimit=10, successiveCompleteFailures=4, totalCompleteFailures=4, successivePartialFailures=0, totalPartialFailures=0)
2024-02-16 10:40:38 platform > Backing off for: 4 minutes 30 seconds.
2024-02-16 10:45:09 platform > Cloud storage job log path: /workspace/23/4/logs.log
2024-02-16 10:45:09 platform > Executing worker wrapper. Airbyte version: 0.50.48
2024-02-16 10:45:09 platform > Attempt 0 to save workflow id for cancellation
2024-02-16 10:45:09 platform > 
2024-02-16 10:45:09 platform > Using default value for environment variable SIDECAR_KUBE_CPU_LIMIT: '2.0'
2024-02-16 10:45:09 platform > ----- START CHECK -----
2024-02-16 10:45:09 platform > Using default value for environment variable SOCAT_KUBE_CPU_LIMIT: '2.0'
2024-02-16 10:45:09 platform > 
2024-02-16 10:45:09 platform > Using default value for environment variable SIDECAR_KUBE_CPU_REQUEST: '0.1'
2024-02-16 10:45:09 platform > Using default value for environment variable SOCAT_KUBE_CPU_REQUEST: '0.1'
2024-02-16 10:45:09 platform > Using default value for environment variable LAUNCHDARKLY_KEY: ''
2024-02-16 10:45:09 platform > Using default value for environment variable FEATURE_FLAG_CLIENT: ''
2024-02-16 10:45:09 platform > Using default value for environment variable OTEL_COLLECTOR_ENDPOINT: ''
2024-02-16 10:45:09 platform > Attempting to start pod = source-bing-ads-check-23-4-gjono for airbyte/source-bing-ads:2.1.4 with resources ConnectorResourceRequirements[main=io.airbyte.config.ResourceRequirements@1c01df19[cpuRequest=1000m,cpuLimit=1000m,memoryRequest=2Gi,memoryLimit=2Gi,additionalProperties={}], heartbeat=io.airbyte.config.ResourceRequirements@74a8b128[cpuRequest=0.1,cpuLimit=2.0,memoryRequest=25Mi,memoryLimit=50Mi,additionalProperties={}], stdErr=io.airbyte.config.ResourceRequirements@371a3e30[cpuRequest=0.25,cpuLimit=2,memoryRequest=25Mi,memoryLimit=50Mi,additionalProperties={}], stdIn=io.airbyte.config.ResourceRequirements@7d6df4b1[cpuRequest=0.1,cpuLimit=2.0,memoryRequest=25Mi,memoryLimit=50Mi,additionalProperties={}], stdOut=io.airbyte.config.ResourceRequirements@7d6df4b1[cpuRequest=0.1,cpuLimit=2.0,memoryRequest=25Mi,memoryLimit=50Mi,additionalProperties={}]] and allowedHosts io.airbyte.config.AllowedHosts@4ce2b5b6[hosts=[bingads.microsoft.com, login.microsoftonline.com, login.live.com, login.windows-ppe.net, ads.microsoft.com, api.ads.microsoft.com, clientcenter.api.bingads.microsoft.com, *.datadoghq.com, *.datadoghq.eu, *.sentry.io],additionalProperties={}]
2024-02-16 10:45:09 platform > source-bing-ads-check-23-4-gjono stdoutLocalPort = 9006
2024-02-16 10:45:09 platform > source-bing-ads-check-23-4-gjono stderrLocalPort = 9007
2024-02-16 10:45:09 platform > Creating stdout socket server...
2024-02-16 10:45:09 platform > Creating stderr socket server...
2024-02-16 10:45:09 platform > Creating pod source-bing-ads-check-23-4-gjono...
2024-02-16 10:45:09 platform > Waiting for init container to be ready before copying files...
2024-02-16 10:45:10 platform > Init container ready..
2024-02-16 10:45:10 platform > Copying files...
2024-02-16 10:45:10 platform > Uploading file: source_config.json
2024-02-16 10:45:10 platform > kubectl cp /tmp/bad6058a-fca5-466b-8205-d0d5cf8cf770/source_config.json airbyte/source-bing-ads-check-23-4-gjono:/config/source_config.json -c init --retries=3
2024-02-16 10:45:10 platform > Waiting for kubectl cp to complete
2024-02-16 10:45:10 platform > kubectl cp complete, closing process
2024-02-16 10:45:10 platform > Uploading file: FINISHED_UPLOADING
2024-02-16 10:45:10 platform > kubectl cp /tmp/b1c73b9f-005c-4aaf-987c-caed0b86d480/FINISHED_UPLOADING airbyte/source-bing-ads-check-23-4-gjono:/config/FINISHED_UPLOADING -c init --retries=3
2024-02-16 10:45:10 platform > Waiting for kubectl cp to complete
2024-02-16 10:45:10 platform > kubectl cp complete, closing process
2024-02-16 10:45:10 platform > Waiting until pod is ready...
2024-02-16 10:45:11 platform > Setting stdout...
2024-02-16 10:45:11 platform > Setting stderr...
2024-02-16 10:45:12 platform > Reading pod IP...
2024-02-16 10:45:12 platform > Pod IP: 10.10.2.170
2024-02-16 10:45:12 platform > Using null stdin output stream...
2024-02-16 10:45:12 platform > Reading messages from protocol version 0.2.0
2024-02-16 10:45:15 platform > Fetching access token ...
2024-02-16 10:45:15 platform > Check succeeded
2024-02-16 10:45:16 platform > (pod: airbyte / source-bing-ads-check-23-4-gjono) - Closed all resources for pod
2024-02-16 10:45:16 platform > Check connection job received output: io.airbyte.config.StandardCheckConnectionOutput@1bd0c87b[status=succeeded,message=<null>,additionalProperties={}]
2024-02-16 10:45:16 platform > 
2024-02-16 10:45:16 platform > ----- END CHECK -----
2024-02-16 10:45:16 platform > 
2024-02-16 10:45:16 platform > Cloud storage job log path: /workspace/23/4/logs.log
2024-02-16 10:45:16 platform > Executing worker wrapper. Airbyte version: 0.50.48
2024-02-16 10:45:16 platform > Attempt 0 to save workflow id for cancellation
2024-02-16 10:45:16 platform > Using default value for environment variable SIDECAR_KUBE_CPU_LIMIT: '2.0'
2024-02-16 10:45:16 platform > 
2024-02-16 10:45:16 platform > Using default value for environment variable SOCAT_KUBE_CPU_LIMIT: '2.0'
2024-02-16 10:45:16 platform > ----- START CHECK -----
2024-02-16 10:45:16 platform > 
2024-02-16 10:45:16 platform > Using default value for environment variable SIDECAR_KUBE_CPU_REQUEST: '0.1'
2024-02-16 10:45:16 platform > Using default value for environment variable SOCAT_KUBE_CPU_REQUEST: '0.1'
2024-02-16 10:45:16 platform > Using default value for environment variable LAUNCHDARKLY_KEY: ''
2024-02-16 10:45:16 platform > Using default value for environment variable FEATURE_FLAG_CLIENT: ''
2024-02-16 10:45:16 platform > Using default value for environment variable OTEL_COLLECTOR_ENDPOINT: ''
2024-02-16 10:45:16 platform > Attempting to start pod = destination-redshift-check-23-4-vyski for airbyte/destination-redshift:2.1.7 with resources ConnectorResourceRequirements[main=io.airbyte.config.ResourceRequirements@175150d5[cpuRequest=1000m,cpuLimit=1000m,memoryRequest=2Gi,memoryLimit=2Gi,additionalProperties={}], heartbeat=io.airbyte.config.ResourceRequirements@74a8b128[cpuRequest=0.1,cpuLimit=2.0,memoryRequest=25Mi,memoryLimit=50Mi,additionalProperties={}], stdErr=io.airbyte.config.ResourceRequirements@371a3e30[cpuRequest=0.25,cpuLimit=2,memoryRequest=25Mi,memoryLimit=50Mi,additionalProperties={}], stdIn=io.airbyte.config.ResourceRequirements@7d6df4b1[cpuRequest=0.1,cpuLimit=2.0,memoryRequest=25Mi,memoryLimit=50Mi,additionalProperties={}], stdOut=io.airbyte.config.ResourceRequirements@7d6df4b1[cpuRequest=0.1,cpuLimit=2.0,memoryRequest=25Mi,memoryLimit=50Mi,additionalProperties={}]] and allowedHosts null
2024-02-16 10:45:16 platform > destination-redshift-check-23-4-vyski stdoutLocalPort = 9008
2024-02-16 10:45:16 platform > destination-redshift-check-23-4-vyski stderrLocalPort = 9009
2024-02-16 10:45:16 platform > Creating stdout socket server...
2024-02-16 10:45:16 platform > Creating stderr socket server...
2024-02-16 10:45:16 platform > Creating pod destination-redshift-check-23-4-vyski...
2024-02-16 10:45:16 platform > Waiting for init container to be ready before copying files...
2024-02-16 10:45:19 platform > Init container ready..
2024-02-16 10:45:19 platform > Copying files...
2024-02-16 10:45:19 platform > Uploading file: source_config.json
2024-02-16 10:45:19 platform > kubectl cp /tmp/19d997f7-769d-4290-946d-7451aa85b212/source_config.json airbyte/destination-redshift-check-23-4-vyski:/config/source_config.json -c init --retries=3
2024-02-16 10:45:19 platform > Waiting for kubectl cp to complete
2024-02-16 10:45:19 platform > kubectl cp complete, closing process
2024-02-16 10:45:19 platform > Uploading file: FINISHED_UPLOADING
2024-02-16 10:45:19 platform > kubectl cp /tmp/08dad9ac-ae71-4963-93b4-befde0bfb7fe/FINISHED_UPLOADING airbyte/destination-redshift-check-23-4-vyski:/config/FINISHED_UPLOADING -c init --retries=3
2024-02-16 10:45:19 platform > Waiting for kubectl cp to complete
2024-02-16 10:45:19 platform > kubectl cp complete, closing process
2024-02-16 10:45:19 platform > Waiting until pod is ready...
2024-02-16 10:45:20 platform > Setting stdout...
2024-02-16 10:45:20 platform > Setting stderr...
2024-02-16 10:45:21 platform > Reading pod IP...
2024-02-16 10:45:21 platform > Pod IP: 10.10.2.108
2024-02-16 10:45:21 platform > Using null stdin output stream...
2024-02-16 10:45:21 platform > Reading messages from protocol version 0.2.0
2024-02-16 10:45:21 platform > WARN StatusConsoleListener The use of package scanning to locate plugins is deprecated and will be removed in a future release
2024-02-16 10:45:21 platform > WARN StatusConsoleListener The use of package scanning to locate plugins is deprecated and will be removed in a future release
2024-02-16 10:45:21 platform > WARN StatusConsoleListener The use of package scanning to locate plugins is deprecated and will be removed in a future release
2024-02-16 10:45:21 platform > WARN StatusConsoleListener The use of package scanning to locate plugins is deprecated and will be removed in a future release
2024-02-16 10:45:22 platform > 2024-02-16 10:45:22 INFO i.a.i.d.r.RedshiftDestination(main):79 - starting destination: class io.airbyte.integrations.destination.redshift.RedshiftDestination
2024-02-16 10:45:22 platform > 2024-02-16 10:45:22 INFO i.a.c.i.b.IntegrationCliParser(parseOptions):126 - integration args: {check=null, config=source_config.json}
2024-02-16 10:45:22 platform > 2024-02-16 10:45:22 INFO i.a.c.i.b.IntegrationRunner(runInternal):132 - Running integration: io.airbyte.integrations.destination.redshift.RedshiftDestination
2024-02-16 10:45:22 platform > 2024-02-16 10:45:22 INFO i.a.c.i.b.IntegrationRunner(runInternal):133 - Command: CHECK
2024-02-16 10:45:22 platform > 2024-02-16 10:45:22 INFO i.a.c.i.b.IntegrationRunner(runInternal):134 - Integration config: IntegrationConfig{command=CHECK, configPath='source_config.json', catalogPath='null', statePath='null'}
2024-02-16 10:45:22 platform > 2024-02-16 10:45:22 WARN c.n.s.JsonMetaSchema(newValidator):278 - Unknown keyword groups - you should define your own Meta Schema. If the keyword is irrelevant for validation, just use a NonValidationKeyword
2024-02-16 10:45:22 platform > 2024-02-16 10:45:22 WARN c.n.s.JsonMetaSchema(newValidator):278 - Unknown keyword group - you should define your own Meta Schema. If the keyword is irrelevant for validation, just use a NonValidationKeyword
2024-02-16 10:45:22 platform > 2024-02-16 10:45:22 WARN c.n.s.JsonMetaSchema(newValidator):278 - Unknown keyword order - you should define your own Meta Schema. If the keyword is irrelevant for validation, just use a NonValidationKeyword
2024-02-16 10:45:22 platform > 2024-02-16 10:45:22 WARN c.n.s.JsonMetaSchema(newValidator):278 - Unknown keyword airbyte_secret - you should define your own Meta Schema. If the keyword is irrelevant for validation, just use a NonValidationKeyword
2024-02-16 10:45:22 platform > 2024-02-16 10:45:22 WARN c.n.s.JsonMetaSchema(newValidator):278 - Unknown keyword display_type - you should define your own Meta Schema. If the keyword is irrelevant for validation, just use a NonValidationKeyword
2024-02-16 10:45:22 platform > 2024-02-16 10:45:22 INFO i.a.c.i.d.j.c.SwitchingDestination(check):56 - Using destination type: COPY_S3
2024-02-16 10:45:22 platform > 2024-02-16 10:45:22 INFO i.a.c.i.b.s.SshTunnel(getInstance):252 - Starting connection with method: NO_TUNNEL
2024-02-16 10:45:22 platform > 2024-02-16 10:45:22 INFO i.a.c.i.d.s.S3DestinationConfig(createS3Client):239 - Creating S3 client...
2024-02-16 10:45:24 platform > 2024-02-16 10:45:24 INFO i.a.c.i.d.s.S3BaseChecks(testIAMUserHasListObjectPermission):131 - Started testing if IAM user can call listObjects on the destination bucket
2024-02-16 10:45:24 platform > 2024-02-16 10:45:24 INFO i.a.c.i.d.s.S3BaseChecks(testIAMUserHasListObjectPermission):134 - Finished checking for listObjects permission
2024-02-16 10:45:24 platform > 2024-02-16 10:45:24 INFO c.z.h.HikariDataSource(<init>):79 - HikariPool-1 - Starting...
2024-02-16 10:45:24 platform > 2024-02-16 10:45:24 WARN c.z.h.u.DriverDataSource(<init>):68 - Registered driver with driverClassName=com.amazon.redshift.jdbc.Driver was not found, trying direct instantiation.
2024-02-16 10:45:24 platform > 2024-02-16 10:45:24 INFO c.z.h.HikariDataSource(<init>):81 - HikariPool-1 - Start completed.
2024-02-16 10:45:26 platform > 2024-02-16 10:45:26 INFO i.a.i.d.r.u.RedshiftUtil(checkSvvTableAccess):43 - checking SVV_TABLE_INFO permissions
2024-02-16 10:45:27 platform > 2024-02-16 10:45:27 INFO i.a.c.d.j.DefaultJdbcDatabase(lambda$unsafeQuery$1):132 - closing connection
2024-02-16 10:45:27 platform > 2024-02-16 10:45:27 INFO c.z.h.HikariDataSource(close):349 - HikariPool-1 - Shutdown initiated...
2024-02-16 10:45:27 platform > 2024-02-16 10:45:27 INFO c.z.h.HikariDataSource(close):351 - HikariPool-1 - Shutdown completed.
2024-02-16 10:45:27 platform > 2024-02-16 10:45:27 INFO i.a.c.i.b.IntegrationRunner(runInternal):231 - Completed integration: io.airbyte.integrations.destination.redshift.RedshiftDestination
2024-02-16 10:45:27 platform > 2024-02-16 10:45:27 INFO i.a.i.d.r.RedshiftDestination(main):81 - completed destination: class io.airbyte.integrations.destination.redshift.RedshiftDestination
2024-02-16 10:45:28 platform > (pod: airbyte / destination-redshift-check-23-4-vyski) - Closed all resources for pod
2024-02-16 10:45:28 platform > Check connection job received output: io.airbyte.config.StandardCheckConnectionOutput@7afbc2c0[status=succeeded,message=<null>,additionalProperties={}]
2024-02-16 10:45:28 platform > 
2024-02-16 10:45:28 platform > ----- END CHECK -----
2024-02-16 10:45:28 platform > 
2024-02-16 10:45:29 platform > Cloud storage job log path: /workspace/23/4/logs.log
2024-02-16 10:45:29 platform > Executing worker wrapper. Airbyte version: 0.50.48
2024-02-16 10:45:29 platform > Attempt 0 to save workflow id for cancellation
2024-02-16 10:45:29 platform > Creating orchestrator-repl-job-23-attempt-4 for attempt number: 4
2024-02-16 10:45:29 platform > Successfully deleted all running pods for the connection!
2024-02-16 10:45:29 platform > Waiting for pod to be running...
2024-02-16 10:45:31 platform > Pod airbyte/orchestrator-repl-job-23-attempt-4 is running on 10.10.2.228
2024-02-16 10:45:31 platform > Uploading file: envMap.json
2024-02-16 10:45:31 platform > kubectl cp /tmp/82c17528-8c31-4a76-8dc6-1b5d5ecd6777/envMap.json airbyte/orchestrator-repl-job-23-attempt-4:/config/envMap.json -c init --retries=3
2024-02-16 10:45:31 platform > Waiting for kubectl cp to complete
2024-02-16 10:45:31 platform > kubectl cp complete, closing process
2024-02-16 10:45:31 platform > Uploading file: application.txt
2024-02-16 10:45:31 platform > kubectl cp /tmp/aa26e9b2-57ff-4953-aa54-b70f971f74bf/application.txt airbyte/orchestrator-repl-job-23-attempt-4:/config/application.txt -c init --retries=3
2024-02-16 10:45:31 platform > Waiting for kubectl cp to complete
2024-02-16 10:45:31 platform > kubectl cp complete, closing process
2024-02-16 10:45:31 platform > Uploading file: jobRunConfig.json
2024-02-16 10:45:31 platform > kubectl cp /tmp/9a342626-1bb3-4719-a3fa-c8a1767fa285/jobRunConfig.json airbyte/orchestrator-repl-job-23-attempt-4:/config/jobRunConfig.json -c init --retries=3
2024-02-16 10:45:31 platform > Waiting for kubectl cp to complete
2024-02-16 10:45:31 platform > kubectl cp complete, closing process
2024-02-16 10:45:31 platform > Uploading file: destinationLauncherConfig.json
2024-02-16 10:45:31 platform > kubectl cp /tmp/93a00515-ee41-4326-b45d-1f549d52a695/destinationLauncherConfig.json airbyte/orchestrator-repl-job-23-attempt-4:/config/destinationLauncherConfig.json -c init --retries=3
2024-02-16 10:45:31 platform > Waiting for kubectl cp to complete
2024-02-16 10:45:31 platform > kubectl cp complete, closing process
2024-02-16 10:45:31 platform > Uploading file: sourceLauncherConfig.json
2024-02-16 10:45:31 platform > kubectl cp /tmp/cffe93b2-41ca-4264-a142-23c66e298ed2/sourceLauncherConfig.json airbyte/orchestrator-repl-job-23-attempt-4:/config/sourceLauncherConfig.json -c init --retries=3
2024-02-16 10:45:32 platform > Waiting for kubectl cp to complete
2024-02-16 10:45:32 platform > kubectl cp complete, closing process
2024-02-16 10:45:32 platform > Uploading file: input.json
2024-02-16 10:45:32 platform > kubectl cp /tmp/b10c6203-cb45-4418-ac2e-9da8838c259f/input.json airbyte/orchestrator-repl-job-23-attempt-4:/config/input.json -c init --retries=3
2024-02-16 10:45:32 platform > Waiting for kubectl cp to complete
2024-02-16 10:45:32 platform > kubectl cp complete, closing process
2024-02-16 10:45:32 platform > Uploading file: KUBE_POD_INFO
2024-02-16 10:45:32 platform > kubectl cp /tmp/3cd6c461-bd7f-4a19-ad42-cd4ec5d1ad34/KUBE_POD_INFO airbyte/orchestrator-repl-job-23-attempt-4:/config/KUBE_POD_INFO -c init --retries=3
2024-02-16 10:45:32 platform > Waiting for kubectl cp to complete
2024-02-16 10:45:32 platform > kubectl cp complete, closing process
2024-02-16 10:45:32 platform > Uploading file: FINISHED_UPLOADING
2024-02-16 10:45:32 platform > kubectl cp /tmp/f5501974-4c89-405b-aaed-7287b5c3226a/FINISHED_UPLOADING airbyte/orchestrator-repl-job-23-attempt-4:/config/FINISHED_UPLOADING -c init --retries=3
2024-02-16 10:45:32 platform > Waiting for kubectl cp to complete
2024-02-16 10:45:32 platform > kubectl cp complete, closing process
2024-02-16 10:45:37 INFO i.m.r.Micronaut(lambda$start$2):98 - Startup completed in 2593ms. Server Running: http://orchestrator-repl-job-23-attempt-4:9000
2024-02-16 10:45:40 replication-orchestrator > Writing async status INITIALIZING for KubePodInfo[namespace=airbyte, name=orchestrator-repl-job-23-attempt-4, mainContainerInfo=KubeContainerInfo[image=airbyte/container-orchestrator:0.50.48, pullPolicy=IfNotPresent]]...
2024-02-16 10:45:38 INFO i.a.f.ConfigFileClient(<init>):105 - path /flags does not exist, will return default flag values
2024-02-16 10:45:39 INFO i.a.c.EnvConfigs(getEnvOrDefault):694 - Using default value for environment variable METRIC_CLIENT: ''
2024-02-16 10:45:39 INFO i.a.c.EnvConfigs(getEnvOrDefault):694 - Using default value for environment variable METRIC_CLIENT: ''
2024-02-16 10:45:39 WARN i.a.m.l.MetricClientFactory(initialize):74 - MetricClient was not recognized or not provided. Accepted values are `datadog` or `otel`. 
2024-02-16 10:46:22 INFO i.a.a.SegmentAnalyticsClient(close):226 - Closing Segment analytics client...
2024-02-16 10:46:22 INFO i.a.a.BlockingShutdownAnalyticsPlugin(waitForFlush):281 - Waiting for Segment analytic client to flush enqueued messages...
2024-02-16 10:46:22 INFO i.a.a.BlockingShutdownAnalyticsPlugin(waitForFlush):293 - Segment analytic client flush complete.
2024-02-16 10:46:22 INFO i.a.a.SegmentAnalyticsClient(close):230 - Segment analytics client closed.  No new events will be accepted.
2024-02-16 10:45:40 replication-orchestrator > sourceLauncherConfig is: io.airbyte.persistence.job.models.IntegrationLauncherConfig@727dcc64[jobId=23,attemptId=4,connectionId=0e706487-7b0e-44e5-98b4-ba3fa3dfea44,workspaceId=edfa088e-e32a-462c-a6f6-07e7fc2cca36,dockerImage=airbyte/source-bing-ads:2.1.4,normalizationDockerImage=<null>,supportsDbt=false,normalizationIntegrationType=<null>,protocolVersion=Version{version='0.2.0', major='0', minor='2', patch='0'},isCustomConnector=false,allowedHosts=io.airbyte.config.AllowedHosts@49e62b17[hosts=[bingads.microsoft.com, login.microsoftonline.com, login.live.com, login.windows-ppe.net, ads.microsoft.com, api.ads.microsoft.com, clientcenter.api.bingads.microsoft.com, *.datadoghq.com, *.datadoghq.eu, *.sentry.io],additionalProperties={}],additionalEnvironmentVariables=<null>,additionalLabels=<null>,additionalProperties={}]
2024-02-16 10:45:40 replication-orchestrator > Attempt 0 to get the source definition for feature flag checks
2024-02-16 10:45:40 replication-orchestrator > Attempt 0 to get the source definition
2024-02-16 10:45:41 replication-orchestrator > Concurrent stream read enabled? false
2024-02-16 10:45:41 replication-orchestrator > Setting up source...
2024-02-16 10:45:41 replication-orchestrator > Using default value for environment variable SIDECAR_KUBE_MEMORY_LIMIT: '50Mi'
2024-02-16 10:45:41 replication-orchestrator > Using default value for environment variable SIDECAR_MEMORY_REQUEST: '25Mi'
2024-02-16 10:45:41 replication-orchestrator > Using default value for environment variable SIDECAR_KUBE_CPU_LIMIT: '2.0'
2024-02-16 10:45:41 replication-orchestrator > Using default value for environment variable SIDECAR_KUBE_CPU_REQUEST: '0.1'
2024-02-16 10:45:41 replication-orchestrator > Using default value for environment variable SIDECAR_KUBE_MEMORY_LIMIT: '50Mi'
2024-02-16 10:45:41 replication-orchestrator > Using default value for environment variable SIDECAR_MEMORY_REQUEST: '25Mi'
2024-02-16 10:45:41 replication-orchestrator > Using default value for environment variable SIDECAR_KUBE_CPU_LIMIT: '2.0'
2024-02-16 10:45:41 replication-orchestrator > Using default value for environment variable SIDECAR_KUBE_CPU_REQUEST: '0.1'
2024-02-16 10:45:41 replication-orchestrator > Using default value for environment variable SIDECAR_KUBE_MEMORY_LIMIT: '50Mi'
2024-02-16 10:45:41 replication-orchestrator > Using default value for environment variable SIDECAR_MEMORY_REQUEST: '25Mi'
2024-02-16 10:45:41 replication-orchestrator > Setting up destination...
2024-02-16 10:45:41 replication-orchestrator > Setting up replication worker...
2024-02-16 10:45:41 replication-orchestrator > Running replication worker...
2024-02-16 10:45:41 replication-orchestrator > start sync worker. job id: 23 attempt id: 4
2024-02-16 10:45:41 replication-orchestrator > 
2024-02-16 10:45:41 replication-orchestrator > ----- START REPLICATION -----
2024-02-16 10:45:41 replication-orchestrator > 
2024-02-16 10:45:41 replication-orchestrator > Using default value for environment variable SIDECAR_KUBE_CPU_LIMIT: '2.0'
2024-02-16 10:45:41 replication-orchestrator > Running destination...
2024-02-16 10:45:41 replication-orchestrator > Using default value for environment variable SIDECAR_KUBE_CPU_REQUEST: '0.1'
2024-02-16 10:45:41 replication-orchestrator > Using default value for environment variable LAUNCHDARKLY_KEY: ''
2024-02-16 10:45:41 replication-orchestrator > Using default value for environment variable FEATURE_FLAG_CLIENT: ''
2024-02-16 10:45:41 replication-orchestrator > Using default value for environment variable OTEL_COLLECTOR_ENDPOINT: ''
2024-02-16 10:45:41 replication-orchestrator > Using default value for environment variable SIDECAR_KUBE_CPU_LIMIT: '2.0'
2024-02-16 10:45:41 replication-orchestrator > Using default value for environment variable SIDECAR_KUBE_CPU_REQUEST: '0.1'
2024-02-16 10:45:41 replication-orchestrator > Using default value for environment variable LAUNCHDARKLY_KEY: ''
2024-02-16 10:45:41 replication-orchestrator > Using default value for environment variable FEATURE_FLAG_CLIENT: ''
2024-02-16 10:45:41 replication-orchestrator > Using default value for environment variable OTEL_COLLECTOR_ENDPOINT: ''
2024-02-16 10:45:41 replication-orchestrator > Attempting to start pod = destination-redshift-write-23-4-haodd for airbyte/destination-redshift:2.1.7 with resources ConnectorResourceRequirements[main=io.airbyte.config.ResourceRequirements@3b724f73[cpuRequest=1000m,cpuLimit=1000m,memoryRequest=2Gi,memoryLimit=2Gi,additionalProperties={}], heartbeat=io.airbyte.config.ResourceRequirements@419ce7a0[cpuRequest=0.05,cpuLimit=0.2,memoryRequest=25Mi,memoryLimit=50Mi,additionalProperties={}], stdErr=io.airbyte.config.ResourceRequirements@65ce152b[cpuRequest=0.01,cpuLimit=0.5,memoryRequest=25Mi,memoryLimit=50Mi,additionalProperties={}], stdIn=io.airbyte.config.ResourceRequirements@53d6003d[cpuRequest=0.1,cpuLimit=1,memoryRequest=25Mi,memoryLimit=50Mi,additionalProperties={}], stdOut=io.airbyte.config.ResourceRequirements@7af6c4e3[cpuRequest=0.01,cpuLimit=0.5,memoryRequest=25Mi,memoryLimit=50Mi,additionalProperties={}]] and allowedHosts null
2024-02-16 10:45:41 replication-orchestrator > Attempting to start pod = source-bing-ads-read-23-4-qmdqm for airbyte/source-bing-ads:2.1.4 with resources ConnectorResourceRequirements[main=io.airbyte.config.ResourceRequirements@1307c9af[cpuRequest=1000m,cpuLimit=1000m,memoryRequest=2Gi,memoryLimit=2Gi,additionalProperties={}], heartbeat=io.airbyte.config.ResourceRequirements@419ce7a0[cpuRequest=0.05,cpuLimit=0.2,memoryRequest=25Mi,memoryLimit=50Mi,additionalProperties={}], stdErr=io.airbyte.config.ResourceRequirements@5a363cc0[cpuRequest=0.01,cpuLimit=0.5,memoryRequest=25Mi,memoryLimit=50Mi,additionalProperties={}], stdIn=null, stdOut=io.airbyte.config.ResourceRequirements@48ecca8a[cpuRequest=0.2,cpuLimit=1,memoryRequest=25Mi,memoryLimit=50Mi,additionalProperties={}]] and allowedHosts io.airbyte.config.AllowedHosts@49e62b17[hosts=[bingads.microsoft.com, login.microsoftonline.com, login.live.com, login.windows-ppe.net, ads.microsoft.com, api.ads.microsoft.com, clientcenter.api.bingads.microsoft.com, *.datadoghq.com, *.datadoghq.eu, *.sentry.io],additionalProperties={}]
2024-02-16 10:45:41 replication-orchestrator > destination-redshift-write-23-4-haodd stdoutLocalPort = 9879
2024-02-16 10:45:41 replication-orchestrator > source-bing-ads-read-23-4-qmdqm stdoutLocalPort = 9878
2024-02-16 10:45:41 replication-orchestrator > destination-redshift-write-23-4-haodd stderrLocalPort = 9880
2024-02-16 10:45:41 replication-orchestrator > source-bing-ads-read-23-4-qmdqm stderrLocalPort = 9877
2024-02-16 10:45:41 replication-orchestrator > Using default value for environment variable SYNC_JOB_INIT_RETRY_TIMEOUT_MINUTES: '5'
2024-02-16 10:45:41 replication-orchestrator > Creating stdout socket server...
2024-02-16 10:45:41 replication-orchestrator > Creating stdout socket server...
2024-02-16 10:45:41 replication-orchestrator > Creating stderr socket server...
2024-02-16 10:45:41 replication-orchestrator > Creating stderr socket server...
2024-02-16 10:45:41 replication-orchestrator > Creating pod source-bing-ads-read-23-4-qmdqm...
2024-02-16 10:45:41 replication-orchestrator > Creating pod destination-redshift-write-23-4-haodd...
2024-02-16 10:45:42 replication-orchestrator > Waiting for init container to be ready before copying files...
2024-02-16 10:45:42 replication-orchestrator > Waiting for init container to be ready before copying files...
2024-02-16 10:46:00 replication-orchestrator > Init container ready..
2024-02-16 10:46:00 replication-orchestrator > Copying files...
2024-02-16 10:46:00 replication-orchestrator > Uploading file: destination_config.json
2024-02-16 10:46:00 replication-orchestrator > kubectl cp /tmp/f1cc64a8-180c-43a9-823f-0eb7c2808c0b/destination_config.json airbyte/destination-redshift-write-23-4-haodd:/config/destination_config.json -c init --retries=3
2024-02-16 10:46:00 replication-orchestrator > Waiting for kubectl cp to complete
2024-02-16 10:46:00 replication-orchestrator > (pod: airbyte / destination-redshift-write-23-4-haodd) - Destroying Kube process.
2024-02-16 10:46:00 replication-orchestrator > (pod: airbyte / destination-redshift-write-23-4-haodd) - Closed all resources for pod
2024-02-16 10:46:00 replication-orchestrator > (pod: airbyte / destination-redshift-write-23-4-haodd) - Destroyed Kube process.
2024-02-16 10:46:22 replication-orchestrator > Init container ready..
2024-02-16 10:46:22 replication-orchestrator > Copying files...
2024-02-16 10:46:22 replication-orchestrator > Uploading file: source_config.json
2024-02-16 10:46:22 replication-orchestrator > kubectl cp /tmp/e0491f98-2d7d-4f2e-b308-caf1d46948be/source_config.json airbyte/source-bing-ads-read-23-4-qmdqm:/config/source_config.json -c init --retries=3
2024-02-16 10:46:22 replication-orchestrator > Waiting for kubectl cp to complete
2024-02-16 10:46:22 replication-orchestrator > (pod: airbyte / source-bing-ads-read-23-4-qmdqm) - Destroying Kube process.
2024-02-16 10:46:22 replication-orchestrator > (pod: airbyte / source-bing-ads-read-23-4-qmdqm) - Closed all resources for pod
2024-02-16 10:46:22 replication-orchestrator > (pod: airbyte / source-bing-ads-read-23-4-qmdqm) - Destroyed Kube process.
2024-02-16 10:46:22 replication-orchestrator > thread status... timeout thread: false , replication thread: true
2024-02-16 10:46:22 replication-orchestrator > sync summary: {
  "status" : "failed",
  "startTime" : 1708080341140,
  "endTime" : 1708080382146,
  "totalStats" : {
    "bytesEmitted" : 0,
    "destinationStateMessagesEmitted" : 0,
    "destinationWriteEndTime" : 0,
    "destinationWriteStartTime" : 1708080341147,
    "meanSecondsBeforeSourceStateMessageEmitted" : 0,
    "maxSecondsBeforeSourceStateMessageEmitted" : 0,
    "meanSecondsBetweenStateMessageEmittedandCommitted" : 0,
    "recordsEmitted" : 0,
    "replicationEndTime" : 1708080382144,
    "replicationStartTime" : 1708080341140,
    "sourceReadEndTime" : 0,
    "sourceReadStartTime" : 1708080341148,
    "sourceStateMessagesEmitted" : 0
  },
  "streamStats" : [ ],
  "performanceMetrics" : {
    "processFromSource" : {
      "elapsedTimeInNanos" : 0,
      "executionCount" : 0,
      "avgExecTimeInNanos" : "NaN"
    },
    "readFromSource" : {
      "elapsedTimeInNanos" : 0,
      "executionCount" : 0,
      "avgExecTimeInNanos" : "NaN"
    },
    "processFromDest" : {
      "elapsedTimeInNanos" : 0,
      "executionCount" : 0,
      "avgExecTimeInNanos" : "NaN"
    },
    "writeToDest" : {
      "elapsedTimeInNanos" : 0,
      "executionCount" : 0,
      "avgExecTimeInNanos" : "NaN"
    },
    "readFromDest" : {
      "elapsedTimeInNanos" : 0,
      "executionCount" : 0,
      "avgExecTimeInNanos" : "NaN"
    }
  }
}
2024-02-16 10:46:22 replication-orchestrator > failures: [ {
  "failureOrigin" : "replication",
  "internalMessage" : "io.airbyte.workers.exception.WorkerException: Failed to create pod for write step",
  "externalMessage" : "Something went wrong during replication",
  "metadata" : {
    "attemptNumber" : 4,
    "jobId" : 23
  },
  "stacktrace" : "java.lang.RuntimeException: io.airbyte.workers.exception.WorkerException: Failed to create pod for write step\n\tat io.airbyte.workers.general.ReplicationWorkerHelper.startDestination(ReplicationWorkerHelper.kt:196)\n\tat io.airbyte.workers.general.BufferedReplicationWorker.lambda$run$0(BufferedReplicationWorker.java:152)\n\tat io.airbyte.workers.general.BufferedReplicationWorker.lambda$runAsync$2(BufferedReplicationWorker.java:228)\n\tat java.base/java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1804)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)\n\tat java.base/java.lang.Thread.run(Thread.java:1583)\nCaused by: io.airbyte.workers.exception.WorkerException: Failed to create pod for write step\n\tat io.airbyte.workers.process.KubeProcessFactory.create(KubeProcessFactory.java:197)\n\tat io.airbyte.workers.process.AirbyteIntegrationLauncher.write(AirbyteIntegrationLauncher.java:264)\n\tat io.airbyte.workers.internal.DefaultAirbyteDestination.start(DefaultAirbyteDestination.java:101)\n\tat io.airbyte.workers.general.ReplicationWorkerHelper.startDestination(ReplicationWorkerHelper.kt:194)\n\t... 6 more\nCaused by: java.lang.RuntimeException: java.io.IOException: kubectl cp failed with exit code 1\n\tat io.airbyte.workers.process.KubePodProcess.copyFilesToKubeConfigVolume(KubePodProcess.java:356)\n\tat io.airbyte.workers.process.KubePodProcess.<init>(KubePodProcess.java:655)\n\tat io.airbyte.workers.process.KubeProcessFactory.create(KubeProcessFactory.java:193)\n\t... 9 more\nCaused by: java.io.IOException: kubectl cp failed with exit code 1\n\tat io.airbyte.workers.process.KubePodProcess.copyFilesToKubeConfigVolume(KubePodProcess.java:350)\n\t... 11 more\n",
  "timestamp" : 1708080360954
}, {
  "failureOrigin" : "replication",
  "internalMessage" : "io.airbyte.workers.exception.WorkerException: Failed to create pod for read step",
  "externalMessage" : "Something went wrong during replication",
  "metadata" : {
    "attemptNumber" : 4,
    "jobId" : 23
  },
  "stacktrace" : "java.lang.RuntimeException: io.airbyte.workers.exception.WorkerException: Failed to create pod for read step\n\tat io.airbyte.workers.general.ReplicationWorkerHelper.startSource(ReplicationWorkerHelper.kt:214)\n\tat io.airbyte.workers.general.BufferedReplicationWorker.lambda$run$1(BufferedReplicationWorker.java:153)\n\tat io.airbyte.workers.general.BufferedReplicationWorker.lambda$runAsync$2(BufferedReplicationWorker.java:228)\n\tat java.base/java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1804)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)\n\tat java.base/java.lang.Thread.run(Thread.java:1583)\nCaused by: io.airbyte.workers.exception.WorkerException: Failed to create pod for read step\n\tat io.airbyte.workers.process.KubeProcessFactory.create(KubeProcessFactory.java:197)\n\tat io.airbyte.workers.process.AirbyteIntegrationLauncher.read(AirbyteIntegrationLauncher.java:226)\n\tat io.airbyte.workers.internal.DefaultAirbyteSource.start(DefaultAirbyteSource.java:84)\n\tat io.airbyte.workers.general.ReplicationWorkerHelper.startSource(ReplicationWorkerHelper.kt:212)\n\t... 6 more\nCaused by: java.lang.RuntimeException: java.io.IOException: kubectl cp failed with exit code 1\n\tat io.airbyte.workers.process.KubePodProcess.copyFilesToKubeConfigVolume(KubePodProcess.java:356)\n\tat io.airbyte.workers.process.KubePodProcess.<init>(KubePodProcess.java:655)\n\tat io.airbyte.workers.process.KubeProcessFactory.create(KubeProcessFactory.java:193)\n\t... 9 more\nCaused by: java.io.IOException: kubectl cp failed with exit code 1\n\tat io.airbyte.workers.process.KubePodProcess.copyFilesToKubeConfigVolume(KubePodProcess.java:350)\n\t... 11 more\n",
  "timestamp" : 1708080382140
} ]
2024-02-16 10:46:22 replication-orchestrator > Returning output...
2024-02-16 10:46:22 replication-orchestrator > 
2024-02-16 10:46:22 replication-orchestrator > ----- END REPLICATION -----
2024-02-16 10:46:22 replication-orchestrator > 
2024-02-16 10:46:22 replication-orchestrator > Writing async status SUCCEEDED for KubePodInfo[namespace=airbyte, name=orchestrator-repl-job-23-attempt-4, mainContainerInfo=KubeContainerInfo[image=airbyte/container-orchestrator:0.50.48, pullPolicy=IfNotPresent]]...
2024-02-16 10:46:23 platform > State Store reports orchestrator pod orchestrator-repl-job-23-attempt-4 succeeded
2024-02-16 10:46:23 platform > Retry State: RetryManager(completeFailureBackoffPolicy=BackoffPolicy(minInterval=PT10S, maxInterval=PT30M, base=3), partialFailureBackoffPolicy=null, successiveCompleteFailureLimit=5, totalCompleteFailureLimit=10, successivePartialFailureLimit=1000, totalPartialFailureLimit=10, successiveCompleteFailures=5, totalCompleteFailures=5, successivePartialFailures=0, totalPartialFailures=0)
 Backoff before next attempt: 13 minutes 30 seconds
2024-02-16 10:46:23 platform > Failing job: 23, reason: Job failed after too many retries for connection 0e706487-7b0e-44e5-98b4-ba3fa3dfea44

alexremn · 2024-02-21T13:12:51Z

alexremn
Feb 21, 2024
Author

Good day, colleagues!
Any news on this?

0 replies

alexremn · 2024-03-06T13:17:39Z

alexremn
Mar 6, 2024
Author

@marcosmarxm good day. Any news?

0 replies

aakashg7 · 2024-04-08T06:29:38Z

aakashg7
Apr 8, 2024

Hey guys any update on this?

0 replies

GideonStowell · 2024-08-28T15:19:02Z

GideonStowell
Aug 28, 2024

@alexremn Did you ever find a workaround or solution to this? I am running in what I think is the same issue on an identical setup:
OSS Airbyte on EKS via helm
External postres database (rds)
Karpenter spinning up new nodes for airbyte jobs

I see errors like this on the orchestrator-repl-job pod:

2024-08-28 14:42:09 replication-orchestrator > Waiting for kubectl cp to complete
2024-08-28 14:42:09 replication-orchestrator > (pod: airbyte / destination-bigquery-write-6902-0-ljmra) - Destroying Kube process.
2024-08-28 14:42:09 replication-orchestrator > (pod: airbyte / destination-bigquery-write-6902-0-ljmra) - Closed all resources for pod
2024-08-28 14:42:09 replication-orchestrator > (pod: airbyte / destination-bigquery-write-6902-0-ljmra) - Destroyed Kube process.
java.net.SocketException: Socket closed
	at java.base/sun.nio.ch.NioSocketImpl.endAccept(NioSocketImpl.java:682)
	at java.base/sun.nio.ch.NioSocketImpl.accept(NioSocketImpl.java:755)
	at java.base/java.net.ServerSocket.implAccept(ServerSocket.java:698)
	at java.base/java.net.ServerSocket.platformImplAccept(ServerSocket.java:663)
	at java.base/java.net.ServerSocket.implAccept(ServerSocket.java:639)
	at java.base/java.net.ServerSocket.implAccept(ServerSocket.java:585)
	at java.base/java.net.ServerSocket.accept(ServerSocket.java:543)
	at io.airbyte.workers.process.KubePodProcess.lambda$setupStdOutAndStdErrListeners$10(KubePodProcess.java:706)
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
	at java.base/java.lang.Thread.run(Thread.java:1583)
java.net.SocketException: Socket closed
	at java.base/sun.nio.ch.NioSocketImpl.endAccept(NioSocketImpl.java:682)
	at java.base/sun.nio.ch.NioSocketImpl.accept(NioSocketImpl.java:755)
	at java.base/java.net.ServerSocket.implAccept(ServerSocket.java:698)
	at java.base/java.net.ServerSocket.platformImplAccept(ServerSocket.java:663)
	at java.base/java.net.ServerSocket.implAccept(ServerSocket.java:639)
	at java.base/java.net.ServerSocket.implAccept(ServerSocket.java:585)
	at java.base/java.net.ServerSocket.accept(ServerSocket.java:543)
	at io.airbyte.workers.process.KubePodProcess.lambda$setupStdOutAndStdErrListeners$11(KubePodProcess.java:724)
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
	at java.base/java.lang.Thread.run(Thread.java:1583)

0 replies

alexremn · 2024-08-31T09:23:36Z

alexremn
Aug 31, 2024
Author

Good day, @GideonStowell !
I ended up with static nodes for airbyte workers, but faced with tolerations issue for orchestrator (another issue).
Though, in versions i had problems with workers startup, i had lots of extra envs to help, currently i mostly use default setup.

1 reply

GideonStowell Sep 3, 2024

Thanks for responding!
Do you mind describing the orchestrator toleration issue you experienced?

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Airbyte OSS via Helm on EKS: sync jobs are failing constantly #35301

{{title}}

Replies: 7 comments 8 replies

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

Select a reply

Airbyte OSS via Helm on EKS: sync jobs are failing constantly #35301

alexremn Feb 14, 2024

Replies: 7 comments · 8 replies

marcosmarxm Feb 14, 2024 Maintainer

marcosmarxm Feb 14, 2024 Maintainer

alexremn Feb 14, 2024 Author

marcosmarxm Feb 15, 2024 Maintainer

marcosmarxm Feb 15, 2024 Maintainer

alexremn Feb 15, 2024 Author

alexremn Feb 16, 2024 Author

alexremn Feb 16, 2024 Author

alexremn Feb 21, 2024 Author

alexremn Mar 6, 2024 Author

aakashg7 Apr 8, 2024

GideonStowell Aug 28, 2024

alexremn Aug 31, 2024 Author

GideonStowell Sep 3, 2024

alexremn
Feb 14, 2024

Replies: 7 comments 8 replies

marcosmarxm
Feb 14, 2024
Maintainer

marcosmarxm Feb 14, 2024
Maintainer

alexremn Feb 14, 2024
Author

marcosmarxm Feb 15, 2024
Maintainer

marcosmarxm Feb 15, 2024
Maintainer

alexremn Feb 15, 2024
Author

alexremn
Feb 16, 2024
Author

alexremn Feb 16, 2024
Author

alexremn
Feb 21, 2024
Author

alexremn
Mar 6, 2024
Author

aakashg7
Apr 8, 2024

GideonStowell
Aug 28, 2024

alexremn
Aug 31, 2024
Author