[helm] S3 Exceptions: Wrong region and malformed authorization header

[joeybenamy]

Helm Chart Version

1.2.0

What step the error happened?

During the Sync

Relevant information

With Airbyte 1.2.0 and Helm chart 1.2.0, syncs and checks on sources and destinations fail with an HTTP 504. Relevant Helm values:

global:
  storage:
    bucket:
      log: airbyte-logs-uat
      state: airbyte-logs-uat
    s3:
      region: us-east-1

With the Helm chart configured to use S3 storage for logs and state, starting any sync or running a check for any connector causes the below exception. Example:

1. In Airbyte UI, go to Destinations
2. Select the Snowflake connector
3. Under Settings go to "Test the destination"
4. Click "Retest saved destination"
5. Test fails with error An unknown error occurred. (HTTP 504)

Relevant log output

connector-sidecar 2024-11-07 18:27:51,123 [main]    ERROR    i.a.c.ConnectorWatcher(handleException):177 - Error performing operation: io.airbyte.workers.workload.exception.DocStoreAccessException
connector-sidecar io.airbyte.workers.workload.exception.DocStoreAccessException: Unable to write output for 424892c4-daac-4491-b35d-c6688ba547ba_54a33d37-88f3-4af7-b16f-91af2df2fb40_0_check
connector-sidecar     at io.airbyte.workers.workload.JobOutputDocStore.writeOutput(JobOutputDocStore.kt:72)
connector-sidecar     at io.airbyte.workers.workload.JobOutputDocStore.write(JobOutputDocStore.kt:38)
connector-sidecar     at io.airbyte.connectorSidecar.ConnectorWatcher.saveConnectorOutput(ConnectorWatcher.kt:165)
connector-sidecar     at io.airbyte.connectorSidecar.ConnectorWatcher.run(ConnectorWatcher.kt:74)
connector-sidecar     at io.airbyte.connectorSidecar.$ConnectorWatcher$Definition.initialize(Unknown Source)
connector-sidecar     at io.airbyte.connectorSidecar.$ConnectorWatcher$Definition.instantiate(Unknown Source)
connector-sidecar     at io.micronaut.context.DefaultBeanContext.resolveByBeanFactory(DefaultBeanContext.java:2330)
connector-sidecar     at io.micronaut.context.DefaultBeanContext.doCreateBean(DefaultBeanContext.java:2300)
connector-sidecar     at io.micronaut.context.DefaultBeanContext.doCreateBean(DefaultBeanContext.java:2312)
connector-sidecar     at io.micronaut.context.DefaultBeanContext.createRegistration(DefaultBeanContext.java:3123)
connector-sidecar     at io.micronaut.context.SingletonScope.getOrCreate(SingletonScope.java:80)
connector-sidecar     at io.micronaut.context.DefaultBeanContext.findOrCreateSingletonBeanRegistration(DefaultBeanContext.java:3025)
connector-sidecar     at io.micronaut.context.DefaultBeanContext.initializeEagerBean(DefaultBeanContext.java:2698)
connector-sidecar     at io.micronaut.context.DefaultBeanContext.initializeContext(DefaultBeanContext.java:1990)
connector-sidecar     at io.micronaut.context.DefaultApplicationContext.initializeContext(DefaultApplicationContext.java:307)
connector-sidecar     at io.micronaut.context.DefaultBeanContext.configureAndStartContext(DefaultBeanContext.java:3314)
connector-sidecar     at io.micronaut.context.DefaultBeanContext.start(DefaultBeanContext.java:345)
connector-sidecar     at io.micronaut.context.DefaultApplicationContext.start(DefaultApplicationContext.java:215)
connector-sidecar     at io.micronaut.context.ApplicationContextBuilder.start(ApplicationContextBuilder.java:286)
connector-sidecar     at io.airbyte.connectorSidecar.ApplicationKt.main(Application.kt:16)
connector-sidecar     at io.airbyte.connectorSidecar.ApplicationKt.main(Application.kt)
connector-sidecar Caused by: software.amazon.awssdk.services.s3.model.S3Exception: The authorization header is malformed; the region 'us-east-1' is wrong; expecting 'eu-west-1' (Service: S3, Status Code: 400, Request ID: MRAX1XEYMWCP7HGK, Extended Request ID: b8msqNSMDnZE28rVdes2W7eX/ScYpaq/HxnE3o315ABm9rByXNY5nYAH1+NPYDagVXnUi5MeTNM=)
connector-sidecar     at software.amazon.awssdk.protocols.xml.internal.unmarshall.AwsXmlPredicatedResponseHandler.handleErrorResponse(AwsXmlPredicatedResponseHandler.java:156)
connector-sidecar     at software.amazon.awssdk.protocols.xml.internal.unmarshall.AwsXmlPredicatedResponseHandler.handleResponse(AwsXmlPredicatedResponseHandler.java:108)
connector-sidecar     at software.amazon.awssdk.protocols.xml.internal.unmarshall.AwsXmlPredicatedResponseHandler.handle(AwsXmlPredicatedResponseHandler.java:85)
connector-sidecar     at software.amazon.awssdk.protocols.xml.internal.unmarshall.AwsXmlPredicatedResponseHandler.handle(AwsXmlPredicatedResponseHandler.java:43)
connector-sidecar     at software.amazon.awssdk.awscore.client.handler.AwsSyncClientHandler$Crc32ValidationResponseHandler.handle(AwsSyncClientHandler.java:93)
connector-sidecar     at software.amazon.awssdk.core.internal.handler.BaseClientHandler.lambda$successTransformationResponseHandler$7(BaseClientHandler.java:279)
connector-sidecar     at software.amazon.awssdk.core.internal.http.pipeline.stages.HandleResponseStage.execute(HandleResponseStage.java:50)
connector-sidecar     at software.amazon.awssdk.core.internal.http.pipeline.stages.HandleResponseStage.execute(HandleResponseStage.java:38)
connector-sidecar     at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
connector-sidecar     at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptTimeoutTrackingStage.execute(ApiCallAttemptTimeoutTrackingStage.java:74)
connector-sidecar     at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptTimeoutTrackingStage.execute(ApiCallAttemptTimeoutTrackingStage.java:43)
connector-sidecar     at software.amazon.awssdk.core.internal.http.pipeline.stages.TimeoutExceptionHandlingStage.execute(TimeoutExceptionHandlingStage.java:79)
connector-sidecar     at software.amazon.awssdk.core.internal.http.pipeline.stages.TimeoutExceptionHandlingStage.execute(TimeoutExceptionHandlingStage.java:41)
connector-sidecar     at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptMetricCollectionStage.execute(ApiCallAttemptMetricCollectionStage.java:55)
connector-sidecar     at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptMetricCollectionStage.execute(ApiCallAttemptMetricCollectionStage.java:39)
connector-sidecar     at software.amazon.awssdk.core.internal.http.pipeline.stages.RetryableStage2.executeRequest(RetryableStage2.java:93)
connector-sidecar     at software.amazon.awssdk.core.internal.http.pipeline.stages.RetryableStage2.execute(RetryableStage2.java:56)
connector-sidecar     at software.amazon.awssdk.core.internal.http.pipeline.stages.RetryableStage2.execute(RetryableStage2.java:36)
connector-sidecar     at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
connector-sidecar     at software.amazon.awssdk.core.internal.http.StreamManagingStage.execute(StreamManagingStage.java:53)
connector-sidecar     at software.amazon.awssdk.core.internal.http.StreamManagingStage.execute(StreamManagingStage.java:35)
connector-sidecar     at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.executeWithTimer(ApiCallTimeoutTrackingStage.java:82)
connector-sidecar     at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.execute(ApiCallTimeoutTrackingStage.java:62)
connector-sidecar     at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.execute(ApiCallTimeoutTrackingStage.java:43)
connector-sidecar     at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallMetricCollectionStage.execute(ApiCallMetricCollectionStage.java:50)
connector-sidecar     at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallMetricCollectionStage.execute(ApiCallMetricCollectionStage.java:32)
connector-sidecar     at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
connector-sidecar     at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
connector-sidecar     at software.amazon.awssdk.core.internal.http.pipeline.stages.ExecutionFailureExceptionReportingStage.execute(ExecutionFailureExceptionReportingStage.java:37)
connector-sidecar     at software.amazon.awssdk.core.internal.http.pipeline.stages.ExecutionFailureExceptionReportingStage.execute(ExecutionFailureExceptionReportingStage.java:26)
connector-sidecar     at software.amazon.awssdk.core.internal.http.AmazonSyncHttpClient$RequestExecutionBuilderImpl.execute(AmazonSyncHttpClient.java:210)
connector-sidecar     at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.invoke(BaseSyncClientHandler.java:103)
connector-sidecar     at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.doExecute(BaseSyncClientHandler.java:173)
connector-sidecar     at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.lambda$execute$1(BaseSyncClientHandler.java:80)
connector-sidecar     at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.measureApiCallSuccess(BaseSyncClientHandler.java:182)
connector-sidecar     at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.execute(BaseSyncClientHandler.java:74)
connector-sidecar     at software.amazon.awssdk.core.client.handler.SdkSyncClientHandler.execute(SdkSyncClientHandler.java:45)
connector-sidecar     at software.amazon.awssdk.awscore.client.handler.AwsSyncClientHandler.execute(AwsSyncClientHandler.java:53)
connector-sidecar     at software.amazon.awssdk.services.s3.DefaultS3Client.putObject(DefaultS3Client.java:10191)
connector-sidecar     at io.airbyte.commons.storage.AbstractS3StorageClient.write(StorageClient.kt:447)
connector-sidecar     at io.airbyte.workers.workload.JobOutputDocStore.writeOutput(JobOutputDocStore.kt:68)
connector-sidecar     ... 20 common frames omitted

[marcosmarxm]

@airbytehq/platform-deployments can someone take a look in this issue?

[marcosmarxm]

@joeybenamy can you confirm the bucket is in us-east-1 and not us-west-1?

connector-sidecar Caused by: software.amazon.awssdk.services.s3.model.S3Exception: 
**The authorization header is malformed; the region 'us-east-1' is wrong; expecting 'eu-west-1'** 
(Service: S3, Status Code: 400, Request ID: MRAX1XEYMWCP7HGK, Extended Request ID: b8msqNSMDnZE28rVdes2W7eX/ScYpaq/HxnE3o315ABm9rByXNY5nYAH1+NPYDagVXnUi5MeTNM=)

[joeybenamy]

@joeybenamy can you confirm the bucket is in us-east-1 and not us-west-1?

connector-sidecar Caused by: software.amazon.awssdk.services.s3.model.S3Exception: 
**The authorization header is malformed; the region 'us-east-1' is wrong; expecting 'eu-west-1'** 
(Service: S3, Status Code: 400, Request ID: MRAX1XEYMWCP7HGK, Extended Request ID: b8msqNSMDnZE28rVdes2W7eX/ScYpaq/HxnE3o315ABm9rByXNY5nYAH1+NPYDagVXnUi5MeTNM=)

The bucket is in us-east-1.

[joeybenamy]

Added some more detail @marcosmarxm