airbyte_source_custom configuration - never-ending drift

[robertomczak]

Recent PR #80 introduced airbyte_source_custom.

We are testing it with AIrbyte 0.50.45 (HELM deployed), provider version 0.4.1.

Terraform definition:

resource "airbyte_source_custom" "float_api" {
  name = "Float API"

  definition_id = "0c08da44-45a1-4668-b827-f42cafcf6497"
  workspace_id  = local.workspace_id

  configuration = jsonencode({
    api_key = local.float_api_key
  })
}

Above definition creates custom source correctly, unfortunately it is getting into unresorvable drift state.

Terraform will perform the following actions:

  # airbyte_source_custom.float_api will be updated in-place
  ~ resource "airbyte_source_custom" "float_api" {
      ~ configuration = (sensitive value)
        name          = "Float API"
        # (3 unchanged attributes hidden)
    }

Plan: 0 to add, 1 to change, 0 to destroy.
airbyte_source_custom.float_api: Modifying... [name=Float API]
airbyte_source_custom.float_api: Modifications complete after 0s [name=Float API]

Apply complete! Resources: 0 added, 1 changed, 0 destroyed.

Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
  ~ update in-place

Terraform will perform the following actions:

  # airbyte_source_custom.float_api will be updated in-place
  ~ resource "airbyte_source_custom" "float_api" {
      ~ configuration = (sensitive value)
        name          = "Float API"
        # (3 unchanged attributes hidden)
    }

Plan: 0 to add, 1 to change, 0 to destroy.

[robertomczak]

Looks that is might be an API that returns ***** instead of configuration value

[robertomczak]

@terencecho is that expected provider behavior?

[ThomasRooney]

Does the airbyte API mask api keys when they're returned in the configuration object? If so drift detection/determining of "if we need to update this field" unfortunately won't ever really work with an untyped SourceConfiguration object. We could fix this by writeOnly: true to the SourceConfiguration json schema to disable drift detection on that attribute.

[szemek]

@ThomasRooney yes, Airbyte API returns masked keys, tokens, credentials, etc. used by connector
https://github.com/airbytehq/airbyte/blob/5fc6ca213121305c5413b5d288e4205d8b4fddce/airbyte-cdk/java/airbyte-cdk/dependencies/src/main/java/io/airbyte/commons/constants/AirbyteSecretConstants.java#L20

As a workaround we use

  lifecycle {
    ignore_changes = [configuration]
  }

https://developer.hashicorp.com/terraform/language/meta-arguments/lifecycle#ignore_changes

[neurotroph]

We are facing the same issue and using a lot of custom sources in our connections. @szemek 's workaround works for now, but makes our CI/CD pipeline a bit problematic.

Is there any update? Anyone working on this issue?

[rwask]

Current Status and Solutions

The Airbyte team has identified the root cause and is exploring potential solutions, including enhancements for value-based object comparisons and sorting streams in API responses. In the interim, a temporary workaround involves using the ignore_changes lifecycle argument in Terraform, though this affects CI/CD pipelines. The community is awaiting a permanent fix from the provider team.

[szemek]

Current Status and Solutions

The Airbyte team has identified the root cause and is exploring potential solutions, including enhancements for value-based object comparisons and sorting streams in API responses. In the interim, a temporary workaround involves using the ignore_changes lifecycle argument in Terraform, though this affects CI/CD pipelines. The community is awaiting a permanent fix from the provider team.

@rwask fix for streams is here #111