Datadog source setup error Configuration check failed

[BGireesh]

Error Title: Configuration check failed
Error Description: "Unable to connect to stream logs - time data '' does not match format '%Y-%m-%dT%H:%M:%SZ'"

2023-12-06 15:21:18 �[32mINFO�[m i.a.w.t.TemporalAttemptExecution(get):129 - Executing worker wrapper. Airbyte version: 0.50.34
2023-12-06 15:21:18 �[32mINFO�[m i.a.a.c.AirbyteApiClient(retryWithJitterThrows):290 - Attempt 0 to save workflow id for cancellation
2023-12-06 15:21:18 �[32mINFO�[m i.a.c.EnvConfigs(getEnvOrDefault):1158 - Using default value for environment variable SIDECAR_KUBE_CPU_LIMIT: '2.0'
2023-12-06 15:21:18 �[32mINFO�[m i.a.c.i.LineGobbler(voidCall):149 - 
2023-12-06 15:21:18 �[32mINFO�[m i.a.c.EnvConfigs(getEnvOrDefault):1158 - Using default value for environment variable SOCAT_KUBE_CPU_LIMIT: '2.0'
2023-12-06 15:21:18 �[32mINFO�[m i.a.c.EnvConfigs(getEnvOrDefault):1158 - Using default value for environment variable SIDECAR_KUBE_CPU_REQUEST: '0.1'
2023-12-06 15:21:18 �[32mINFO�[m i.a.c.i.LineGobbler(voidCall):149 - ----- START CHECK -----
2023-12-06 15:21:18 �[32mINFO�[m i.a.c.EnvConfigs(getEnvOrDefault):1158 - Using default value for environment variable SOCAT_KUBE_CPU_REQUEST: '0.1'
2023-12-06 15:21:18 �[32mINFO�[m i.a.c.EnvConfigs(getEnvOrDefault):1158 - Using default value for environment variable LAUNCHDARKLY_KEY: ''
2023-12-06 15:21:18 �[32mINFO�[m i.a.c.i.LineGobbler(voidCall):149 - 
2023-12-06 15:21:18 �[32mINFO�[m i.a.c.i.LineGobbler(voidCall):149 - Checking if airbyte/source-datadog:0.3.0 exists...
2023-12-06 15:21:18 �[32mINFO�[m i.a.c.i.LineGobbler(voidCall):149 - airbyte/source-datadog:0.3.0 was found locally.
2023-12-06 15:21:18 �[32mINFO�[m i.a.w.p.DockerProcessFactory(create):140 - Creating docker container = source-datadog-check-9c4a61d7-daaf-4534-9075-4a4b31a7a53d-0-qnrxv with resources io.airbyte.config.ResourceRequirements@5d4b469a[cpuRequest=,cpuLimit=,memoryRequest=,memoryLimit=,additionalProperties={}] and allowedHosts null
2023-12-06 15:21:18 �[32mINFO�[m i.a.w.p.DockerProcessFactory(create):187 - Preparing command: docker run --rm --init -i -w /data/9c4a61d7-daaf-4534-9075-4a4b31a7a53d/0 --log-driver none --name source-datadog-check-9c4a61d7-daaf-4534-9075-4a4b31a7a53d-0-qnrxv --network host -v airbyte_workspace:/data -v /tmp/airbyte_local:/local -e DEPLOYMENT_MODE=OSS -e WORKER_CONNECTOR_IMAGE=airbyte/source-datadog:0.3.0 -e AUTO_DETECT_SCHEMA=true -e LAUNCHDARKLY_KEY= -e SOCAT_KUBE_CPU_REQUEST=0.1 -e SOCAT_KUBE_CPU_LIMIT=2.0 -e FIELD_SELECTION_WORKSPACES= -e USE_STREAM_CAPABLE_STATE=true -e AIRBYTE_ROLE= -e WORKER_ENVIRONMENT=DOCKER -e APPLY_FIELD_SELECTION=false -e WORKER_JOB_ATTEMPT=0 -e OTEL_COLLECTOR_ENDPOINT=http://host.docker.internal:4317 -e FEATURE_FLAG_CLIENT=config -e AIRBYTE_VERSION=0.50.34 -e WORKER_JOB_ID=9c4a61d7-daaf-4534-9075-4a4b31a7a53d airbyte/source-datadog:0.3.0 check --config source_config.json
2023-12-06 15:21:18 �[32mINFO�[m i.a.w.i.VersionedAirbyteStreamFactory(create):186 - Reading messages from protocol version 0.2.0
2023-12-06 15:21:21 �[1;31mERROR�[m i.a.w.i.VersionedAirbyteStreamFactory(internalLog):313 - Encountered an error trying to connect to stream logs. Error: 
 Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/declarative/checks/check_stream.py", line 42, in check_connection
    stream_is_available, reason = availability_strategy.check_availability(stream, logger, source)
  File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/streams/http/availability_strategy.py", line 36, in check_availability
    stream_slice = get_first_stream_slice(stream)
  File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/streams/utils/stream_helper.py", line 21, in get_first_stream_slice
    stream.stream_slices(
  File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/declarative/declarative_stream.py", line 126, in stream_slices
    return self.retriever.stream_slices()
  File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/declarative/retrievers/simple_retriever.py", line 348, in stream_slices
    return self.stream_slicer.stream_slices()
  File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/declarative/incremental/datetime_based_cursor.py", line 140, in stream_slices
    end_datetime = self._select_best_end_datetime()
  File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/declarative/incremental/datetime_based_cursor.py", line 154, in _select_best_end_datetime
    return min(self.end_datetime.get_datetime(self.config), now)
  File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/declarative/datetime/min_max_datetime.py", line 58, in get_datetime
    time = self._parser.parse(str(self.datetime.eval(config, **additional_parameters)), datetime_format)
  File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/declarative/datetime/datetime_parser.py", line 33, in parse
    parsed_datetime = datetime.datetime.strptime(str(date), format)
  File "/usr/local/lib/python3.9/_strptime.py", line 568, in _strptime_datetime
    tt, fraction, gmtoff_fraction = _strptime(data_string, format)
  File "/usr/local/lib/python3.9/_strptime.py", line 349, in _strptime
    raise ValueError("time data %r does not match format %r" %
ValueError: time data '' does not match format '%Y-%m-%dT%H:%M:%SZ'

2023-12-06 15:21:21 �[1;31mERROR�[m i.a.w.i.VersionedAirbyteStreamFactory(internalLog):313 - Check failed
2023-12-06 15:21:21 �[32mINFO�[m i.a.w.g.DefaultCheckConnectionWorker(run):117 - Check connection job received output: io.airbyte.config.StandardCheckConnectionOutput@58389f3[status=failed,message="Unable to connect to stream logs - time data '' does not match format '%Y-%m-%dT%H:%M:%SZ'",additionalProperties={}]
2023-12-06 15:21:21 �[32mINFO�[m i.a.c.i.LineGobbler(voidCall):149 - 
2023-12-06 15:21:21 �[32mINFO�[m i.a.c.i.LineGobbler(voidCall):149 - ----- END CHECK -----
2023-12-06 15:21:21 �[32mINFO�[m i.a.c.i.LineGobbler(voidCall):149 - ```


Is this a known issue? How to troubleshoot and fix?

[sajarin]

hey @BGireesh looks like this is a bug with the incremental sync of the logs stream, can you confirm if other streams are facing the same issues?

[dlecocq]

I'm also experiencing the same issue when even establishing my source. With the DataDog source @ 0.2.2, I'm able to set up a connection, but with 0.3.0+, even testing the connection yields an error for me:

"Unable to connect to stream logs - time data '' does not match format '%Y-%m-%dT%H:%M:%SZ'"

Unfortunately, this is on Airbyte 0.42, and we've got a non-trivial upgrade path.

[dlecocq]

By way of an update, I no longer get that error if I also include both "Start date" and "End date" configurations, both of which are marked "optional."

[datasurfergtx]

Confirming I got this error using the Braintree connector using an incremental sync. Adding in a start date fixed the problem.
Airbyte v0.50.53
Braintree v0.2.1
Redshift v2.1.8

[ghatdev]

I'm facing same error:

2024-04-24 06:23:39 platform > Docker volume job log path: /tmp/workspace/b3ea5a7b-e860-4119-abe1-fe7622befda4/0/logs.log
2024-04-24 06:23:39 platform > Executing worker wrapper. Airbyte version: 0.57.3
2024-04-24 06:23:39 platform > Attempt 0 to save workflow id for cancellation
2024-04-24 06:23:39 platform > Using default value for environment variable SIDECAR_KUBE_CPU_LIMIT: '2.0'
2024-04-24 06:23:39 platform > Using default value for environment variable SOCAT_KUBE_CPU_LIMIT: '2.0'
2024-04-24 06:23:39 platform > Using default value for environment variable SIDECAR_KUBE_CPU_REQUEST: '0.1'
2024-04-24 06:23:39 platform > Using default value for environment variable SOCAT_KUBE_CPU_REQUEST: '0.1'
2024-04-24 06:23:39 platform >
2024-04-24 06:23:39 platform > ----- START CHECK -----
2024-04-24 06:23:39 platform >
2024-04-24 06:23:39 platform > Checking if airbyte/source-datadog:0.4.0 exists...
2024-04-24 06:23:39 platform > airbyte/source-datadog:0.4.0 was found locally.
2024-04-24 06:23:39 platform > Creating docker container = source-datadog-check-b3ea5a7b-e860-4119-abe1-fe7622befda4-0-teaii with resources io.airbyte.config.ResourceRequirements@401edeed[cpuRequest=,cpuLimit=,memoryRequest=,memoryLimit=,additionalProperties={}] and allowedHosts null
2024-04-24 06:23:39 platform > Preparing command: docker run --rm --init -i -w /data/b3ea5a7b-e860-4119-abe1-fe7622befda4/0 --log-driver none --name source-datadog-check-b3ea5a7b-e860-4119-abe1-fe7622befda4-0-teaii --network host -v airbyte_workspace:/data -v oss_local_root:/local -e DEPLOYMENT_MODE=OSS -e WORKER_CONNECTOR_IMAGE=airbyte/source-datadog:0.4.0 -e AUTO_DETECT_SCHEMA=true -e LAUNCHDARKLY_KEY= -e SOCAT_KUBE_CPU_REQUEST=0.1 -e SOCAT_KUBE_CPU_LIMIT=2.0 -e FIELD_SELECTION_WORKSPACES= -e USE_STREAM_CAPABLE_STATE=true -e AIRBYTE_ROLE=dev -e WORKER_ENVIRONMENT=DOCKER -e APPLY_FIELD_SELECTION=false -e WORKER_JOB_ATTEMPT=0 -e OTEL_COLLECTOR_ENDPOINT=http://host.docker.internal:4317 -e FEATURE_FLAG_CLIENT=config -e AIRBYTE_VERSION=0.57.3 -e WORKER_JOB_ID=b3ea5a7b-e860-4119-abe1-fe7622befda4 airbyte/source-datadog:0.4.0 check --config source_config.json
2024-04-24 06:23:39 platform > Reading messages from protocol version 0.2.0
2024-04-24 06:23:42 platform > Successfully connected to stream audit_logs, but got 0 records.
2024-04-24 06:23:44 platform > Successfully connected to stream downtimes, but got 0 records.
2024-04-24 06:23:45 platform > Successfully connected to stream incident_teams, but got 0 records.
2024-04-24 06:23:46 platform > Successfully connected to stream incidents, but got 0 records.
2024-04-24 06:23:46 platform > Encountered an error trying to connect to stream logs. Error:
Traceback (most recent call last):
File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/declarative/checks/check_stream.py", line 42, in check_connection

Airbyte OSS version: 0.57.3
Datadog connector version: 0.4.0

[sluo2024]

I encountered the same error.
any hints how we can work around it? thanks

2024-07-09 17:07:55 platform > Docker volume job log path: /tmp/workspace/1d4bccde-1a0d-444b-831c-b28be83943ac/0/logs.log
2024-07-09 17:07:55 platform > Executing worker wrapper. Airbyte version: 0.63.1
2024-07-09 17:07:55 platform > Using default value for environment variable SIDECAR_KUBE_CPU_LIMIT: '2.0'
2024-07-09 17:07:55 platform >
2024-07-09 17:07:55 platform > Using default value for environment variable SOCAT_KUBE_CPU_LIMIT: '2.0'
2024-07-09 17:07:55 platform > ----- START CHECK -----
2024-07-09 17:07:55 platform > Using default value for environment variable SIDECAR_KUBE_CPU_REQUEST: '0.1'
2024-07-09 17:07:55 platform >
2024-07-09 17:07:55 platform > Using default value for environment variable SOCAT_KUBE_CPU_REQUEST: '0.1'
2024-07-09 17:07:55 platform > Checking if airbyte/source-datadog:0.4.4 exists...
2024-07-09 17:07:55 platform > airbyte/source-datadog:0.4.4 was found locally.
2024-07-09 17:07:55 platform > Creating docker container = source-datadog-check-1d4bccde-1a0d-444b-831c-b28be83943ac-0-zikig with resources io.airbyte.config.ResourceRequirements@2ac11237[cpuRequest=,cpuLimit=,memoryRequest=,memoryLimit=,additionalProperties={}] and allowedHosts null
2024-07-09 17:07:55 platform > Preparing command: docker run --rm --init -i -w /data/1d4bccde-1a0d-444b-831c-b28be83943ac/0 --log-driver none --name source-datadog-check-1d4bccde-1a0d-444b-831c-b28be83943ac-0-zikig --network host -v airbyte_workspace:/data -v oss_local_root:/local -e DEPLOYMENT_MODE=OSS -e WORKER_CONNECTOR_IMAGE=airbyte/source-datadog:0.4.4 -e AUTO_DETECT_SCHEMA=true -e LAUNCHDARKLY_KEY= -e SOCAT_KUBE_CPU_REQUEST=0.1 -e SOCAT_KUBE_CPU_LIMIT=2.0 -e FIELD_SELECTION_WORKSPACES= -e USE_STREAM_CAPABLE_STATE=true -e AIRBYTE_ROLE=dev -e WORKER_ENVIRONMENT=DOCKER -e APPLY_FIELD_SELECTION=false -e WORKER_JOB_ATTEMPT=0 -e OTEL_COLLECTOR_ENDPOINT=http://host.docker.internal:4317 -e FEATURE_FLAG_CLIENT=config -e AIRBYTE_VERSION=0.63.1 -e WORKER_JOB_ID=1d4bccde-1a0d-444b-831c-b28be83943ac airbyte/source-datadog:0.4.4 check --config source_config.json
2024-07-09 17:07:55 platform > Reading messages from protocol version 0.2.0
2024-07-09 17:07:59 platform > Successfully connected to stream incident_teams, but got 0 records.
2024-07-09 17:07:59 platform > Encountered an error trying to connect to stream logs. Error:
Traceback (most recent call last):
File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/declarative/checks/check_stream.py", line 42, in check_connection
stream_is_available, reason = availability_strategy.check_availability(stream, logger, source)
File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/streams/http/availability_strategy.py", line 36, in check_availability
stream_slice = get_first_stream_slice(stream)
File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/streams/utils/stream_helper.py", line 21, in get_first_stream_slice