Connect Iceberg to Azure Blob Storage (via s3proxy) #41668

trupus · 2024-07-12T15:24:25Z

trupus
Jul 12, 2024

I will first state my goals and current state of things, and then provide the solution I came up with and the problem with it. Please feel free to point me in a better direction.

Current state

I use azure blob storage to store json files and I need a way to query this data with SQL. I came upon the hive metastore, and later discovered the apache iceberg as a better alternative. While doing the research for the PoC, I learned that s3 is the de facto standard today, but azure blob storage doesn't support this protocol.

Solution I came up with

Use s3proxy to create an s3 interface for azure blob storage.
Add a spark with iceberg image and point it to the s3proxy instead of the minio.
Use airbyte to register iceberg as the destination.

This is my `docker-compose.yaml`

version: "3.8"
#https://github.com/compose-spec/compose-spec/blob/master/spec.md#using-extensions-as-fragments
x-logging: &default-logging
  options:
    max-size: "100m"
    max-file: "5"
  driver: json-file
services:
  docker-proxy:
      image: alpine/socat
      command: -t 900 TCP-LISTEN:2375,fork,reuseaddr UNIX-CONNECT:/var/run/docker.sock
      ports: 
        - "2375"
      user: root
      volumes: 
        - ${DOCKER_SOCKET:-/var/run/docker.sock}:/var/run/docker.sock
      networks:
        - airbyte_internal
  # hook in case we need to add init behavior
  # every root service (no depends_on) should depend on init
  init:
    image: airbyte/init:${VERSION}
    logging: *default-logging
    container_name: init
    command: /bin/sh -c "./scripts/create_mount_directories.sh /local_parent ${HACK_LOCAL_ROOT_PARENT} ${LOCAL_ROOT}"
    environment:
      - LOCAL_ROOT=${LOCAL_ROOT}
      - HACK_LOCAL_ROOT_PARENT=${HACK_LOCAL_ROOT_PARENT}
    volumes:
      - ${HACK_LOCAL_ROOT_PARENT}:/local_parent
  bootloader:
    image: airbyte/bootloader:${VERSION}
    logging: *default-logging
    container_name: airbyte-bootloader
    environment:
      - AIRBYTE_VERSION=${VERSION}
      - CONNECTOR_REGISTRY_BASE_URL=${CONNECTOR_REGISTRY_BASE_URL:-}
      - CONNECTOR_REGISTRY_SEED_PROVIDER=${CONNECTOR_REGISTRY_SEED_PROVIDER:-}
      - DATABASE_PASSWORD=${DATABASE_PASSWORD}
      - DATABASE_URL=${DATABASE_URL}
      - DATABASE_USER=${DATABASE_USER}
      - LOCAL_CONNECTOR_CATALOG_PATH=${LOCAL_CONNECTOR_CATALOG_PATH}
      - LOG_LEVEL=${LOG_LEVEL}
      - DD_LOGS_INJECTION=${DD_LOGS_INJECTION}
      - JAVA_OPTS=${JAVA_OPTS}
      - CONTAINER_ORCHESTRATOR_JAVA_OPTS=${CONTAINER_ORCHESTRATOR_JAVA_OPTS}
      - DD_DOGSTATSD_PORT=${DD_DOGSTATSD_PORT}
      - DD_TRACE_AGENT_PORT=${DD_TRACE_AGENT_PORT}
      - DD_AGENT_HOST=${DD_AGENT_HOST}
    networks:
      - airbyte_internal
    depends_on:
      init:
        condition: service_completed_successfully
  db:
    image: airbyte/db:${VERSION}
    logging: *default-logging
    container_name: airbyte-db
    restart: unless-stopped
    environment:
      - CONFIG_DATABASE_PASSWORD=${CONFIG_DATABASE_PASSWORD:-}
      - CONFIG_DATABASE_URL=${CONFIG_DATABASE_URL:-}
      - CONFIG_DATABASE_USER=${CONFIG_DATABASE_USER:-}
      - DATABASE_PASSWORD=${DATABASE_PASSWORD}
      - DATABASE_URL=${DATABASE_URL}
      - DATABASE_USER=${DATABASE_USER}
      - POSTGRES_PASSWORD=${DATABASE_PASSWORD}
      - POSTGRES_USER=${DATABASE_USER}
      - DD_LOGS_INJECTION=${DD_LOGS_INJECTION}
      - JAVA_OPTS=${JAVA_OPTS}
      - CONTAINER_ORCHESTRATOR_JAVA_OPTS=${CONTAINER_ORCHESTRATOR_JAVA_OPTS}
      - DD_DOGSTATSD_PORT=${DD_DOGSTATSD_PORT}
      - DD_TRACE_AGENT_PORT=${DD_TRACE_AGENT_PORT}
      - DD_AGENT_HOST=${DD_AGENT_HOST}
    volumes:
      - db:/var/lib/postgresql/data
    networks:
      - airbyte_internal
  worker:
    image: airbyte/worker:${VERSION}
    logging: *default-logging
    container_name: airbyte-worker
    restart: unless-stopped
    environment:
      - ACTIVITY_INITIAL_DELAY_BETWEEN_ATTEMPTS_SECONDS=${ACTIVITY_INITIAL_DELAY_BETWEEN_ATTEMPTS_SECONDS}
      - ACTIVITY_MAX_ATTEMPT=${ACTIVITY_MAX_ATTEMPT}
      - ACTIVITY_MAX_DELAY_BETWEEN_ATTEMPTS_SECONDS=${ACTIVITY_MAX_DELAY_BETWEEN_ATTEMPTS_SECONDS}
      - AIRBYTE_INSTALLATION_ID=${AIRBYTE_INSTALLATION_ID:-}
      - AIRBYTE_ROLE=${AIRBYTE_ROLE:-}
      - AIRBYTE_VERSION=${VERSION}
      - APPLY_FIELD_SELECTION=${APPLY_FIELD_SELECTION}
      - AUTO_DETECT_SCHEMA=${AUTO_DETECT_SCHEMA}
      - AUTO_DISABLE_FAILING_CONNECTIONS=${AUTO_DISABLE_FAILING_CONNECTIONS}
      - CONFIGS_DATABASE_MINIMUM_FLYWAY_MIGRATION_VERSION=${CONFIGS_DATABASE_MINIMUM_FLYWAY_MIGRATION_VERSION:-}
      - CONFIG_DATABASE_PASSWORD=${CONFIG_DATABASE_PASSWORD:-}
      - CONFIG_DATABASE_URL=${CONFIG_DATABASE_URL:-}
      - CONFIG_DATABASE_USER=${CONFIG_DATABASE_USER:-}
      - CONFIG_ROOT=${CONFIG_ROOT}
      - CONNECTOR_REGISTRY_BASE_URL=${CONNECTOR_REGISTRY_BASE_URL:-}
      - DATABASE_PASSWORD=${DATABASE_PASSWORD}
      - DATABASE_URL=${DATABASE_URL}
      - DATABASE_USER=${DATABASE_USER}
      - DEPLOYMENT_MODE=${DEPLOYMENT_MODE}
      - DD_AGENT_HOST=${DD_AGENT_HOST}
      - DD_DOGSTATSD_PORT=${DD_DOGSTATSD_PORT}
      - DOCKER_HOST=docker-proxy:2375
      - FEATURE_FLAG_CLIENT=${FEATURE_FLAG_CLIENT}
      - FIELD_SELECTION_WORKSPACES=${FIELD_SELECTION_WORKSPACES}
      - INTERNAL_API_HOST=${INTERNAL_API_HOST}
      - JOBS_DATABASE_MINIMUM_FLYWAY_MIGRATION_VERSION=${JOBS_DATABASE_MINIMUM_FLYWAY_MIGRATION_VERSION:-}
      - JOB_ERROR_REPORTING_SENTRY_DSN=${JOB_ERROR_REPORTING_SENTRY_DSN}
      - JOB_ERROR_REPORTING_STRATEGY=${JOB_ERROR_REPORTING_STRATEGY}
      - JOB_MAIN_CONTAINER_CPU_LIMIT=${JOB_MAIN_CONTAINER_CPU_LIMIT}
      - JOB_MAIN_CONTAINER_CPU_REQUEST=${JOB_MAIN_CONTAINER_CPU_REQUEST}
      - JOB_MAIN_CONTAINER_MEMORY_LIMIT=${JOB_MAIN_CONTAINER_MEMORY_LIMIT}
      - JOB_MAIN_CONTAINER_MEMORY_REQUEST=${JOB_MAIN_CONTAINER_MEMORY_REQUEST}
      - LAUNCHDARKLY_KEY=${LAUNCHDARKLY_KEY}
      - LOCAL_DOCKER_MOUNT=${LOCAL_DOCKER_MOUNT}
      - LOCAL_ROOT=${LOCAL_ROOT}
      - LOG_CONNECTOR_MESSAGES=${LOG_CONNECTOR_MESSAGES}
      - LOG_LEVEL=${LOG_LEVEL}
      - MAX_CHECK_WORKERS=${MAX_CHECK_WORKERS}
      - MAX_DISCOVER_WORKERS=${MAX_DISCOVER_WORKERS}
      - MAX_NOTIFY_WORKERS=${MAX_NOTIFY_WORKERS}
      - MAX_SPEC_WORKERS=${MAX_SPEC_WORKERS}
      - MAX_SYNC_WORKERS=${MAX_SYNC_WORKERS}
      - METRIC_CLIENT=${METRIC_CLIENT}
      - MICROMETER_METRICS_ENABLED=${MICROMETER_METRICS_ENABLED}
      - MICROMETER_METRICS_STATSD_FLAVOR=${MICROMETER_METRICS_STATSD_FLAVOR}
      - MICRONAUT_ENVIRONMENTS=${WORKERS_MICRONAUT_ENVIRONMENTS}
      - OTEL_COLLECTOR_ENDPOINT=${OTEL_COLLECTOR_ENDPOINT}
      - PUBLISH_METRICS=${PUBLISH_METRICS}
      - SECRET_PERSISTENCE=${SECRET_PERSISTENCE}
      - SEGMENT_WRITE_KEY=${SEGMENT_WRITE_KEY}
      - SHOULD_RUN_NOTIFY_WORKFLOWS=${SHOULD_RUN_NOTIFY_WORKFLOWS}
      - STATSD_HOST=${STATSD_HOST}
      - STATSD_PORT=${STATSD_PORT}
      - SYNC_JOB_INIT_RETRY_TIMEOUT_MINUTES=${SYNC_JOB_INIT_RETRY_TIMEOUT_MINUTES}
      - SYNC_JOB_MAX_ATTEMPTS=${SYNC_JOB_MAX_ATTEMPTS}
      - SYNC_JOB_MAX_TIMEOUT_DAYS=${SYNC_JOB_MAX_TIMEOUT_DAYS}
      - TEMPORAL_HOST=${TEMPORAL_HOST}
      - TRACKING_STRATEGY=${TRACKING_STRATEGY}
      - WEBAPP_URL=${WEBAPP_URL}
      - STORAGE_BUCKET_ACTIVITY_PAYLOAD=${STORAGE_BUCKET_ACTIVITY_PAYLOAD}
      - STORAGE_BUCKET_LOG=${STORAGE_BUCKET_LOG}
      - STORAGE_BUCKET_STATE=${STORAGE_BUCKET_STATE}
      - STORAGE_BUCKET_WORKLOAD_OUTPUT=${STORAGE_BUCKET_WORKLOAD_OUTPUT}
      - STORAGE_TYPE=${STORAGE_TYPE}
      - WORKFLOW_FAILURE_RESTART_DELAY_SECONDS=${WORKFLOW_FAILURE_RESTART_DELAY_SECONDS}
      - WORKLOAD_API_HOST=${WORKLOAD_API_URL}
      - WORKSPACE_DOCKER_MOUNT=${WORKSPACE_DOCKER_MOUNT}
      - WORKSPACE_ROOT=${WORKSPACE_ROOT}
      - DD_LOGS_INJECTION=${DD_LOGS_INJECTION}
      - JAVA_OPTS=${JAVA_OPTS}
      - CONTAINER_ORCHESTRATOR_JAVA_OPTS=${CONTAINER_ORCHESTRATOR_JAVA_OPTS}
      - DD_TRACE_AGENT_PORT=${DD_TRACE_AGENT_PORT}
    configs:
      - flags
    volumes:
      - workspace:${WORKSPACE_ROOT}
      - local_root:${LOCAL_ROOT}
    ports:
      - "9000"
    networks:
      - airbyte_internal
    depends_on:
      bootloader:
        condition: service_completed_successfully
  server:
    image: airbyte/server:${VERSION}
    logging: *default-logging
    container_name: airbyte-server
    restart: unless-stopped
    environment:
      - AIRBYTE_API_HOST=${AIRBYTE_API_HOST}
      - AIRBYTE_INSTALLATION_ID=${AIRBYTE_INSTALLATION_ID:-}
      - AIRBYTE_ROLE=${AIRBYTE_ROLE:-}
      - AIRBYTE_VERSION=${VERSION}
      - AUTO_DETECT_SCHEMA=${AUTO_DETECT_SCHEMA}
      - CONFIGS_DATABASE_MINIMUM_FLYWAY_MIGRATION_VERSION=${CONFIGS_DATABASE_MINIMUM_FLYWAY_MIGRATION_VERSION:-}
      - CONFIG_DATABASE_PASSWORD=${CONFIG_DATABASE_PASSWORD:-}
      - CONFIG_DATABASE_URL=${CONFIG_DATABASE_URL:-}
      - CONFIG_DATABASE_USER=${CONFIG_DATABASE_USER:-}
      - CONFIG_ROOT=${CONFIG_ROOT}
      - CONNECTOR_REGISTRY_BASE_URL=${CONNECTOR_REGISTRY_BASE_URL:-}
      - DATABASE_PASSWORD=${DATABASE_PASSWORD}
      - DATABASE_URL=${DATABASE_URL}
      - DATABASE_USER=${DATABASE_USER}
      - DD_AGENT_HOST=${DD_AGENT_HOST}
      - DD_DOGSTATSD_PORT=${DD_DOGSTATSD_PORT}
      - FEATURE_FLAG_CLIENT=${FEATURE_FLAG_CLIENT}
      - GITHUB_STORE_BRANCH=${GITHUB_STORE_BRANCH}
      - JOBS_DATABASE_MINIMUM_FLYWAY_MIGRATION_VERSION=${JOBS_DATABASE_MINIMUM_FLYWAY_MIGRATION_VERSION:-}
      - JOB_ERROR_REPORTING_SENTRY_DSN=${JOB_ERROR_REPORTING_SENTRY_DSN}
      - JOB_ERROR_REPORTING_STRATEGY=${JOB_ERROR_REPORTING_STRATEGY}
      - JOB_MAIN_CONTAINER_CPU_LIMIT=${JOB_MAIN_CONTAINER_CPU_LIMIT}
      - JOB_MAIN_CONTAINER_CPU_REQUEST=${JOB_MAIN_CONTAINER_CPU_REQUEST}
      - JOB_MAIN_CONTAINER_MEMORY_LIMIT=${JOB_MAIN_CONTAINER_MEMORY_LIMIT}
      - JOB_MAIN_CONTAINER_MEMORY_REQUEST=${JOB_MAIN_CONTAINER_MEMORY_REQUEST}
      - LAUNCHDARKLY_KEY=${LAUNCHDARKLY_KEY}
      - LOG_LEVEL=${LOG_LEVEL}
      - METRIC_CLIENT=${METRIC_CLIENT}
      - MAX_NOTIFY_WORKERS=5
      - MICROMETER_METRICS_ENABLED=${MICROMETER_METRICS_ENABLED}
      - MICROMETER_METRICS_STATSD_FLAVOR=${MICROMETER_METRICS_STATSD_FLAVOR}
      - MICRONAUT_ENVIRONMENTS=${SERVER_MICRONAUT_ENVIRONMENTS}
      - NEW_SCHEDULER=${NEW_SCHEDULER}
      - PUBLISH_METRICS=${PUBLISH_METRICS}
      - SECRET_PERSISTENCE=${SECRET_PERSISTENCE}
      - SEGMENT_WRITE_KEY=${SEGMENT_WRITE_KEY}
      - SHOULD_RUN_NOTIFY_WORKFLOWS=${SHOULD_RUN_NOTIFY_WORKFLOWS}
      - STATSD_HOST=${STATSD_HOST}
      - STATSD_PORT=${STATSD_PORT}
      - STORAGE_BUCKET_ACTIVITY_PAYLOAD=${STORAGE_BUCKET_ACTIVITY_PAYLOAD}
      - STORAGE_BUCKET_LOG=${STORAGE_BUCKET_LOG}
      - STORAGE_BUCKET_STATE=${STORAGE_BUCKET_STATE}
      - STORAGE_BUCKET_WORKLOAD_OUTPUT=${STORAGE_BUCKET_WORKLOAD_OUTPUT}
      - STORAGE_TYPE=${STORAGE_TYPE}
      - TEMPORAL_HOST=${TEMPORAL_HOST}
      - TRACKING_STRATEGY=${TRACKING_STRATEGY}
      - WEBAPP_URL=${WEBAPP_URL}
      - WORKER_ENVIRONMENT=${WORKER_ENVIRONMENT}
      - WORKSPACE_ROOT=${WORKSPACE_ROOT}
      - CONNECTOR_BUILDER_SERVER_API_HOST=${CONNECTOR_BUILDER_SERVER_API_HOST}
      - DD_LOGS_INJECTION=${DD_LOGS_INJECTION}
      - JAVA_OPTS=${JAVA_OPTS}
      - CONTAINER_ORCHESTRATOR_JAVA_OPTS=${CONTAINER_ORCHESTRATOR_JAVA_OPTS}
      - DD_TRACE_AGENT_PORT=${DD_TRACE_AGENT_PORT}
    ports:
      - "8001"
    configs:
      - flags
    volumes:
      - workspace:${WORKSPACE_ROOT}
      - data:${CONFIG_ROOT}
      - local_root:${LOCAL_ROOT}
      - ./configs:/app/configs:ro
    networks:
      - airbyte_internal
    depends_on:
      bootloader:
        condition: service_completed_successfully
  webapp:
    image: airbyte/webapp:${VERSION}
    logging: *default-logging
    container_name: airbyte-webapp
    restart: unless-stopped
    environment:
      - AIRBYTE_SERVER_HOST=${AIRBYTE_SERVER_HOST}
      - CONNECTOR_BUILDER_API_HOST=${CONNECTOR_BUILDER_API_HOST}
      - KEYCLOAK_INTERNAL_HOST=localhost # placeholder to ensure the webapp's nginx config is valid
    networks:
      - airbyte_internal
    depends_on:
      bootloader:
        condition: service_completed_successfully
  airbyte-temporal:
    image: airbyte/temporal:${VERSION}
    logging: *default-logging
    container_name: airbyte-temporal
    restart: unless-stopped
    environment:
      - DB=postgresql
      - DB_PORT=${DATABASE_PORT}
      - DYNAMIC_CONFIG_FILE_PATH=config/dynamicconfig/development.yaml
      - LOG_LEVEL=${LOG_LEVEL}
      - POSTGRES_PWD=${DATABASE_PASSWORD}
      - POSTGRES_SEEDS=${DATABASE_HOST}
      - POSTGRES_USER=${DATABASE_USER}
      - DD_LOGS_INJECTION=${DD_LOGS_INJECTION}
      - JAVA_OPTS=${JAVA_OPTS}
      - CONTAINER_ORCHESTRATOR_JAVA_OPTS=${CONTAINER_ORCHESTRATOR_JAVA_OPTS}
      - DD_DOGSTATSD_PORT=${DD_DOGSTATSD_PORT}
      - DD_TRACE_AGENT_PORT=${DD_TRACE_AGENT_PORT}
      - DD_AGENT_HOST=${DD_AGENT_HOST}
    volumes:
      - ./temporal/dynamicconfig:/etc/temporal/config/dynamicconfig
    networks:
      - airbyte_internal
  airbyte-cron:
    image: airbyte/cron:${VERSION}
    logging: *default-logging
    container_name: airbyte-cron
    restart: unless-stopped
    environment:
      - AIRBYTE_INSTALLATION_ID=${AIRBYTE_INSTALLATION_ID:-}
      - AIRBYTE_VERSION=${VERSION}
      - CONFIGS_DATABASE_MINIMUM_FLYWAY_MIGRATION_VERSION=${CONFIGS_DATABASE_MINIMUM_FLYWAY_MIGRATION_VERSION}
      - CONNECTOR_REGISTRY_BASE_URL=${CONNECTOR_REGISTRY_BASE_URL:-}
      - CONNECTOR_REGISTRY_SEED_PROVIDER=${CONNECTOR_REGISTRY_SEED_PROVIDER:-}
      - DATABASE_PASSWORD=${DATABASE_PASSWORD}
      - DATABASE_URL=${DATABASE_URL}
      - DATABASE_USER=${DATABASE_USER}
      - DD_AGENT_HOST=${DD_AGENT_HOST}
      - DD_DOGSTATSD_PORT=${DD_DOGSTATSD_PORT}
      - DEPLOYMENT_MODE=${DEPLOYMENT_MODE}
      - LOG_LEVEL=${LOG_LEVEL}
      - METRIC_CLIENT=${METRIC_CLIENT}
      - MICROMETER_METRICS_ENABLED=${MICROMETER_METRICS_ENABLED}
      - MICROMETER_METRICS_STATSD_FLAVOR=${MICROMETER_METRICS_STATSD_FLAVOR}
      - MICRONAUT_ENVIRONMENTS=${CRON_MICRONAUT_ENVIRONMENTS}
      - PUBLISH_METRICS=${PUBLISH_METRICS}
      - SEGMENT_WRITE_KEY=${SEGMENT_WRITE_KEY}
      - STATSD_HOST=${STATSD_HOST}
      - STATSD_PORT=${STATSD_PORT}
      - TEMPORAL_HISTORY_RETENTION_IN_DAYS=${TEMPORAL_HISTORY_RETENTION_IN_DAYS}
      - TRACKING_STRATEGY=${TRACKING_STRATEGY}
      - UPDATE_DEFINITIONS_CRON_ENABLED=${UPDATE_DEFINITIONS_CRON_ENABLED}
      - WORKSPACE_ROOT=${WORKSPACE_ROOT}
      - DD_LOGS_INJECTION=${DD_LOGS_INJECTION}
      - JAVA_OPTS=${JAVA_OPTS}
      - CONTAINER_ORCHESTRATOR_JAVA_OPTS=${CONTAINER_ORCHESTRATOR_JAVA_OPTS}
      - DD_TRACE_AGENT_PORT=${DD_TRACE_AGENT_PORT}
    configs:
      - flags
    volumes:
      - workspace:${WORKSPACE_ROOT}
    networks:
      - airbyte_internal
    depends_on:
      bootloader:
        condition: service_completed_successfully
  airbyte-api-server:
    image: airbyte/airbyte-api-server:${VERSION}
    logging: *default-logging
    container_name: airbyte-api-server
    restart: unless-stopped
    ports:
      - "8006"
    environment:
      - AIRBYTE_API_HOST=${AIRBYTE_API_HOST}
      - AIRBYTE_VERSION=${VERSION}
      - DEPLOYMENT_MODE=${DEPLOYMENT_MODE}
      - INTERNAL_API_HOST=${INTERNAL_API_HOST} # Non-ideal, but the existing URL can't have https:// added because it's used for nginx conf
      - LOG_LEVEL=${LOG_LEVEL}
      - MICROMETER_METRICS_ENABLED=${MICROMETER_METRICS_ENABLED}
      - MICROMETER_METRICS_STATSD_FLAVOR=${MICROMETER_METRICS_STATSD_FLAVOR}
      - SEGMENT_WRITE_KEY=${SEGMENT_WRITE_KEY}
      - STATSD_HOST=${STATSD_HOST}
      - STATSD_PORT=${STATSD_PORT}
      - TRACKING_STRATEGY=${TRACKING_STRATEGY}
      - DD_LOGS_INJECTION=${DD_LOGS_INJECTION}
      - JAVA_OPTS=${JAVA_OPTS}
      - CONTAINER_ORCHESTRATOR_JAVA_OPTS=${CONTAINER_ORCHESTRATOR_JAVA_OPTS}
      - DD_DOGSTATSD_PORT=${DD_DOGSTATSD_PORT}
      - DD_TRACE_AGENT_PORT=${DD_TRACE_AGENT_PORT}
      - DD_AGENT_HOST=${DD_AGENT_HOST}
    networks:
      - airbyte_internal
    depends_on:
      bootloader:
        condition: service_completed_successfully
  airbyte-connector-builder-server:
    image: airbyte/connector-builder-server:${VERSION}
    logging: *default-logging
    container_name: airbyte-connector-builder-server
    restart: unless-stopped
    ports:
      - 8080
    environment:
      - AIRBYTE_VERSION=${VERSION}
      - CDK_VERSION=${CDK_VERSION}
      - DEPLOYMENT_MODE=${DEPLOYMENT_MODE}
      - METRIC_CLIENT=${METRIC_CLIENT}
      - MICROMETER_METRICS_ENABLED=${MICROMETER_METRICS_ENABLED}
      - MICROMETER_METRICS_STATSD_FLAVOR=${MICROMETER_METRICS_STATSD_FLAVOR}
      - PYTHON_VERSION=${PYTHON_VERSION}
      - SEGMENT_WRITE_KEY=${SEGMENT_WRITE_KEY}
      - STATSD_HOST=${STATSD_HOST}
      - STATSD_PORT=${STATSD_PORT}
      - TRACKING_STRATEGY=${TRACKING_STRATEGY}
      - DD_LOGS_INJECTION=${DD_LOGS_INJECTION}
      - JAVA_OPTS=${JAVA_OPTS}
      - CONTAINER_ORCHESTRATOR_JAVA_OPTS=${CONTAINER_ORCHESTRATOR_JAVA_OPTS}
      - DD_DOGSTATSD_PORT=${DD_DOGSTATSD_PORT}
      - DD_TRACE_AGENT_PORT=${DD_TRACE_AGENT_PORT}
      - DD_AGENT_HOST=${DD_AGENT_HOST}
    networks:
      - airbyte_internal
    depends_on:
      bootloader:
        condition: service_completed_successfully
  airbyte-proxy:
    image: airbyte/proxy:${VERSION}
    container_name: airbyte-proxy
    restart: unless-stopped
    ports:
      - "8000:8000"
      - "8001:8001"
      - "8003:8003"
      - "8006:8006"
    environment:
      - BASIC_AUTH_USERNAME=${BASIC_AUTH_USERNAME}
      - BASIC_AUTH_PASSWORD=${BASIC_AUTH_PASSWORD}
      - BASIC_AUTH_PROXY_TIMEOUT=${BASIC_AUTH_PROXY_TIMEOUT}
      - DD_LOGS_INJECTION=${DD_LOGS_INJECTION}
      - JAVA_OPTS=${JAVA_OPTS}
      - CONTAINER_ORCHESTRATOR_JAVA_OPTS=${CONTAINER_ORCHESTRATOR_JAVA_OPTS}
      - DD_DOGSTATSD_PORT=${DD_DOGSTATSD_PORT}
      - DD_TRACE_AGENT_PORT=${DD_TRACE_AGENT_PORT}
      - DD_AGENT_HOST=${DD_AGENT_HOST}
    networks:
      - airbyte_internal
      - airbyte_public
    depends_on:
      - webapp
      - server
      - airbyte-api-server
  
  s3proxy:
    image: andrewgaul/s3proxy
    restart: always
    ports:
      - "9042:8080"
    environment:
      - S3PROXY_AUTHORIZATION=none
      - S3PROXY_ENDPOINT=http://0.0.0.0:8080
      - JCLOUDS_AZUREBLOB_AUTH=azureKey
      - JCLOUDS_PROVIDER=azureblob
      - JCLOUDS_IDENTITY=name
      - JCLOUDS_CREDENTIAL=secret
      - JCLOUDS_ENDPOINT=https://name.blob.core.windows.net
      - JCLOUDS_REGION=eu-west-1
      - JCLOUDS_REGIONS=eu-west-1
    networks:
      - airbyte_internal

  spark-iceberg:
    image: tabulario/spark-iceberg
    container_name: spark-iceberg
    build: spark/
    networks:
      airbyte_internal:
    depends_on:
      - rest
    volumes:
      - ./warehouse:/home/iceberg/warehouse
      - ./notebooks:/home/iceberg/notebooks/notebooks
    environment:
      - AWS_ACCESS_KEY_ID=name
      - AWS_SECRET_ACCESS_KEY=secret
      - AWS_REGION=eu-west-1
    ports:
      - 8888:8888
      - 9080:8080
      - 10000:10000
      - 10001:10001
  rest:
    image: tabulario/iceberg-rest
    container_name: iceberg-rest
    networks:
      airbyte_internal:
    ports:
      - 8181:8181
    environment:
      - AWS_ACCESS_KEY_ID=name
      - AWS_SECRET_ACCESS_KEY=secret
      - AWS_REGION=eu-west-1
      - CATALOG_WAREHOUSE=s3://iceberg/
      - CATALOG_IO__IMPL=org.apache.iceberg.aws.s3.S3FileIO
      - CATALOG_S3_ENDPOINT=http://s3proxy:8080

volumes:
  workspace:
    name: ${WORKSPACE_DOCKER_MOUNT}
  local_root:
    name: ${LOCAL_DOCKER_MOUNT}
  # the data volume is only needed for backward compatibility; when users upgrade
  # from an old Airbyte version that relies on file-based configs, the server needs
  # to read this volume to copy their configs to the database
  data:
    name: ${DATA_DOCKER_MOUNT}
  db:
    name: ${DB_DOCKER_MOUNT}
configs:
  flags:
    file: ./flags.yml
networks:
  airbyte_public:
  airbyte_internal:

Problem with the solution

I cannot create the connection

Error Logs

2024-07-12 14:44:17 �[46mplatform�[0m > Docker volume job log path: /tmp/workspace/8f161986-e281-44db-aaab-bf148b77a980/0/logs.log
2024-07-12 14:44:17 �[46mplatform�[0m > Executing worker wrapper. Airbyte version: 0.63.6
2024-07-12 14:44:17 �[46mplatform�[0m > 
2024-07-12 14:44:17 �[46mplatform�[0m > Using default value for environment variable SIDECAR_KUBE_CPU_LIMIT: '2.0'
2024-07-12 14:44:17 �[46mplatform�[0m > Using default value for environment variable SOCAT_KUBE_CPU_LIMIT: '2.0'
2024-07-12 14:44:17 �[46mplatform�[0m > ----- START CHECK -----
2024-07-12 14:44:17 �[46mplatform�[0m > 
2024-07-12 14:44:17 �[46mplatform�[0m > Using default value for environment variable SIDECAR_KUBE_CPU_REQUEST: '0.1'
2024-07-12 14:44:17 �[46mplatform�[0m > Using default value for environment variable SOCAT_KUBE_CPU_REQUEST: '0.1'
2024-07-12 14:44:17 �[46mplatform�[0m > Checking if airbyte/destination-iceberg:0.1.7 exists...
2024-07-12 14:44:17 �[46mplatform�[0m > airbyte/destination-iceberg:0.1.7 was found locally.
2024-07-12 14:44:17 �[46mplatform�[0m > Creating docker container = destination-iceberg-check-8f161986-e281-44db-aaab-bf148b77a980-0-jshin with resources io.airbyte.config.ResourceRequirements@46863ea4[cpuRequest=,cpuLimit=,memoryRequest=,memoryLimit=,additionalProperties={}] and allowedHosts null
2024-07-12 14:44:17 �[46mplatform�[0m > Preparing command: docker run --rm --init -i -w /data/8f161986-e281-44db-aaab-bf148b77a980/0 --log-driver none --name destination-iceberg-check-8f161986-e281-44db-aaab-bf148b77a980-0-jshin --network host -v airbyte_workspace:/data -v oss_local_root:/local -e DEPLOYMENT_MODE=OSS -e WORKER_CONNECTOR_IMAGE=airbyte/destination-iceberg:0.1.7 -e AUTO_DETECT_SCHEMA=true -e LAUNCHDARKLY_KEY= -e SOCAT_KUBE_CPU_REQUEST=0.1 -e SOCAT_KUBE_CPU_LIMIT=2.0 -e FIELD_SELECTION_WORKSPACES= -e USE_STREAM_CAPABLE_STATE=true -e WORKER_ENVIRONMENT=DOCKER -e AIRBYTE_ROLE=dev -e APPLY_FIELD_SELECTION=false -e WORKER_JOB_ATTEMPT=0 -e OTEL_COLLECTOR_ENDPOINT=http://host.docker.internal:4317 -e FEATURE_FLAG_CLIENT=config -e AIRBYTE_VERSION=0.63.6 -e WORKER_JOB_ID=8f161986-e281-44db-aaab-bf148b77a980 airbyte/destination-iceberg:0.1.7 check --config source_config.json
2024-07-12 14:44:17 �[46mplatform�[0m > Reading messages from protocol version 0.2.0
2024-07-12 14:44:18 �[46mplatform�[0m > WARN StatusConsoleListener The use of package scanning to locate plugins is deprecated and will be removed in a future release
2024-07-12 14:44:18 �[46mplatform�[0m > WARN StatusConsoleListener The use of package scanning to locate plugins is deprecated and will be removed in a future release
2024-07-12 14:44:18 �[46mplatform�[0m > WARN StatusConsoleListener The use of package scanning to locate plugins is deprecated and will be removed in a future release
2024-07-12 14:44:18 �[46mplatform�[0m > WARN StatusConsoleListener The use of package scanning to locate plugins is deprecated and will be removed in a future release
2024-07-12 14:44:18 �[46mplatform�[0m > 2024-07-12 14:44:18 �[32mINFO�[m i.a.c.i.b.IntegrationCliParser(parseOptions):126 - integration args: {check=null, config=source_config.json}
2024-07-12 14:44:18 �[46mplatform�[0m > 2024-07-12 14:44:18 �[32mINFO�[m i.a.c.i.b.IntegrationRunner(runInternal):132 - Running integration: io.airbyte.integrations.destination.iceberg.IcebergDestination
2024-07-12 14:44:18 �[46mplatform�[0m > 2024-07-12 14:44:18 �[32mINFO�[m i.a.c.i.b.IntegrationRunner(runInternal):133 - Command: CHECK
2024-07-12 14:44:18 �[46mplatform�[0m > 2024-07-12 14:44:18 �[32mINFO�[m i.a.c.i.b.IntegrationRunner(runInternal):134 - Integration config: IntegrationConfig{command=CHECK, configPath='source_config.json', catalogPath='null', statePath='null'}
2024-07-12 14:44:18 �[46mplatform�[0m > 2024-07-12 14:44:18 �[33mWARN�[m c.n.s.JsonMetaSchema(newValidator):278 - Unknown keyword order - you should define your own Meta Schema. If the keyword is irrelevant for validation, just use a NonValidationKeyword
2024-07-12 14:44:18 �[46mplatform�[0m > 2024-07-12 14:44:18 �[33mWARN�[m c.n.s.JsonMetaSchema(newValidator):278 - Unknown keyword airbyte_secret - you should define your own Meta Schema. If the keyword is irrelevant for validation, just use a NonValidationKeyword
2024-07-12 14:44:19 �[46mplatform�[0m > 2024-07-12 14:44:19 �[1;31mERROR�[m i.a.i.d.i.IcebergDestination(check):54 - Exception attempting to access the Iceberg catalog: 
2024-07-12 14:44:19 �[46mplatform�[0m > org.apache.iceberg.exceptions.RESTException: Error occurred while processing GET request
2024-07-12 14:44:19 �[46mplatform�[0m > 	at org.apache.iceberg.rest.HTTPClient.execute(HTTPClient.java:332) ~[iceberg-spark-runtime-3.5_2.13-1.5.2.jar:?]
2024-07-12 14:44:19 �[46mplatform�[0m > 	at org.apache.iceberg.rest.HTTPClient.execute(HTTPClient.java:252) ~[iceberg-spark-runtime-3.5_2.13-1.5.2.jar:?]
2024-07-12 14:44:19 �[46mplatform�[0m > 	at org.apache.iceberg.rest.HTTPClient.get(HTTPClient.java:348) ~[iceberg-spark-runtime-3.5_2.13-1.5.2.jar:?]
2024-07-12 14:44:19 �[46mplatform�[0m > 	at org.apache.iceberg.rest.RESTSessionCatalog.fetchConfig(RESTSessionCatalog.java:909) ~[iceberg-spark-runtime-3.5_2.13-1.5.2.jar:?]
2024-07-12 14:44:19 �[46mplatform�[0m > 	at org.apache.iceberg.rest.RESTSessionCatalog.initialize(RESTSessionCatalog.java:195) ~[iceberg-spark-runtime-3.5_2.13-1.5.2.jar:?]
2024-07-12 14:44:19 �[46mplatform�[0m > 	at org.apache.iceberg.rest.RESTCatalog.initialize(RESTCatalog.java:78) ~[iceberg-spark-runtime-3.5_2.13-1.5.2.jar:?]
2024-07-12 14:44:19 �[46mplatform�[0m > 	at io.airbyte.integrations.destination.iceberg.config.catalog.RESTCatalogConfig.genCatalog(RESTCatalogConfig.java:79) ~[io.airbyte.airbyte-integrations.connectors-destination-iceberg.jar:?]
2024-07-12 14:44:19 �[46mplatform�[0m > 	at io.airbyte.integrations.destination.iceberg.config.catalog.IcebergCatalogConfig.check(IcebergCatalogConfig.java:43) ~[io.airbyte.airbyte-integrations.connectors-destination-iceberg.jar:?]
2024-07-12 14:44:19 �[46mplatform�[0m > 	at io.airbyte.integrations.destination.iceberg.IcebergDestination.check(IcebergDestination.java:49) [io.airbyte.airbyte-integrations.connectors-destination-iceberg.jar:?]
2024-07-12 14:44:19 �[46mplatform�[0m > 	at io.airbyte.cdk.integrations.base.IntegrationRunner.runInternal(IntegrationRunner.java:150) [airbyte-cdk-core-0.2.0.jar:?]
2024-07-12 14:44:19 �[46mplatform�[0m > 	at io.airbyte.cdk.integrations.base.IntegrationRunner.run(IntegrationRunner.java:125) [airbyte-cdk-core-0.2.0.jar:?]
2024-07-12 14:44:19 �[46mplatform�[0m > 	at io.airbyte.integrations.destination.iceberg.IcebergDestination.main(IcebergDestination.java:42) [io.airbyte.airbyte-integrations.connectors-destination-iceberg.jar:?]
2024-07-12 14:44:19 �[46mplatform�[0m > Caused by: java.net.UnknownHostException: iceberg-rest: System error
2024-07-12 14:44:19 �[46mplatform�[0m > 	at java.base/java.net.Inet6AddressImpl.lookupAllHostAddr(Native Method) ~[?:?]
2024-07-12 14:44:19 �[46mplatform�[0m > 	at java.base/java.net.Inet6AddressImpl.lookupAllHostAddr(Inet6AddressImpl.java:52) ~[?:?]
2024-07-12 14:44:19 �[46mplatform�[0m > 	at java.base/java.net.InetAddress$PlatformResolver.lookupByName(InetAddress.java:1211) ~[?:?]
2024-07-12 14:44:19 �[46mplatform�[0m > 	at java.base/java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1828) ~[?:?]
2024-07-12 14:44:19 �[46mplatform�[0m > 	at java.base/java.net.InetAddress$NameServiceAddresses.get(InetAddress.java:1139) ~[?:?]
2024-07-12 14:44:19 �[46mplatform�[0m > 	at java.base/java.net.InetAddress.getAllByName0(InetAddress.java:1818) ~[?:?]
2024-07-12 14:44:19 �[46mplatform�[0m > 	at java.base/java.net.InetAddress.getAllByName(InetAddress.java:1688) ~[?:?]
2024-07-12 14:44:19 �[46mplatform�[0m > 	at org.apache.iceberg.shaded.org.apache.hc.client5.http.SystemDefaultDnsResolver.resolve(SystemDefaultDnsResolver.java:45) ~[iceberg-spark-runtime-3.5_2.13-1.5.2.jar:?]
2024-07-12 14:44:19 �[46mplatform�[0m > 	at org.apache.iceberg.shaded.org.apache.hc.client5.http.impl.io.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:144) ~[iceberg-spark-runtime-3.5_2.13-1.5.2.jar:?]
2024-07-12 14:44:19 �[46mplatform�[0m > 	at org.apache.iceberg.shaded.org.apache.hc.client5.http.impl.io.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:450) ~[iceberg-spark-runtime-3.5_2.13-1.5.2.jar:?]
2024-07-12 14:44:19 �[46mplatform�[0m > 	at org.apache.iceberg.shaded.org.apache.hc.client5.http.impl.classic.InternalExecRuntime.connectEndpoint(InternalExecRuntime.java:162) ~[iceberg-spark-runtime-3.5_2.13-1.5.2.jar:?]
2024-07-12 14:44:19 �[46mplatform�[0m > 	at org.apache.iceberg.shaded.org.apache.hc.client5.http.impl.classic.InternalExecRuntime.connectEndpoint(InternalExecRuntime.java:172) ~[iceberg-spark-runtime-3.5_2.13-1.5.2.jar:?]
2024-07-12 14:44:19 �[46mplatform�[0m > 	at org.apache.iceberg.shaded.org.apache.hc.client5.http.impl.classic.ConnectExec.execute(ConnectExec.java:142) ~[iceberg-spark-runtime-3.5_2.13-1.5.2.jar:?]
2024-07-12 14:44:19 �[46mplatform�[0m > 	at org.apache.iceberg.shaded.org.apache.hc.client5.http.impl.classic.ExecChainElement.execute(ExecChainElement.java:51) ~[iceberg-spark-runtime-3.5_2.13-1.5.2.jar:?]
2024-07-12 14:44:19 �[46mplatform�[0m > 	at org.apache.iceberg.shaded.org.apache.hc.client5.http.impl.classic.ProtocolExec.execute(ProtocolExec.java:192) ~[iceberg-spark-runtime-3.5_2.13-1.5.2.jar:?]
2024-07-12 14:44:19 �[46mplatform�[0m > 	at org.apache.iceberg.shaded.org.apache.hc.client5.http.impl.classic.ExecChainElement.execute(ExecChainElement.java:51) ~[iceberg-spark-runtime-3.5_2.13-1.5.2.jar:?]
2024-07-12 14:44:19 �[46mplatform�[0m > 	at org.apache.iceberg.shaded.org.apache.hc.client5.http.impl.classic.HttpRequestRetryExec.execute(HttpRequestRetryExec.java:113) ~[iceberg-spark-runtime-3.5_2.13-1.5.2.jar:?]
2024-07-12 14:44:19 �[46mplatform�[0m > 	at org.apache.iceberg.shaded.org.apache.hc.client5.http.impl.classic.ExecChainElement.execute(ExecChainElement.java:51) ~[iceberg-spark-runtime-3.5_2.13-1.5.2.jar:?]
2024-07-12 14:44:19 �[46mplatform�[0m > 	at org.apache.iceberg.shaded.org.apache.hc.client5.http.impl.classic.ContentCompressionExec.execute(ContentCompressionExec.java:152) ~[iceberg-spark-runtime-3.5_2.13-1.5.2.jar:?]
2024-07-12 14:44:19 �[46mplatform�[0m > 	at org.apache.iceberg.shaded.org.apache.hc.client5.http.impl.classic.ExecChainElement.execute(ExecChainElement.java:51) ~[iceberg-spark-runtime-3.5_2.13-1.5.2.jar:?]
2024-07-12 14:44:19 �[46mplatform�[0m > 	at org.apache.iceberg.shaded.org.apache.hc.client5.http.impl.classic.RedirectExec.execute(RedirectExec.java:116) ~[iceberg-spark-runtime-3.5_2.13-1.5.2.jar:?]
2024-07-12 14:44:19 �[46mplatform�[0m > 	at org.apache.iceberg.shaded.org.apache.hc.client5.http.impl.classic.ExecChainElement.execute(ExecChainElement.java:51) ~[iceberg-spark-runtime-3.5_2.13-1.5.2.jar:?]
2024-07-12 14:44:19 �[46mplatform�[0m > 	at org.apache.iceberg.shaded.org.apache.hc.client5.http.impl.classic.InternalHttpClient.doExecute(InternalHttpClient.java:170) ~[iceberg-spark-runtime-3.5_2.13-1.5.2.jar:?]
2024-07-12 14:44:19 �[46mplatform�[0m > 	at org.apache.iceberg.shaded.org.apache.hc.client5.http.impl.classic.CloseableHttpClient.execute(CloseableHttpClient.java:123) ~[iceberg-spark-runtime-3.5_2.13-1.5.2.jar:?]
2024-07-12 14:44:19 �[46mplatform�[0m > 	at org.apache.iceberg.rest.HTTPClient.execute(HTTPClient.java:295) ~[iceberg-spark-runtime-3.5_2.13-1.5.2.jar:?]
2024-07-12 14:44:19 �[46mplatform�[0m > 	... 11 more
2024-07-12 14:44:19 �[46mplatform�[0m > 2024-07-12 14:44:19 �[32mINFO�[m i.a.c.i.b.IntegrationRunner(runInternal):228 - Completed integration: io.airbyte.integrations.destination.iceberg.IcebergDestination
2024-07-12 14:44:19 �[46mplatform�[0m > Check connection job received output: io.airbyte.config.StandardCheckConnectionOutput@2b576ac5[status=failed,message=Could not connect to the Iceberg catalog with the provided configuration. 
Error occurred while processing GET request, root cause: UnknownHostException(iceberg-rest: System error),additionalProperties={}]
2024-07-12 14:44:19 �[46mplatform�[0m > 
2024-07-12 14:44:19 �[46mplatform�[0m > ----- END CHECK -----
2024-07-12 14:44:19 �[46mplatform�[0m >

What works

The s3proxy works. I can use aws cli to interact with the azure blob storage
Airbyte works

I think the problem is in the configuration somewhere. But either way, If you know a better way to query with a SQL(like) large amounts of data stored in azure blob storage, I'd be glad to read your opinion

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Connect Iceberg to Azure Blob Storage (via s3proxy) #41668

{{title}}

Replies: 0 comments

Select a reply

Connect Iceberg to Azure Blob Storage (via s3proxy) #41668

trupus Jul 12, 2024

Current state

Solution I came up with

Problem with the solution

What works

Replies: 0 comments

trupus
Jul 12, 2024