Can't create kafka_topic

[fregate]

Hello! I'm very new to terraform and trying to set up whole environment with docker, kafka and topics to versioning it and provide consistent environment to devs.

And I need help with kafka_topic config

I have error when apply following config:

# Define provider and variables
terraform {
  required_providers {
    docker = {
      source = "kreuzwerker/docker"
    }
    kafka = {
      source = "Mongey/kafka"
    }
  }
}

variable "zookeeper_container_name" {
  description = "Name of the Zookeeper Docker container"
  default     = "zookeeper-container"
}

variable "zookeeper_host_port" {
  description = "Host port to map to Zookeeper's container port"
  default     = 2181
}

variable "kafka_container_name" {
  description = "Name of the Kafka Docker container"
  default     = "kafka-container"
}

variable "kafka_host_port" {
  description = "Host port to map to Kafka's container port"
  default     = 9092
}

# Zookeeper Docker container
resource "docker_container" "zookeeper_container" {
  name  = var.zookeeper_container_name
  image = "bitnami/zookeeper:latest"

  env = ["ALLOW_ANONYMOUS_LOGIN=yes"]

  ports {
    internal = 2181
    external = var.zookeeper_host_port
  }
}

# Kafka Docker container
resource "docker_container" "kafka_container" {
  name  = var.kafka_container_name
  image = "bitnami/kafka:latest"

  depends_on = [docker_container.zookeeper_container]

  env = ["KAFKA_ZOOKEEPER_CONNECT=localhost:${var.zookeeper_host_port}", "ALLOW_PLAINTEXT_LISTENER=yes"]

  ports {
    internal = 9092
    external = var.kafka_host_port
  }
}

provider "kafka" {
  bootstrap_servers = ["localhost:${var.kafka_host_port}"]
  tls_enabled = false
}

resource "kafka_topic" "test-topic" {
  # depends_on         = [docker_container.kafka_container]
  name               = "test-topic"
  replication_factor = 1
  partitions         = 1
}

Error: kafka: client has run out of available brokers to talk to: write tcp 127.0.0.1:55868->127.0.0.1:9092: write: broken pipe
│ 
│   with kafka_topic.test-topic,
│   on main.tf line 66, in resource "kafka_topic" "test-topic":
│   66: resource "kafka_topic" "test-topic" {

127.0.0.1:55868->127.0.0.1:9092
first port always random

there is TF_LOG=1 terraform apply file
terraform-apply.txt

[OneCricketeer]

You're missing several environment variables for the Kafka container to have the correct port mapping. Also, localhost is not the correct zookeeper connection host between two containers, but the latest bitnami images dont use zookeeper anyway... But I suspect your Kafka container isn't actually starting, which is why you get a generic Kafka client error.

You can create topics within a compose file, by the way

kafka:
    image: &kafka-image bitnami/kafka:3.5.1
    restart: unless-stopped
    ports:
      - "29092:29092"
    environment:
      BITNAMI_DEBUG: yes
      ALLOW_PLAINTEXT_LISTENER: yes
      KAFKA_ENABLE_KRAFT: yes
      KAFKA_CFG_PROCESS_ROLES: controller,broker
      KAFKA_CFG_CONTROLLER_LISTENER_NAMES: CONTROLLER
      KAFKA_CFG_NODE_ID: 1
      KAFKA_CFG_CONTROLLER_QUORUM_VOTERS: 1@kafka:9093
      KAFKA_CFG_DELETE_TOPIC_ENABLE: 'true'
      KAFKA_CFG_LOG_RETENTION_HOURS: 72  # 3 days of retention for local testing
      # https://rmoff.net/2018/08/02/kafka-listeners-explained/
      KAFKA_CFG_LISTENERS: INTERNAL://:9092,CONTROLLER://:9093,EXTERNAL://0.0.0.0:29092
      KAFKA_CFG_INTER_BROKER_LISTENER_NAME: INTERNAL
      KAFKA_CFG_ADVERTISED_LISTENERS: INTERNAL://kafka:9092,EXTERNAL://127.0.0.1:29092
      KAFKA_CFG_LISTENER_SECURITY_PROTOCOL_MAP: INTERNAL:PLAINTEXT,EXTERNAL:PLAINTEXT,CONTROLLER:PLAINTEXT
    healthcheck:
      test: ["CMD", "kafka-topics.sh", "--bootstrap-server=localhost:9092", "--list"]
      start_period: 15s
      interval: 30s

  init-kafka:
    image: *kafka-image
    working_dir: /opt/bitnami/kafka/bin
    entrypoint: /bin/bash
    depends_on:
      kafka:
        condition: service_healthy
    command: |
      kafka-topics.sh --create --if-not-exists --topic foo --replication-factor=1 --partitions=1 --bootstrap-server kafka:9092