Add terraform module for ECS (#4765)

* Add terraform module for ECS

* Make module id customizable

* More example and docs

* Address review comments

distribution/ecs/.gitignore

@@ -0,0 +1,4 @@
+.terraform
+terraform.tfstate*
+.terraform.tfstate*
+terraform.tfvars

distribution/ecs/README.md

@@ -0,0 +1,113 @@
+# ECS deployment for quickwit
+
+## Run Quickwit in your infrastructure
+
+Create a Quickwit module using:
+
+```terraform
+module "quickwit" {
+  source = "github.com/quickwit-oss/quickwit/distribution/ecs/quickwit"
+
+  vpc_id                       =       # VPC in which all resources will be created
+  subnet_ids                   = [...] # At least 2 private subnets must be specified
+  quickwit_ingress_cidr_blocks = [...] # List of CIDR blocks allowed to access to the Quickwit API
+}
+```
+
+The Quickwit cluster is running on a private subnet. For ECS to pull the image:
+- if using the default Docker Hub image `quickwit/quickwit`, the subnets
+specified must be configured with a NAT Gateway (no public IPs are attached to
+the tasks)
+- if using an image hosted on ECR, a VPC endpoint for ECR can be used instead of
+a NAT Gateway
+
+
+## Module configurations
+
+To get the list of available configurations, check the `./quickwit/variables.tf`
+file.
+
+### Tips
+
+Metastore database backups are disabled as restoring one would lead to
+inconsistencies with the index store on S3. To ensure high availability, you
+should enable `rds_config.multi_az` instead. The module currently doesn't allow
+using an externally provided metastore.
+
+Using NAT Gateways for the image registry is quite costly (~$0.05/hour/AZ). If
+you are not already using NAT Gateways in the AZs where Quickwit will be
+deployed, you should probably push the Quickwit image to ECR and use ECR
+interface VPC endpoints instead (~$0.01/hour/AZ).
+
+When using the default image, you will quickly run into the Docker Hub rate
+limiting. We recommand pushing the Quickwit image to ECR and configure that as
+`quickwit_image`. Note that the architecture of the image that you push to ECR
+must match the `quickwit_cpu_architecture` variable (`ARM64` by default).
+
+Sidecar container and custom logging configurations can be configured using the
+variables `sidecar_container_definitions`, `sidecar_container_dependencies`,
+`log_configuration`, `enable_cloudwatch_logging`. See [custom log
+routing](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/using_firelens.html).
+
+You can use sidecars to inject additional secrets as files. This can be
+useful for configuring sources such as Kafka. See `./example/kafka.tf` for an
+example.
+
+## Running the example stack
+
+We provide an example of self contained deployment with an ad-hoc VPC. 
+
+> [!IMPORTANT] This stack costs ~$150/month to run (Fargate tasks, NAT Gateways
+> and RDS)
+
+To make it easy to access your the Quickwit cluster, this stack includes a
+bastion instance. Access is secured using an SSH key pair that you need to
+provide (e.g generated with `ssh-keygen -t ed25519`).
+
+In the `./example` directory create a `terraform.tfvars` file with the public
+key of your RSA key pair:
+
+```terraform
+bastion_public_key = "ssh-ed25519 ..."
+```
+
+> [!NOTE] You can skip the creation of the bastion by not specifying the
+> `bastion_public_key` variable, but that would make it hard to access and
+> experiment with the created Quickwit cluster.
+
+In the same directory (`./example`) run:
+
+```bash
+terraform init
+terraform apply
+```
+
+The successful `apply` command should output the IP of the bastion EC2 instance.
+You can port forward Quickwit's search UI using:
+
+```bash
+ssh -N -L 7280:searcher.quickwit:7280 -i {your-private-key-file} ubuntu@{bastion_ip}
+```
+
+To ingest some example dataset, log into the bastion:
+
+```bash
+ssh -i {your-private-key-file} ubuntu@{bastion_ip}
+
+# create the log index
+wget https://raw.githubusercontent.com/quickwit-oss/quickwit/main/config/tutorials/hdfs-logs/index-config.yaml
+curl -X POST \
+  -H "content-type: application/yaml" \
+  --data-binary @index-config.yaml \
+  http://indexer.quickwit:7280/api/v1/indexes
+
+# import some data
+wget https://quickwit-datasets-public.s3.amazonaws.com/hdfs-logs-multitenants-10000.json
+curl -X POST \
+  -H "content-type: application/json" \
+  --data-binary @hdfs-logs-multitenants-10000.json \
+  http://indexer.quickwit:7280/api/v1/hdfs-logs/ingest?commit=force
+```
+
+If your SSH tunnel to the searcher is still running, you should be able to see
+the ingested data in the UI.

distribution/ecs/example/bastion.tf

@@ -0,0 +1,65 @@
+variable "bastion_public_key" {
+  description = "The public key used to connect to the bastion host. If empty, no bastion is created."
+  default     = ""
+}
+
+output "bastion_ip" {
+  value = var.bastion_public_key != "" ? aws_instance.bastion[0].public_ip : null
+}
+
+data "aws_ami" "ubuntu" {
+  most_recent = true
+
+  filter {
+    name   = "name"
+    values = ["ubuntu/images/hvm-ssd/ubuntu-jammy-22.04-amd64-server-*"]
+  }
+
+  filter {
+    name   = "virtualization-type"
+    values = ["hvm"]
+  }
+
+  owners = ["099720109477"] # Canonical
+}
+
+resource "aws_security_group" "allow_ssh" {
+  count       = var.bastion_public_key != "" ? 1 : 0
+  name        = "qw_ecs_bastion_allow_ssh"
+  description = "Allow SSH inbound traffic from everywhere"
+  vpc_id      = module.vpc.vpc_id
+
+  ingress {
+    from_port   = 22
+    to_port     = 22
+    protocol    = "tcp"
+    cidr_blocks = ["0.0.0.0/0"]
+  }
+
+  egress {
+    from_port   = 0
+    to_port     = 0
+    protocol    = "-1"
+    cidr_blocks = ["0.0.0.0/0"]
+  }
+}
+
+resource "aws_instance" "bastion" {
+  count                       = var.bastion_public_key != "" ? 1 : 0
+  ami                         = data.aws_ami.ubuntu.id
+  instance_type               = "t3.nano"
+  key_name                    = aws_key_pair.bastion_key[0].key_name
+  subnet_id                   = module.vpc.public_subnets[0]
+  associate_public_ip_address = true
+  vpc_security_group_ids      = [aws_security_group.allow_ssh[0].id]
+
+  tags = {
+    Name = "quickwit-ecs-bastion"
+  }
+}
+
+resource "aws_key_pair" "bastion_key" {
+  count      = var.bastion_public_key != "" ? 1 : 0
+  key_name   = "quickwit-ecs-bastion-key"
+  public_key = var.bastion_public_key
+}

distribution/ecs/example/kafka.tf

@@ -0,0 +1,58 @@
+# Example configuration for injecting SSL keys for securing a Kafka connection
+# You can then create a secured Kafka source along these lines:
+# 
+# version: 0.8
+# source_id: kafka-source
+# source_type: kafka
+# num_pipelines: 2
+# params:
+#   topic: your-topic
+#   client_params:
+#     bootstrap.servers: "your-kafka-broker.com"
+#     security.protocol: "SSL"
+#     ssl.ca.location: "/quickwit/keys/ca.pem"
+#     ssl.certificate.location: "/quickwit/keys/service.cert"
+#     ssl.key.location: "/quickwit/keys/service.key"
+
+
+locals {
+  ca_pem       = "echo \"$CA_PEM\" > /quickwit/cfg/ca.pem"
+  service_cert = "echo \"$SERVICE_CERT\" > /quickwit/cfg/service.cert"
+  service_key  = "echo \"$SERVICE_KEY\" > /quickwit/cfg/service.key"
+  example_kafka_sidecar_container_definitions = {
+    kafka_key_init = {
+      name                      = "kafka_key_init"
+      essential                 = false
+      image                     = "busybox"
+      command                   = ["sh", "-c", "${local.ca_pem} && ${local.service_cert} && ${local.service_key}"]
+      enable_cloudwatch_logging = true
+      mount_points = [
+        {
+          sourceVolume  = "quickwit-keys"
+          containerPath = "/quickwit/keys"
+        }
+      ]
+      secrets = [
+        {
+          name      = "CA_PEM"
+          valueFrom = "arn:aws:secretsmanager:eu-west-1:123456789:secret:your_kafka_ca_pem"
+        },
+        {
+          name      = "SERVICE_CERT"
+          valueFrom = "arn:aws:secretsmanager:eu-west-1:123456789:secret:your_kafka_service_cert"
+        },
+        {
+          name      = "SERVICE_KEY"
+          valueFrom = "arn:aws:secretsmanager:eu-west-1:123456789:secret:your_kafka_service_key"
+        }
+      ]
+    }
+  }
+
+  example_kafka_sidecar_container_dependencies = [
+    {
+      condition     = "SUCCESS"
+      containerName = "kafka_key_init"
+    }
+  ]
+}

distribution/ecs/example/terraform.tf

@@ -0,0 +1,92 @@
+terraform {
+  backend "local" {}
+  required_providers {
+    aws = {
+      source  = "hashicorp/aws"
+      version = "~> 5.39.1"
+    }
+  }
+}
+
+provider "aws" {
+  region = "eu-west-1"
+  default_tags {
+    tags = {
+      provisioner = "terraform"
+    }
+  }
+}
+
+# resource "aws_ecr_repository" "quickwit" {
+#   name                 = "quickwit"
+#   force_delete         = true
+#   image_tag_mutability = "MUTABLE"
+# }
+
+module "quickwit" {
+  source                       = "../quickwit"
+  vpc_id                       = module.vpc.vpc_id
+  subnet_ids                   = module.vpc.private_subnets
+  quickwit_ingress_cidr_blocks = [module.vpc.vpc_cidr_block]
+
+  ## Optional configurations:
+
+  # quickwit_index_s3_prefix  = "my-bucket/my-prefix"
+  # quickwit_domain           = "quickwit"
+  # quickwit_image = aws_ecr_repository.quickwit.repository_url
+  # quickwit_cpu_architecture = "ARM64"
+
+  # quickwit_indexer = {
+  #   desired_count = 1
+  #   memory        = 2048
+  #   cpu           = 1024
+  # }
+
+  # quickwit_metastore = {
+  #   desired_count = 1
+  #   memory        = 512
+  #   cpu           = 256
+  # }
+
+  # quickwit_searcher = {
+  #   desired_count = 1
+  #   memory        = 2048
+  #   cpu           = 1024
+  # }
+
+  # quickwit_control_plane = {
+  #   memory = 512
+  #   cpu    = 256
+  # }
+
+  # quickwit_janitor = {
+  #   memory = 512
+  #   cpu    = 256
+  # }
+
+  # rds_config = {
+  #   instance_class = "db.t4g.micro"
+  #   multi_az       = false
+  # }
+
+  ## Example logging configuration 
+  # sidecar_container_definitions  = {
+  #   my_sidecar_container = see http://docs.aws.amazon.com/AmazonECS/latest/APIReference/API_ContainerDefinition.html
+  # }
+  # sidecar_container_dependencies = [{condition = "START", containerName = "my_sidecar_container"}]
+  # log_configuration              = see https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/ecs_service#log_configuration
+  # enable_cloudwatch_logging      = false
+
+  ## Example Kafka key injection (see kafka.tf)
+  # sidecar_container_definitions  = local.example_kafka_sidecar_container_definitions
+  # sidecar_container_dependencies = local.example_kafka_sidecar_container_dependencies
+}
+
+
+output "indexer_service_name" {
+  value = module.quickwit.indexer_service_name
+}
+
+output "searcher_service_name" {
+  value = module.quickwit.searcher_service_name
+}

distribution/ecs/example/vpc.tf

@@ -0,0 +1,13 @@
+module "vpc" {
+  source  = "terraform-aws-modules/vpc/aws"
+  version = "5.5.3"
+
+  name = "quickwit-ecs"
+  cidr = "10.0.0.0/16"
+
+  azs             = ["eu-west-1a", "eu-west-1b"]
+  private_subnets = ["10.0.1.0/24", "10.0.2.0/24"]
+  public_subnets  = ["10.0.101.0/24", "10.0.102.0/24"]
+
+  enable_nat_gateway = true
+}