From 3301c27a01319a719b4e4ca78e15406c92615d90 Mon Sep 17 00:00:00 2001 From: alex-treebeard Date: Mon, 26 Feb 2024 11:13:49 +0000 Subject: [PATCH 01/56] script --- scripts/setup-krew.sh | 2 -- 1 file changed, 2 deletions(-) diff --git a/scripts/setup-krew.sh b/scripts/setup-krew.sh index b281eb4..66342d7 100755 --- a/scripts/setup-krew.sh +++ b/scripts/setup-krew.sh @@ -5,5 +5,3 @@ KREW="krew-${OS}_${ARCH}" && curl -fsSLO "https://github.com/kubernetes-sigs/krew/releases/latest/download/${KREW}.tar.gz" && tar zxvf "${KREW}.tar.gz" && ./"${KREW}" install krew - -bash -c 'kubectl krew install resource-capacity' From f459d3deb2ab095793d3e867695b72b717ef9726 Mon Sep 17 00:00:00 2001 From: alex-treebeard Date: Mon, 26 Feb 2024 12:57:56 +0000 Subject: [PATCH 02/56] contributing/security docs --- CODE_OF_CONDUCT.md | 76 ++++++++++++++++++++++++++++++++++++++++++++++ CONTRIBUTING.md | 27 ++++++++++++++++ SECURITY.md | 11 +++++++ 3 files changed, 114 insertions(+) create mode 100644 CODE_OF_CONDUCT.md create mode 100644 CONTRIBUTING.md create mode 100644 SECURITY.md diff --git a/CODE_OF_CONDUCT.md b/CODE_OF_CONDUCT.md new file mode 100644 index 0000000..128483e --- /dev/null +++ b/CODE_OF_CONDUCT.md @@ -0,0 +1,76 @@ +# Contributor Covenant Code of Conduct + +## Our Pledge + +In the interest of fostering an open and welcoming environment, we as +contributors and maintainers pledge to making participation in our project and +our community a harassment-free experience for everyone, regardless of age, body +size, disability, ethnicity, sex characteristics, gender identity and expression, +level of experience, education, socio-economic status, nationality, personal +appearance, race, religion, or sexual identity and orientation. + +## Our Standards + +Examples of behavior that contributes to creating a positive environment +include: + +* Using welcoming and inclusive language +* Being respectful of differing viewpoints and experiences +* Gracefully accepting constructive criticism +* Focusing on what is best for the community +* Showing empathy towards other community members + +Examples of unacceptable behavior by participants include: + +* The use of sexualized language or imagery and unwelcome sexual attention or + advances +* Trolling, insulting/derogatory comments, and personal or political attacks +* Public or private harassment +* Publishing others' private information, such as a physical or electronic + address, without explicit permission +* Other conduct which could reasonably be considered inappropriate in a + professional setting + +## Our Responsibilities + +Project maintainers are responsible for clarifying the standards of acceptable +behavior and are expected to take appropriate and fair corrective action in +response to any instances of unacceptable behavior. + +Project maintainers have the right and responsibility to remove, edit, or +reject comments, commits, code, wiki edits, issues, and other contributions +that are not aligned to this Code of Conduct, or to ban temporarily or +permanently any contributor for other behaviors that they deem inappropriate, +threatening, offensive, or harmful. + +## Scope + +This Code of Conduct applies both within project spaces and in public spaces +when an individual is representing the project or its community. Examples of +representing a project or community include using an official project e-mail +address, posting via an official social media account, or acting as an appointed +representative at an online or offline event. Representation of a project may be +further defined and clarified by project maintainers. + +## Enforcement + +Instances of abusive, harassing, or otherwise unacceptable behavior may be +reported by contacting the project team at alex@treebeard.io. All +complaints will be reviewed and investigated and will result in a response that +is deemed necessary and appropriate to the circumstances. The project team is +obligated to maintain confidentiality with regard to the reporter of an incident. +Further details of specific enforcement policies may be posted separately. + +Project maintainers who do not follow or enforce the Code of Conduct in good +faith may face temporary or permanent repercussions as determined by other +members of the project's leadership. + +## Attribution + +This Code of Conduct is adapted from the [Contributor Covenant][homepage], version 1.4, +available at https://www.contributor-covenant.org/version/1/4/code-of-conduct.html + +[homepage]: https://www.contributor-covenant.org + +For answers to common questions about this code of conduct, see +https://www.contributor-covenant.org/faq \ No newline at end of file diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md new file mode 100644 index 0000000..f4ad83f --- /dev/null +++ b/CONTRIBUTING.md @@ -0,0 +1,27 @@ +# Contributing + +Thanks for considering contributing to Kubeflow Terraform Modules. We are open to contributions but please check via GitHub issues that we don't already provide the functionality you are requesting and that we are open to it being contributed. + +## Creating a development environment + +We recommend an Ubuntu environment running docker with 2 cpus and 16G of memory. + +This can be achieved using VSCode and the `.devcontainer` directory in this repo. A cloud VM gives the best performance. + +* [devcontainer docs](https://containers.dev/) +* See the `.devcontainer` directory for details on dev dependencies, the main ones are: + * k3d + * helm CLI + * terraform CLI + +## Install from source + +While developing your change, install the K3s example into a local K3d cluster. Follow [the example](./examples/k3s/README.md) to do this. + +## Make a change + +You can change terraform files in the project then `terraform apply` to see them in your dev environment. + +## Ensure changes work in non K3s environments + +This can be validated by following examples other K8s providers such as EKS. diff --git a/SECURITY.md b/SECURITY.md new file mode 100644 index 0000000..b1af40b --- /dev/null +++ b/SECURITY.md @@ -0,0 +1,11 @@ +# Security Policy + +## Supported Versions + +| Version | Supported | +| ------- | ------------------ | +| 0.1.x | :white_check_mark: | + +## Reporting a Vulnerability + +alex@treebeard.io \ No newline at end of file From 9c55f13318ffd8b9fb2255a48c9f72c3d886ee28 Mon Sep 17 00:00:00 2001 From: alex-treebeard Date: Tue, 27 Feb 2024 09:08:08 +0000 Subject: [PATCH 03/56] example with istio disabled --- examples/k3s-existing-istio/main.tf | 121 ++++++++++++++++++++++++++++ main.tf | 33 ++++++-- 2 files changed, 149 insertions(+), 5 deletions(-) create mode 100644 examples/k3s-existing-istio/main.tf diff --git a/examples/k3s-existing-istio/main.tf b/examples/k3s-existing-istio/main.tf new file mode 100644 index 0000000..01523cb --- /dev/null +++ b/examples/k3s-existing-istio/main.tf @@ -0,0 +1,121 @@ + +terraform { + required_providers { + kustomization = { + source = "kbst/kustomization" + version = "~> 0.9.5" + } + helm = { + source = "hashicorp/helm" + version = "~> 2.12.1" + } + kubernetes = { + source = "hashicorp/kubernetes" + version = "~> 2.25.2" + } + } + backend "local" { + } +} + +variable "kubeconfig" { + type = string +} + +provider "kustomization" { + kubeconfig_path = var.kubeconfig +} + +provider "helm" { + kubernetes { + config_path = var.kubeconfig + } +} + +provider "kubernetes" { + config_path = var.kubeconfig +} + +resource "helm_release" "cert-manager" { + name = "cert-manager" + namespace = "cert-manager" + chart = "cert-manager" + repository = "https://charts.jetstack.io" + version = "1.14.3" + create_namespace = true + depends_on = [] + values = [ + < Date: Tue, 27 Feb 2024 09:08:26 +0000 Subject: [PATCH 04/56] fmt --- examples/k3s-existing-istio/main.tf | 50 ++++++++++++++--------------- main.tf | 14 ++++---- 2 files changed, 32 insertions(+), 32 deletions(-) diff --git a/examples/k3s-existing-istio/main.tf b/examples/k3s-existing-istio/main.tf index 01523cb..1f6b624 100644 --- a/examples/k3s-existing-istio/main.tf +++ b/examples/k3s-existing-istio/main.tf @@ -37,13 +37,13 @@ provider "kubernetes" { } resource "helm_release" "cert-manager" { - name = "cert-manager" - namespace = "cert-manager" - chart = "cert-manager" - repository = "https://charts.jetstack.io" - version = "1.14.3" + name = "cert-manager" + namespace = "cert-manager" + chart = "cert-manager" + repository = "https://charts.jetstack.io" + version = "1.14.3" create_namespace = true - depends_on = [] + depends_on = [] values = [ < Date: Tue, 27 Feb 2024 09:09:23 +0000 Subject: [PATCH 05/56] container --- .devcontainer/Dockerfile | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/.devcontainer/Dockerfile b/.devcontainer/Dockerfile index dfac67d..6124d0a 100644 --- a/.devcontainer/Dockerfile +++ b/.devcontainer/Dockerfile @@ -14,5 +14,7 @@ ENV DEBIAN_FRONTEND=noninteractive RUN sudo apt-get update -y && sudo apt-get install -y \ iputils-ping \ traceroute \ - kmod -ENV PATH="/home/vscode/.local/bin/:$PATH" + kmod \ + vim + ENV PATH="/home/vscode/.local/bin/:$PATH" +ENV EDITOR=vim \ No newline at end of file From 3acf3e071669bb29e800b932b4c767d783938425 Mon Sep 17 00:00:00 2001 From: alex-treebeard Date: Tue, 27 Feb 2024 10:29:11 +0000 Subject: [PATCH 06/56] completed variable --- examples/k3s-existing-istio/main.tf | 19 ++++++++++++++----- main.tf | 14 +++++++++++--- util.tf | 3 +++ 3 files changed, 28 insertions(+), 8 deletions(-) diff --git a/examples/k3s-existing-istio/main.tf b/examples/k3s-existing-istio/main.tf index 1f6b624..f7a2cae 100644 --- a/examples/k3s-existing-istio/main.tf +++ b/examples/k3s-existing-istio/main.tf @@ -102,6 +102,19 @@ resource "helm_release" "istio_ingressgateway" { ] } +# resource "time_sleep" "wait_30_seconds" { +# depends_on = [ +# helm_release.istio_ingressgateway +# ] +# create_duration = "30s" +# } + +resource null_resource "completed" { + depends_on = [ + helm_release.istio_ingressgateway + ] +} + module "treebeardkf" { source = "../.." hostname = "kf.example.com" @@ -113,9 +126,5 @@ module "treebeardkf" { enable_istiod = false enable_istio_resources = true enable_cert_manager = false - depends_on = [ - helm_release.istio_base, - helm_release.istiod, - helm_release.istio_ingressgateway - ] + completed = null_resource.completed.id } \ No newline at end of file diff --git a/main.tf b/main.tf index 8a2db02..15478d5 100644 --- a/main.tf +++ b/main.tf @@ -15,6 +15,11 @@ terraform { } } +variable "completed" { + type = string + default = false +} + variable "hostname" { type = string default = "localhost" @@ -62,7 +67,8 @@ module "kubeflow_issuer" { source = "./modules/kust" build = data.kustomization_build.kubeflow_issuer depends_on = [ - module.cert_manager + module.cert_manager, + var.completed ] } @@ -163,7 +169,8 @@ module "oidc_authservice" { source = "./modules/kust" build = data.kustomization_overlay.oidc_authservice depends_on = [ - module.istio_install + module.istio_install, + var.completed ] } @@ -215,7 +222,8 @@ module "dex" { source = "./modules/kust" build = data.kustomization_overlay.dex depends_on = [ - module.istio_install + module.istio_install, + module.oidc_authservice ] } diff --git a/util.tf b/util.tf index 1df90b6..b663e54 100644 --- a/util.tf +++ b/util.tf @@ -5,6 +5,9 @@ resource "kubernetes_namespace" "tkf_system" { "admission.gatekeeper.sh/ignore" = "no-self-managing" # this gets added by gatekeeper otherwise } } + depends_on = [ + var.completed + ] } variable "enable_gatekeeper" { From 61b72950c9d8750dd35c8a2dd838c5b35829517c Mon Sep 17 00:00:00 2001 From: alex-treebeard Date: Tue, 27 Feb 2024 10:29:31 +0000 Subject: [PATCH 07/56] completed --- examples/k3s-existing-istio/main.tf | 4 ++-- main.tf | 2 +- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/examples/k3s-existing-istio/main.tf b/examples/k3s-existing-istio/main.tf index f7a2cae..8d2b64e 100644 --- a/examples/k3s-existing-istio/main.tf +++ b/examples/k3s-existing-istio/main.tf @@ -109,7 +109,7 @@ resource "helm_release" "istio_ingressgateway" { # create_duration = "30s" # } -resource null_resource "completed" { +resource "null_resource" "completed" { depends_on = [ helm_release.istio_ingressgateway ] @@ -126,5 +126,5 @@ module "treebeardkf" { enable_istiod = false enable_istio_resources = true enable_cert_manager = false - completed = null_resource.completed.id + completed = null_resource.completed.id } \ No newline at end of file diff --git a/main.tf b/main.tf index 15478d5..a92cf85 100644 --- a/main.tf +++ b/main.tf @@ -16,7 +16,7 @@ terraform { } variable "completed" { - type = string + type = string default = false } From 3137fd121b390b9f7fcc5fe46673a5aa1db89c02 Mon Sep 17 00:00:00 2001 From: alex-treebeard Date: Tue, 27 Feb 2024 10:39:28 +0000 Subject: [PATCH 08/56] doc --- Makefile | 8 ++++---- README.md | 15 ++++++++++++++- 2 files changed, 18 insertions(+), 5 deletions(-) diff --git a/Makefile b/Makefile index 9c98d82..d34fdad 100644 --- a/Makefile +++ b/Makefile @@ -10,10 +10,10 @@ post-start: ./scripts/setup-krew.sh docs: - terraform-docs markdown . > ./README.md - terraform-docs markdown examples/k3s > examples/k3s/README.md - terraform-docs markdown examples/eks > examples/eks/README.md - terraform-docs markdown examples/aks > examples/aks/README.md + terraform-docs markdown . > ./README.md.new + terraform-docs markdown examples/k3s > examples/k3s/README.md.new + terraform-docs markdown examples/eks > examples/eks/README.md.new + terraform-docs markdown examples/aks > examples/aks/README.md.new fmt: terraform fmt . modules/* examples/* diff --git a/README.md b/README.md index 6e8012d..33d4327 100644 --- a/README.md +++ b/README.md @@ -7,7 +7,6 @@ Terraform module which creates a Kubeflow instance in a Kubernetes cluster dagster logo ---- > [!Note] > This repository is currently suitable for development environments only. Please report any problem you might have by opening a GitHub issue, feature requests welcome. @@ -62,6 +61,20 @@ module "treebeardkf" { } ``` +### Install in an existing cluster + +You can incrementally add Kubeflow to your K8s cluster by installing the terraform module. + +Some considerations: +1. If you are calling this Terraform module from your own module, pass in a string to the `completed` variable in order to manage Kubeflow *after* changes to your other resources. (Note that `depends_on` does not work with this module) +2. If you already have Istio and Cert Manager installed, you will need to ensure Kubeflow works with them. See [examples/k3s-existing-istio](examples/k3s-existing-istio) for a configuration that we have tested like this. + +### Teardown + +1. Manually remove any manually created Kubeflow resources, e.g. Notebook Servers and Volumes +2. Remove the terraform module, e.g. with `terraform destroy` if you have installed directly from CLI +3. Clean up remaining resources, e.g. Istio leaves behind some secrets that can prevent successful re-installation. + ## Architecture This module is built on top of the official [Kubeflow Manifests repo](https://github.com/kubeflow/manifests) which contains _Kustomizations_ for the various components of Kubeflow. From 6c2055efab01dbf941bb8c80def9e7cd030d84cc Mon Sep 17 00:00:00 2001 From: alex-treebeard Date: Tue, 27 Feb 2024 10:44:19 +0000 Subject: [PATCH 09/56] docs --- Makefile | 10 ++++++- README.md | 4 +++ examples/k3s-existing-istio/README.md | 40 +++++++++++++++++++++++++++ examples/k3s/README.md | 34 ++++++++++++++++++++++- 4 files changed, 86 insertions(+), 2 deletions(-) create mode 100644 examples/k3s-existing-istio/README.md diff --git a/Makefile b/Makefile index d34fdad..1709664 100644 --- a/Makefile +++ b/Makefile @@ -9,12 +9,20 @@ post-start: @echo "post-start" ./scripts/setup-krew.sh +.PHONY: docs docs: terraform-docs markdown . > ./README.md.new terraform-docs markdown examples/k3s > examples/k3s/README.md.new - terraform-docs markdown examples/eks > examples/eks/README.md.new + terraform-docs markdown examples/k3s-existing-istio > examples/k3s-existing-istio/README.md.new terraform-docs markdown examples/aks > examples/aks/README.md.new +.PHONY: docs-rm-new +docs-rm-new: + rm -f ./README.md.new + rm -f examples/k3s/README.md.new + rm -f examples/eks/README.md.new + rm -f examples/aks/README.md.new + fmt: terraform fmt . modules/* examples/* .PHONY: build \ No newline at end of file diff --git a/README.md b/README.md index 33d4327..0ac6048 100644 --- a/README.md +++ b/README.md @@ -167,11 +167,15 @@ This module is built on top of the official [Kubeflow Manifests repo](https://gi | Name | Description | Type | Default | Required | |------|-------------|------|---------|:--------:| +| [completed](#input\_completed) | n/a | `string` | `false` | no | +| [enable\_cert\_manager](#input\_enable\_cert\_manager) | n/a | `bool` | `true` | no | | [enable\_external\_dns](#input\_enable\_external\_dns) | n/a | `bool` | `true` | no | | [enable\_external\_secrets](#input\_enable\_external\_secrets) | n/a | `bool` | `false` | no | | [enable\_gatekeeper](#input\_enable\_gatekeeper) | n/a | `bool` | `false` | no | | [enable\_gpu\_operator](#input\_enable\_gpu\_operator) | n/a | `bool` | `false` | no | +| [enable\_istio\_base](#input\_enable\_istio\_base) | n/a | `bool` | `true` | no | | [enable\_istio\_resources](#input\_enable\_istio\_resources) | Enable istio resources for clusters with pre-existing istio | `bool` | `true` | no | +| [enable\_istiod](#input\_enable\_istiod) | n/a | `bool` | `true` | no | | [enable\_kserve](#input\_enable\_kserve) | n/a | `bool` | `false` | no | | [enable\_kubeflow\_pipelines](#input\_enable\_kubeflow\_pipelines) | n/a | `bool` | `false` | no | | [enable\_kuberay](#input\_enable\_kuberay) | n/a | `bool` | `false` | no | diff --git a/examples/k3s-existing-istio/README.md b/examples/k3s-existing-istio/README.md new file mode 100644 index 0000000..8090121 --- /dev/null +++ b/examples/k3s-existing-istio/README.md @@ -0,0 +1,40 @@ +## Requirements + +| Name | Version | +|------|---------| +| [helm](#requirement\_helm) | ~> 2.12.1 | +| [kubernetes](#requirement\_kubernetes) | ~> 2.25.2 | +| [kustomization](#requirement\_kustomization) | ~> 0.9.5 | + +## Providers + +| Name | Version | +|------|---------| +| [helm](#provider\_helm) | 2.12.1 | +| [null](#provider\_null) | 3.2.2 | + +## Modules + +| Name | Source | Version | +|------|--------|---------| +| [treebeardkf](#module\_treebeardkf) | ../.. | n/a | + +## Resources + +| Name | Type | +|------|------| +| [helm_release.cert-manager](https://registry.terraform.io/providers/hashicorp/helm/latest/docs/resources/release) | resource | +| [helm_release.istio_base](https://registry.terraform.io/providers/hashicorp/helm/latest/docs/resources/release) | resource | +| [helm_release.istio_ingressgateway](https://registry.terraform.io/providers/hashicorp/helm/latest/docs/resources/release) | resource | +| [helm_release.istiod](https://registry.terraform.io/providers/hashicorp/helm/latest/docs/resources/release) | resource | +| [null_resource.completed](https://registry.terraform.io/providers/hashicorp/null/latest/docs/resources/resource) | resource | + +## Inputs + +| Name | Description | Type | Default | Required | +|------|-------------|------|---------|:--------:| +| [kubeconfig](#input\_kubeconfig) | n/a | `string` | n/a | yes | + +## Outputs + +No outputs. diff --git a/examples/k3s/README.md b/examples/k3s/README.md index f23bb93..7b3df83 100644 --- a/examples/k3s/README.md +++ b/examples/k3s/README.md @@ -100,4 +100,36 @@ Customisation and deployment for your team is another matter and will be discuss ```sh k3d cluster delete demo -``` \ No newline at end of file +``` + +## Requirements + +| Name | Version | +|------|---------| +| [helm](#requirement\_helm) | ~> 2.12.1 | +| [kubernetes](#requirement\_kubernetes) | ~> 2.25.2 | +| [kustomization](#requirement\_kustomization) | ~> 0.9.5 | + +## Providers + +No providers. + +## Modules + +| Name | Source | Version | +|------|--------|---------| +| [treebeardkf](#module\_treebeardkf) | ../.. | n/a | + +## Resources + +No resources. + +## Inputs + +| Name | Description | Type | Default | Required | +|------|-------------|------|---------|:--------:| +| [kubeconfig](#input\_kubeconfig) | n/a | `string` | n/a | yes | + +## Outputs + +No outputs. From efd5031eef1b4a94f8b1edd317de1a84f568d40b Mon Sep 17 00:00:00 2001 From: alex-treebeard Date: Tue, 27 Feb 2024 15:18:41 +0000 Subject: [PATCH 10/56] dns refactor: variables --- README.md | 20 ++ examples/k3s-sso-https-dns/README.md | 0 examples/k3s-sso-https-dns/dex_conf.yaml | 34 +++ examples/k3s-sso-https-dns/main.tf | 295 +++++++++++++++++++++++ examples/k3s-sso-https-dns/profiles.yaml | 0 main.tf | 184 ++++++++++++-- 6 files changed, 511 insertions(+), 22 deletions(-) create mode 100644 examples/k3s-sso-https-dns/README.md create mode 100644 examples/k3s-sso-https-dns/dex_conf.yaml create mode 100644 examples/k3s-sso-https-dns/main.tf create mode 100644 examples/k3s-sso-https-dns/profiles.yaml diff --git a/README.md b/README.md index 0ac6048..a6d0859 100644 --- a/README.md +++ b/README.md @@ -31,6 +31,9 @@ To do so, follow the [k3s tutorial](https://github.com/treebeardtech/terraform-k ## Guides +In order to integrate Kubeflow with your production systems there are some changes you +may want to make: + ### Add Kubeflow to your Terraform module ```hcl @@ -69,6 +72,23 @@ Some considerations: 1. If you are calling this Terraform module from your own module, pass in a string to the `completed` variable in order to manage Kubeflow *after* changes to your other resources. (Note that `depends_on` does not work with this module) 2. If you already have Istio and Cert Manager installed, you will need to ensure Kubeflow works with them. See [examples/k3s-existing-istio](examples/k3s-existing-istio) for a configuration that we have tested like this. +### Enable Single-Sign-On (SSO) + + + +### Make Kubeflow available securely on a network using HTTPS + + + +### Host your Kubeflow on a domain name using DNS + +This is best done by using the external DNS operator. + +If you are new to external DNS, follow the [docs](https://kubernetes-sigs.github.io/external-dns/v0.14.0/) for setting up a deployment, then +use [this guide](https://kubernetes-sigs.github.io/external-dns/v0.14.0/tutorials/istio/) to connect external DNS to the istio gateway *service* for your Kubeflow deployment. + +### Create Profiles for your users + ### Teardown 1. Manually remove any manually created Kubeflow resources, e.g. Notebook Servers and Volumes diff --git a/examples/k3s-sso-https-dns/README.md b/examples/k3s-sso-https-dns/README.md new file mode 100644 index 0000000..e69de29 diff --git a/examples/k3s-sso-https-dns/dex_conf.yaml b/examples/k3s-sso-https-dns/dex_conf.yaml new file mode 100644 index 0000000..eee2058 --- /dev/null +++ b/examples/k3s-sso-https-dns/dex_conf.yaml @@ -0,0 +1,34 @@ +apiVersion: v1 +kind: ConfigMap +metadata: + name: dex + namespace: auth +data: + config.yaml: |- + issuer: http://dex.auth.svc.cluster.local:5556/dex + storage: + type: kubernetes + config: + inCluster: true + web: + http: 0.0.0.0:5556 + logger: + level: "debug" + format: text + oauth2: + skipApprovalScreen: false + enablePasswordDB: true + staticPasswords: + - email: user@example.com + hash: $2y$12$4K/VkmDd1q1Orb3xAt82zu8gk7Ad6ReFR4LCP9UeYE90NLiN9Df72 + # https://github.com/dexidp/dex/pull/1601/commits + # FIXME: Use hashFromEnv instead + username: user + userID: "15841185641784" + staticClients: + # https://github.com/dexidp/dex/pull/1664 + - idEnv: OIDC_CLIENT_ID + redirectURIs: ["/authservice/oidc/callback"] + name: 'Dex Login Application' + secretEnv: OIDC_CLIENT_SECRET + connectors: [] \ No newline at end of file diff --git a/examples/k3s-sso-https-dns/main.tf b/examples/k3s-sso-https-dns/main.tf new file mode 100644 index 0000000..252aec6 --- /dev/null +++ b/examples/k3s-sso-https-dns/main.tf @@ -0,0 +1,295 @@ + +terraform { + required_providers { + kustomization = { + source = "kbst/kustomization" + version = "~> 0.9.5" + } + helm = { + source = "hashicorp/helm" + version = "~> 2.12.1" + } + kubernetes = { + source = "hashicorp/kubernetes" + version = "~> 2.25.2" + } + } + backend "local" { + } +} + +variable "kubeconfig" { + type = string +} + +provider "kustomization" { + kubeconfig_path = var.kubeconfig +} + +provider "helm" { + kubernetes { + config_path = var.kubeconfig + } +} + +provider "kubernetes" { + config_path = var.kubeconfig +} + +## DNS Setup + +variable "aws_region" { + description = "AWS region" + type = string + default = "us-east-1" +} + +variable "aws_access_key_id" { + description = "AWS access key id" + type = string +} + +variable "aws_secret_access_key" { + description = "AWS secret access key" + type = string +} + +resource "kubernetes_namespace" "external_dns" { + metadata { + name = "external-dns" + } +} + +resource "kubernetes_secret" "aws_credentials" { + metadata { + name = "aws-credentials" + namespace = "external-dns" + } + data = { + aws_access_key_id = var.aws_access_key_id + aws_secret_access_key = var.aws_secret_access_key + } + type = "Opaque" + depends_on = [ + kubernetes_namespace.external_dns + ] +} + +resource "helm_release" "external_dns" { + name = "external-dns" + chart = "external-dns" + repository = "https://kubernetes-sigs.github.io/external-dns/" + namespace = "external-dns" + version = "1.14.0" + values = [ + <<-EOF + sources: + - istio-gateway + provider: aws + env: + - name: AWS_REGION + value: ${var.aws_region} + - name: AWS_ACCESS_KEY_ID + valueFrom: + secretKeyRef: + name: aws-credentials + key: aws_access_key_id + - name: AWS_SECRET_ACCESS_KEY + valueFrom: + secretKeyRef: + name: aws-credentials + key: aws_secret_access_key + EOF + ] + depends_on = [ + kubernetes_secret.aws_credentials + ] +} + +locals { + issuer_spec = <<-EOF +spec: + acme: + email: alex@treebeard.io + preferredChain: '' + privateKeySecretRef: + name: treebeard-issuer-account-key + server: https://acme-v02.api.letsencrypt.org/directory + solvers: + - dns01: + route53: + region: eu-west-1 + hostedZoneID: Z1026422YPHVFN3SS6AV + accessKeyIDSecretRef: + name: aws-credentials + key: aws_access_key_id + secretAccessKeySecretRef: + name: aws-credentials + key: aws_secret_access_key +EOF +} + +## HTTPS Setup (depends on DNS setup) + +variable "issuer_ref" { + description = "Issuer reference for cert-manager" + type = object({ + name = string + kind = string + group = string + }) + default = null +} + +## OIDC Setup + +variable "dex_config" { + type = object({ + oauth2 = object({ + skipApprovalScreen = bool + }) + enablePasswordDB = bool + staticPasswords = list(object({ + email = string + hash = string + username = string + userID = string + })) + staticClients = list(object({ + idEnv = string + redirectURIs = list(string) + name = string + secretEnv = string + })) + connectors = list(object({ + type = string + id = string + name = string + config = object({ + clientID = string + clientSecret = string + redirectURI = string + orgs = list(object({ + name = string + })) + loadAllGroups = bool + teamNameField = string + useLoginAsID = bool + }) + })) + }) + default = { + oauth2 = { + skipApprovalScreen = false + } + enablePasswordDB = true + staticPasswords = [] + staticClients = [ + { + idEnv = "OIDC_CLIENT_ID" + redirectURIs = ["/authservice/oidc/callback"] + name = "Dex Login Application" + secretEnv = "OIDC_CLIENT_SECRET" + } + ] + connectors = [ + { + type = "github" + id = "github" + name = "GitHub" + config = { + clientID = "" + clientSecret = "" + redirectURI = "" + orgs = [ + { + name = "" + } + ] + loadAllGroups = false + teamNameField = "slug" + useLoginAsID = true + } + } + ] + } +} + +## Authorization Setup + +variable "profile_configuration" { + type = object({ + users = list(object({ + id = string + email = string + })) + groups = list(object({ + id = string + users = list(string) + })) + profiles = list(object({ + name = string + members = list(object({ + group = string + access = object({ + role = string + notebooksAccess = bool + }) + })) + })) + }) + default = { + users = [ + { id = "user-1", email = "user1@example.com" }, + { id = "user-2", email = "user2@example.com" }, + { id = "user-3", email = "user3@example.com" } + ], + groups = [ + { id = "team-1--admins", users = ["user-1"] }, + { id = "team-1--users", users = ["user-1", "user-2", "user-3"] } + ], + profiles = [ + { + name = "team-1", + members = [ + { + group = "team-1--users", + access = { role = "edit", notebooksAccess = true } + } + ] + }, + { + name = "team-1-prod", + members = [ + { + group = "team-1--admins", + access = { role = "edit", notebooksAccess = true } + }, + { + group = "team-1--users", + access = { role = "view", notebooksAccess = false } + } + ] + } + ] + } +} + +resource null_resource "completed" { + depends_on = [ + helm_release.external_dns + ] +} + +module "treebeardkf" { + source = "../.." + hostname = "kf.example.com" + protocol = "https://" + port = "" + enable_kuberay = false + enable_mlflow = false + dex_config = var.dex_config + profile_configuration = var.profile_configuration + issuer_spec = local.issuer_spec + completed = null_resource.completed.id +} \ No newline at end of file diff --git a/examples/k3s-sso-https-dns/profiles.yaml b/examples/k3s-sso-https-dns/profiles.yaml new file mode 100644 index 0000000..e69de29 diff --git a/main.tf b/main.tf index a92cf85..2fb7ab9 100644 --- a/main.tf +++ b/main.tf @@ -15,6 +15,149 @@ terraform { } } +variable "enable_istio_ingressgateway_loadbalancer" { + type = bool + default = false +} + +variable "dex_config" { + type = object({ + oauth2 = object({ + skipApprovalScreen = bool + }) + enablePasswordDB = bool + staticPasswords = list(object({ + email = string + hash = string + username = string + userID = string + })) + staticClients = list(object({ + idEnv = string + redirectURIs = list(string) + name = string + secretEnv = string + })) + connectors = list(object({ + type = string + id = string + name = string + config = object({ + clientID = string + clientSecret = string + redirectURI = string + orgs = list(object({ + name = string + })) + loadAllGroups = bool + teamNameField = string + useLoginAsID = bool + }) + })) + }) + default = { + oauth2 = { + skipApprovalScreen = false + } + enablePasswordDB = true + staticPasswords = [] + staticClients = [ + { + idEnv = "OIDC_CLIENT_ID" + redirectURIs = ["/authservice/oidc/callback"] + name = "Dex Login Application" + secretEnv = "OIDC_CLIENT_SECRET" + } + ] + connectors = [ + { + type = "github" + id = "github" + name = "GitHub" + config = { + clientID = "" + clientSecret = "" + redirectURI = "" + orgs = [ + { + name = "" + } + ] + loadAllGroups = false + teamNameField = "slug" + useLoginAsID = true + } + } + ] + } +} + +variable "profile_configuration" { + type = object({ + users = list(object({ + id = string + email = string + })) + groups = list(object({ + id = string + users = list(string) + })) + profiles = list(object({ + name = string + members = list(object({ + group = string + access = object({ + role = string + notebooksAccess = bool + }) + })) + })) + }) + default = { + users = [ + { id = "user-1", email = "user1@example.com" }, + { id = "user-2", email = "user2@example.com" }, + { id = "user-3", email = "user3@example.com" } + ], + groups = [ + { id = "team-1--admins", users = ["user-1"] }, + { id = "team-1--users", users = ["user-1", "user-2", "user-3"] } + ], + profiles = [ + { + name = "team-1", + members = [ + { + group = "team-1--users", + access = { role = "edit", notebooksAccess = true } + } + ] + }, + { + name = "team-1-prod", + members = [ + { + group = "team-1--admins", + access = { role = "edit", notebooksAccess = true } + }, + { + group = "team-1--users", + access = { role = "view", notebooksAccess = false } + } + ] + } + ] + } +} + +variable "issuer_spec" { + type = string +} + +# variable "extra_manifests" { +# type = list(string) +# } + variable "completed" { type = string default = false @@ -118,28 +261,25 @@ data "kustomization_overlay" "istio_install" { ] - # dynamic "patches" { - # for_each = var.enable_external_dns ? [1] : [] - # content { - # target { - # kind = "Service" - # name = "istio-ingressgateway" - # namespace = "istio-system" - # } - # patch = < Date: Wed, 28 Feb 2024 15:30:08 +0000 Subject: [PATCH 11/56] https and dns --- .gitignore | 4 +- Makefile | 8 +- examples/k3s-sso-https-dns/.tfvars | 5 + examples/k3s-sso-https-dns/issuer/Chart.yaml | 3 + .../issuer/templates/issuer.yaml | 19 + examples/k3s-sso-https-dns/main.tf | 389 ++++++++++++++---- main.tf | 67 ++- overlays/istio-resources/kustomization.yaml | 4 +- 8 files changed, 396 insertions(+), 103 deletions(-) create mode 100644 examples/k3s-sso-https-dns/.tfvars create mode 100644 examples/k3s-sso-https-dns/issuer/Chart.yaml create mode 100644 examples/k3s-sso-https-dns/issuer/templates/issuer.yaml diff --git a/.gitignore b/.gitignore index 5547ba8..6a05f53 100644 --- a/.gitignore +++ b/.gitignore @@ -13,8 +13,8 @@ crash.*.log # password, private keys, and other secrets. These should not be part of version # control as they are data points which are potentially sensitive and subject # to change depending on the environment. -*.tfvars -*.tfvars.json +# *.tfvars +# *.tfvars.json # Ignore override files as they are usually used to override resources locally and so # are not checked in diff --git a/Makefile b/Makefile index 1709664..2e0769e 100644 --- a/Makefile +++ b/Makefile @@ -25,4 +25,10 @@ docs-rm-new: fmt: terraform fmt . modules/* examples/* -.PHONY: build \ No newline at end of file + +.PHONY: k3d-create +k3d-create: + k3d cluster create dev \ + -p "80:80@loadbalancer" \ + -p "443:443@loadbalancer" \ + --k3s-arg '--disable=traefik@server:0' \ No newline at end of file diff --git a/examples/k3s-sso-https-dns/.tfvars b/examples/k3s-sso-https-dns/.tfvars new file mode 100644 index 0000000..1dfaab7 --- /dev/null +++ b/examples/k3s-sso-https-dns/.tfvars @@ -0,0 +1,5 @@ +host = "eks1.dev.treebeard.io" +aws_profile = "treebeard-dev" +aws_region = "us-east-1" +cert_email_owner = "alex+dev@treebeard.io" +hosted_zone_id = "Z00570851YIWRU2E633DM" diff --git a/examples/k3s-sso-https-dns/issuer/Chart.yaml b/examples/k3s-sso-https-dns/issuer/Chart.yaml new file mode 100644 index 0000000..153fe68 --- /dev/null +++ b/examples/k3s-sso-https-dns/issuer/Chart.yaml @@ -0,0 +1,3 @@ +apiVersion: v2 +name: issuer +version: 0.1.0 diff --git a/examples/k3s-sso-https-dns/issuer/templates/issuer.yaml b/examples/k3s-sso-https-dns/issuer/templates/issuer.yaml new file mode 100644 index 0000000..b69e257 --- /dev/null +++ b/examples/k3s-sso-https-dns/issuer/templates/issuer.yaml @@ -0,0 +1,19 @@ +apiVersion: cert-manager.io/v1 +kind: Issuer +metadata: + name: treebeard-issuer + namespace: istio-system +spec: + # selfSigned: {} + + acme: + email: {{ .Values.certEmailOwner | required "Please provide a valid email address" }} + preferredChain: '' + privateKeySecretRef: + name: treebeard-issuer-account-key + server: https://acme-v02.api.letsencrypt.org/directory + solvers: + - dns01: + route53: + region: us-east-1 + hostedZoneID: {{ .Values.hostedZoneId | required "Please provide a valid hosted zone ID" }} diff --git a/examples/k3s-sso-https-dns/main.tf b/examples/k3s-sso-https-dns/main.tf index 252aec6..687f7e8 100644 --- a/examples/k3s-sso-https-dns/main.tf +++ b/examples/k3s-sso-https-dns/main.tf @@ -18,68 +18,223 @@ terraform { } } -variable "kubeconfig" { - type = string +variable host { } -provider "kustomization" { - kubeconfig_path = var.kubeconfig +variable cert_email_owner { } -provider "helm" { - kubernetes { - config_path = var.kubeconfig +variable hosted_zone_id { +} + +data "aws_availability_zones" "available" {} + +locals { + name = basename(path.cwd) + azs = slice(data.aws_availability_zones.available.names, 0, 3) + tags = { + "tf" : "true" + "Name" : local.name } + cloud_cidr = "10.0.0.0/16" } -provider "kubernetes" { - config_path = var.kubeconfig +variable "aws_region" { + description = "AWS region to launch servers." } -## DNS Setup +variable "aws_profile" { + description = "AWS profile to use for authentication." +} -variable "aws_region" { - description = "AWS region" - type = string - default = "us-east-1" +provider "aws" { + region = var.aws_region + profile = var.aws_profile + default_tags { + tags = { + "tf" : "true" + "Name" : local.name + } + } } -variable "aws_access_key_id" { - description = "AWS access key id" - type = string + +module "vpc" { + source = "terraform-aws-modules/vpc/aws" + version = "5.1.2" + + name = "kubeflow-vpc" + cidr = local.cloud_cidr + map_public_ip_on_launch = true + + azs = local.azs + public_subnets = ["10.0.1.0/24", "10.0.2.0/24"] + private_subnets = ["10.0.3.0/24", "10.0.4.0/24"] + + enable_nat_gateway = true + single_nat_gateway = true + + enable_dns_hostnames = true + enable_dns_support = true } -variable "aws_secret_access_key" { - description = "AWS secret access key" - type = string +module "eks" { + source = "terraform-aws-modules/eks/aws" + version = "~> 19.20" + + cluster_name = "cluster" + cluster_version = "1.28" + cluster_endpoint_public_access = true + + vpc_id = module.vpc.vpc_id + subnet_ids = module.vpc.public_subnets + create_cloudwatch_log_group = false + cluster_encryption_config = {} + + node_security_group_additional_rules = { + ssh = { + type = "ingress" + description = "Allow SSH inbound traffic from anywhere" + from_port = 22 + to_port = 22 + protocol = "tcp" + cidr_blocks = ["0.0.0.0/0"] + }, + istio_injection_webhook = { + description = "Allow istio injection" + protocol = "tcp" + from_port = "15017" + to_port = "15017" + type = "ingress" + source_cluster_security_group = true + } + } + eks_managed_node_groups = { + amzn_linux = { + instance_types = ["t3.xlarge"] + min_size = 0 + max_size = 1 + desired_size = 1 + } + } + tags = local.tags } -resource "kubernetes_namespace" "external_dns" { - metadata { - name = "external-dns" +provider "kubernetes" { + host = module.eks.cluster_endpoint + cluster_ca_certificate = base64decode(module.eks.cluster_certificate_authority_data) + exec { + command = "aws" + api_version = "client.authentication.k8s.io/v1beta1" + args = [ + "--region", + var.aws_region, + "eks", + "get-token", + "--cluster-name", + module.eks.cluster_name + ] + env = { + name = "AWS_PROFILE" + value = var.aws_profile + } + } +} + +provider "helm" { + kubernetes { + host = module.eks.cluster_endpoint + cluster_ca_certificate = base64decode(module.eks.cluster_certificate_authority_data) + exec { + command = "aws" + api_version = "client.authentication.k8s.io/v1beta1" + args = [ + "--region", + var.aws_region, + "eks", + "get-token", + "--cluster-name", + module.eks.cluster_name + ] + env = { + name = "AWS_PROFILE" + value = var.aws_profile + } + } } } -resource "kubernetes_secret" "aws_credentials" { +provider "kustomization" { + kubeconfig_path = "/home/vscode/.kube/eks.yaml" +} + +module "ebs_csi_role" { + source = "terraform-aws-modules/iam/aws//modules/iam-role-for-service-accounts-eks" + version = "5.34.0" + role_name = "irsa-ebs-csi" + attach_ebs_csi_policy = true + oidc_providers = { + main = { + provider_arn = module.eks.oidc_provider_arn + namespace_service_accounts = ["kube-system:ebs-csi-controller-sa"] + } + } +} + +resource "kubernetes_annotations" "default-storageclass" { + api_version = "storage.k8s.io/v1" + kind = "StorageClass" + force = "true" + metadata { - name = "aws-credentials" - namespace = "external-dns" + name = "gp2" } - data = { - aws_access_key_id = var.aws_access_key_id - aws_secret_access_key = var.aws_secret_access_key + annotations = { + "storageclass.kubernetes.io/is-default-class" = "false" } - type = "Opaque" - depends_on = [ - kubernetes_namespace.external_dns +} + +resource "helm_release" "ebs_csi_driver" { + name = "aws-ebs-csi-driver" + chart = "aws-ebs-csi-driver" + repository = "https://kubernetes-sigs.github.io/aws-ebs-csi-driver" + namespace = "kube-system" + version = "2.28.1" + values = [ + < Date: Wed, 28 Feb 2024 15:32:11 +0000 Subject: [PATCH 12/56] fmt --- examples/k3s-sso-https-dns/main.tf | 100 ++++++++++++++--------------- main.tf | 64 +++++++++--------- 2 files changed, 82 insertions(+), 82 deletions(-) diff --git a/examples/k3s-sso-https-dns/main.tf b/examples/k3s-sso-https-dns/main.tf index 687f7e8..b4e82ae 100644 --- a/examples/k3s-sso-https-dns/main.tf +++ b/examples/k3s-sso-https-dns/main.tf @@ -18,13 +18,13 @@ terraform { } } -variable host { +variable "host" { } -variable cert_email_owner { +variable "cert_email_owner" { } -variable hosted_zone_id { +variable "hosted_zone_id" { } data "aws_availability_zones" "available" {} @@ -217,9 +217,9 @@ storageClasses: ## DNS Setup module "external_dns_role" { - source = "terraform-aws-modules/iam/aws//modules/iam-role-for-service-accounts-eks" - version = "5.34.0" - role_name = "external-dns" + source = "terraform-aws-modules/iam/aws//modules/iam-role-for-service-accounts-eks" + version = "5.34.0" + role_name = "external-dns" attach_external_dns_policy = true oidc_providers = { main = { @@ -230,12 +230,12 @@ module "external_dns_role" { } resource "helm_release" "external_dns" { - name = "external-dns" - chart = "external-dns" - repository = "https://kubernetes-sigs.github.io/external-dns/" - namespace = "external-dns" + name = "external-dns" + chart = "external-dns" + repository = "https://kubernetes-sigs.github.io/external-dns/" + namespace = "external-dns" create_namespace = true - version = "1.14.0" + version = "1.14.0" values = [ <<-EOF sources: @@ -258,9 +258,9 @@ resource "helm_release" "external_dns" { ## HTTPS Setup module "cert_manager_role" { - source = "terraform-aws-modules/iam/aws//modules/iam-role-for-service-accounts-eks" - version = "5.34.0" - role_name = "cert-manager" + source = "terraform-aws-modules/iam/aws//modules/iam-role-for-service-accounts-eks" + version = "5.34.0" + role_name = "cert-manager" attach_cert_manager_policy = true oidc_providers = { main = { @@ -277,7 +277,7 @@ resource "helm_release" "cert_manager" { repository = "https://charts.jetstack.io" version = "1.12.8" create_namespace = true - depends_on = [ + depends_on = [ helm_release.external_dns ] values = [ @@ -296,9 +296,9 @@ resource "helm_release" "cert_manager" { } resource "helm_release" "issuer" { - name = "issuer" - namespace = "cert-manager" - chart = "${path.module}/issuer" + name = "issuer" + namespace = "cert-manager" + chart = "${path.module}/issuer" values = [ < Date: Wed, 28 Feb 2024 16:32:43 +0000 Subject: [PATCH 13/56] eks example deps --- examples/k3s-sso-https-dns/main.tf | 92 +++++++++++++++++----- main.tf | 10 +-- util.tf | 121 ----------------------------- 3 files changed, 76 insertions(+), 147 deletions(-) delete mode 100644 util.tf diff --git a/examples/k3s-sso-https-dns/main.tf b/examples/k3s-sso-https-dns/main.tf index b4e82ae..b04b501 100644 --- a/examples/k3s-sso-https-dns/main.tf +++ b/examples/k3s-sso-https-dns/main.tf @@ -168,6 +168,20 @@ provider "kustomization" { kubeconfig_path = "/home/vscode/.kube/eks.yaml" } +resource "null_resource" "cluster_ready" { + triggers = { + always_run = "${timestamp()}" + } + + provisioner "local-exec" { + command = "echo 'Cluster is ready!'" + } + depends_on = [ + module.vpc, + module.eks + ] +} + module "ebs_csi_role" { source = "terraform-aws-modules/iam/aws//modules/iam-role-for-service-accounts-eks" version = "5.34.0" @@ -179,6 +193,9 @@ module "ebs_csi_role" { namespace_service_accounts = ["kube-system:ebs-csi-controller-sa"] } } + depends_on = [ + null_resource.cluster_ready + ] } resource "kubernetes_annotations" "default-storageclass" { @@ -192,6 +209,9 @@ resource "kubernetes_annotations" "default-storageclass" { annotations = { "storageclass.kubernetes.io/is-default-class" = "false" } + depends_on = [ + null_resource.cluster_ready + ] } resource "helm_release" "ebs_csi_driver" { @@ -212,6 +232,9 @@ storageClasses: storageclass.kubernetes.io/is-default-class: "true" EOF ] + depends_on = [ + null_resource.cluster_ready + ] } ## DNS Setup @@ -227,6 +250,9 @@ module "external_dns_role" { namespace_service_accounts = ["external-dns:external-dns"] } } + depends_on = [ + null_resource.cluster_ready + ] } resource "helm_release" "external_dns" { @@ -251,7 +277,7 @@ resource "helm_release" "external_dns" { EOF ] depends_on = [ - module.external_dns_role + null_resource.cluster_ready ] } @@ -295,6 +321,7 @@ resource "helm_release" "cert_manager" { ] } + resource "helm_release" "issuer" { name = "issuer" namespace = "cert-manager" @@ -310,17 +337,20 @@ resource "helm_release" "issuer" { ] } -# data "kustomization_build" "issuer" { -# path = "${path.module}/issuer" -# } +resource "null_resource" "core_addons" { + triggers = { + always_run = "${timestamp()}" + } + + provisioner "local-exec" { + when = destroy + command = "echo 'Waiting for addons to cleanup DNS/Loadbalancers' && sleep 60" + } -# module "issuer" { -# source = "../../modules/kust" -# build = data.kustomization_build.issuer -# depends_on = [ -# helm_release.cert_manager -# ] -# } + depends_on = [ + helm_release.issuer + ] +} resource "helm_release" "istio_base" { name = "istio-base" @@ -329,13 +359,13 @@ resource "helm_release" "istio_base" { repository = "https://istio-release.storage.googleapis.com/charts" version = "1.18.7" create_namespace = true - depends_on = [ - helm_release.issuer - ] values = [ < -n kubeflow and adding the commented out vendors -# updating the notebook runtimeClassName to 'nvidia' - -variable "enable_gpu_operator" { - type = bool - default = false -} - -resource "helm_release" "gpu_operator" { - count = var.enable_gpu_operator ? 1 : 0 - name = "gpu-operator" - chart = "gpu-operator" - repository = "https://helm.ngc.nvidia.com/nvidia" - namespace = kubernetes_namespace.tkf_system.metadata[0].name - depends_on = [ - kubernetes_namespace.tkf_system - ] - values = [ - < Date: Wed, 28 Feb 2024 16:47:30 +0000 Subject: [PATCH 14/56] fix eks example, rm mlflow, kuberay --- .../issuer/templates/issuer.yaml | 1 - examples/k3s-sso-https-dns/main.tf | 175 ++-------------- main.tf | 196 ------------------ mlflow.tf | 36 ---- 4 files changed, 17 insertions(+), 391 deletions(-) delete mode 100644 mlflow.tf diff --git a/examples/k3s-sso-https-dns/issuer/templates/issuer.yaml b/examples/k3s-sso-https-dns/issuer/templates/issuer.yaml index b69e257..5faf650 100644 --- a/examples/k3s-sso-https-dns/issuer/templates/issuer.yaml +++ b/examples/k3s-sso-https-dns/issuer/templates/issuer.yaml @@ -2,7 +2,6 @@ apiVersion: cert-manager.io/v1 kind: Issuer metadata: name: treebeard-issuer - namespace: istio-system spec: # selfSigned: {} diff --git a/examples/k3s-sso-https-dns/main.tf b/examples/k3s-sso-https-dns/main.tf index b04b501..aaadd7b 100644 --- a/examples/k3s-sso-https-dns/main.tf +++ b/examples/k3s-sso-https-dns/main.tf @@ -321,22 +321,6 @@ resource "helm_release" "cert_manager" { ] } - -resource "helm_release" "issuer" { - name = "issuer" - namespace = "cert-manager" - chart = "${path.module}/issuer" - values = [ - < Date: Wed, 28 Feb 2024 17:07:01 +0000 Subject: [PATCH 15/56] split into files --- examples/k3s-sso-https-dns/main.tf | 18 +- kf_apps.tf | 88 +++++ kf_auth.tf | 77 +++++ kf_core.tf | 90 +++++ kf_dependencies.tf | 94 ++++++ kf_kserve.tf | 84 +++++ kf_pipelines.tf | 28 ++ main.tf | 509 ----------------------------- variables.tf | 51 +++ 9 files changed, 521 insertions(+), 518 deletions(-) create mode 100644 kf_apps.tf create mode 100644 kf_auth.tf create mode 100644 kf_core.tf create mode 100644 kf_dependencies.tf create mode 100644 kf_kserve.tf create mode 100644 kf_pipelines.tf diff --git a/examples/k3s-sso-https-dns/main.tf b/examples/k3s-sso-https-dns/main.tf index aaadd7b..cde22cc 100644 --- a/examples/k3s-sso-https-dns/main.tf +++ b/examples/k3s-sso-https-dns/main.tf @@ -169,9 +169,9 @@ provider "kustomization" { } resource "null_resource" "cluster_ready" { - triggers = { - always_run = "${timestamp()}" - } + # triggers = { + # always_run = "${timestamp()}" + # } provisioner "local-exec" { command = "echo 'Cluster is ready!'" @@ -322,9 +322,9 @@ resource "helm_release" "cert_manager" { } resource "null_resource" "core_addons" { - triggers = { - always_run = "${timestamp()}" - } + # triggers = { + # always_run = "${timestamp()}" + # } provisioner "local-exec" { when = destroy @@ -403,9 +403,9 @@ resource "helm_release" "istio_ingressgateway" { } resource "null_resource" "istio" { - triggers = { - always_run = "${timestamp()}" - } + # triggers = { + # always_run = "${timestamp()}" + # } provisioner "local-exec" { command = "echo 'Istio is ready!'" diff --git a/kf_apps.tf b/kf_apps.tf new file mode 100644 index 0000000..94a0560 --- /dev/null +++ b/kf_apps.tf @@ -0,0 +1,88 @@ + +data "kustomization_build" "central_dashboard" { + path = "${path.module}/overlays/centraldashboard" +} + +module "central_dashboard" { + source = "./modules/kust" + build = data.kustomization_build.central_dashboard + depends_on = [ + module.models_web_app, + module.kubeflow_istio_resources, + module.kubeflow_pipelines, + module.kserve + ] +} + +data "kustomization_build" "admission_webhook" { + path = "${path.module}/submodules/manifests/apps/admission-webhook/upstream/overlays/cert-manager" +} + +module "admission_webhook" { + source = "./modules/kust" + build = data.kustomization_build.admission_webhook + depends_on = [ + module.central_dashboard + ] +} + +data "kustomization_build" "notebook_controller" { + path = "${path.module}/submodules/manifests/apps/jupyter/notebook-controller/upstream/overlays/kubeflow" +} + +module "notebook_controller" { + source = "./modules/kust" + build = data.kustomization_build.notebook_controller + depends_on = [ + module.admission_webhook + ] +} + +data "kustomization_build" "jupyter_web_app" { + path = "${path.module}/submodules/manifests/apps/jupyter/jupyter-web-app/upstream/overlays/istio" +} + +module "jupyter_web_app" { + source = "./modules/kust" + build = data.kustomization_build.jupyter_web_app + depends_on = [ + module.notebook_controller + ] +} + +data "kustomization_build" "pvc_viewer_controller" { + path = "${path.module}/submodules/manifests/apps/pvcviewer-controller/upstream/default" +} + +module "pvc_viewer_controller" { + source = "./modules/kust" + build = data.kustomization_build.pvc_viewer_controller + depends_on = [ + module.jupyter_web_app + ] +} + +data "kustomization_build" "profiles_kfam" { + path = "${path.module}/submodules/manifests/apps/profiles/upstream/overlays/kubeflow" +} + +module "profiles_kfam" { + source = "./modules/kust" + build = data.kustomization_build.profiles_kfam + depends_on = [ + module.pvc_viewer_controller + ] +} + +data "kustomization_build" "volumes_web_app" { + path = "${path.module}/submodules/manifests/apps/volumes-web-app/upstream/overlays/istio" +} + +module "volumes_web_app" { + source = "./modules/kust" + build = data.kustomization_build.volumes_web_app + depends_on = [ + module.profiles_kfam + ] +} + diff --git a/kf_auth.tf b/kf_auth.tf new file mode 100644 index 0000000..98c6542 --- /dev/null +++ b/kf_auth.tf @@ -0,0 +1,77 @@ + + +data "kustomization_overlay" "oidc_authservice" { + config_map_generator { + name = "oidc-authservice-parameters" + behavior = "merge" + literals = [ + # "OIDC_PROVIDER=${var.protocol}${var.hostname}${var.port}/dex" + ] + } + + resources = [ + "${path.module}/submodules/manifests/common/oidc-client/oidc-authservice/base", + ] +} + +module "oidc_authservice" { + source = "./modules/kust" + build = data.kustomization_overlay.oidc_authservice + depends_on = [ + module.istio_install, + var.dependency + ] +} + +data "kustomization_overlay" "dex" { + resources = [ + "${path.module}/submodules/manifests/common/dex/overlays/istio" + ] + patches { + patch = < Date: Wed, 28 Feb 2024 17:07:17 +0000 Subject: [PATCH 16/56] fmt --- examples/k3s-sso-https-dns/main.tf | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/examples/k3s-sso-https-dns/main.tf b/examples/k3s-sso-https-dns/main.tf index cde22cc..09a35b6 100644 --- a/examples/k3s-sso-https-dns/main.tf +++ b/examples/k3s-sso-https-dns/main.tf @@ -327,8 +327,8 @@ resource "null_resource" "core_addons" { # } provisioner "local-exec" { - when = destroy - command = "echo 'Waiting for addons to cleanup DNS/Loadbalancers' && sleep 60" + when = destroy + command = "echo 'Waiting for addons to cleanup DNS/Loadbalancers' && sleep 60" } depends_on = [ @@ -428,12 +428,12 @@ variable "enable_treebeardkf" { } module "treebeardkf" { - count = var.enable_treebeardkf ? 1 : 0 + count = var.enable_treebeardkf ? 1 : 0 source = "../.." hostname = var.host enable_istio_base = false enable_istiod = false enable_istio_resources = true enable_cert_manager = false - dependency = null_resource.istio.id + dependency = null_resource.istio.id } \ No newline at end of file From 90718d19113781ef47b496bbedfb2507ec007c7c Mon Sep 17 00:00:00 2001 From: alex-treebeard Date: Wed, 28 Feb 2024 17:08:48 +0000 Subject: [PATCH 17/56] rm comments --- kf_core.tf | 8 -------- kf_kserve.tf | 3 --- 2 files changed, 11 deletions(-) diff --git a/kf_core.tf b/kf_core.tf index 6cf0f32..319e191 100644 --- a/kf_core.tf +++ b/kf_core.tf @@ -1,7 +1,3 @@ - - -## kubeflow namespace - data "kustomization_build" "kubeflow_namespace" { path = "${path.module}/submodules/manifests/common/kubeflow-namespace/base" } @@ -14,8 +10,6 @@ module "kubeflow_namespace" { ] } -## kubeflow roles - data "kustomization_build" "kubeflow_roles" { path = "${path.module}/submodules/manifests/common/kubeflow-roles/base" } @@ -28,8 +22,6 @@ module "kubeflow_roles" { ] } -## kubeflow istio resources - data "kustomization_overlay" "kubeflow_istio_resources" { count = var.enable_istio_resources ? 1 : 0 resources = [ diff --git a/kf_kserve.tf b/kf_kserve.tf index 87455a3..56e00ac 100644 --- a/kf_kserve.tf +++ b/kf_kserve.tf @@ -1,6 +1,3 @@ - -## knative - data "kustomization_overlay" "knative_serving" { count = var.enable_kserve ? 1 : 0 resources = [ From 86092477a467ed008744077a7865a2a8abd15070 Mon Sep 17 00:00:00 2001 From: alex-treebeard Date: Wed, 28 Feb 2024 17:14:36 +0000 Subject: [PATCH 18/56] refactor main --- kf_apps.tf | 30 ++++++++++++++++++++++++++++++ kf_auth.tf | 2 +- kubeflow.tf | 30 ------------------------------ 3 files changed, 31 insertions(+), 31 deletions(-) delete mode 100644 kubeflow.tf diff --git a/kf_apps.tf b/kf_apps.tf index 94a0560..91dda06 100644 --- a/kf_apps.tf +++ b/kf_apps.tf @@ -86,3 +86,33 @@ module "volumes_web_app" { ] } +data "kustomization_overlay" "kubeflow_profile" { + resources = [ + "${path.module}/overlays/profile" + ] + + patches { + target { + kind = "Profile" + name = "prod" + } + patch = < Date: Wed, 28 Feb 2024 17:19:39 +0000 Subject: [PATCH 19/56] refactor eks example --- examples/k3s-sso-https-dns/addons.tf | 154 +++++++++ examples/k3s-sso-https-dns/eks.tf | 148 +++++++++ examples/k3s-sso-https-dns/istio.tf | 78 +++++ examples/k3s-sso-https-dns/kubeflow.tf | 10 + examples/k3s-sso-https-dns/main.tf | 420 ------------------------ examples/k3s-sso-https-dns/variables.tf | 23 ++ 6 files changed, 413 insertions(+), 420 deletions(-) create mode 100644 examples/k3s-sso-https-dns/addons.tf create mode 100644 examples/k3s-sso-https-dns/eks.tf create mode 100644 examples/k3s-sso-https-dns/istio.tf create mode 100644 examples/k3s-sso-https-dns/kubeflow.tf create mode 100644 examples/k3s-sso-https-dns/variables.tf diff --git a/examples/k3s-sso-https-dns/addons.tf b/examples/k3s-sso-https-dns/addons.tf new file mode 100644 index 0000000..828e31a --- /dev/null +++ b/examples/k3s-sso-https-dns/addons.tf @@ -0,0 +1,154 @@ + +module "ebs_csi_role" { + source = "terraform-aws-modules/iam/aws//modules/iam-role-for-service-accounts-eks" + version = "5.34.0" + role_name = "irsa-ebs-csi" + attach_ebs_csi_policy = true + oidc_providers = { + main = { + provider_arn = module.eks.oidc_provider_arn + namespace_service_accounts = ["kube-system:ebs-csi-controller-sa"] + } + } + depends_on = [ + null_resource.cluster_ready + ] +} + +resource "kubernetes_annotations" "default-storageclass" { + api_version = "storage.k8s.io/v1" + kind = "StorageClass" + force = "true" + + metadata { + name = "gp2" + } + annotations = { + "storageclass.kubernetes.io/is-default-class" = "false" + } + depends_on = [ + null_resource.cluster_ready + ] +} + +resource "helm_release" "ebs_csi_driver" { + name = "aws-ebs-csi-driver" + chart = "aws-ebs-csi-driver" + repository = "https://kubernetes-sigs.github.io/aws-ebs-csi-driver" + namespace = "kube-system" + version = "2.28.1" + values = [ + < Date: Wed, 28 Feb 2024 17:25:56 +0000 Subject: [PATCH 20/56] rename chart dir --- examples/k3s-sso-https-dns/{ => charts}/issuer/Chart.yaml | 0 .../k3s-sso-https-dns/{ => charts}/issuer/templates/issuer.yaml | 0 examples/k3s-sso-https-dns/istio.tf | 2 +- 3 files changed, 1 insertion(+), 1 deletion(-) rename examples/k3s-sso-https-dns/{ => charts}/issuer/Chart.yaml (100%) rename examples/k3s-sso-https-dns/{ => charts}/issuer/templates/issuer.yaml (100%) diff --git a/examples/k3s-sso-https-dns/issuer/Chart.yaml b/examples/k3s-sso-https-dns/charts/issuer/Chart.yaml similarity index 100% rename from examples/k3s-sso-https-dns/issuer/Chart.yaml rename to examples/k3s-sso-https-dns/charts/issuer/Chart.yaml diff --git a/examples/k3s-sso-https-dns/issuer/templates/issuer.yaml b/examples/k3s-sso-https-dns/charts/issuer/templates/issuer.yaml similarity index 100% rename from examples/k3s-sso-https-dns/issuer/templates/issuer.yaml rename to examples/k3s-sso-https-dns/charts/issuer/templates/issuer.yaml diff --git a/examples/k3s-sso-https-dns/istio.tf b/examples/k3s-sso-https-dns/istio.tf index bedb211..6d9c43e 100644 --- a/examples/k3s-sso-https-dns/istio.tf +++ b/examples/k3s-sso-https-dns/istio.tf @@ -17,7 +17,7 @@ resource "helm_release" "istio_base" { resource "helm_release" "issuer" { name = "issuer" namespace = "istio-system" - chart = "${path.module}/issuer" + chart = "${path.module}/charts/issuer" values = [ < Date: Wed, 28 Feb 2024 19:00:31 +0000 Subject: [PATCH 21/56] dependency fix --- examples/k3s-sso-https-dns/addons.tf | 1 + examples/k3s-sso-https-dns/kubeflow.tf | 10 +++++++++- 2 files changed, 10 insertions(+), 1 deletion(-) diff --git a/examples/k3s-sso-https-dns/addons.tf b/examples/k3s-sso-https-dns/addons.tf index 828e31a..7c81f39 100644 --- a/examples/k3s-sso-https-dns/addons.tf +++ b/examples/k3s-sso-https-dns/addons.tf @@ -149,6 +149,7 @@ resource "null_resource" "core_addons" { } depends_on = [ + helm_release.ebs_csi_driver, helm_release.cert_manager ] } \ No newline at end of file diff --git a/examples/k3s-sso-https-dns/kubeflow.tf b/examples/k3s-sso-https-dns/kubeflow.tf index a9799d9..9940f86 100644 --- a/examples/k3s-sso-https-dns/kubeflow.tf +++ b/examples/k3s-sso-https-dns/kubeflow.tf @@ -1,3 +1,11 @@ +resource "null_resource" "dependency" { + depends_on = [ + null_resource.cluster_ready, + null_resource.core_addons, + null_resource.istio, + ] +} + module "treebeardkf" { count = var.enable_treebeardkf ? 1 : 0 source = "../.." @@ -6,5 +14,5 @@ module "treebeardkf" { enable_istiod = false enable_istio_resources = true enable_cert_manager = false - dependency = null_resource.istio.id + dependency = null_resource.dependency.id } \ No newline at end of file From 7e01584263d9cd60f2a6bf9a8ee5c0b2d9f3b350 Mon Sep 17 00:00:00 2001 From: alex-treebeard Date: Wed, 28 Feb 2024 20:37:47 +0000 Subject: [PATCH 22/56] fix examples --- examples/aks/main.tf | 12 ++- examples/eks/main.tf | 19 ++-- examples/k3s-sso-https-dns/kubeflow.tf | 1 + examples/k3s/main.tf | 7 +- kf_core.tf | 89 +++++++++++-------- overlays/istio-resources/cert.yaml | 24 ----- overlays/istio-resources/kustomization.yaml | 4 +- .../istio-resources/patches/resources.yaml | 18 ---- variables.tf | 7 +- 9 files changed, 82 insertions(+), 99 deletions(-) delete mode 100644 overlays/istio-resources/patches/resources.yaml diff --git a/examples/aks/main.tf b/examples/aks/main.tf index 5e7edb6..c544dfb 100644 --- a/examples/aks/main.tf +++ b/examples/aks/main.tf @@ -61,11 +61,15 @@ provider "kustomization" { kubeconfig_raw = azurerm_kubernetes_cluster.aks.kube_config_raw } +variable "enable_treebeardkf" { + description = "Enable Treebeard" + type = bool + default = false +} + module "treebeardkf" { + count = var.enable_treebeardkf ? 1 : 0 source = "../.." hostname = "kf.example.com" - protocol = "https://" - port = "" - enable_kuberay = false - enable_mlflow = false + enable_https = true } \ No newline at end of file diff --git a/examples/eks/main.tf b/examples/eks/main.tf index e3cdb69..4dcf964 100644 --- a/examples/eks/main.tf +++ b/examples/eks/main.tf @@ -156,8 +156,13 @@ provider "helm" { } } +variable "kubeconfig" { + description = "Path to the kubeconfig file" + type = string +} + provider "kustomization" { - kubeconfig_path = "/home/vscode/.kube/eks.yaml" + kubeconfig_raw = file(var.kubeconfig) } module "iam_eks_role" { @@ -206,11 +211,15 @@ storageClasses: ] } +variable "enable_treebeardkf" { + description = "Enable Treebeard" + type = bool + default = false +} + module "treebeardkf" { + count = var.enable_treebeardkf ? 1 : 0 source = "../.." hostname = "kf.example.com" - protocol = "https://" - port = "" - enable_kuberay = false - enable_mlflow = false + enable_https = true } \ No newline at end of file diff --git a/examples/k3s-sso-https-dns/kubeflow.tf b/examples/k3s-sso-https-dns/kubeflow.tf index 9940f86..2dc3eac 100644 --- a/examples/k3s-sso-https-dns/kubeflow.tf +++ b/examples/k3s-sso-https-dns/kubeflow.tf @@ -10,6 +10,7 @@ module "treebeardkf" { count = var.enable_treebeardkf ? 1 : 0 source = "../.." hostname = var.host + enable_https = true enable_istio_base = false enable_istiod = false enable_istio_resources = true diff --git a/examples/k3s/main.tf b/examples/k3s/main.tf index c595954..6e98a57 100644 --- a/examples/k3s/main.tf +++ b/examples/k3s/main.tf @@ -38,9 +38,6 @@ provider "kubernetes" { module "treebeardkf" { source = "../.." - hostname = "kf.example.com" - protocol = "https://" - port = "" - enable_kuberay = false - enable_mlflow = false + enable_https = true + hostname = "kubeflow.example.com" } \ No newline at end of file diff --git a/kf_core.tf b/kf_core.tf index 319e191..b202dab 100644 --- a/kf_core.tf +++ b/kf_core.tf @@ -1,3 +1,49 @@ +locals { + cert_resource = var.enable_https ? yamlencode({ + apiVersion: "cert-manager.io/v1", + kind: "Certificate", + metadata: { + name: "gateway-cert", + namespace: "istio-system" + }, + spec: { + commonName: var.hostname, + dnsNames: [var.hostname] + issuerRef: { + kind: "Issuer", + name: "treebeard-issuer" + } + secretName: "gateway-cert" + } + }) : "" + + gateway_patch = yamlencode({ + apiVersion: "networking.istio.io/v1alpha3", + kind: "Gateway", + metadata: { + name: "kubeflow-gateway", + namespace: "kubeflow", + }, + spec: { + selector: { + istio: "ingressgateway", + }, + servers: [{ + hosts: [var.hostname], + port: { + name: var.enable_https ? "https" : "http", + number: var.enable_https ? 443 : 80, + protocol: var.enable_https ? "HTTPS" : "HTTP", + }, + tls: var.enable_https ? { + credentialName: "gateway-cert", + mode: "SIMPLE", + } : null, + }], + }, + }) +} + data "kustomization_build" "kubeflow_namespace" { path = "${path.module}/submodules/manifests/common/kubeflow-namespace/base" } @@ -27,48 +73,13 @@ data "kustomization_overlay" "kubeflow_istio_resources" { resources = [ "${path.module}/overlays/istio-resources" ] + patches { - patch = < -# preferredChain: '' -# privateKeySecretRef: -# name: treebeard-issuer-account-key -# server: https://acme-v02.api.letsencrypt.org/directory -# solvers: -# - dns01: -# route53: -# region: -# hostedZoneID: -# accessKeyIDSecretRef: -# name: aws-credentials -# key: aws_access_key_id -# secretAccessKeySecretRef: -# name: aws-credentials -# key: aws_secret_access_key ---- apiVersion: cert-manager.io/v1 kind: Certificate metadata: diff --git a/overlays/istio-resources/kustomization.yaml b/overlays/istio-resources/kustomization.yaml index 911d8b9..3898782 100644 --- a/overlays/istio-resources/kustomization.yaml +++ b/overlays/istio-resources/kustomization.yaml @@ -2,6 +2,4 @@ apiVersion: kustomize.config.k8s.io/v1beta1 kind: Kustomization resources: - ../../submodules/manifests/common/istio-1-17/kubeflow-istio-resources/base -- cert.yaml -patches: # only needed when using TLS loadbalancer -- path: patches/resources.yaml \ No newline at end of file +- cert.yaml \ No newline at end of file diff --git a/overlays/istio-resources/patches/resources.yaml b/overlays/istio-resources/patches/resources.yaml deleted file mode 100644 index 461e781..0000000 --- a/overlays/istio-resources/patches/resources.yaml +++ /dev/null @@ -1,18 +0,0 @@ -apiVersion: networking.istio.io/v1alpha3 -kind: Gateway -metadata: - name: kubeflow-gateway - namespace: kubeflow -spec: - selector: - istio: ingressgateway - servers: - - hosts: - - '*' - port: - name: https - number: 443 - protocol: HTTPS - tls: - credentialName: gateway-cert - mode: SIMPLE \ No newline at end of file diff --git a/variables.tf b/variables.tf index 7b4080d..4a11b7c 100644 --- a/variables.tf +++ b/variables.tf @@ -11,7 +11,12 @@ variable "dependency" { variable "hostname" { type = string - default = "localhost" + default = "*" +} + +variable enable_https { + type = bool + default = false } variable "enable_cert_manager" { From 3f75516d192e14ce9fddfdbe726f830326002a1d Mon Sep 17 00:00:00 2001 From: alex-treebeard Date: Wed, 28 Feb 2024 20:38:20 +0000 Subject: [PATCH 23/56] fmt --- examples/aks/main.tf | 6 ++--- examples/eks/main.tf | 6 ++--- examples/k3s/main.tf | 6 ++--- kf_core.tf | 60 ++++++++++++++++++++++---------------------- variables.tf | 2 +- 5 files changed, 40 insertions(+), 40 deletions(-) diff --git a/examples/aks/main.tf b/examples/aks/main.tf index c544dfb..37b0c37 100644 --- a/examples/aks/main.tf +++ b/examples/aks/main.tf @@ -68,8 +68,8 @@ variable "enable_treebeardkf" { } module "treebeardkf" { - count = var.enable_treebeardkf ? 1 : 0 - source = "../.." - hostname = "kf.example.com" + count = var.enable_treebeardkf ? 1 : 0 + source = "../.." + hostname = "kf.example.com" enable_https = true } \ No newline at end of file diff --git a/examples/eks/main.tf b/examples/eks/main.tf index 4dcf964..91e0c8f 100644 --- a/examples/eks/main.tf +++ b/examples/eks/main.tf @@ -218,8 +218,8 @@ variable "enable_treebeardkf" { } module "treebeardkf" { - count = var.enable_treebeardkf ? 1 : 0 - source = "../.." - hostname = "kf.example.com" + count = var.enable_treebeardkf ? 1 : 0 + source = "../.." + hostname = "kf.example.com" enable_https = true } \ No newline at end of file diff --git a/examples/k3s/main.tf b/examples/k3s/main.tf index 6e98a57..89c15a0 100644 --- a/examples/k3s/main.tf +++ b/examples/k3s/main.tf @@ -37,7 +37,7 @@ provider "kubernetes" { } module "treebeardkf" { - source = "../.." - enable_https = true - hostname = "kubeflow.example.com" + source = "../.." + enable_https = true + hostname = "kubeflow.example.com" } \ No newline at end of file diff --git a/kf_core.tf b/kf_core.tf index b202dab..cfed0ee 100644 --- a/kf_core.tf +++ b/kf_core.tf @@ -1,43 +1,43 @@ locals { cert_resource = var.enable_https ? yamlencode({ - apiVersion: "cert-manager.io/v1", - kind: "Certificate", - metadata: { - name: "gateway-cert", - namespace: "istio-system" + apiVersion : "cert-manager.io/v1", + kind : "Certificate", + metadata : { + name : "gateway-cert", + namespace : "istio-system" }, - spec: { - commonName: var.hostname, - dnsNames: [var.hostname] - issuerRef: { - kind: "Issuer", - name: "treebeard-issuer" + spec : { + commonName : var.hostname, + dnsNames : [var.hostname] + issuerRef : { + kind : "Issuer", + name : "treebeard-issuer" } - secretName: "gateway-cert" + secretName : "gateway-cert" } }) : "" gateway_patch = yamlencode({ - apiVersion: "networking.istio.io/v1alpha3", - kind: "Gateway", - metadata: { - name: "kubeflow-gateway", - namespace: "kubeflow", + apiVersion : "networking.istio.io/v1alpha3", + kind : "Gateway", + metadata : { + name : "kubeflow-gateway", + namespace : "kubeflow", }, - spec: { - selector: { - istio: "ingressgateway", + spec : { + selector : { + istio : "ingressgateway", }, - servers: [{ - hosts: [var.hostname], - port: { - name: var.enable_https ? "https" : "http", - number: var.enable_https ? 443 : 80, - protocol: var.enable_https ? "HTTPS" : "HTTP", + servers : [{ + hosts : [var.hostname], + port : { + name : var.enable_https ? "https" : "http", + number : var.enable_https ? 443 : 80, + protocol : var.enable_https ? "HTTPS" : "HTTP", }, - tls: var.enable_https ? { - credentialName: "gateway-cert", - mode: "SIMPLE", + tls : var.enable_https ? { + credentialName : "gateway-cert", + mode : "SIMPLE", } : null, }], }, @@ -77,7 +77,7 @@ data "kustomization_overlay" "kubeflow_istio_resources" { patches { patch = local.gateway_patch } - + patches { patch = local.cert_resource } diff --git a/variables.tf b/variables.tf index 4a11b7c..5700cf1 100644 --- a/variables.tf +++ b/variables.tf @@ -14,7 +14,7 @@ variable "hostname" { default = "*" } -variable enable_https { +variable "enable_https" { type = bool default = false } From 7ca34c99f3725179c8f3e5292ff21ebd1e739374 Mon Sep 17 00:00:00 2001 From: alex-treebeard Date: Thu, 29 Feb 2024 11:33:53 +0000 Subject: [PATCH 24/56] move kf apps to argo --- .devcontainer/devcontainer.json | 2 + Makefile | 6 +- charts/argo_app/Chart.yaml | 3 + charts/argo_app/templates/all.yaml | 15 +++ charts/profile/Chart.yaml | 3 + charts/profile/templates/all.yaml | 8 ++ examples/k3s-existing-istio/main.tf | 7 +- kf_apps.tf | 190 +++++++++++++++------------- kf_dependencies.tf | 19 ++- kf_kserve.tf | 81 ------------ kf_pipelines.tf | 28 ---- overlays/profile/all.yaml | 8 -- variables.tf | 10 -- 13 files changed, 157 insertions(+), 223 deletions(-) create mode 100644 charts/argo_app/Chart.yaml create mode 100644 charts/argo_app/templates/all.yaml create mode 100644 charts/profile/Chart.yaml create mode 100644 charts/profile/templates/all.yaml delete mode 100644 kf_kserve.tf delete mode 100644 kf_pipelines.tf diff --git a/.devcontainer/devcontainer.json b/.devcontainer/devcontainer.json index e77d57d..3cc71f5 100644 --- a/.devcontainer/devcontainer.json +++ b/.devcontainer/devcontainer.json @@ -68,6 +68,8 @@ }, "ghcr.io/devcontainers-contrib/features/istioctl:1": { }, + "ghcr.io/devcontainers-contrib/features/argo-cd:1": { + }, }, "mounts": [ "source=${localEnv:HOME}${localEnv:USERPROFILE}/.aws,target=/home/vscode/.aws,type=bind,consistency=cached", diff --git a/Makefile b/Makefile index 2e0769e..ac54af8 100644 --- a/Makefile +++ b/Makefile @@ -31,4 +31,8 @@ k3d-create: k3d cluster create dev \ -p "80:80@loadbalancer" \ -p "443:443@loadbalancer" \ - --k3s-arg '--disable=traefik@server:0' \ No newline at end of file + --k3s-arg '--disable=traefik@server:0' + +.PHONY: k3d-delete +k3d-delete: + k3d cluster delete dev diff --git a/charts/argo_app/Chart.yaml b/charts/argo_app/Chart.yaml new file mode 100644 index 0000000..6124dd4 --- /dev/null +++ b/charts/argo_app/Chart.yaml @@ -0,0 +1,3 @@ +apiVersion: v2 +name: argo-app +version: 0.1.0 diff --git a/charts/argo_app/templates/all.yaml b/charts/argo_app/templates/all.yaml new file mode 100644 index 0000000..54c3db5 --- /dev/null +++ b/charts/argo_app/templates/all.yaml @@ -0,0 +1,15 @@ +apiVersion: argoproj.io/v1alpha1 +kind: Application +metadata: + name: {{ .Values.name }} + finalizers: + - resources-finalizer.argocd.argoproj.io +spec: + destination: + namespace: kubeflow + server: https://kubernetes.default.svc + project: default + source: + path: {{ .Values.path }} + repoURL: {{ .Values.repoURL }} + targetRevision: {{ .Values.targetRevision }} diff --git a/charts/profile/Chart.yaml b/charts/profile/Chart.yaml new file mode 100644 index 0000000..6bca490 --- /dev/null +++ b/charts/profile/Chart.yaml @@ -0,0 +1,3 @@ +apiVersion: v2 +name: profile +version: 0.1.0 diff --git a/charts/profile/templates/all.yaml b/charts/profile/templates/all.yaml new file mode 100644 index 0000000..a704fcc --- /dev/null +++ b/charts/profile/templates/all.yaml @@ -0,0 +1,8 @@ +apiVersion: kubeflow.org/v1 +kind: Profile +metadata: + name: prod +spec: + owner: + kind: User + name: user@example.com \ No newline at end of file diff --git a/examples/k3s-existing-istio/main.tf b/examples/k3s-existing-istio/main.tf index 8d2b64e..b70b105 100644 --- a/examples/k3s-existing-istio/main.tf +++ b/examples/k3s-existing-istio/main.tf @@ -117,14 +117,9 @@ resource "null_resource" "completed" { module "treebeardkf" { source = "../.." - hostname = "kf.example.com" - protocol = "https://" - port = "" - enable_kuberay = false - enable_mlflow = false enable_istio_base = false enable_istiod = false enable_istio_resources = true enable_cert_manager = false - completed = null_resource.completed.id + dependency = null_resource.completed.id } \ No newline at end of file diff --git a/kf_apps.tf b/kf_apps.tf index 91dda06..2a97bc5 100644 --- a/kf_apps.tf +++ b/kf_apps.tf @@ -1,118 +1,132 @@ -data "kustomization_build" "central_dashboard" { - path = "${path.module}/overlays/centraldashboard" -} - -module "central_dashboard" { - source = "./modules/kust" - build = data.kustomization_build.central_dashboard +resource "helm_release" "central_dashboard" { + name = "centraldashboard" + namespace = "argo-cd" + chart = "${path.module}/charts/argo_app" depends_on = [ - module.models_web_app, module.kubeflow_istio_resources, - module.kubeflow_pipelines, - module.kserve ] -} - -data "kustomization_build" "admission_webhook" { - path = "${path.module}/submodules/manifests/apps/admission-webhook/upstream/overlays/cert-manager" -} + values = [ + < Date: Thu, 29 Feb 2024 11:34:05 +0000 Subject: [PATCH 25/56] fmt --- kf_apps.tf | 54 +++++++++++++++++++++++----------------------- kf_dependencies.tf | 10 ++++----- 2 files changed, 32 insertions(+), 32 deletions(-) diff --git a/kf_apps.tf b/kf_apps.tf index 2a97bc5..349a975 100644 --- a/kf_apps.tf +++ b/kf_apps.tf @@ -1,8 +1,8 @@ resource "helm_release" "central_dashboard" { - name = "centraldashboard" - namespace = "argo-cd" - chart = "${path.module}/charts/argo_app" + name = "centraldashboard" + namespace = "argo-cd" + chart = "${path.module}/charts/argo_app" depends_on = [ module.kubeflow_istio_resources, ] @@ -14,12 +14,12 @@ resource "helm_release" "central_dashboard" { targetRevision: 776d4f4 EOF ] -} +} resource "helm_release" "admission_webhook" { - name = "admission-webhook" - namespace = "argo-cd" - chart = "${path.module}/charts/argo_app" + name = "admission-webhook" + namespace = "argo-cd" + chart = "${path.module}/charts/argo_app" depends_on = [ helm_release.central_dashboard ] @@ -31,12 +31,12 @@ resource "helm_release" "admission_webhook" { targetRevision: 776d4f4 EOF ] -} +} resource "helm_release" "notebook_controller" { - name = "notebook-controller" - namespace = "argo-cd" - chart = "${path.module}/charts/argo_app" + name = "notebook-controller" + namespace = "argo-cd" + chart = "${path.module}/charts/argo_app" depends_on = [ helm_release.admission_webhook ] @@ -48,12 +48,12 @@ resource "helm_release" "notebook_controller" { targetRevision: 776d4f4 EOF ] -} +} resource "helm_release" "jupyter_web_app" { - name = "jupyter-web-app" - namespace = "argo-cd" - chart = "${path.module}/charts/argo_app" + name = "jupyter-web-app" + namespace = "argo-cd" + chart = "${path.module}/charts/argo_app" depends_on = [ helm_release.notebook_controller ] @@ -68,9 +68,9 @@ resource "helm_release" "jupyter_web_app" { } resource "helm_release" "pvc_viewer_controller" { - name = "pvcviewer-controller" - namespace = "argo-cd" - chart = "${path.module}/charts/argo_app" + name = "pvcviewer-controller" + namespace = "argo-cd" + chart = "${path.module}/charts/argo_app" depends_on = [ helm_release.jupyter_web_app ] @@ -85,9 +85,9 @@ resource "helm_release" "pvc_viewer_controller" { } resource "helm_release" "profiles_kfam" { - name = "profiles-kfam" - namespace = "argo-cd" - chart = "${path.module}/charts/argo_app" + name = "profiles-kfam" + namespace = "argo-cd" + chart = "${path.module}/charts/argo_app" depends_on = [ helm_release.pvc_viewer_controller ] @@ -102,9 +102,9 @@ resource "helm_release" "profiles_kfam" { } resource "helm_release" "volumes_web_app" { - name = "volumes-web-app" - namespace = "argo-cd" - chart = "${path.module}/charts/argo_app" + name = "volumes-web-app" + namespace = "argo-cd" + chart = "${path.module}/charts/argo_app" depends_on = [ helm_release.profiles_kfam ] @@ -119,9 +119,9 @@ resource "helm_release" "volumes_web_app" { } resource "helm_release" "kubeflow_profile" { - name = "kubeflow-profile" - namespace = "argo-cd" - chart = "${path.module}/charts/profile" + name = "kubeflow-profile" + namespace = "argo-cd" + chart = "${path.module}/charts/profile" depends_on = [ helm_release.volumes_web_app ] diff --git a/kf_dependencies.tf b/kf_dependencies.tf index 6785114..847e2f7 100644 --- a/kf_dependencies.tf +++ b/kf_dependencies.tf @@ -1,10 +1,10 @@ resource "helm_release" "argo_cd" { - name = "argo-cd" - namespace = "argo-cd" - chart = "argo-cd" - repository = "https://argoproj.github.io/argo-helm" - version = "6.4.1" + name = "argo-cd" + namespace = "argo-cd" + chart = "argo-cd" + repository = "https://argoproj.github.io/argo-helm" + version = "6.4.1" create_namespace = true depends_on = [ var.dependency From 69ace4e1a70fbd2cfbdad65c026800df5dcc31ce Mon Sep 17 00:00:00 2001 From: alex-treebeard Date: Thu, 29 Feb 2024 11:47:42 +0000 Subject: [PATCH 26/56] rm kustomize from auth --- kf_apps.tf | 2 +- kf_auth.tf | 87 +++++++++++++++--------------------------------------- kf_core.tf | 2 +- 3 files changed, 25 insertions(+), 66 deletions(-) diff --git a/kf_apps.tf b/kf_apps.tf index 349a975..b6e13f2 100644 --- a/kf_apps.tf +++ b/kf_apps.tf @@ -78,7 +78,7 @@ resource "helm_release" "pvc_viewer_controller" { < Date: Thu, 29 Feb 2024 13:00:26 +0000 Subject: [PATCH 27/56] remove kustomization dep --- charts/issuer/Chart.yaml | 3 + charts/issuer/templates/all.yaml | 6 + examples/k3s-existing-istio/main.tf | 15 --- kf_apps.tf | 62 +++++++--- kf_auth.tf | 3 +- kf_core.tf | 79 ++++++++----- kf_dependencies.tf | 172 +++++++++++++++------------- main.tf | 4 - variables.tf | 5 + 9 files changed, 207 insertions(+), 142 deletions(-) create mode 100644 charts/issuer/Chart.yaml create mode 100644 charts/issuer/templates/all.yaml diff --git a/charts/issuer/Chart.yaml b/charts/issuer/Chart.yaml new file mode 100644 index 0000000..153fe68 --- /dev/null +++ b/charts/issuer/Chart.yaml @@ -0,0 +1,3 @@ +apiVersion: v2 +name: issuer +version: 0.1.0 diff --git a/charts/issuer/templates/all.yaml b/charts/issuer/templates/all.yaml new file mode 100644 index 0000000..ccbc1ad --- /dev/null +++ b/charts/issuer/templates/all.yaml @@ -0,0 +1,6 @@ +apiVersion: cert-manager.io/v1 +kind: ClusterIssuer +metadata: + name: kubeflow-self-signing-issuer +spec: + selfSigned: {} diff --git a/examples/k3s-existing-istio/main.tf b/examples/k3s-existing-istio/main.tf index b70b105..a261e70 100644 --- a/examples/k3s-existing-istio/main.tf +++ b/examples/k3s-existing-istio/main.tf @@ -1,10 +1,6 @@ terraform { required_providers { - kustomization = { - source = "kbst/kustomization" - version = "~> 0.9.5" - } helm = { source = "hashicorp/helm" version = "~> 2.12.1" @@ -22,10 +18,6 @@ variable "kubeconfig" { type = string } -provider "kustomization" { - kubeconfig_path = var.kubeconfig -} - provider "helm" { kubernetes { config_path = var.kubeconfig @@ -102,13 +94,6 @@ resource "helm_release" "istio_ingressgateway" { ] } -# resource "time_sleep" "wait_30_seconds" { -# depends_on = [ -# helm_release.istio_ingressgateway -# ] -# create_duration = "30s" -# } - resource "null_resource" "completed" { depends_on = [ helm_release.istio_ingressgateway diff --git a/kf_apps.tf b/kf_apps.tf index b6e13f2..67c364f 100644 --- a/kf_apps.tf +++ b/kf_apps.tf @@ -1,10 +1,19 @@ +resource "null_resource" "kf_apps_start" { + provisioner "local-exec" { + command = "echo '⏳ Installing Kubeflow apps...'" + } + + depends_on = [ + null_resource.kf_core_end + ] +} resource "helm_release" "central_dashboard" { name = "centraldashboard" namespace = "argo-cd" chart = "${path.module}/charts/argo_app" depends_on = [ - module.kubeflow_istio_resources, + null_resource.kf_apps_start ] values = [ < Date: Thu, 29 Feb 2024 14:25:58 +0000 Subject: [PATCH 28/56] wait for argo --- charts/argo_app/templates/all.yaml | 33 ++++++++++++++++++++++++++++++ kf_apps.tf | 30 ++++++++++++--------------- 2 files changed, 46 insertions(+), 17 deletions(-) diff --git a/charts/argo_app/templates/all.yaml b/charts/argo_app/templates/all.yaml index 54c3db5..1f5ba9f 100644 --- a/charts/argo_app/templates/all.yaml +++ b/charts/argo_app/templates/all.yaml @@ -13,3 +13,36 @@ spec: path: {{ .Values.path }} repoURL: {{ .Values.repoURL }} targetRevision: {{ .Values.targetRevision }} +operation: + sync: + syncStrategy: + apply: {} +--- +apiVersion: batch/v1 +kind: Job +metadata: + name: {{ .Values.name }}-wait-for-sync + namespace: argo-cd # Change this to your actual namespace +spec: + template: + spec: + serviceAccount: argocd-server + containers: + - name: check-sync + image: bitnami/kubectl # This image is an example; any image with kubectl will work + command: ["/bin/sh","-c"] + args: + - | + set -e + APP_NAME="volumes-web-app" # Change this to your actual app name + NAMESPACE="argo-cd" # Change this to your actual namespace + until kubectl get -n $NAMESPACE app $APP_NAME -o jsonpath='{.status.operationState.phase}' | grep -q "Succeeded"; do + echo "Waiting for app $APP_NAME to be successfully synced..." + sleep 3 # Check every 30 seconds + done + echo "App $APP_NAME has been successfully synced." + # Optionally, output the entire operationState or a specific part of it + kubectl get -n $NAMESPACE app $APP_NAME -o json | jq '.status.operationState' + restartPolicy: Never + backoffLimit: 1 # Number of retries before considering the Job failed + activeDeadlineSeconds: 100 \ No newline at end of file diff --git a/kf_apps.tf b/kf_apps.tf index 67c364f..45fd64a 100644 --- a/kf_apps.tf +++ b/kf_apps.tf @@ -100,6 +100,7 @@ resource "helm_release" "profiles_kfam" { depends_on = [ null_resource.kf_apps_start ] + wait_for_jobs = true values = [ < Date: Thu, 29 Feb 2024 14:32:03 +0000 Subject: [PATCH 29/56] argocd wait for jobs --- charts/argo_app/templates/all.yaml | 3 +- kf_apps.tf | 47 +++++++++++++++++------------- kf_auth.tf | 2 ++ kf_core.tf | 6 ++++ 4 files changed, 36 insertions(+), 22 deletions(-) diff --git a/charts/argo_app/templates/all.yaml b/charts/argo_app/templates/all.yaml index 1f5ba9f..c938fbf 100644 --- a/charts/argo_app/templates/all.yaml +++ b/charts/argo_app/templates/all.yaml @@ -45,4 +45,5 @@ spec: kubectl get -n $NAMESPACE app $APP_NAME -o json | jq '.status.operationState' restartPolicy: Never backoffLimit: 1 # Number of retries before considering the Job failed - activeDeadlineSeconds: 100 \ No newline at end of file + activeDeadlineSeconds: 100 + ttlSecondsAfterFinished: 30 \ No newline at end of file diff --git a/kf_apps.tf b/kf_apps.tf index 45fd64a..0ed570d 100644 --- a/kf_apps.tf +++ b/kf_apps.tf @@ -12,9 +12,7 @@ resource "helm_release" "central_dashboard" { name = "centraldashboard" namespace = "argo-cd" chart = "${path.module}/charts/argo_app" - depends_on = [ - null_resource.kf_apps_start - ] + wait_for_jobs = true values = [ < Date: Thu, 29 Feb 2024 14:46:11 +0000 Subject: [PATCH 30/56] argo spinup --- charts/argo_app/templates/all.yaml | 14 +++---- .../{templates/all.yaml => templates.yaml} | 0 kf_auth.tf | 37 ------------------- kf_core.tf | 37 +++++++++++++++++++ kf_dependencies.tf | 5 +++ 5 files changed, 49 insertions(+), 44 deletions(-) rename charts/issuer/{templates/all.yaml => templates.yaml} (100%) delete mode 100644 kf_auth.tf diff --git a/charts/argo_app/templates/all.yaml b/charts/argo_app/templates/all.yaml index c938fbf..4101597 100644 --- a/charts/argo_app/templates/all.yaml +++ b/charts/argo_app/templates/all.yaml @@ -22,28 +22,28 @@ apiVersion: batch/v1 kind: Job metadata: name: {{ .Values.name }}-wait-for-sync - namespace: argo-cd # Change this to your actual namespace + namespace: argo-cd spec: template: spec: serviceAccount: argocd-server containers: - name: check-sync - image: bitnami/kubectl # This image is an example; any image with kubectl will work + image: bitnami/kubectl command: ["/bin/sh","-c"] args: - | set -e - APP_NAME="volumes-web-app" # Change this to your actual app name - NAMESPACE="argo-cd" # Change this to your actual namespace + APP_NAME="{{ .Values.name }}" + NAMESPACE="argo-cd" until kubectl get -n $NAMESPACE app $APP_NAME -o jsonpath='{.status.operationState.phase}' | grep -q "Succeeded"; do echo "Waiting for app $APP_NAME to be successfully synced..." - sleep 3 # Check every 30 seconds + sleep 3 done echo "App $APP_NAME has been successfully synced." - # Optionally, output the entire operationState or a specific part of it + kubectl get -n $NAMESPACE app $APP_NAME -o json | jq '.status.operationState' restartPolicy: Never - backoffLimit: 1 # Number of retries before considering the Job failed + backoffLimit: 1 activeDeadlineSeconds: 100 ttlSecondsAfterFinished: 30 \ No newline at end of file diff --git a/charts/issuer/templates/all.yaml b/charts/issuer/templates.yaml similarity index 100% rename from charts/issuer/templates/all.yaml rename to charts/issuer/templates.yaml diff --git a/kf_auth.tf b/kf_auth.tf deleted file mode 100644 index 013fc22..0000000 --- a/kf_auth.tf +++ /dev/null @@ -1,37 +0,0 @@ - - -resource "helm_release" "oidc_authservice" { - name = "oidc-authservice" - namespace = "argo-cd" - chart = "${path.module}/charts/argo_app" - wait_for_jobs = true - values = [ - < Date: Thu, 29 Feb 2024 15:00:41 +0000 Subject: [PATCH 31/56] fix wait --- kf_dependencies.tf | 17 +++++++++++++++-- 1 file changed, 15 insertions(+), 2 deletions(-) diff --git a/kf_dependencies.tf b/kf_dependencies.tf index 59e2756..0d0ec68 100644 --- a/kf_dependencies.tf +++ b/kf_dependencies.tf @@ -113,17 +113,30 @@ resource "helm_release" "argo_cd" { ] } +resource "time_sleep" "wait" { + depends_on = [ + helm_release.istiod, + helm_release.argo_cd, + ] + + create_duration = "10s" + destroy_duration = "10s" +} + resource "null_resource" "kf_dependencies_end" { provisioner "local-exec" { + when = create command = "echo '✅ Kubeflow dependencies installed'" } provisioner "local-exec" { - when = destroy - command = "echo 'Tearing down kf_dependencies soon' && sleep 20s" + when = destroy # note, this only runs when the root module is destroyed + # https://github.com/hashicorp/terraform/issues/13549 + command = "echo 'Tearing down kf_dependencies'" } depends_on = [ + time_sleep.wait, helm_release.cert_manager, helm_release.istio_base, helm_release.istiod, From 0d14423624415266e6fe56822cd2dfe5bbd2bb07 Mon Sep 17 00:00:00 2001 From: alex-treebeard Date: Thu, 29 Feb 2024 15:11:02 +0000 Subject: [PATCH 32/56] update k3s ex --- examples/k3s/main.tf | 10 ---------- 1 file changed, 10 deletions(-) diff --git a/examples/k3s/main.tf b/examples/k3s/main.tf index 89c15a0..0fd5352 100644 --- a/examples/k3s/main.tf +++ b/examples/k3s/main.tf @@ -1,10 +1,6 @@ terraform { required_providers { - kustomization = { - source = "kbst/kustomization" - version = "~> 0.9.5" - } helm = { source = "hashicorp/helm" version = "~> 2.12.1" @@ -22,10 +18,6 @@ variable "kubeconfig" { type = string } -provider "kustomization" { - kubeconfig_path = var.kubeconfig -} - provider "helm" { kubernetes { config_path = var.kubeconfig @@ -38,6 +30,4 @@ provider "kubernetes" { module "treebeardkf" { source = "../.." - enable_https = true - hostname = "kubeflow.example.com" } \ No newline at end of file From 9d60b0e1965277953372912380c246dd6b21ca59 Mon Sep 17 00:00:00 2001 From: alex-treebeard Date: Thu, 29 Feb 2024 16:23:44 +0000 Subject: [PATCH 33/56] central dashboard links --- charts/argo_app/templates/all.yaml | 32 +-------- .../templates/central-dashboard/app.yaml | 64 +++++++++++++++++ .../argo_app/templates/wait-for-sync/job.yaml | 29 ++++++++ overlays/istio-resources/cert.yaml | 14 ---- overlays/istio-resources/kustomization.yaml | 5 -- overlays/kuberay-rbac/install.yaml | 68 ------------------- overlays/kuberay-rbac/kustomization.yaml | 4 -- overlays/mlflow-istio/all.yaml | 41 ----------- overlays/mlflow-istio/kustomization.yaml | 4 -- 9 files changed, 95 insertions(+), 166 deletions(-) create mode 100644 charts/argo_app/templates/central-dashboard/app.yaml create mode 100644 charts/argo_app/templates/wait-for-sync/job.yaml delete mode 100644 overlays/istio-resources/cert.yaml delete mode 100644 overlays/istio-resources/kustomization.yaml delete mode 100644 overlays/kuberay-rbac/install.yaml delete mode 100644 overlays/kuberay-rbac/kustomization.yaml delete mode 100644 overlays/mlflow-istio/all.yaml delete mode 100644 overlays/mlflow-istio/kustomization.yaml diff --git a/charts/argo_app/templates/all.yaml b/charts/argo_app/templates/all.yaml index 4101597..453132a 100644 --- a/charts/argo_app/templates/all.yaml +++ b/charts/argo_app/templates/all.yaml @@ -1,3 +1,4 @@ +{{- if ne .Values.name "centraldashboard" }} apiVersion: argoproj.io/v1alpha1 kind: Application metadata: @@ -17,33 +18,4 @@ operation: sync: syncStrategy: apply: {} ---- -apiVersion: batch/v1 -kind: Job -metadata: - name: {{ .Values.name }}-wait-for-sync - namespace: argo-cd -spec: - template: - spec: - serviceAccount: argocd-server - containers: - - name: check-sync - image: bitnami/kubectl - command: ["/bin/sh","-c"] - args: - - | - set -e - APP_NAME="{{ .Values.name }}" - NAMESPACE="argo-cd" - until kubectl get -n $NAMESPACE app $APP_NAME -o jsonpath='{.status.operationState.phase}' | grep -q "Succeeded"; do - echo "Waiting for app $APP_NAME to be successfully synced..." - sleep 3 - done - echo "App $APP_NAME has been successfully synced." - - kubectl get -n $NAMESPACE app $APP_NAME -o json | jq '.status.operationState' - restartPolicy: Never - backoffLimit: 1 - activeDeadlineSeconds: 100 - ttlSecondsAfterFinished: 30 \ No newline at end of file +{{- end }} diff --git a/charts/argo_app/templates/central-dashboard/app.yaml b/charts/argo_app/templates/central-dashboard/app.yaml new file mode 100644 index 0000000..1d90709 --- /dev/null +++ b/charts/argo_app/templates/central-dashboard/app.yaml @@ -0,0 +1,64 @@ +{{- if eq .Values.name "centraldashboard" }} +apiVersion: argoproj.io/v1alpha1 +kind: Application +metadata: + name: {{ .Values.name }} + annotations: + test: "test" + finalizers: + - resources-finalizer.argocd.argoproj.io +spec: + destination: + namespace: kubeflow + server: https://kubernetes.default.svc + project: default + source: + path: {{ .Values.path }} + repoURL: {{ .Values.repoURL }} + targetRevision: {{ .Values.targetRevision }} + kustomize: + patches: + - target: + kind: ConfigMap + name: centraldashboard-config + patch: |- + - op: replace + path: /data/links + value: | + { + "menuLinks": [ + { + "type": "item", + "link": "/jupyter/", + "text": "Notebooks", + "icon": "book" + }, + { + "type": "item", + "link": "/volumes/", + "text": "Volumes", + "icon": "device:storage" + } + ], + "externalLinks": [ ], + "quickLinks": [ + { + "text": "Create a new Notebook server", + "desc": "Notebook Servers", + "link": "/jupyter/new?namespace=kubeflow" + } + ], + "documentationItems": [ + { + "text": "Getting Started with Kubeflow", + "desc": "Get your machine-learning workflow up and running on Kubeflow", + "link": "https://www.kubeflow.org/docs/started/getting-started/" + } + ] + } + +operation: + sync: + syncStrategy: + apply: {} +{{- end }} diff --git a/charts/argo_app/templates/wait-for-sync/job.yaml b/charts/argo_app/templates/wait-for-sync/job.yaml new file mode 100644 index 0000000..e9f1456 --- /dev/null +++ b/charts/argo_app/templates/wait-for-sync/job.yaml @@ -0,0 +1,29 @@ +apiVersion: batch/v1 +kind: Job +metadata: + name: {{ .Values.name }}-wait-for-sync + namespace: argo-cd +spec: + template: + spec: + serviceAccount: argocd-server + containers: + - name: check-sync + image: bitnami/kubectl + command: ["/bin/sh","-c"] + args: + - | + set -e + APP_NAME="{{ .Values.name }}" + NAMESPACE="argo-cd" + until kubectl get -n $NAMESPACE app $APP_NAME -o jsonpath='{.status.operationState.phase}' | grep -q "Succeeded"; do + echo "Waiting for app $APP_NAME to be successfully synced..." + sleep 3 + done + echo "App $APP_NAME has been successfully synced." + + kubectl get -n $NAMESPACE app $APP_NAME -o json | jq '.status.operationState' + restartPolicy: Never + backoffLimit: 1 + activeDeadlineSeconds: 100 + ttlSecondsAfterFinished: 30 \ No newline at end of file diff --git a/overlays/istio-resources/cert.yaml b/overlays/istio-resources/cert.yaml deleted file mode 100644 index d3aadc3..0000000 --- a/overlays/istio-resources/cert.yaml +++ /dev/null @@ -1,14 +0,0 @@ -apiVersion: cert-manager.io/v1 -kind: Certificate -metadata: - name: gateway-cert - namespace: istio-system -spec: - commonName: - dnsNames: - - - issuerRef: - kind: Issuer - name: treebeard-issuer - secretName: gateway-cert ---- \ No newline at end of file diff --git a/overlays/istio-resources/kustomization.yaml b/overlays/istio-resources/kustomization.yaml deleted file mode 100644 index 3898782..0000000 --- a/overlays/istio-resources/kustomization.yaml +++ /dev/null @@ -1,5 +0,0 @@ -apiVersion: kustomize.config.k8s.io/v1beta1 -kind: Kustomization -resources: -- ../../submodules/manifests/common/istio-1-17/kubeflow-istio-resources/base -- cert.yaml \ No newline at end of file diff --git a/overlays/kuberay-rbac/install.yaml b/overlays/kuberay-rbac/install.yaml deleted file mode 100644 index e16b5c9..0000000 --- a/overlays/kuberay-rbac/install.yaml +++ /dev/null @@ -1,68 +0,0 @@ -apiVersion: rbac.authorization.k8s.io/v1 -kind: ClusterRole -metadata: - name: kubeflow-ray-admin - labels: - rbac.authorization.kubeflow.org/aggregate-to-kubeflow-admin: 'true' -rules: -- apiGroups: - - ray.io - verbs: ["*"] - resources: ["*"] ---- -apiVersion: rbac.authorization.k8s.io/v1 -kind: ClusterRole -metadata: - name: kubeflow-ray-editor - labels: - rbac.authorization.kubeflow.org/aggregate-to-kubeflow-edit: 'true' -rules: -- apiGroups: - - ray.io - verbs: ["get", "list", "watch", "create", "update", "patch", "delete"] - resources: ["*"] ---- -apiVersion: rbac.authorization.k8s.io/v1 -kind: ClusterRole -metadata: - name: kubeflow-ray-viewer - labels: - rbac.authorization.kubeflow.org/aggregate-to-kubeflow-view: 'true' -rules: -- apiGroups: - - ray.io - verbs: ["get", "list", "watch"] - resources: ["*"] ---- -apiVersion: security.istio.io/v1beta1 -kind: AuthorizationPolicy -metadata: - name: kuberay - namespace: istio-system -spec: - action: ALLOW - selector: - matchLabels: - ray.io/is-ray-node: 'yes' - rules: - - from: - - source: - principals: - - cluster.local/ns/kubeflow/sa/kuberay-operator ---- -# this seems to be unnecessary for workspace -> ray service -# apiVersion: security.istio.io/v1beta1 -# kind: AuthorizationPolicy -# metadata: -# name: raycluster-editor -# namespace: kubeflow-user-example-com -# spec: -# action: ALLOW -# rules: -# - from: -# - source: -# principals: -# - cluster.local/ns/kubeflow-user-example-com/sa/default-editor -# selector: -# matchLabels: -# ray.io/is-ray-node: 'yes' \ No newline at end of file diff --git a/overlays/kuberay-rbac/kustomization.yaml b/overlays/kuberay-rbac/kustomization.yaml deleted file mode 100644 index 41a86bb..0000000 --- a/overlays/kuberay-rbac/kustomization.yaml +++ /dev/null @@ -1,4 +0,0 @@ -apiVersion: kustomize.config.k8s.io/v1beta1 -kind: Kustomization -resources: -- install.yaml \ No newline at end of file diff --git a/overlays/mlflow-istio/all.yaml b/overlays/mlflow-istio/all.yaml deleted file mode 100644 index 0ec4f72..0000000 --- a/overlays/mlflow-istio/all.yaml +++ /dev/null @@ -1,41 +0,0 @@ -apiVersion: networking.istio.io/v1alpha3 -kind: VirtualService -metadata: - name: mlflow - namespace: kubeflow -spec: - gateways: - - kubeflow-gateway - hosts: - - '*' - http: - - headers: - request: - add: - x-forwarded-prefix: /mlflow - match: - - uri: - prefix: /mlflow - rewrite: - uri: / - route: - - destination: - host: my-mlflow.kubeflow.svc.cluster.local - port: - number: 5000 ---- -apiVersion: security.istio.io/v1beta1 -kind: AuthorizationPolicy -metadata: - name: mlflow - namespace: kubeflow -spec: - action: ALLOW - rules: - - from: - - source: - principals: - - cluster.local/ns/istio-system/sa/istio-ingressgateway-service-account - selector: - matchLabels: - app: mlflow diff --git a/overlays/mlflow-istio/kustomization.yaml b/overlays/mlflow-istio/kustomization.yaml deleted file mode 100644 index a7a55ae..0000000 --- a/overlays/mlflow-istio/kustomization.yaml +++ /dev/null @@ -1,4 +0,0 @@ -apiVersion: kustomize.config.k8s.io/v1beta1 -kind: Kustomization -resources: -- all.yaml \ No newline at end of file From 0cec1e0130060adcd819399268d97922f35993b7 Mon Sep 17 00:00:00 2001 From: alex-treebeard Date: Thu, 29 Feb 2024 17:07:24 +0000 Subject: [PATCH 34/56] enable https --- charts/argo_app/templates/all.yaml | 3 +- .../templates/istio-resources/app.yaml | 43 +++++++++++++++++++ kf_core.tf | 28 +----------- variables.tf | 5 --- 4 files changed, 47 insertions(+), 32 deletions(-) create mode 100644 charts/argo_app/templates/istio-resources/app.yaml diff --git a/charts/argo_app/templates/all.yaml b/charts/argo_app/templates/all.yaml index 453132a..bf2bdd1 100644 --- a/charts/argo_app/templates/all.yaml +++ b/charts/argo_app/templates/all.yaml @@ -1,4 +1,5 @@ -{{- if ne .Values.name "centraldashboard" }} +{{- $nameList := list "kubeflow-istio-resources" "centraldashboard" }} +{{- if not (has .Values.name $nameList) }} apiVersion: argoproj.io/v1alpha1 kind: Application metadata: diff --git a/charts/argo_app/templates/istio-resources/app.yaml b/charts/argo_app/templates/istio-resources/app.yaml new file mode 100644 index 0000000..e581432 --- /dev/null +++ b/charts/argo_app/templates/istio-resources/app.yaml @@ -0,0 +1,43 @@ +{{- if eq .Values.name "kubeflow-istio-resources" }} +apiVersion: argoproj.io/v1alpha1 +kind: Application +metadata: + name: {{ .Values.name }} + finalizers: + - resources-finalizer.argocd.argoproj.io +spec: + destination: + namespace: kubeflow + server: https://kubernetes.default.svc + project: default + source: + path: {{ .Values.path }} + repoURL: {{ .Values.repoURL }} + targetRevision: {{ .Values.targetRevision }} + kustomize: + patches: + - target: + kind: Gateway + name: kubeflow-gateway + patch: |- + - op: replace + path: /spec/servers/0 + value: + hosts: + - {{ .Values.hostname }} + port: + name: {{ eq .Values.enableHttps "true" | ternary "https" "http" }} + number: {{ eq .Values.enableHttps "true" | ternary 443 80 }} + protocol: {{ eq .Values.enableHttps "true" | ternary "HTTPS" "HTTP" }} + {{- if eq .Values.enableHttps "true" }} + tls: + credentialName: gateway-cert + mode: SIMPLE + {{- end }} + credentialName: gateway-cert + mode: SIMPLE +operation: + sync: + syncStrategy: + apply: {} +{{- end }} \ No newline at end of file diff --git a/kf_core.tf b/kf_core.tf index f32546f..975366c 100644 --- a/kf_core.tf +++ b/kf_core.tf @@ -16,32 +16,6 @@ locals { secretName : "gateway-cert" } }) : "" - - gateway_patch = yamlencode({ - apiVersion : "networking.istio.io/v1alpha3", - kind : "Gateway", - metadata : { - name : "kubeflow-gateway", - namespace : "kubeflow", - }, - spec : { - selector : { - istio : "ingressgateway", - }, - servers : [{ - hosts : [var.hostname], - port : { - name : var.enable_https ? "https" : "http", - number : var.enable_https ? 443 : 80, - protocol : var.enable_https ? "HTTPS" : "HTTP", - }, - tls : var.enable_https ? { - credentialName : "gateway-cert", - mode : "SIMPLE", - } : null, - }], - }, - }) } resource "null_resource" "kf_core_start" { @@ -138,6 +112,8 @@ resource "helm_release" "kubeflow_istio_resources" { repoURL: https://github.com/kubeflow/manifests path: common/istio-1-17/kubeflow-istio-resources/base targetRevision: 776d4f4 + hostname: '${var.hostname}' + enableHttps: "false" EOF ] depends_on = [ diff --git a/variables.tf b/variables.tf index d9fa8b8..81019f8 100644 --- a/variables.tf +++ b/variables.tf @@ -39,11 +39,6 @@ variable "enable_istiod" { default = true } -variable "enable_tensorboard" { - type = bool - default = false -} - variable "enable_istio_resources" { type = bool default = true From 00abc0c3ebe975bce2c557fa98d08a6c4f9ac595 Mon Sep 17 00:00:00 2001 From: alex-treebeard Date: Thu, 29 Feb 2024 17:29:37 +0000 Subject: [PATCH 35/56] issuer and cert --- .../argo_app/templates/istio-resources/app.yaml | 2 +- .../templates/istio-resources/cert.yaml | 17 +++++++++++++++++ .../argo_app/templates/wait-for-sync/job.yaml | 2 +- kf_core.tf | 3 ++- variables.tf | 6 ++++++ 5 files changed, 27 insertions(+), 3 deletions(-) create mode 100644 charts/argo_app/templates/istio-resources/cert.yaml diff --git a/charts/argo_app/templates/istio-resources/app.yaml b/charts/argo_app/templates/istio-resources/app.yaml index e581432..a46a6e3 100644 --- a/charts/argo_app/templates/istio-resources/app.yaml +++ b/charts/argo_app/templates/istio-resources/app.yaml @@ -24,7 +24,7 @@ spec: path: /spec/servers/0 value: hosts: - - {{ .Values.hostname }} + - "{{ .Values.hostname }}" port: name: {{ eq .Values.enableHttps "true" | ternary "https" "http" }} number: {{ eq .Values.enableHttps "true" | ternary 443 80 }} diff --git a/charts/argo_app/templates/istio-resources/cert.yaml b/charts/argo_app/templates/istio-resources/cert.yaml new file mode 100644 index 0000000..3c491ee --- /dev/null +++ b/charts/argo_app/templates/istio-resources/cert.yaml @@ -0,0 +1,17 @@ +{{- if eq .Values.name "kubeflow-istio-resources" }} +{{- if eq .Values.enableHttps "true" }} +apiVersion: cert-manager.io/v1 +kind: Certificate +metadata: + name: gateway-cert + namespace: istio-system +spec: + commonName: '{{ .Values.hostname }}' + dnsNames: + - '{{ .Values.hostname }}' + issuerRef: + kind: Issuer + name: '{{ .Values.issuerName }}' + secretName: gateway-cert +{{- end }} +{{- end }} diff --git a/charts/argo_app/templates/wait-for-sync/job.yaml b/charts/argo_app/templates/wait-for-sync/job.yaml index e9f1456..5a934ea 100644 --- a/charts/argo_app/templates/wait-for-sync/job.yaml +++ b/charts/argo_app/templates/wait-for-sync/job.yaml @@ -25,5 +25,5 @@ spec: kubectl get -n $NAMESPACE app $APP_NAME -o json | jq '.status.operationState' restartPolicy: Never backoffLimit: 1 - activeDeadlineSeconds: 100 + activeDeadlineSeconds: 20 ttlSecondsAfterFinished: 30 \ No newline at end of file diff --git a/kf_core.tf b/kf_core.tf index 975366c..b2bcd2d 100644 --- a/kf_core.tf +++ b/kf_core.tf @@ -113,7 +113,8 @@ resource "helm_release" "kubeflow_istio_resources" { path: common/istio-1-17/kubeflow-istio-resources/base targetRevision: 776d4f4 hostname: '${var.hostname}' - enableHttps: "false" + enableHttps: '${var.enable_https}' + issuerName: '${var.issuer_name}' EOF ] depends_on = [ diff --git a/variables.tf b/variables.tf index 81019f8..c78ad2d 100644 --- a/variables.tf +++ b/variables.tf @@ -19,6 +19,12 @@ variable "enable_https" { default = false } +variable "issuer_name" { + type = string + default = "null" + description = "Required if enable_https is true" +} + variable "enable_argocd" { type = bool default = true From 236a141b05b4d3c5c3c537ff025f450368aaa2e4 Mon Sep 17 00:00:00 2001 From: alex-treebeard Date: Thu, 29 Feb 2024 17:30:38 +0000 Subject: [PATCH 36/56] rm unused --- examples/aks/main.tf | 2 -- kf_core.tf | 20 -------------------- 2 files changed, 22 deletions(-) diff --git a/examples/aks/main.tf b/examples/aks/main.tf index 37b0c37..c0d30e4 100644 --- a/examples/aks/main.tf +++ b/examples/aks/main.tf @@ -70,6 +70,4 @@ variable "enable_treebeardkf" { module "treebeardkf" { count = var.enable_treebeardkf ? 1 : 0 source = "../.." - hostname = "kf.example.com" - enable_https = true } \ No newline at end of file diff --git a/kf_core.tf b/kf_core.tf index b2bcd2d..9a74fc7 100644 --- a/kf_core.tf +++ b/kf_core.tf @@ -1,23 +1,3 @@ -locals { - cert_resource = var.enable_https ? yamlencode({ - apiVersion : "cert-manager.io/v1", - kind : "Certificate", - metadata : { - name : "gateway-cert", - namespace : "istio-system" - }, - spec : { - commonName : var.hostname, - dnsNames : [var.hostname] - issuerRef : { - kind : "Issuer", - name : "treebeard-issuer" - } - secretName : "gateway-cert" - } - }) : "" -} - resource "null_resource" "kf_core_start" { provisioner "local-exec" { command = "echo '⏳ Installing Kubeflow core...'" From 831ebb5b3b6992f2125a1fc2b8b92550fd6964d8 Mon Sep 17 00:00:00 2001 From: alex-treebeard Date: Thu, 29 Feb 2024 17:31:08 +0000 Subject: [PATCH 37/56] fmt --- examples/aks/main.tf | 4 ++-- examples/k3s/main.tf | 2 +- kf_apps.tf | 42 +++++++++++++++++++++--------------------- kf_core.tf | 32 ++++++++++++++++---------------- kf_dependencies.tf | 16 ++++++++-------- variables.tf | 4 ++-- 6 files changed, 50 insertions(+), 50 deletions(-) diff --git a/examples/aks/main.tf b/examples/aks/main.tf index c0d30e4..a8f4665 100644 --- a/examples/aks/main.tf +++ b/examples/aks/main.tf @@ -68,6 +68,6 @@ variable "enable_treebeardkf" { } module "treebeardkf" { - count = var.enable_treebeardkf ? 1 : 0 - source = "../.." + count = var.enable_treebeardkf ? 1 : 0 + source = "../.." } \ No newline at end of file diff --git a/examples/k3s/main.tf b/examples/k3s/main.tf index 0fd5352..1ebfab2 100644 --- a/examples/k3s/main.tf +++ b/examples/k3s/main.tf @@ -29,5 +29,5 @@ provider "kubernetes" { } module "treebeardkf" { - source = "../.." + source = "../.." } \ No newline at end of file diff --git a/kf_apps.tf b/kf_apps.tf index 0ed570d..037da8e 100644 --- a/kf_apps.tf +++ b/kf_apps.tf @@ -9,9 +9,9 @@ resource "null_resource" "kf_apps_start" { } resource "helm_release" "central_dashboard" { - name = "centraldashboard" - namespace = "argo-cd" - chart = "${path.module}/charts/argo_app" + name = "centraldashboard" + namespace = "argo-cd" + chart = "${path.module}/charts/argo_app" wait_for_jobs = true values = [ < Date: Thu, 29 Feb 2024 17:35:23 +0000 Subject: [PATCH 38/56] lint --- main.tf | 14 ++++++++++++-- variables.tf | 6 ------ 2 files changed, 12 insertions(+), 8 deletions(-) diff --git a/main.tf b/main.tf index 02461e6..d940da3 100644 --- a/main.tf +++ b/main.tf @@ -1,12 +1,22 @@ terraform { + required_version = ">= 1.3" + required_providers { helm = { source = "hashicorp/helm" - version = "~> 2.12.1" + version = ">= 2.12" } kubernetes = { source = "hashicorp/kubernetes" - version = "~> 2.25.2" + version = ">= 2.25.2" + } + null = { + source = "hashicorp/null" + version = ">= 3.0" + } + time = { + source = "hashicorp/time" + version = ">= 0.9" } } } diff --git a/variables.tf b/variables.tf index afe5937..30c067a 100644 --- a/variables.tf +++ b/variables.tf @@ -1,9 +1,3 @@ - -variable "enable_istio_ingressgateway_loadbalancer" { - type = bool - default = false -} - variable "dependency" { type = string default = "null" From f15456c885974cb97d73772ae3989867ef362fce Mon Sep 17 00:00:00 2001 From: alex-treebeard Date: Thu, 29 Feb 2024 17:43:34 +0000 Subject: [PATCH 39/56] some cleanup --- .../.tfvars | 0 .../README.md | 0 .../addons.tf | 0 .../charts/issuer/Chart.yaml | 0 .../charts/issuer/templates/issuer.yaml | 0 .../dex_conf.yaml | 0 .../eks.tf | 0 .../istio.tf | 0 .../kubeflow.tf | 0 .../main.tf | 0 .../profiles.yaml | 0 .../variables.tf | 0 modules/kust/README.md | 33 - modules/kust/main.tf | 63 - overlays/centraldashboard/kustomization.yaml | 6 - overlays/centraldashboard/patches/config.yaml | 93 -- overlays/istio-install/kustomization.yaml | 7 - overlays/istio-install/patches/config.yaml | 1297 ----------------- overlays/istio-install/patches/resources.yaml | 14 - overlays/profile/all.yaml | 0 overlays/profile/kustomization.yaml | 4 - 21 files changed, 1517 deletions(-) rename examples/{k3s-sso-https-dns => eks-https-loadbalancer}/.tfvars (100%) rename examples/{k3s-sso-https-dns => eks-https-loadbalancer}/README.md (100%) rename examples/{k3s-sso-https-dns => eks-https-loadbalancer}/addons.tf (100%) rename examples/{k3s-sso-https-dns => eks-https-loadbalancer}/charts/issuer/Chart.yaml (100%) rename examples/{k3s-sso-https-dns => eks-https-loadbalancer}/charts/issuer/templates/issuer.yaml (100%) rename examples/{k3s-sso-https-dns => eks-https-loadbalancer}/dex_conf.yaml (100%) rename examples/{k3s-sso-https-dns => eks-https-loadbalancer}/eks.tf (100%) rename examples/{k3s-sso-https-dns => eks-https-loadbalancer}/istio.tf (100%) rename examples/{k3s-sso-https-dns => eks-https-loadbalancer}/kubeflow.tf (100%) rename examples/{k3s-sso-https-dns => eks-https-loadbalancer}/main.tf (100%) rename examples/{k3s-sso-https-dns => eks-https-loadbalancer}/profiles.yaml (100%) rename examples/{k3s-sso-https-dns => eks-https-loadbalancer}/variables.tf (100%) delete mode 100644 modules/kust/README.md delete mode 100644 modules/kust/main.tf delete mode 100644 overlays/centraldashboard/kustomization.yaml delete mode 100644 overlays/centraldashboard/patches/config.yaml delete mode 100644 overlays/istio-install/kustomization.yaml delete mode 100644 overlays/istio-install/patches/config.yaml delete mode 100644 overlays/istio-install/patches/resources.yaml delete mode 100644 overlays/profile/all.yaml delete mode 100644 overlays/profile/kustomization.yaml diff --git a/examples/k3s-sso-https-dns/.tfvars b/examples/eks-https-loadbalancer/.tfvars similarity index 100% rename from examples/k3s-sso-https-dns/.tfvars rename to examples/eks-https-loadbalancer/.tfvars diff --git a/examples/k3s-sso-https-dns/README.md b/examples/eks-https-loadbalancer/README.md similarity index 100% rename from examples/k3s-sso-https-dns/README.md rename to examples/eks-https-loadbalancer/README.md diff --git a/examples/k3s-sso-https-dns/addons.tf b/examples/eks-https-loadbalancer/addons.tf similarity index 100% rename from examples/k3s-sso-https-dns/addons.tf rename to examples/eks-https-loadbalancer/addons.tf diff --git a/examples/k3s-sso-https-dns/charts/issuer/Chart.yaml b/examples/eks-https-loadbalancer/charts/issuer/Chart.yaml similarity index 100% rename from examples/k3s-sso-https-dns/charts/issuer/Chart.yaml rename to examples/eks-https-loadbalancer/charts/issuer/Chart.yaml diff --git a/examples/k3s-sso-https-dns/charts/issuer/templates/issuer.yaml b/examples/eks-https-loadbalancer/charts/issuer/templates/issuer.yaml similarity index 100% rename from examples/k3s-sso-https-dns/charts/issuer/templates/issuer.yaml rename to examples/eks-https-loadbalancer/charts/issuer/templates/issuer.yaml diff --git a/examples/k3s-sso-https-dns/dex_conf.yaml b/examples/eks-https-loadbalancer/dex_conf.yaml similarity index 100% rename from examples/k3s-sso-https-dns/dex_conf.yaml rename to examples/eks-https-loadbalancer/dex_conf.yaml diff --git a/examples/k3s-sso-https-dns/eks.tf b/examples/eks-https-loadbalancer/eks.tf similarity index 100% rename from examples/k3s-sso-https-dns/eks.tf rename to examples/eks-https-loadbalancer/eks.tf diff --git a/examples/k3s-sso-https-dns/istio.tf b/examples/eks-https-loadbalancer/istio.tf similarity index 100% rename from examples/k3s-sso-https-dns/istio.tf rename to examples/eks-https-loadbalancer/istio.tf diff --git a/examples/k3s-sso-https-dns/kubeflow.tf b/examples/eks-https-loadbalancer/kubeflow.tf similarity index 100% rename from examples/k3s-sso-https-dns/kubeflow.tf rename to examples/eks-https-loadbalancer/kubeflow.tf diff --git a/examples/k3s-sso-https-dns/main.tf b/examples/eks-https-loadbalancer/main.tf similarity index 100% rename from examples/k3s-sso-https-dns/main.tf rename to examples/eks-https-loadbalancer/main.tf diff --git a/examples/k3s-sso-https-dns/profiles.yaml b/examples/eks-https-loadbalancer/profiles.yaml similarity index 100% rename from examples/k3s-sso-https-dns/profiles.yaml rename to examples/eks-https-loadbalancer/profiles.yaml diff --git a/examples/k3s-sso-https-dns/variables.tf b/examples/eks-https-loadbalancer/variables.tf similarity index 100% rename from examples/k3s-sso-https-dns/variables.tf rename to examples/eks-https-loadbalancer/variables.tf diff --git a/modules/kust/README.md b/modules/kust/README.md deleted file mode 100644 index 74a8788..0000000 --- a/modules/kust/README.md +++ /dev/null @@ -1,33 +0,0 @@ -## Requirements - -| Name | Version | -|------|---------| -| [kustomization](#requirement\_kustomization) | 0.9.5 | - -## Providers - -| Name | Version | -|------|---------| -| [kustomization](#provider\_kustomization) | 0.9.5 | - -## Modules - -No modules. - -## Resources - -| Name | Type | -|------|------| -| [kustomization_resource.p0](https://registry.terraform.io/providers/kbst/kustomization/0.9.5/docs/resources/resource) | resource | -| [kustomization_resource.p1](https://registry.terraform.io/providers/kbst/kustomization/0.9.5/docs/resources/resource) | resource | -| [kustomization_resource.p2](https://registry.terraform.io/providers/kbst/kustomization/0.9.5/docs/resources/resource) | resource | - -## Inputs - -| Name | Description | Type | Default | Required | -|------|-------------|------|---------|:--------:| -| [build](#input\_build) | description | 
object({
    ids       = list(string)
    ids_prio  = list(list(string))
    manifests = map(string)
  })
| n/a | yes | - -## Outputs - -No outputs. diff --git a/modules/kust/main.tf b/modules/kust/main.tf deleted file mode 100644 index 4420e72..0000000 --- a/modules/kust/main.tf +++ /dev/null @@ -1,63 +0,0 @@ -terraform { - required_providers { - kustomization = { - source = "kbst/kustomization" - version = "0.9.5" - } - } -} - -variable "build" { - type = object({ - ids = list(string) - ids_prio = list(list(string)) - manifests = map(string) - }) - - description = "description" -} - - -resource "kustomization_resource" "p0" { - for_each = toset(var.build.ids_prio[0]) - - manifest = ( - contains(["_/Secret"], regex("(?P.*/.*)/.*/.*", each.value)["group_kind"]) - ? sensitive(var.build.manifests[each.value]) - : var.build.manifests[each.value] - ) -} - -# then loop through resources in ids_prio[1] -# and set an explicit depends_on on kustomization_resource.p0 -# wait 2 minutes for any deployment or daemonset to become ready -resource "kustomization_resource" "p1" { - for_each = toset(var.build.ids_prio[1]) - - manifest = ( - contains(["_/Secret"], regex("(?P.*/.*)/.*/.*", each.value)["group_kind"]) - ? sensitive(var.build.manifests[each.value]) - : var.build.manifests[each.value] - ) - wait = true - timeouts { - create = "5m" - update = "2m" - } - - depends_on = [kustomization_resource.p0] -} - -# finally, loop through resources in ids_prio[2] -# and set an explicit depends_on on kustomization_resource.p1 -resource "kustomization_resource" "p2" { - for_each = toset(var.build.ids_prio[2]) - - manifest = ( - contains(["_/Secret"], regex("(?P.*/.*)/.*/.*", each.value)["group_kind"]) - ? sensitive(var.build.manifests[each.value]) - : var.build.manifests[each.value] - ) - - depends_on = [kustomization_resource.p1] -} \ No newline at end of file diff --git a/overlays/centraldashboard/kustomization.yaml b/overlays/centraldashboard/kustomization.yaml deleted file mode 100644 index b443168..0000000 --- a/overlays/centraldashboard/kustomization.yaml +++ /dev/null @@ -1,6 +0,0 @@ -apiVersion: kustomize.config.k8s.io/v1beta1 -kind: Kustomization -resources: -- ../../submodules/manifests/apps/centraldashboard/upstream/overlays/kserve -patches: -- path: patches/config.yaml \ No newline at end of file diff --git a/overlays/centraldashboard/patches/config.yaml b/overlays/centraldashboard/patches/config.yaml deleted file mode 100644 index 03903bd..0000000 --- a/overlays/centraldashboard/patches/config.yaml +++ /dev/null @@ -1,93 +0,0 @@ -apiVersion: v1 -data: - settings: |- - { - "DASHBOARD_FORCE_IFRAME": true - } - links: |- - { - "menuLinks": [ - { - "type": "item", - "link": "/jupyter/", - "text": "Notebooks", - "icon": "book" - }, - { - "type": "item", - "link": "/volumes/", - "text": "Volumes", - "icon": "device:storage" - }, - { - "type": "item", - "link": "/kserve-endpoints/", - "text": "Endpoints", - "icon": "kubeflow:models" - }, - { - "type": "item", - "text": "Experiments (KFP)", - "link": "/pipeline/#/experiments", - "icon": "done-all" - }, - { - "type": "item", - "link": "/pipeline/#/pipelines", - "text": "Pipelines", - "icon": "kubeflow:pipeline-centered" - }, - { - "type": "item", - "link": "/pipeline/#/runs", - "text": "Runs", - "icon": "maps:directions-run" - }, - { - "type": "item", - "link": "/pipeline/#/recurringruns", - "text": "Recurring Runs", - "icon": "device:access-alarm" - }, - { - "type": "item", - "link": "/pipeline/#/artifacts", - "text": "Artifacts", - "icon": "editor:bubble-chart" - }, - { - "type": "item", - "link": "/pipeline/#/executions", - "text": "Executions", - "icon": "av:play-arrow" - } - ], - "externalLinks": [ ], - "quickLinks": [ - { - "text": "Upload a pipeline", - "desc": "Pipelines", - "link": "/pipeline/" - }, - { - "text": "View all pipeline runs", - "desc": "Pipelines", - "link": "/pipeline/#/runs" - }, - { - "text": "Create a new Notebook server", - "desc": "Notebook Servers", - "link": "/jupyter/new?namespace=kubeflow" - } - ], - "documentationItems": [ - { - "text": "Getting Started with Kubeflow", - "desc": "Get your machine-learning workflow up and running on Kubeflow", - "link": "https://www.kubeflow.org/docs/started/getting-started/" - } - ] - } -kind: ConfigMap -metadata: - name: centraldashboard-config diff --git a/overlays/istio-install/kustomization.yaml b/overlays/istio-install/kustomization.yaml deleted file mode 100644 index 553094f..0000000 --- a/overlays/istio-install/kustomization.yaml +++ /dev/null @@ -1,7 +0,0 @@ -apiVersion: kustomize.config.k8s.io/v1beta1 -kind: Kustomization -resources: -- ../../submodules/manifests/common/istio-1-17/istio-install/base -patches: -- path: patches/resources.yaml -- path: patches/config.yaml \ No newline at end of file diff --git a/overlays/istio-install/patches/config.yaml b/overlays/istio-install/patches/config.yaml deleted file mode 100644 index 350639f..0000000 --- a/overlays/istio-install/patches/config.yaml +++ /dev/null @@ -1,1297 +0,0 @@ -apiVersion: v1 -kind: ConfigMap -metadata: - name: istio-sidecar-injector - namespace: istio-system - labels: - istio.io/rev: default - install.operator.istio.io/owning-resource: unknown - operator.istio.io/component: Pilot - release: istio -data: - - values: |- - { - "global": { - "autoscalingv2API": true, - "caAddress": "", - "caName": "", - "certSigners": [], - "configCluster": false, - "configValidation": true, - "defaultNodeSelector": {}, - "defaultPodDisruptionBudget": { - "enabled": true - }, - "defaultResources": { - "requests": { - "cpu": "10m" - } - }, - "enabled": true, - "externalIstiod": false, - "hub": "docker.io/istio", - "imagePullPolicy": "", - "imagePullSecrets": [], - "istioNamespace": "istio-system", - "istiod": { - "enableAnalysis": false - }, - "jwtPolicy": "third-party-jwt", - "logAsJson": false, - "logging": { - "level": "default:info" - }, - "meshID": "", - "meshNetworks": {}, - "mountMtlsCerts": false, - "multiCluster": { - "clusterName": "", - "enabled": false - }, - "namespace": "istio-system", - "network": "", - "omitSidecarInjectorConfigMap": false, - "oneNamespace": false, - "operatorManageWebhooks": false, - "pilotCertProvider": "istiod", - "priorityClassName": "", - "proxy": { - "autoInject": "enabled", - "clusterDomain": "cluster.local", - "componentLogLevel": "misc:error", - "enableCoreDump": false, - "excludeIPRanges": "", - "excludeInboundPorts": "", - "excludeOutboundPorts": "", - "holdApplicationUntilProxyStarts": false, - "image": "proxyv2", - "includeIPRanges": "*", - "includeInboundPorts": "*", - "includeOutboundPorts": "", - "logLevel": "warning", - "privileged": false, - "readinessFailureThreshold": 30, - "readinessInitialDelaySeconds": 1, - "readinessPeriodSeconds": 2, - "resources": { - "limits": { - "cpu": "2000m", - "memory": "1024Mi" - }, - "requests": { - "cpu": "0", - "memory": "64Mi" - } - }, - "statusPort": 15020, - "tracer": "zipkin" - }, - "proxy_init": { - "image": "proxyv2", - "resources": { - "limits": { - "cpu": "2000m", - "memory": "1024Mi" - }, - "requests": { - "cpu": "10m", - "memory": "10Mi" - } - } - }, - "remotePilotAddress": "", - "sds": { - "token": { - "aud": "istio-ca" - } - }, - "sts": { - "servicePort": 0 - }, - "tag": "1.17.5", - "tracer": { - "datadog": { - "address": "$(HOST_IP):8126" - }, - "lightstep": { - "accessToken": "", - "address": "" - }, - "stackdriver": { - "debug": false, - "maxNumberOfAnnotations": 200, - "maxNumberOfAttributes": 200, - "maxNumberOfMessageEvents": 200 - }, - "zipkin": { - "address": "" - } - }, - "useMCP": false, - "variant": "" - }, - "istio_cni": { - "enabled": false - }, - "revision": "", - "sidecarInjectorWebhook": { - "alwaysInjectSelector": [], - "defaultTemplates": [], - "enableNamespacesByDefault": false, - "injectedAnnotations": {}, - "neverInjectSelector": [], - "rewriteAppHTTPProbe": true, - "templates": {} - } - } - - # To disable injection: use omitSidecarInjectorConfigMap, which disables the webhook patching - # and istiod webhook functionality. - # - # New fields should not use Values - it is a 'primary' config object, users should be able - # to fine tune it or use it with kube-inject. - config: |- - # defaultTemplates defines the default template to use for pods that do not explicitly specify a template - defaultTemplates: [sidecar] - policy: enabled - alwaysInjectSelector: - [] - neverInjectSelector: - [] - injectedAnnotations: - template: "{{ Template_Version_And_Istio_Version_Mismatched_Check_Installation }}" - templates: - sidecar: | - {{- define "resources" }} - {{- if or (isset .ObjectMeta.Annotations `sidecar.istio.io/proxyCPU`) (isset .ObjectMeta.Annotations `sidecar.istio.io/proxyMemory`) (isset .ObjectMeta.Annotations `sidecar.istio.io/proxyCPULimit`) (isset .ObjectMeta.Annotations `sidecar.istio.io/proxyMemoryLimit`) }} - {{- if or (isset .ObjectMeta.Annotations `sidecar.istio.io/proxyCPU`) (isset .ObjectMeta.Annotations `sidecar.istio.io/proxyMemory`) }} - requests: - {{ if (isset .ObjectMeta.Annotations `sidecar.istio.io/proxyCPU`) -}} - cpu: "{{ index .ObjectMeta.Annotations `sidecar.istio.io/proxyCPU` }}" - {{ end }} - {{ if (isset .ObjectMeta.Annotations `sidecar.istio.io/proxyMemory`) -}} - memory: "{{ index .ObjectMeta.Annotations `sidecar.istio.io/proxyMemory` }}" - {{ end }} - {{- end }} - {{- if or (isset .ObjectMeta.Annotations `sidecar.istio.io/proxyCPULimit`) (isset .ObjectMeta.Annotations `sidecar.istio.io/proxyMemoryLimit`) }} - limits: - {{ if (isset .ObjectMeta.Annotations `sidecar.istio.io/proxyCPULimit`) -}} - cpu: "{{ index .ObjectMeta.Annotations `sidecar.istio.io/proxyCPULimit` }}" - {{ end }} - {{ if (isset .ObjectMeta.Annotations `sidecar.istio.io/proxyMemoryLimit`) -}} - memory: "{{ index .ObjectMeta.Annotations `sidecar.istio.io/proxyMemoryLimit` }}" - {{ end }} - {{- end }} - {{- else }} - {{- if .Values.global.proxy.resources }} - {{ toYaml .Values.global.proxy.resources | indent 6 }} - {{- end }} - {{- end }} - {{- end }} - {{- $containers := list }} - {{- range $index, $container := .Spec.Containers }}{{ if not (eq $container.Name "istio-proxy") }}{{ $containers = append $containers $container.Name }}{{end}}{{- end}} - metadata: - labels: - security.istio.io/tlsMode: {{ index .ObjectMeta.Labels `security.istio.io/tlsMode` | default "istio" | quote }} - {{- if eq (index .ProxyConfig.ProxyMetadata "ISTIO_META_ENABLE_HBONE") "true" }} - networking.istio.io/tunnel: {{ index .ObjectMeta.Labels `networking.istio.io/tunnel` | default "http" | quote }} - {{- end }} - service.istio.io/canonical-name: {{ index .ObjectMeta.Labels `service.istio.io/canonical-name` | default (index .ObjectMeta.Labels `app.kubernetes.io/name`) | default (index .ObjectMeta.Labels `app`) | default .DeploymentMeta.Name | quote }} - service.istio.io/canonical-revision: {{ index .ObjectMeta.Labels `service.istio.io/canonical-revision` | default (index .ObjectMeta.Labels `app.kubernetes.io/version`) | default (index .ObjectMeta.Labels `version`) | default "latest" | quote }} - annotations: { - {{- if ge (len $containers) 1 }} - {{- if not (isset .ObjectMeta.Annotations `kubectl.kubernetes.io/default-logs-container`) }} - kubectl.kubernetes.io/default-logs-container: "{{ index $containers 0 }}", - {{- end }} - {{- if not (isset .ObjectMeta.Annotations `kubectl.kubernetes.io/default-container`) }} - kubectl.kubernetes.io/default-container: "{{ index $containers 0 }}", - {{- end }} - {{- end }} - {{- if .Values.istio_cni.enabled }} - {{- if not .Values.istio_cni.chained }} - k8s.v1.cni.cncf.io/networks: '{{ appendMultusNetwork (index .ObjectMeta.Annotations `k8s.v1.cni.cncf.io/networks`) `istio-cni` }}', - {{- end }} - sidecar.istio.io/interceptionMode: "{{ annotation .ObjectMeta `sidecar.istio.io/interceptionMode` .ProxyConfig.InterceptionMode }}", - {{ with annotation .ObjectMeta `traffic.sidecar.istio.io/includeOutboundIPRanges` .Values.global.proxy.includeIPRanges }}traffic.sidecar.istio.io/includeOutboundIPRanges: "{{.}}",{{ end }} - {{ with annotation .ObjectMeta `traffic.sidecar.istio.io/excludeOutboundIPRanges` .Values.global.proxy.excludeIPRanges }}traffic.sidecar.istio.io/excludeOutboundIPRanges: "{{.}}",{{ end }} - {{ with annotation .ObjectMeta `traffic.sidecar.istio.io/includeInboundPorts` .Values.global.proxy.includeInboundPorts }}traffic.sidecar.istio.io/includeInboundPorts: "{{.}}",{{ end }} - traffic.sidecar.istio.io/excludeInboundPorts: "{{ excludeInboundPort (annotation .ObjectMeta `status.sidecar.istio.io/port` .Values.global.proxy.statusPort) (annotation .ObjectMeta `traffic.sidecar.istio.io/excludeInboundPorts` .Values.global.proxy.excludeInboundPorts) }}", - {{ if or (isset .ObjectMeta.Annotations `traffic.sidecar.istio.io/includeOutboundPorts`) (ne (valueOrDefault .Values.global.proxy.includeOutboundPorts "") "") }} - traffic.sidecar.istio.io/includeOutboundPorts: "{{ annotation .ObjectMeta `traffic.sidecar.istio.io/includeOutboundPorts` .Values.global.proxy.includeOutboundPorts }}", - {{- end }} - {{ if or (isset .ObjectMeta.Annotations `traffic.sidecar.istio.io/excludeOutboundPorts`) (ne .Values.global.proxy.excludeOutboundPorts "") }} - traffic.sidecar.istio.io/excludeOutboundPorts: "{{ annotation .ObjectMeta `traffic.sidecar.istio.io/excludeOutboundPorts` .Values.global.proxy.excludeOutboundPorts }}", - {{- end }} - {{ with index .ObjectMeta.Annotations `traffic.sidecar.istio.io/kubevirtInterfaces` }}traffic.sidecar.istio.io/kubevirtInterfaces: "{{.}}",{{ end }} - {{ with index .ObjectMeta.Annotations `traffic.sidecar.istio.io/excludeInterfaces` }}traffic.sidecar.istio.io/excludeInterfaces: "{{.}}",{{ end }} - {{- end }} - } - spec: - {{- $holdProxy := or .ProxyConfig.HoldApplicationUntilProxyStarts.GetValue .Values.global.proxy.holdApplicationUntilProxyStarts }} - initContainers: - {{ if ne (annotation .ObjectMeta `sidecar.istio.io/interceptionMode` .ProxyConfig.InterceptionMode) `NONE` }} - {{ if .Values.istio_cni.enabled -}} - - name: istio-validation - {{ else -}} - - name: istio-init - {{ end -}} - {{- if contains "/" (annotation .ObjectMeta `sidecar.istio.io/proxyImage` .Values.global.proxy_init.image) }} - image: "{{ annotation .ObjectMeta `sidecar.istio.io/proxyImage` .Values.global.proxy_init.image }}" - {{- else }} - image: "{{ .ProxyImage }}" - {{- end }} - args: - - istio-iptables - - "-p" - - {{ .MeshConfig.ProxyListenPort | default "15001" | quote }} - - "-z" - - "15006" - - "-u" - - "1337" - - "-m" - - "{{ annotation .ObjectMeta `sidecar.istio.io/interceptionMode` .ProxyConfig.InterceptionMode }}" - - "-i" - - "{{ annotation .ObjectMeta `traffic.sidecar.istio.io/includeOutboundIPRanges` .Values.global.proxy.includeIPRanges }}" - - "-x" - - "{{ annotation .ObjectMeta `traffic.sidecar.istio.io/excludeOutboundIPRanges` .Values.global.proxy.excludeIPRanges }}" - - "-b" - - "{{ annotation .ObjectMeta `traffic.sidecar.istio.io/includeInboundPorts` .Values.global.proxy.includeInboundPorts }}" - - "-d" - {{- if excludeInboundPort (annotation .ObjectMeta `status.sidecar.istio.io/port` .Values.global.proxy.statusPort) (annotation .ObjectMeta `traffic.sidecar.istio.io/excludeInboundPorts` .Values.global.proxy.excludeInboundPorts) }} - - "15090,15021,{{ excludeInboundPort (annotation .ObjectMeta `status.sidecar.istio.io/port` .Values.global.proxy.statusPort) (annotation .ObjectMeta `traffic.sidecar.istio.io/excludeInboundPorts` .Values.global.proxy.excludeInboundPorts) }}" - {{- else }} - - "15090,15021" - {{- end }} - {{ if or (isset .ObjectMeta.Annotations `traffic.sidecar.istio.io/includeOutboundPorts`) (ne (valueOrDefault .Values.global.proxy.includeOutboundPorts "") "") -}} - - "-q" - - "{{ annotation .ObjectMeta `traffic.sidecar.istio.io/includeOutboundPorts` .Values.global.proxy.includeOutboundPorts }}" - {{ end -}} - {{ if or (isset .ObjectMeta.Annotations `traffic.sidecar.istio.io/excludeOutboundPorts`) (ne (valueOrDefault .Values.global.proxy.excludeOutboundPorts "") "") -}} - - "-o" - - "{{ annotation .ObjectMeta `traffic.sidecar.istio.io/excludeOutboundPorts` .Values.global.proxy.excludeOutboundPorts }}" - {{ end -}} - {{ if (isset .ObjectMeta.Annotations `traffic.sidecar.istio.io/kubevirtInterfaces`) -}} - - "-k" - - "{{ index .ObjectMeta.Annotations `traffic.sidecar.istio.io/kubevirtInterfaces` }}" - {{ end -}} - {{ if (isset .ObjectMeta.Annotations `traffic.sidecar.istio.io/excludeInterfaces`) -}} - - "-c" - - "{{ index .ObjectMeta.Annotations `traffic.sidecar.istio.io/excludeInterfaces` }}" - {{ end -}} - - "--log_output_level={{ annotation .ObjectMeta `sidecar.istio.io/agentLogLevel` .Values.global.logging.level }}" - {{ if .Values.global.logAsJson -}} - - "--log_as_json" - {{ end -}} - {{ if .Values.istio_cni.enabled -}} - - "--run-validation" - - "--skip-rule-apply" - {{ end -}} - {{with .Values.global.imagePullPolicy }}imagePullPolicy: "{{.}}"{{end}} - {{- if .ProxyConfig.ProxyMetadata }} - env: - {{- range $key, $value := .ProxyConfig.ProxyMetadata }} - - name: {{ $key }} - value: "{{ $value }}" - {{- end }} - {{- end }} - resources: - {{ template "resources" . }} - securityContext: - allowPrivilegeEscalation: {{ .Values.global.proxy.privileged }} - privileged: {{ .Values.global.proxy.privileged }} - capabilities: - {{- if not .Values.istio_cni.enabled }} - add: - - NET_ADMIN - - NET_RAW - {{- end }} - drop: - - ALL - {{- if not .Values.istio_cni.enabled }} - readOnlyRootFilesystem: false - runAsGroup: 0 - runAsNonRoot: false - runAsUser: 0 - {{- else }} - readOnlyRootFilesystem: true - runAsGroup: 1337 - runAsUser: 1337 - runAsNonRoot: true - {{- end }} - {{ end -}} - {{- if eq (annotation .ObjectMeta `sidecar.istio.io/enableCoreDump` .Values.global.proxy.enableCoreDump) "true" }} - - name: enable-core-dump - args: - - -c - - sysctl -w kernel.core_pattern=/var/lib/istio/data/core.proxy && ulimit -c unlimited - command: - - /bin/sh - {{- if contains "/" (annotation .ObjectMeta `sidecar.istio.io/proxyImage` .Values.global.proxy_init.image) }} - image: "{{ annotation .ObjectMeta `sidecar.istio.io/proxyImage` .Values.global.proxy_init.image }}" - {{- else }} - image: "{{ .ProxyImage }}" - {{- end }} - {{with .Values.global.imagePullPolicy }}imagePullPolicy: "{{.}}"{{end}} - resources: - {{ template "resources" . }} - securityContext: - allowPrivilegeEscalation: true - capabilities: - add: - - SYS_ADMIN - drop: - - ALL - privileged: true - readOnlyRootFilesystem: false - runAsGroup: 0 - runAsNonRoot: false - runAsUser: 0 - {{ end }} - containers: - - name: istio-proxy - {{- if contains "/" (annotation .ObjectMeta `sidecar.istio.io/proxyImage` .Values.global.proxy.image) }} - image: "{{ annotation .ObjectMeta `sidecar.istio.io/proxyImage` .Values.global.proxy.image }}" - {{- else }} - image: "{{ .ProxyImage }}" - {{- end }} - ports: - - containerPort: 15090 - protocol: TCP - name: http-envoy-prom - args: - - proxy - - sidecar - - --domain - - $(POD_NAMESPACE).svc.{{ .Values.global.proxy.clusterDomain }} - - --proxyLogLevel={{ annotation .ObjectMeta `sidecar.istio.io/logLevel` .Values.global.proxy.logLevel }} - - --proxyComponentLogLevel={{ annotation .ObjectMeta `sidecar.istio.io/componentLogLevel` .Values.global.proxy.componentLogLevel }} - - --log_output_level={{ annotation .ObjectMeta `sidecar.istio.io/agentLogLevel` .Values.global.logging.level }} - {{- if .Values.global.sts.servicePort }} - - --stsPort={{ .Values.global.sts.servicePort }} - {{- end }} - {{- if .Values.global.logAsJson }} - - --log_as_json - {{- end }} - {{- if gt .EstimatedConcurrency 0 }} - - --concurrency - - "{{ .EstimatedConcurrency }}" - {{- end -}} - {{- if .Values.global.proxy.lifecycle }} - lifecycle: - {{ toYaml .Values.global.proxy.lifecycle | indent 6 }} - {{- else if $holdProxy }} - lifecycle: - postStart: - exec: - command: - - pilot-agent - - wait - {{- end }} - env: - {{- if eq (env "PILOT_ENABLE_INBOUND_PASSTHROUGH" "true") "false" }} - - name: REWRITE_PROBE_LEGACY_LOCALHOST_DESTINATION - value: "true" - {{- end }} - - name: JWT_POLICY - value: {{ .Values.global.jwtPolicy }} - - name: PILOT_CERT_PROVIDER - value: {{ .Values.global.pilotCertProvider }} - - name: CA_ADDR - {{- if .Values.global.caAddress }} - value: {{ .Values.global.caAddress }} - {{- else }} - value: istiod{{- if not (eq .Values.revision "") }}-{{ .Values.revision }}{{- end }}.{{ .Values.global.istioNamespace }}.svc:15012 - {{- end }} - - name: POD_NAME - valueFrom: - fieldRef: - fieldPath: metadata.name - - name: POD_NAMESPACE - valueFrom: - fieldRef: - fieldPath: metadata.namespace - - name: INSTANCE_IP - valueFrom: - fieldRef: - fieldPath: status.podIP - - name: SERVICE_ACCOUNT - valueFrom: - fieldRef: - fieldPath: spec.serviceAccountName - - name: HOST_IP - valueFrom: - fieldRef: - fieldPath: status.hostIP - - name: PROXY_CONFIG - value: | - {{ protoToJSON .ProxyConfig }} - - name: ISTIO_META_POD_PORTS - value: |- - [ - {{- $first := true }} - {{- range $index1, $c := .Spec.Containers }} - {{- range $index2, $p := $c.Ports }} - {{- if (structToJSON $p) }} - {{if not $first}},{{end}}{{ structToJSON $p }} - {{- $first = false }} - {{- end }} - {{- end}} - {{- end}} - ] - - name: ISTIO_META_APP_CONTAINERS - value: "{{ $containers | join "," }}" - - name: ISTIO_META_CLUSTER_ID - value: "{{ valueOrDefault .Values.global.multiCluster.clusterName `Kubernetes` }}" - - name: ISTIO_META_NODE_NAME - valueFrom: - fieldRef: - fieldPath: spec.nodeName - - name: ISTIO_META_INTERCEPTION_MODE - value: "{{ or (index .ObjectMeta.Annotations `sidecar.istio.io/interceptionMode`) .ProxyConfig.InterceptionMode.String }}" - {{- if .Values.global.network }} - - name: ISTIO_META_NETWORK - value: "{{ .Values.global.network }}" - {{- end }} - {{- if .DeploymentMeta.Name }} - - name: ISTIO_META_WORKLOAD_NAME - value: "{{ .DeploymentMeta.Name }}" - {{ end }} - {{- if and .TypeMeta.APIVersion .DeploymentMeta.Name }} - - name: ISTIO_META_OWNER - value: kubernetes://apis/{{ .TypeMeta.APIVersion }}/namespaces/{{ valueOrDefault .DeploymentMeta.Namespace `default` }}/{{ toLower .TypeMeta.Kind}}s/{{ .DeploymentMeta.Name }} - {{- end}} - {{- if (isset .ObjectMeta.Annotations `sidecar.istio.io/bootstrapOverride`) }} - - name: ISTIO_BOOTSTRAP_OVERRIDE - value: "/etc/istio/custom-bootstrap/custom_bootstrap.json" - {{- end }} - {{- if .Values.global.meshID }} - - name: ISTIO_META_MESH_ID - value: "{{ .Values.global.meshID }}" - {{- else if (valueOrDefault .MeshConfig.TrustDomain .Values.global.trustDomain) }} - - name: ISTIO_META_MESH_ID - value: "{{ (valueOrDefault .MeshConfig.TrustDomain .Values.global.trustDomain) }}" - {{- end }} - {{- with (valueOrDefault .MeshConfig.TrustDomain .Values.global.trustDomain) }} - - name: TRUST_DOMAIN - value: "{{ . }}" - {{- end }} - {{- if and (eq .Values.global.proxy.tracer "datadog") (isset .ObjectMeta.Annotations `apm.datadoghq.com/env`) }} - {{- range $key, $value := fromJSON (index .ObjectMeta.Annotations `apm.datadoghq.com/env`) }} - - name: {{ $key }} - value: "{{ $value }}" - {{- end }} - {{- end }} - {{- range $key, $value := .ProxyConfig.ProxyMetadata }} - - name: {{ $key }} - value: "{{ $value }}" - {{- end }} - {{with .Values.global.imagePullPolicy }}imagePullPolicy: "{{.}}"{{end}} - {{ if ne (annotation .ObjectMeta `status.sidecar.istio.io/port` .Values.global.proxy.statusPort) `0` }} - readinessProbe: - httpGet: - path: /healthz/ready - port: 15021 - initialDelaySeconds: {{ annotation .ObjectMeta `readiness.status.sidecar.istio.io/initialDelaySeconds` .Values.global.proxy.readinessInitialDelaySeconds }} - periodSeconds: {{ annotation .ObjectMeta `readiness.status.sidecar.istio.io/periodSeconds` .Values.global.proxy.readinessPeriodSeconds }} - timeoutSeconds: 3 - failureThreshold: {{ annotation .ObjectMeta `readiness.status.sidecar.istio.io/failureThreshold` .Values.global.proxy.readinessFailureThreshold }} - {{ end -}} - securityContext: - {{- if eq (index .ProxyConfig.ProxyMetadata "IPTABLES_TRACE_LOGGING") "true" }} - allowPrivilegeEscalation: true - capabilities: - add: - - NET_ADMIN - drop: - - ALL - privileged: true - readOnlyRootFilesystem: {{ ne (annotation .ObjectMeta `sidecar.istio.io/enableCoreDump` .Values.global.proxy.enableCoreDump) "true" }} - runAsGroup: 1337 - runAsNonRoot: false - runAsUser: 0 - {{- else }} - allowPrivilegeEscalation: {{ .Values.global.proxy.privileged }} - capabilities: - {{ if or (eq (annotation .ObjectMeta `sidecar.istio.io/interceptionMode` .ProxyConfig.InterceptionMode) `TPROXY`) (eq (annotation .ObjectMeta `sidecar.istio.io/capNetBindService` .Values.global.proxy.capNetBindService) `true`) -}} - add: - {{ if eq (annotation .ObjectMeta `sidecar.istio.io/interceptionMode` .ProxyConfig.InterceptionMode) `TPROXY` -}} - - NET_ADMIN - {{- end }} - {{ if eq (annotation .ObjectMeta `sidecar.istio.io/capNetBindService` .Values.global.proxy.capNetBindService) `true` -}} - - NET_BIND_SERVICE - {{- end }} - {{- end }} - drop: - - ALL - privileged: {{ .Values.global.proxy.privileged }} - readOnlyRootFilesystem: {{ ne (annotation .ObjectMeta `sidecar.istio.io/enableCoreDump` .Values.global.proxy.enableCoreDump) "true" }} - runAsGroup: 1337 - {{ if or (eq (annotation .ObjectMeta `sidecar.istio.io/interceptionMode` .ProxyConfig.InterceptionMode) `TPROXY`) (eq (annotation .ObjectMeta `sidecar.istio.io/capNetBindService` .Values.global.proxy.capNetBindService) `true`) -}} - runAsNonRoot: false - runAsUser: 0 - {{- else -}} - runAsNonRoot: true - runAsUser: 1337 - {{- end }} - {{- end }} - resources: - {{ template "resources" . }} - volumeMounts: - - name: workload-socket - mountPath: /var/run/secrets/workload-spiffe-uds - - name: credential-socket - mountPath: /var/run/secrets/credential-uds - {{- if eq .Values.global.caName "GkeWorkloadCertificate" }} - - name: gke-workload-certificate - mountPath: /var/run/secrets/workload-spiffe-credentials - readOnly: true - {{- else }} - - name: workload-certs - mountPath: /var/run/secrets/workload-spiffe-credentials - {{- end }} - {{- if eq .Values.global.pilotCertProvider "istiod" }} - - mountPath: /var/run/secrets/istio - name: istiod-ca-cert - {{- end }} - {{- if eq .Values.global.pilotCertProvider "kubernetes" }} - - mountPath: /var/run/secrets/istio/kubernetes - name: kube-ca-cert - {{- end }} - - mountPath: /var/lib/istio/data - name: istio-data - {{ if (isset .ObjectMeta.Annotations `sidecar.istio.io/bootstrapOverride`) }} - - mountPath: /etc/istio/custom-bootstrap - name: custom-bootstrap-volume - {{- end }} - # SDS channel between istioagent and Envoy - - mountPath: /etc/istio/proxy - name: istio-envoy - {{- if eq .Values.global.jwtPolicy "third-party-jwt" }} - - mountPath: /var/run/secrets/tokens - name: istio-token - {{- end }} - {{- if .Values.global.mountMtlsCerts }} - # Use the key and cert mounted to /etc/certs/ for the in-cluster mTLS communications. - - mountPath: /etc/certs/ - name: istio-certs - readOnly: true - {{- end }} - - name: istio-podinfo - mountPath: /etc/istio/pod - {{- if and (eq .Values.global.proxy.tracer "lightstep") .ProxyConfig.GetTracing.GetTlsSettings }} - - mountPath: {{ directory .ProxyConfig.GetTracing.GetTlsSettings.GetCaCertificates }} - name: lightstep-certs - readOnly: true - {{- end }} - {{- if isset .ObjectMeta.Annotations `sidecar.istio.io/userVolumeMount` }} - {{ range $index, $value := fromJSON (index .ObjectMeta.Annotations `sidecar.istio.io/userVolumeMount`) }} - - name: "{{ $index }}" - {{ toYaml $value | indent 6 }} - {{ end }} - {{- end }} - volumes: - - emptyDir: - name: workload-socket - - emptyDir: - name: credential-socket - {{- if eq .Values.global.caName "GkeWorkloadCertificate" }} - - name: gke-workload-certificate - csi: - driver: workloadcertificates.security.cloud.google.com - {{- else }} - - emptyDir: - name: workload-certs - {{- end }} - {{- if (isset .ObjectMeta.Annotations `sidecar.istio.io/bootstrapOverride`) }} - - name: custom-bootstrap-volume - configMap: - name: {{ annotation .ObjectMeta `sidecar.istio.io/bootstrapOverride` "" }} - {{- end }} - # SDS channel between istioagent and Envoy - - emptyDir: - medium: Memory - name: istio-envoy - - name: istio-data - emptyDir: {} - - name: istio-podinfo - downwardAPI: - items: - - path: "labels" - fieldRef: - fieldPath: metadata.labels - - path: "annotations" - fieldRef: - fieldPath: metadata.annotations - {{- if eq .Values.global.jwtPolicy "third-party-jwt" }} - - name: istio-token - projected: - sources: - - serviceAccountToken: - path: istio-token - expirationSeconds: 43200 - audience: {{ .Values.global.sds.token.aud }} - {{- end }} - {{- if eq .Values.global.pilotCertProvider "istiod" }} - - name: istiod-ca-cert - configMap: - name: istio-ca-root-cert - {{- end }} - {{- if eq .Values.global.pilotCertProvider "kubernetes" }} - - name: kube-ca-cert - configMap: - name: kube-root-ca.crt - {{- end }} - {{- if .Values.global.mountMtlsCerts }} - # Use the key and cert mounted to /etc/certs/ for the in-cluster mTLS communications. - - name: istio-certs - secret: - optional: true - {{ if eq .Spec.ServiceAccountName "" }} - secretName: istio.default - {{ else -}} - secretName: {{ printf "istio.%s" .Spec.ServiceAccountName }} - {{ end -}} - {{- end }} - {{- if isset .ObjectMeta.Annotations `sidecar.istio.io/userVolume` }} - {{range $index, $value := fromJSON (index .ObjectMeta.Annotations `sidecar.istio.io/userVolume`) }} - - name: "{{ $index }}" - {{ toYaml $value | indent 4 }} - {{ end }} - {{ end }} - {{- if and (eq .Values.global.proxy.tracer "lightstep") .ProxyConfig.GetTracing.GetTlsSettings }} - - name: lightstep-certs - secret: - optional: true - secretName: lightstep.cacert - {{- end }} - {{- if .Values.global.imagePullSecrets }} - imagePullSecrets: - {{- range .Values.global.imagePullSecrets }} - - name: {{ . }} - {{- end }} - {{- end }} - {{- if eq (env "ENABLE_LEGACY_FSGROUP_INJECTION" "false") "true" }} - securityContext: - fsGroup: 1337 - {{- end }} - gateway: | - {{- $containers := list }} - {{- range $index, $container := .Spec.Containers }}{{ if not (eq $container.Name "istio-proxy") }}{{ $containers = append $containers $container.Name }}{{end}}{{- end}} - metadata: - labels: - service.istio.io/canonical-name: {{ index .ObjectMeta.Labels `service.istio.io/canonical-name` | default (index .ObjectMeta.Labels `app.kubernetes.io/name`) | default (index .ObjectMeta.Labels `app`) | default .DeploymentMeta.Name | quote }} - service.istio.io/canonical-revision: {{ index .ObjectMeta.Labels `service.istio.io/canonical-revision` | default (index .ObjectMeta.Labels `app.kubernetes.io/version`) | default (index .ObjectMeta.Labels `version`) | default "latest" | quote }} - istio.io/rev: {{ index .ObjectMeta.Labels `istio.io/rev` | default .Revision | default "default" | quote }} - annotations: { - {{- if eq (len $containers) 1 }} - kubectl.kubernetes.io/default-logs-container: "{{ index $containers 0 }}", - kubectl.kubernetes.io/default-container: "{{ index $containers 0 }}", - {{ end }} - } - spec: - containers: - - name: istio-proxy - {{- if contains "/" .Values.global.proxy.image }} - image: "{{ annotation .ObjectMeta `sidecar.istio.io/proxyImage` .Values.global.proxy.image }}" - {{- else }} - image: "{{ .ProxyImage }}" - {{- end }} - ports: - - containerPort: 15090 - protocol: TCP - name: http-envoy-prom - args: - - proxy - - router - - --domain - - $(POD_NAMESPACE).svc.{{ .Values.global.proxy.clusterDomain }} - - --proxyLogLevel={{ annotation .ObjectMeta `sidecar.istio.io/logLevel` .Values.global.proxy.logLevel }} - - --proxyComponentLogLevel={{ annotation .ObjectMeta `sidecar.istio.io/componentLogLevel` .Values.global.proxy.componentLogLevel }} - - --log_output_level={{ annotation .ObjectMeta `sidecar.istio.io/agentLogLevel` .Values.global.logging.level }} - {{- if .Values.global.sts.servicePort }} - - --stsPort={{ .Values.global.sts.servicePort }} - {{- end }} - {{- if .Values.global.logAsJson }} - - --log_as_json - {{- end }} - {{- if .Values.global.proxy.lifecycle }} - lifecycle: - {{ toYaml .Values.global.proxy.lifecycle | indent 6 }} - {{- end }} - env: - - name: JWT_POLICY - value: {{ .Values.global.jwtPolicy }} - - name: PILOT_CERT_PROVIDER - value: {{ .Values.global.pilotCertProvider }} - - name: CA_ADDR - {{- if .Values.global.caAddress }} - value: {{ .Values.global.caAddress }} - {{- else }} - value: istiod{{- if not (eq .Values.revision "") }}-{{ .Values.revision }}{{- end }}.{{ .Values.global.istioNamespace }}.svc:15012 - {{- end }} - - name: POD_NAME - valueFrom: - fieldRef: - fieldPath: metadata.name - - name: POD_NAMESPACE - valueFrom: - fieldRef: - fieldPath: metadata.namespace - - name: INSTANCE_IP - valueFrom: - fieldRef: - fieldPath: status.podIP - - name: SERVICE_ACCOUNT - valueFrom: - fieldRef: - fieldPath: spec.serviceAccountName - - name: HOST_IP - valueFrom: - fieldRef: - fieldPath: status.hostIP - - name: PROXY_CONFIG - value: | - {{ protoToJSON .ProxyConfig }} - - name: ISTIO_META_POD_PORTS - value: |- - [ - {{- $first := true }} - {{- range $index1, $c := .Spec.Containers }} - {{- range $index2, $p := $c.Ports }} - {{- if (structToJSON $p) }} - {{if not $first}},{{end}}{{ structToJSON $p }} - {{- $first = false }} - {{- end }} - {{- end}} - {{- end}} - ] - - name: ISTIO_META_APP_CONTAINERS - value: "{{ $containers | join "," }}" - - name: ISTIO_META_CLUSTER_ID - value: "{{ valueOrDefault .Values.global.multiCluster.clusterName `Kubernetes` }}" - - name: ISTIO_META_NODE_NAME - valueFrom: - fieldRef: - fieldPath: spec.nodeName - - name: ISTIO_META_INTERCEPTION_MODE - value: "{{ .ProxyConfig.InterceptionMode.String }}" - {{- if .Values.global.network }} - - name: ISTIO_META_NETWORK - value: "{{ .Values.global.network }}" - {{- end }} - {{- if .DeploymentMeta.Name }} - - name: ISTIO_META_WORKLOAD_NAME - value: "{{ .DeploymentMeta.Name }}" - {{ end }} - {{- if and .TypeMeta.APIVersion .DeploymentMeta.Name }} - - name: ISTIO_META_OWNER - value: kubernetes://apis/{{ .TypeMeta.APIVersion }}/namespaces/{{ valueOrDefault .DeploymentMeta.Namespace `default` }}/{{ toLower .TypeMeta.Kind}}s/{{ .DeploymentMeta.Name }} - {{- end}} - {{- if .Values.global.meshID }} - - name: ISTIO_META_MESH_ID - value: "{{ .Values.global.meshID }}" - {{- else if (valueOrDefault .MeshConfig.TrustDomain .Values.global.trustDomain) }} - - name: ISTIO_META_MESH_ID - value: "{{ (valueOrDefault .MeshConfig.TrustDomain .Values.global.trustDomain) }}" - {{- end }} - {{- with (valueOrDefault .MeshConfig.TrustDomain .Values.global.trustDomain) }} - - name: TRUST_DOMAIN - value: "{{ . }}" - {{- end }} - {{- range $key, $value := .ProxyConfig.ProxyMetadata }} - - name: {{ $key }} - value: "{{ $value }}" - {{- end }} - {{with .Values.global.imagePullPolicy }}imagePullPolicy: "{{.}}"{{end}} - readinessProbe: - httpGet: - path: /healthz/ready - port: 15021 - initialDelaySeconds: {{.Values.global.proxy.readinessInitialDelaySeconds }} - periodSeconds: {{ .Values.global.proxy.readinessPeriodSeconds }} - timeoutSeconds: 3 - failureThreshold: {{ .Values.global.proxy.readinessFailureThreshold }} - volumeMounts: - - name: workload-socket - mountPath: /var/run/secrets/workload-spiffe-uds - - name: credential-socket - mountPath: /var/run/secrets/credential-uds - {{- if eq .Values.global.caName "GkeWorkloadCertificate" }} - - name: gke-workload-certificate - mountPath: /var/run/secrets/workload-spiffe-credentials - readOnly: true - {{- else }} - - name: workload-certs - mountPath: /var/run/secrets/workload-spiffe-credentials - {{- end }} - {{- if eq .Values.global.pilotCertProvider "istiod" }} - - mountPath: /var/run/secrets/istio - name: istiod-ca-cert - {{- end }} - - mountPath: /var/lib/istio/data - name: istio-data - # SDS channel between istioagent and Envoy - - mountPath: /etc/istio/proxy - name: istio-envoy - {{- if eq .Values.global.jwtPolicy "third-party-jwt" }} - - mountPath: /var/run/secrets/tokens - name: istio-token - {{- end }} - {{- if .Values.global.mountMtlsCerts }} - # Use the key and cert mounted to /etc/certs/ for the in-cluster mTLS communications. - - mountPath: /etc/certs/ - name: istio-certs - readOnly: true - {{- end }} - - name: istio-podinfo - mountPath: /etc/istio/pod - volumes: - - emptyDir: {} - name: workload-socket - - emptyDir: {} - name: credential-socket - {{- if eq .Values.global.caName "GkeWorkloadCertificate" }} - - name: gke-workload-certificate - csi: - driver: workloadcertificates.security.cloud.google.com - {{- else}} - - emptyDir: {} - name: workload-certs - {{- end }} - # SDS channel between istioagent and Envoy - - emptyDir: - medium: Memory - name: istio-envoy - - name: istio-data - emptyDir: {} - - name: istio-podinfo - downwardAPI: - items: - - path: "labels" - fieldRef: - fieldPath: metadata.labels - - path: "annotations" - fieldRef: - fieldPath: metadata.annotations - {{- if eq .Values.global.jwtPolicy "third-party-jwt" }} - - name: istio-token - projected: - sources: - - serviceAccountToken: - path: istio-token - expirationSeconds: 43200 - audience: {{ .Values.global.sds.token.aud }} - {{- end }} - {{- if eq .Values.global.pilotCertProvider "istiod" }} - - name: istiod-ca-cert - configMap: - name: istio-ca-root-cert - {{- end }} - {{- if .Values.global.mountMtlsCerts }} - # Use the key and cert mounted to /etc/certs/ for the in-cluster mTLS communications. - - name: istio-certs - secret: - optional: true - {{ if eq .Spec.ServiceAccountName "" }} - secretName: istio.default - {{ else -}} - secretName: {{ printf "istio.%s" .Spec.ServiceAccountName }} - {{ end -}} - {{- end }} - {{- if .Values.global.imagePullSecrets }} - imagePullSecrets: - {{- range .Values.global.imagePullSecrets }} - - name: {{ . }} - {{- end }} - {{- end }} - {{- if eq (env "ENABLE_LEGACY_FSGROUP_INJECTION" "false") "true" }} - securityContext: - fsGroup: 1337 - {{- end }} - grpc-simple: | - metadata: - annotations: - sidecar.istio.io/rewriteAppHTTPProbers: "false" - spec: - initContainers: - - name: grpc-bootstrap-init - image: busybox:1.28 - volumeMounts: - - mountPath: /var/lib/grpc/data/ - name: grpc-io-proxyless-bootstrap - env: - - name: INSTANCE_IP - valueFrom: - fieldRef: - fieldPath: status.podIP - - name: POD_NAME - valueFrom: - fieldRef: - fieldPath: metadata.name - - name: POD_NAMESPACE - valueFrom: - fieldRef: - fieldPath: metadata.namespace - - name: ISTIO_NAMESPACE - value: | - {{ .Values.global.istioNamespace }} - command: - - sh - - "-c" - - |- - NODE_ID="sidecar~${INSTANCE_IP}~${POD_NAME}.${POD_NAMESPACE}~cluster.local" - SERVER_URI="dns:///istiod.${ISTIO_NAMESPACE}.svc:15010" - echo ' - { - "xds_servers": [ - { - "server_uri": "'${SERVER_URI}'", - "channel_creds": [{"type": "insecure"}], - "server_features" : ["xds_v3"] - } - ], - "node": { - "id": "'${NODE_ID}'", - "metadata": { - "GENERATOR": "grpc" - } - } - }' > /var/lib/grpc/data/bootstrap.json - containers: - {{- range $index, $container := .Spec.Containers }} - - name: {{ $container.Name }} - env: - - name: GRPC_XDS_BOOTSTRAP - value: /var/lib/grpc/data/bootstrap.json - - name: GRPC_GO_LOG_VERBOSITY_LEVEL - value: "99" - - name: GRPC_GO_LOG_SEVERITY_LEVEL - value: info - volumeMounts: - - mountPath: /var/lib/grpc/data/ - name: grpc-io-proxyless-bootstrap - {{- end }} - volumes: - - name: grpc-io-proxyless-bootstrap - emptyDir: {} - grpc-agent: | - {{- define "resources" }} - {{- if or (isset .ObjectMeta.Annotations `sidecar.istio.io/proxyCPU`) (isset .ObjectMeta.Annotations `sidecar.istio.io/proxyMemory`) (isset .ObjectMeta.Annotations `sidecar.istio.io/proxyCPULimit`) (isset .ObjectMeta.Annotations `sidecar.istio.io/proxyMemoryLimit`) }} - {{- if or (isset .ObjectMeta.Annotations `sidecar.istio.io/proxyCPU`) (isset .ObjectMeta.Annotations `sidecar.istio.io/proxyMemory`) }} - requests: - {{ if (isset .ObjectMeta.Annotations `sidecar.istio.io/proxyCPU`) -}} - cpu: "{{ index .ObjectMeta.Annotations `sidecar.istio.io/proxyCPU` }}" - {{ end }} - {{ if (isset .ObjectMeta.Annotations `sidecar.istio.io/proxyMemory`) -}} - memory: "{{ index .ObjectMeta.Annotations `sidecar.istio.io/proxyMemory` }}" - {{ end }} - {{- end }} - {{- if or (isset .ObjectMeta.Annotations `sidecar.istio.io/proxyCPULimit`) (isset .ObjectMeta.Annotations `sidecar.istio.io/proxyMemoryLimit`) }} - limits: - {{ if (isset .ObjectMeta.Annotations `sidecar.istio.io/proxyCPULimit`) -}} - cpu: "{{ index .ObjectMeta.Annotations `sidecar.istio.io/proxyCPULimit` }}" - {{ end }} - {{ if (isset .ObjectMeta.Annotations `sidecar.istio.io/proxyMemoryLimit`) -}} - memory: "{{ index .ObjectMeta.Annotations `sidecar.istio.io/proxyMemoryLimit` }}" - {{ end }} - {{- end }} - {{- else }} - {{- if .Values.global.proxy.resources }} - {{ toYaml .Values.global.proxy.resources | indent 6 }} - {{- end }} - {{- end }} - {{- end }} - {{- $containers := list }} - {{- range $index, $container := .Spec.Containers }}{{ if not (eq $container.Name "istio-proxy") }}{{ $containers = append $containers $container.Name }}{{end}}{{- end}} - metadata: - labels: - {{/* security.istio.io/tlsMode: istio must be set by user, if gRPC is using mTLS initialization code. We can't set it automatically. */}} - service.istio.io/canonical-name: {{ index .ObjectMeta.Labels `service.istio.io/canonical-name` | default (index .ObjectMeta.Labels `app.kubernetes.io/name`) | default (index .ObjectMeta.Labels `app`) | default .DeploymentMeta.Name | quote }} - service.istio.io/canonical-revision: {{ index .ObjectMeta.Labels `service.istio.io/canonical-revision` | default (index .ObjectMeta.Labels `app.kubernetes.io/version`) | default (index .ObjectMeta.Labels `version`) | default "latest" | quote }} - annotations: { - {{- if ge (len $containers) 1 }} - {{- if not (isset .ObjectMeta.Annotations `kubectl.kubernetes.io/default-logs-container`) }} - kubectl.kubernetes.io/default-logs-container: "{{ index $containers 0 }}", - {{- end }} - {{- if not (isset .ObjectMeta.Annotations `kubectl.kubernetes.io/default-container`) }} - kubectl.kubernetes.io/default-container: "{{ index $containers 0 }}", - {{- end }} - {{- end }} - sidecar.istio.io/rewriteAppHTTPProbers: "false", - } - spec: - containers: - - name: istio-proxy - {{- if contains "/" (annotation .ObjectMeta `sidecar.istio.io/proxyImage` .Values.global.proxy.image) }} - image: "{{ annotation .ObjectMeta `sidecar.istio.io/proxyImage` .Values.global.proxy.image }}" - {{- else }} - image: "{{ .ProxyImage }}" - {{- end }} - ports: - - containerPort: 15020 - protocol: TCP - name: mesh-metrics - args: - - proxy - - sidecar - - --domain - - $(POD_NAMESPACE).svc.{{ .Values.global.proxy.clusterDomain }} - - --proxyLogLevel={{ annotation .ObjectMeta `sidecar.istio.io/logLevel` .Values.global.proxy.logLevel }} - - --proxyComponentLogLevel={{ annotation .ObjectMeta `sidecar.istio.io/componentLogLevel` .Values.global.proxy.componentLogLevel }} - - --log_output_level={{ annotation .ObjectMeta `sidecar.istio.io/agentLogLevel` .Values.global.logging.level }} - {{- if .Values.global.sts.servicePort }} - - --stsPort={{ .Values.global.sts.servicePort }} - {{- end }} - {{- if .Values.global.logAsJson }} - - --log_as_json - {{- end }} - lifecycle: - postStart: - exec: - command: - - pilot-agent - - wait - - --url=http://localhost:15020/healthz/ready - env: - - name: ISTIO_META_GENERATOR - value: grpc - - name: OUTPUT_CERTS - value: /var/lib/istio/data - {{- if eq (env "PILOT_ENABLE_INBOUND_PASSTHROUGH" "true") "false" }} - - name: REWRITE_PROBE_LEGACY_LOCALHOST_DESTINATION - value: "true" - {{- end }} - - name: JWT_POLICY - value: {{ .Values.global.jwtPolicy }} - - name: PILOT_CERT_PROVIDER - value: {{ .Values.global.pilotCertProvider }} - - name: CA_ADDR - {{- if .Values.global.caAddress }} - value: {{ .Values.global.caAddress }} - {{- else }} - value: istiod{{- if not (eq .Values.revision "") }}-{{ .Values.revision }}{{- end }}.{{ .Values.global.istioNamespace }}.svc:15012 - {{- end }} - - name: POD_NAME - valueFrom: - fieldRef: - fieldPath: metadata.name - - name: POD_NAMESPACE - valueFrom: - fieldRef: - fieldPath: metadata.namespace - - name: INSTANCE_IP - valueFrom: - fieldRef: - fieldPath: status.podIP - - name: SERVICE_ACCOUNT - valueFrom: - fieldRef: - fieldPath: spec.serviceAccountName - - name: HOST_IP - valueFrom: - fieldRef: - fieldPath: status.hostIP - - name: PROXY_CONFIG - value: | - {{ protoToJSON .ProxyConfig }} - - name: ISTIO_META_POD_PORTS - value: |- - [ - {{- $first := true }} - {{- range $index1, $c := .Spec.Containers }} - {{- range $index2, $p := $c.Ports }} - {{- if (structToJSON $p) }} - {{if not $first}},{{end}}{{ structToJSON $p }} - {{- $first = false }} - {{- end }} - {{- end}} - {{- end}} - ] - - name: ISTIO_META_APP_CONTAINERS - value: "{{ $containers | join "," }}" - - name: ISTIO_META_CLUSTER_ID - value: "{{ valueOrDefault .Values.global.multiCluster.clusterName `Kubernetes` }}" - - name: ISTIO_META_NODE_NAME - valueFrom: - fieldRef: - fieldPath: spec.nodeName - {{- if .Values.global.network }} - - name: ISTIO_META_NETWORK - value: "{{ .Values.global.network }}" - {{- end }} - {{- if .DeploymentMeta.Name }} - - name: ISTIO_META_WORKLOAD_NAME - value: "{{ .DeploymentMeta.Name }}" - {{ end }} - {{- if and .TypeMeta.APIVersion .DeploymentMeta.Name }} - - name: ISTIO_META_OWNER - value: kubernetes://apis/{{ .TypeMeta.APIVersion }}/namespaces/{{ valueOrDefault .DeploymentMeta.Namespace `default` }}/{{ toLower .TypeMeta.Kind}}s/{{ .DeploymentMeta.Name }} - {{- end}} - {{- if .Values.global.meshID }} - - name: ISTIO_META_MESH_ID - value: "{{ .Values.global.meshID }}" - {{- else if (valueOrDefault .MeshConfig.TrustDomain .Values.global.trustDomain) }} - - name: ISTIO_META_MESH_ID - value: "{{ (valueOrDefault .MeshConfig.TrustDomain .Values.global.trustDomain) }}" - {{- end }} - {{- with (valueOrDefault .MeshConfig.TrustDomain .Values.global.trustDomain) }} - - name: TRUST_DOMAIN - value: "{{ . }}" - {{- end }} - {{- range $key, $value := .ProxyConfig.ProxyMetadata }} - - name: {{ $key }} - value: "{{ $value }}" - {{- end }} - # grpc uses xds:/// to resolve – no need to resolve VIP - - name: ISTIO_META_DNS_CAPTURE - value: "false" - - name: DISABLE_ENVOY - value: "true" - {{with .Values.global.imagePullPolicy }}imagePullPolicy: "{{.}}"{{end}} - {{ if ne (annotation .ObjectMeta `status.sidecar.istio.io/port` .Values.global.proxy.statusPort) `0` }} - readinessProbe: - httpGet: - path: /healthz/ready - port: 15020 - initialDelaySeconds: {{ annotation .ObjectMeta `readiness.status.sidecar.istio.io/initialDelaySeconds` .Values.global.proxy.readinessInitialDelaySeconds }} - periodSeconds: {{ annotation .ObjectMeta `readiness.status.sidecar.istio.io/periodSeconds` .Values.global.proxy.readinessPeriodSeconds }} - timeoutSeconds: 3 - failureThreshold: {{ annotation .ObjectMeta `readiness.status.sidecar.istio.io/failureThreshold` .Values.global.proxy.readinessFailureThreshold }} - resources: - {{ template "resources" . }} - volumeMounts: - - name: workload-socket - mountPath: /var/run/secrets/workload-spiffe-uds - {{- if eq .Values.global.caName "GkeWorkloadCertificate" }} - - name: gke-workload-certificate - mountPath: /var/run/secrets/workload-spiffe-credentials - readOnly: true - {{- else }} - - name: workload-certs - mountPath: /var/run/secrets/workload-spiffe-credentials - {{- end }} - {{- if eq .Values.global.pilotCertProvider "istiod" }} - - mountPath: /var/run/secrets/istio - name: istiod-ca-cert - {{- end }} - - mountPath: /var/lib/istio/data - name: istio-data - # UDS channel between istioagent and gRPC client for XDS/SDS - - mountPath: /etc/istio/proxy - name: istio-xds - {{- if eq .Values.global.jwtPolicy "third-party-jwt" }} - - mountPath: /var/run/secrets/tokens - name: istio-token - {{- end }} - {{- if .Values.global.mountMtlsCerts }} - # Use the key and cert mounted to /etc/certs/ for the in-cluster mTLS communications. - - mountPath: /etc/certs/ - name: istio-certs - readOnly: true - {{- end }} - - name: istio-podinfo - mountPath: /etc/istio/pod - {{- end }} - {{- if isset .ObjectMeta.Annotations `sidecar.istio.io/userVolumeMount` }} - {{ range $index, $value := fromJSON (index .ObjectMeta.Annotations `sidecar.istio.io/userVolumeMount`) }} - - name: "{{ $index }}" - {{ toYaml $value | indent 6 }} - {{ end }} - {{- end }} - {{- range $index, $container := .Spec.Containers }} - {{ if not (eq $container.Name "istio-proxy") }} - - name: {{ $container.Name }} - env: - - name: "GRPC_XDS_EXPERIMENTAL_SECURITY_SUPPORT" - value: "true" - - name: "GRPC_XDS_BOOTSTRAP" - value: "/etc/istio/proxy/grpc-bootstrap.json" - volumeMounts: - - mountPath: /var/lib/istio/data - name: istio-data - # UDS channel between istioagent and gRPC client for XDS/SDS - - mountPath: /etc/istio/proxy - name: istio-xds - {{- if eq $.Values.global.caName "GkeWorkloadCertificate" }} - - name: gke-workload-certificate - mountPath: /var/run/secrets/workload-spiffe-credentials - readOnly: true - {{- else }} - - name: workload-certs - mountPath: /var/run/secrets/workload-spiffe-credentials - {{- end }} - {{- end }} - {{- end }} - volumes: - - emptyDir: - name: workload-socket - {{- if eq .Values.global.caName "GkeWorkloadCertificate" }} - - name: gke-workload-certificate - csi: - driver: workloadcertificates.security.cloud.google.com - {{- else }} - - emptyDir: - name: workload-certs - {{- end }} - {{- if (isset .ObjectMeta.Annotations `sidecar.istio.io/bootstrapOverride`) }} - - name: custom-bootstrap-volume - configMap: - name: {{ annotation .ObjectMeta `sidecar.istio.io/bootstrapOverride` "" }} - {{- end }} - # SDS channel between istioagent and Envoy - - emptyDir: - medium: Memory - name: istio-xds - - name: istio-data - emptyDir: {} - - name: istio-podinfo - downwardAPI: - items: - - path: "labels" - fieldRef: - fieldPath: metadata.labels - - path: "annotations" - fieldRef: - fieldPath: metadata.annotations - {{- if eq .Values.global.jwtPolicy "third-party-jwt" }} - - name: istio-token - projected: - sources: - - serviceAccountToken: - path: istio-token - expirationSeconds: 43200 - audience: {{ .Values.global.sds.token.aud }} - {{- end }} - {{- if eq .Values.global.pilotCertProvider "istiod" }} - - name: istiod-ca-cert - configMap: - name: istio-ca-root-cert - {{- end }} - {{- if .Values.global.mountMtlsCerts }} - # Use the key and cert mounted to /etc/certs/ for the in-cluster mTLS communications. - - name: istio-certs - secret: - optional: true - {{ if eq .Spec.ServiceAccountName "" }} - secretName: istio.default - {{ else -}} - secretName: {{ printf "istio.%s" .Spec.ServiceAccountName }} - {{ end -}} - {{- end }} - {{- if isset .ObjectMeta.Annotations `sidecar.istio.io/userVolume` }} - {{range $index, $value := fromJSON (index .ObjectMeta.Annotations `sidecar.istio.io/userVolume`) }} - - name: "{{ $index }}" - {{ toYaml $value | indent 4 }} - {{ end }} - {{ end }} - {{- if .Values.global.imagePullSecrets }} - imagePullSecrets: - {{- range .Values.global.imagePullSecrets }} - - name: {{ . }} - {{- end }} - {{- end }} - {{- if eq (env "ENABLE_LEGACY_FSGROUP_INJECTION" "false") "true" }} - securityContext: - fsGroup: 1337 - {{- end }} diff --git a/overlays/istio-install/patches/resources.yaml b/overlays/istio-install/patches/resources.yaml deleted file mode 100644 index 0c1bf0e..0000000 --- a/overlays/istio-install/patches/resources.yaml +++ /dev/null @@ -1,14 +0,0 @@ -apiVersion: apps/v1 -kind: Deployment -metadata: - name: istiod - namespace: istio-system -spec: - template: - spec: - containers: - - name: discovery - resources: - requests: - cpu: 20m - memory: 128Mi diff --git a/overlays/profile/all.yaml b/overlays/profile/all.yaml deleted file mode 100644 index e69de29..0000000 diff --git a/overlays/profile/kustomization.yaml b/overlays/profile/kustomization.yaml deleted file mode 100644 index a7a55ae..0000000 --- a/overlays/profile/kustomization.yaml +++ /dev/null @@ -1,4 +0,0 @@ -apiVersion: kustomize.config.k8s.io/v1beta1 -kind: Kustomization -resources: -- all.yaml \ No newline at end of file From 9a62aaa65f37f6ae185c0e0198c555204710189b Mon Sep 17 00:00:00 2001 From: alex-treebeard Date: Thu, 29 Feb 2024 18:15:16 +0000 Subject: [PATCH 40/56] eks e2e --- .../argo_app/templates/central-dashboard/app.yaml | 2 -- examples/eks-https-loadbalancer/eks.tf | 4 ---- examples/eks-https-loadbalancer/kubeflow.tf | 15 ++++++--------- examples/eks-https-loadbalancer/main.tf | 4 ---- kf_dependencies.tf | 4 ---- variables.tf | 5 ----- 6 files changed, 6 insertions(+), 28 deletions(-) diff --git a/charts/argo_app/templates/central-dashboard/app.yaml b/charts/argo_app/templates/central-dashboard/app.yaml index 1d90709..eec3841 100644 --- a/charts/argo_app/templates/central-dashboard/app.yaml +++ b/charts/argo_app/templates/central-dashboard/app.yaml @@ -3,8 +3,6 @@ apiVersion: argoproj.io/v1alpha1 kind: Application metadata: name: {{ .Values.name }} - annotations: - test: "test" finalizers: - resources-finalizer.argocd.argoproj.io spec: diff --git a/examples/eks-https-loadbalancer/eks.tf b/examples/eks-https-loadbalancer/eks.tf index 457d4e6..bd119bc 100644 --- a/examples/eks-https-loadbalancer/eks.tf +++ b/examples/eks-https-loadbalancer/eks.tf @@ -129,10 +129,6 @@ provider "helm" { } } -provider "kustomization" { - kubeconfig_path = "/home/vscode/.kube/eks.yaml" -} - resource "null_resource" "cluster_ready" { # triggers = { # always_run = "${timestamp()}" diff --git a/examples/eks-https-loadbalancer/kubeflow.tf b/examples/eks-https-loadbalancer/kubeflow.tf index 2dc3eac..a22d014 100644 --- a/examples/eks-https-loadbalancer/kubeflow.tf +++ b/examples/eks-https-loadbalancer/kubeflow.tf @@ -1,19 +1,16 @@ -resource "null_resource" "dependency" { - depends_on = [ - null_resource.cluster_ready, - null_resource.core_addons, - null_resource.istio, - ] -} - module "treebeardkf" { count = var.enable_treebeardkf ? 1 : 0 source = "../.." hostname = var.host enable_https = true + issuer_name = "treebeard-issuer" enable_istio_base = false enable_istiod = false enable_istio_resources = true enable_cert_manager = false - dependency = null_resource.dependency.id + depends_on = [ + null_resource.cluster_ready, + null_resource.core_addons, + null_resource.istio +] } \ No newline at end of file diff --git a/examples/eks-https-loadbalancer/main.tf b/examples/eks-https-loadbalancer/main.tf index 237490d..13a3842 100644 --- a/examples/eks-https-loadbalancer/main.tf +++ b/examples/eks-https-loadbalancer/main.tf @@ -1,10 +1,6 @@ terraform { required_providers { - kustomization = { - source = "kbst/kustomization" - version = "~> 0.9.5" - } helm = { source = "hashicorp/helm" version = "~> 2.12.1" diff --git a/kf_dependencies.tf b/kf_dependencies.tf index b5dda48..8bde6fc 100644 --- a/kf_dependencies.tf +++ b/kf_dependencies.tf @@ -3,10 +3,6 @@ resource "null_resource" "kf_dependencies_start" { provisioner "local-exec" { command = "echo '⏳ Installing Kubeflow dependencies...'" } - - depends_on = [ - var.dependency - ] } resource "helm_release" "cert_manager" { diff --git a/variables.tf b/variables.tf index 30c067a..9fff31b 100644 --- a/variables.tf +++ b/variables.tf @@ -1,8 +1,3 @@ -variable "dependency" { - type = string - default = "null" -} - variable "hostname" { type = string default = "*" From 92a087e55e916477b06d57bed3e3c513338e6c69 Mon Sep 17 00:00:00 2001 From: alex-treebeard Date: Thu, 29 Feb 2024 18:15:48 +0000 Subject: [PATCH 41/56] fmt --- Makefile | 2 +- examples/eks-https-loadbalancer/kubeflow.tf | 4 ++-- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/Makefile b/Makefile index ac54af8..45b4311 100644 --- a/Makefile +++ b/Makefile @@ -24,7 +24,7 @@ docs-rm-new: rm -f examples/aks/README.md.new fmt: - terraform fmt . modules/* examples/* + terraform fmt . examples/* .PHONY: k3d-create k3d-create: diff --git a/examples/eks-https-loadbalancer/kubeflow.tf b/examples/eks-https-loadbalancer/kubeflow.tf index a22d014..040a461 100644 --- a/examples/eks-https-loadbalancer/kubeflow.tf +++ b/examples/eks-https-loadbalancer/kubeflow.tf @@ -3,7 +3,7 @@ module "treebeardkf" { source = "../.." hostname = var.host enable_https = true - issuer_name = "treebeard-issuer" + issuer_name = "treebeard-issuer" enable_istio_base = false enable_istiod = false enable_istio_resources = true @@ -12,5 +12,5 @@ module "treebeardkf" { null_resource.cluster_ready, null_resource.core_addons, null_resource.istio -] + ] } \ No newline at end of file From ffa437d4e8f0f88256ed6d26704e7db1b883eab9 Mon Sep 17 00:00:00 2001 From: alex-treebeard Date: Thu, 29 Feb 2024 18:31:47 +0000 Subject: [PATCH 42/56] aks fix --- .../argo_app/templates/wait-for-sync/job.yaml | 2 +- examples/aks/main.tf | 4 + examples/eks/README.md | 52 ---- examples/eks/main.tf | 225 ------------------ 4 files changed, 5 insertions(+), 278 deletions(-) delete mode 100644 examples/eks/README.md delete mode 100644 examples/eks/main.tf diff --git a/charts/argo_app/templates/wait-for-sync/job.yaml b/charts/argo_app/templates/wait-for-sync/job.yaml index 5a934ea..069a367 100644 --- a/charts/argo_app/templates/wait-for-sync/job.yaml +++ b/charts/argo_app/templates/wait-for-sync/job.yaml @@ -25,5 +25,5 @@ spec: kubectl get -n $NAMESPACE app $APP_NAME -o json | jq '.status.operationState' restartPolicy: Never backoffLimit: 1 - activeDeadlineSeconds: 20 + activeDeadlineSeconds: 30 ttlSecondsAfterFinished: 30 \ No newline at end of file diff --git a/examples/aks/main.tf b/examples/aks/main.tf index a8f4665..5a0ac4d 100644 --- a/examples/aks/main.tf +++ b/examples/aks/main.tf @@ -70,4 +70,8 @@ variable "enable_treebeardkf" { module "treebeardkf" { count = var.enable_treebeardkf ? 1 : 0 source = "../.." + + depends_on = [ + azurerm_kubernetes_cluster.aks + ] } \ No newline at end of file diff --git a/examples/eks/README.md b/examples/eks/README.md deleted file mode 100644 index 73b9a08..0000000 --- a/examples/eks/README.md +++ /dev/null @@ -1,52 +0,0 @@ -# Kubeflow Terraform EKS Example - -* We recommend deploying the eks cluster alone then adding kubeflow -* When eks is ready you can use the aws CLI to setup kubectl - ```sh - aws eks update-kubeconfig --name cluster --region us-east-1 - ``` - -## Requirements - -| Name | Version | -|------|---------| -| [aws](#requirement\_aws) | 5.28.0 | -| [helm](#requirement\_helm) | ~> 2.12.1 | -| [kubernetes](#requirement\_kubernetes) | ~> 2.25.2 | -| [kustomization](#requirement\_kustomization) | ~> 0.9.5 | - -## Providers - -| Name | Version | -|------|---------| -| [aws](#provider\_aws) | 5.28.0 | -| [helm](#provider\_helm) | 2.12.1 | -| [kubernetes](#provider\_kubernetes) | 2.25.2 | - -## Modules - -| Name | Source | Version | -|------|--------|---------| -| [eks](#module\_eks) | terraform-aws-modules/eks/aws | ~> 19.20 | -| [iam\_eks\_role](#module\_iam\_eks\_role) | terraform-aws-modules/iam/aws//modules/iam-role-for-service-accounts-eks | 5.34.0 | -| [treebeardkf](#module\_treebeardkf) | ../.. | n/a | -| [vpc](#module\_vpc) | terraform-aws-modules/vpc/aws | 5.1.2 | - -## Resources - -| Name | Type | -|------|------| -| [helm_release.ebs_csi_driver](https://registry.terraform.io/providers/hashicorp/helm/latest/docs/resources/release) | resource | -| [kubernetes_annotations.default-storageclass](https://registry.terraform.io/providers/hashicorp/kubernetes/latest/docs/resources/annotations) | resource | -| [aws_availability_zones.available](https://registry.terraform.io/providers/hashicorp/aws/5.28.0/docs/data-sources/availability_zones) | data source | - -## Inputs - -| Name | Description | Type | Default | Required | -|------|-------------|------|---------|:--------:| -| [aws\_profile](#input\_aws\_profile) | AWS profile to use for authentication. | `any` | n/a | yes | -| [aws\_region](#input\_aws\_region) | AWS region to launch servers. | `any` | n/a | yes | - -## Outputs - -No outputs. diff --git a/examples/eks/main.tf b/examples/eks/main.tf deleted file mode 100644 index 91e0c8f..0000000 --- a/examples/eks/main.tf +++ /dev/null @@ -1,225 +0,0 @@ -terraform { - required_providers { - aws = { - source = "hashicorp/aws" - version = "5.28.0" - } - kustomization = { - source = "kbst/kustomization" - version = "~> 0.9.5" - } - helm = { - source = "hashicorp/helm" - version = "~> 2.12.1" - } - kubernetes = { - source = "hashicorp/kubernetes" - version = "~> 2.25.2" - } - } -} - -data "aws_availability_zones" "available" {} - -locals { - name = basename(path.cwd) - azs = slice(data.aws_availability_zones.available.names, 0, 3) - tags = { - "tf" : "true" - "Name" : local.name - } - cloud_cidr = "10.0.0.0/16" -} - -variable "aws_region" { - description = "AWS region to launch servers." -} - -variable "aws_profile" { - description = "AWS profile to use for authentication." -} - -provider "aws" { - region = var.aws_region - profile = var.aws_profile - default_tags { - tags = { - "tf" : "true" - "Name" : local.name - } - } -} - - -module "vpc" { - source = "terraform-aws-modules/vpc/aws" - version = "5.1.2" - - name = "kubeflow-vpc" - cidr = local.cloud_cidr - map_public_ip_on_launch = true - - azs = local.azs - public_subnets = ["10.0.1.0/24", "10.0.2.0/24"] - private_subnets = ["10.0.3.0/24", "10.0.4.0/24"] - - enable_nat_gateway = true - single_nat_gateway = true - - enable_dns_hostnames = true - enable_dns_support = true -} - -module "eks" { - source = "terraform-aws-modules/eks/aws" - version = "~> 19.20" - - cluster_name = "cluster" - cluster_version = "1.28" - cluster_endpoint_public_access = true - - vpc_id = module.vpc.vpc_id - subnet_ids = module.vpc.public_subnets - create_cloudwatch_log_group = false - cluster_encryption_config = {} - - node_security_group_additional_rules = { - ssh = { - type = "ingress" - description = "Allow SSH inbound traffic from anywhere" - from_port = 22 - to_port = 22 - protocol = "tcp" - cidr_blocks = ["0.0.0.0/0"] - }, - istio_injection_webhook = { - description = "Allow istio injection" - protocol = "tcp" - from_port = "15017" - to_port = "15017" - type = "ingress" - source_cluster_security_group = true - } - } - eks_managed_node_groups = { - amzn_linux = { - instance_types = ["t3.xlarge"] - min_size = 0 - max_size = 1 - desired_size = 1 - } - } - tags = local.tags -} - -provider "kubernetes" { - host = module.eks.cluster_endpoint - cluster_ca_certificate = base64decode(module.eks.cluster_certificate_authority_data) - exec { - command = "aws" - api_version = "client.authentication.k8s.io/v1beta1" - args = [ - "--region", - var.aws_region, - "eks", - "get-token", - "--cluster-name", - module.eks.cluster_name - ] - env = { - name = "AWS_PROFILE" - value = var.aws_profile - } - } -} - -provider "helm" { - kubernetes { - host = module.eks.cluster_endpoint - cluster_ca_certificate = base64decode(module.eks.cluster_certificate_authority_data) - exec { - command = "aws" - api_version = "client.authentication.k8s.io/v1beta1" - args = [ - "--region", - var.aws_region, - "eks", - "get-token", - "--cluster-name", - module.eks.cluster_name - ] - env = { - name = "AWS_PROFILE" - value = var.aws_profile - } - } - } -} - -variable "kubeconfig" { - description = "Path to the kubeconfig file" - type = string -} - -provider "kustomization" { - kubeconfig_raw = file(var.kubeconfig) -} - -module "iam_eks_role" { - source = "terraform-aws-modules/iam/aws//modules/iam-role-for-service-accounts-eks" - version = "5.34.0" - role_name = "ebs-csi" - attach_ebs_csi_policy = true - oidc_providers = { - main = { - provider_arn = module.eks.oidc_provider_arn - namespace_service_accounts = ["kube-system:ebs-csi-controller-sa"] - } - } -} - -resource "kubernetes_annotations" "default-storageclass" { - api_version = "storage.k8s.io/v1" - kind = "StorageClass" - force = "true" - - metadata { - name = "gp2" - } - annotations = { - "storageclass.kubernetes.io/is-default-class" = "false" - } -} - -resource "helm_release" "ebs_csi_driver" { - name = "aws-ebs-csi-driver" - chart = "aws-ebs-csi-driver" - repository = "https://kubernetes-sigs.github.io/aws-ebs-csi-driver" - namespace = "kube-system" - version = "2.28.1" - values = [ - < Date: Thu, 29 Feb 2024 18:48:15 +0000 Subject: [PATCH 43/56] istio resources --- kf_dependencies.tf | 26 ++++++++++++++++++++++---- 1 file changed, 22 insertions(+), 4 deletions(-) diff --git a/kf_dependencies.tf b/kf_dependencies.tf index 8bde6fc..2720d54 100644 --- a/kf_dependencies.tf +++ b/kf_dependencies.tf @@ -53,6 +53,17 @@ resource "helm_release" "istiod" { ] values = [ < Date: Thu, 29 Feb 2024 19:51:06 +0000 Subject: [PATCH 44/56] password change --- README.md | 11 +++++- charts/argo_app/templates/all.yaml | 2 +- charts/argo_app/templates/dex/app.yaml | 33 ++++++++++++++++++ .../templates/istio-resources/app.yaml | 2 -- examples/eks-https-loadbalancer/dex_conf.yaml | 34 ------------------- kf_apps.tf | 1 + kf_core.tf | 1 + variables.tf | 12 +++++++ 8 files changed, 58 insertions(+), 38 deletions(-) create mode 100644 charts/argo_app/templates/dex/app.yaml delete mode 100644 examples/eks-https-loadbalancer/dex_conf.yaml diff --git a/README.md b/README.md index a6d0859..28f447e 100644 --- a/README.md +++ b/README.md @@ -72,7 +72,7 @@ Some considerations: 1. If you are calling this Terraform module from your own module, pass in a string to the `completed` variable in order to manage Kubeflow *after* changes to your other resources. (Note that `depends_on` does not work with this module) 2. If you already have Istio and Cert Manager installed, you will need to ensure Kubeflow works with them. See [examples/k3s-existing-istio](examples/k3s-existing-istio) for a configuration that we have tested like this. -### Enable Single-Sign-On (SSO) +### Set a new password @@ -95,6 +95,15 @@ use [this guide](https://kubernetes-sigs.github.io/external-dns/v0.14.0/tutorial 2. Remove the terraform module, e.g. with `terraform destroy` if you have installed directly from CLI 3. Clean up remaining resources, e.g. Istio leaves behind some secrets that can prevent successful re-installation. +## Troubleshooting + +## Reconfiguration challenges + +Moving the deployment between different states of configuration can be challenging +due to the dependencies between components in the cluster. + +If you have made a change to a dependency such as istio, or an auth component such as dex, it can be a good idea to re-create pods such that they re-initialise. This can be done by scaling to 0 then back up again, or simply deleting a pod managed by a deployment. + ## Architecture This module is built on top of the official [Kubeflow Manifests repo](https://github.com/kubeflow/manifests) which contains _Kustomizations_ for the various components of Kubeflow. diff --git a/charts/argo_app/templates/all.yaml b/charts/argo_app/templates/all.yaml index bf2bdd1..a02df92 100644 --- a/charts/argo_app/templates/all.yaml +++ b/charts/argo_app/templates/all.yaml @@ -1,4 +1,4 @@ -{{- $nameList := list "kubeflow-istio-resources" "centraldashboard" }} +{{- $nameList := list "dex" "kubeflow-istio-resources" "centraldashboard" }} {{- if not (has .Values.name $nameList) }} apiVersion: argoproj.io/v1alpha1 kind: Application diff --git a/charts/argo_app/templates/dex/app.yaml b/charts/argo_app/templates/dex/app.yaml new file mode 100644 index 0000000..dd62ad9 --- /dev/null +++ b/charts/argo_app/templates/dex/app.yaml @@ -0,0 +1,33 @@ +{{- if eq .Values.name "dex" }} +apiVersion: argoproj.io/v1alpha1 +kind: Application +metadata: + name: {{ .Values.name }} + annotations: + test: "test" + finalizers: + - resources-finalizer.argocd.argoproj.io +spec: + destination: + namespace: kubeflow + server: https://kubernetes.default.svc + project: default + source: + path: {{ .Values.path }} + repoURL: {{ .Values.repoURL }} + targetRevision: {{ .Values.targetRevision }} + kustomize: + patches: + - target: + kind: Secret + name: dex-passwords + patch: |- + - op: replace + path: /stringData/DEX_USER_PASSWORD + value: {{ .Values.password }} +operation: + sync: + syncStrategy: + apply: {} +{{- end }} + diff --git a/charts/argo_app/templates/istio-resources/app.yaml b/charts/argo_app/templates/istio-resources/app.yaml index a46a6e3..9ffcd84 100644 --- a/charts/argo_app/templates/istio-resources/app.yaml +++ b/charts/argo_app/templates/istio-resources/app.yaml @@ -34,8 +34,6 @@ spec: credentialName: gateway-cert mode: SIMPLE {{- end }} - credentialName: gateway-cert - mode: SIMPLE operation: sync: syncStrategy: diff --git a/examples/eks-https-loadbalancer/dex_conf.yaml b/examples/eks-https-loadbalancer/dex_conf.yaml deleted file mode 100644 index eee2058..0000000 --- a/examples/eks-https-loadbalancer/dex_conf.yaml +++ /dev/null @@ -1,34 +0,0 @@ -apiVersion: v1 -kind: ConfigMap -metadata: - name: dex - namespace: auth -data: - config.yaml: |- - issuer: http://dex.auth.svc.cluster.local:5556/dex - storage: - type: kubernetes - config: - inCluster: true - web: - http: 0.0.0.0:5556 - logger: - level: "debug" - format: text - oauth2: - skipApprovalScreen: false - enablePasswordDB: true - staticPasswords: - - email: user@example.com - hash: $2y$12$4K/VkmDd1q1Orb3xAt82zu8gk7Ad6ReFR4LCP9UeYE90NLiN9Df72 - # https://github.com/dexidp/dex/pull/1601/commits - # FIXME: Use hashFromEnv instead - username: user - userID: "15841185641784" - staticClients: - # https://github.com/dexidp/dex/pull/1664 - - idEnv: OIDC_CLIENT_ID - redirectURIs: ["/authservice/oidc/callback"] - name: 'Dex Login Application' - secretEnv: OIDC_CLIENT_SECRET - connectors: [] \ No newline at end of file diff --git a/kf_apps.tf b/kf_apps.tf index 037da8e..bde04cc 100644 --- a/kf_apps.tf +++ b/kf_apps.tf @@ -135,6 +135,7 @@ resource "helm_release" "volumes_web_app" { } resource "helm_release" "profile" { + count = var.enable_example_profile == true ? 1 : 0 name = "profile" namespace = "kubeflow" chart = "${path.module}/charts/profile" diff --git a/kf_core.tf b/kf_core.tf index fbaa3e9..5af2145 100644 --- a/kf_core.tf +++ b/kf_core.tf @@ -19,6 +19,7 @@ resource "helm_release" "dex" { repoURL: https://github.com/kubeflow/manifests path: common/dex/overlays/istio targetRevision: 776d4f4 + password: ${bcrypt(var.user_password)} EOF ] depends_on = [ diff --git a/variables.tf b/variables.tf index 9fff31b..c375162 100644 --- a/variables.tf +++ b/variables.tf @@ -39,3 +39,15 @@ variable "enable_istio_resources" { default = true description = "Enable istio resources for clusters with pre-existing istio" } + +variable "enable_example_profile" { + type = bool + default = true +} + +variable "user_password" { + type = string + description = "The password for the user" + default = "12341234" + sensitive = true +} From 5b2dbb7e5a83f3353f157a1746b71edcec93e974 Mon Sep 17 00:00:00 2001 From: alex-treebeard Date: Thu, 29 Feb 2024 19:51:46 +0000 Subject: [PATCH 45/56] fmt --- variables.tf | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/variables.tf b/variables.tf index c375162..5aedd24 100644 --- a/variables.tf +++ b/variables.tf @@ -48,6 +48,6 @@ variable "enable_example_profile" { variable "user_password" { type = string description = "The password for the user" - default = "12341234" - sensitive = true + default = "12341234" + sensitive = true } From fc58b62779a953eebdd1658b41baba0edbe15236 Mon Sep 17 00:00:00 2001 From: alex-treebeard Date: Thu, 29 Feb 2024 20:00:04 +0000 Subject: [PATCH 46/56] doc --- Makefile | 4 +- README.md | 118 ++++++++-------------- examples/aks/README.md | 7 +- examples/aks/main.tf | 8 -- examples/eks-https-loadbalancer/README.md | 58 +++++++++++ examples/k3s-existing-istio/README.md | 1 - examples/k3s/README.md | 1 - 7 files changed, 105 insertions(+), 92 deletions(-) diff --git a/Makefile b/Makefile index 45b4311..248f23c 100644 --- a/Makefile +++ b/Makefile @@ -15,13 +15,15 @@ docs: terraform-docs markdown examples/k3s > examples/k3s/README.md.new terraform-docs markdown examples/k3s-existing-istio > examples/k3s-existing-istio/README.md.new terraform-docs markdown examples/aks > examples/aks/README.md.new + terraform-docs markdown examples/eks-https-loadbalancer > examples/eks-https-loadbalancer/README.md.new .PHONY: docs-rm-new docs-rm-new: rm -f ./README.md.new rm -f examples/k3s/README.md.new - rm -f examples/eks/README.md.new + rm -f examples/k3s-existing-istio/README.md.new rm -f examples/aks/README.md.new + terraform-docs markdown examples/eks-https-loadbalancer > examples/eks-https-loadbalancer/README.md.new fmt: terraform fmt . examples/* diff --git a/README.md b/README.md index 28f447e..9c14d4d 100644 --- a/README.md +++ b/README.md @@ -112,107 +112,69 @@ This module is built on top of the official [Kubeflow Manifests repo](https://gi | Name | Version | |------|---------| -| [helm](#requirement\_helm) | ~> 2.12.1 | -| [kubernetes](#requirement\_kubernetes) | ~> 2.25.2 | -| [kustomization](#requirement\_kustomization) | ~> 0.9.5 | +| [terraform](#requirement\_terraform) | >= 1.3 | +| [helm](#requirement\_helm) | >= 2.12 | +| [kubernetes](#requirement\_kubernetes) | >= 2.25.2 | +| [null](#requirement\_null) | >= 3.0 | +| [time](#requirement\_time) | >= 0.9 | ## Providers | Name | Version | |------|---------| | [helm](#provider\_helm) | 2.12.1 | -| [kubernetes](#provider\_kubernetes) | 2.25.2 | -| [kustomization](#provider\_kustomization) | 0.9.5 | +| [null](#provider\_null) | >= 3.0 | +| [time](#provider\_time) | >= 0.9 | ## Modules -| Name | Source | Version | -|------|--------|---------| -| [admission\_webhook](#module\_admission\_webhook) | ./modules/kust | n/a | -| [central\_dashboard](#module\_central\_dashboard) | ./modules/kust | n/a | -| [cert\_manager](#module\_cert\_manager) | ./modules/kust | n/a | -| [cluster\_local\_gateway](#module\_cluster\_local\_gateway) | ./modules/kust | n/a | -| [dex](#module\_dex) | ./modules/kust | n/a | -| [istio\_crds](#module\_istio\_crds) | ./modules/kust | n/a | -| [istio\_install](#module\_istio\_install) | ./modules/kust | n/a | -| [istio\_namespace](#module\_istio\_namespace) | ./modules/kust | n/a | -| [jupyter\_web\_app](#module\_jupyter\_web\_app) | ./modules/kust | n/a | -| [knative\_serving](#module\_knative\_serving) | ./modules/kust | n/a | -| [kserve](#module\_kserve) | ./modules/kust | n/a | -| [kubeflow\_issuer](#module\_kubeflow\_issuer) | ./modules/kust | n/a | -| [kubeflow\_istio\_resources](#module\_kubeflow\_istio\_resources) | ./modules/kust | n/a | -| [kubeflow\_namespace](#module\_kubeflow\_namespace) | ./modules/kust | n/a | -| [kubeflow\_pipelines](#module\_kubeflow\_pipelines) | ./modules/kust | n/a | -| [kubeflow\_profile](#module\_kubeflow\_profile) | ./modules/kust | n/a | -| [kubeflow\_ray\_rbac](#module\_kubeflow\_ray\_rbac) | ./modules/kust | n/a | -| [kubeflow\_roles](#module\_kubeflow\_roles) | ./modules/kust | n/a | -| [mlflow\_istio](#module\_mlflow\_istio) | ./modules/kust | n/a | -| [models\_web\_app](#module\_models\_web\_app) | ./modules/kust | n/a | -| [notebook\_controller](#module\_notebook\_controller) | ./modules/kust | n/a | -| [oidc\_authservice](#module\_oidc\_authservice) | ./modules/kust | n/a | -| [profiles\_kfam](#module\_profiles\_kfam) | ./modules/kust | n/a | -| [pvc\_viewer\_controller](#module\_pvc\_viewer\_controller) | ./modules/kust | n/a | -| [tensorboard\_web\_app](#module\_tensorboard\_web\_app) | ./modules/kust | n/a | -| [volumes\_web\_app](#module\_volumes\_web\_app) | ./modules/kust | n/a | +No modules. ## Resources | Name | Type | |------|------| -| [helm_release.external_secrets](https://registry.terraform.io/providers/hashicorp/helm/latest/docs/resources/release) | resource | -| [helm_release.gatekeeper](https://registry.terraform.io/providers/hashicorp/helm/latest/docs/resources/release) | resource | -| [helm_release.gpu_operator](https://registry.terraform.io/providers/hashicorp/helm/latest/docs/resources/release) | resource | -| [helm_release.kuberay_operator](https://registry.terraform.io/providers/hashicorp/helm/latest/docs/resources/release) | resource | -| [helm_release.mlflow](https://registry.terraform.io/providers/hashicorp/helm/latest/docs/resources/release) | resource | -| [kubernetes_namespace.tkf_system](https://registry.terraform.io/providers/hashicorp/kubernetes/latest/docs/resources/namespace) | resource | -| [kustomization_build.admission_webhook](https://registry.terraform.io/providers/kbst/kustomization/latest/docs/data-sources/build) | data source | -| [kustomization_build.central_dashboard](https://registry.terraform.io/providers/kbst/kustomization/latest/docs/data-sources/build) | data source | -| [kustomization_build.cert_manager](https://registry.terraform.io/providers/kbst/kustomization/latest/docs/data-sources/build) | data source | -| [kustomization_build.cluster_local_gateway](https://registry.terraform.io/providers/kbst/kustomization/latest/docs/data-sources/build) | data source | -| [kustomization_build.istio_crds](https://registry.terraform.io/providers/kbst/kustomization/latest/docs/data-sources/build) | data source | -| [kustomization_build.istio_namespace](https://registry.terraform.io/providers/kbst/kustomization/latest/docs/data-sources/build) | data source | -| [kustomization_build.jupyter_web_app](https://registry.terraform.io/providers/kbst/kustomization/latest/docs/data-sources/build) | data source | -| [kustomization_build.kserve](https://registry.terraform.io/providers/kbst/kustomization/latest/docs/data-sources/build) | data source | -| [kustomization_build.kubeflow_issuer](https://registry.terraform.io/providers/kbst/kustomization/latest/docs/data-sources/build) | data source | -| [kustomization_build.kubeflow_namespace](https://registry.terraform.io/providers/kbst/kustomization/latest/docs/data-sources/build) | data source | -| [kustomization_build.kubeflow_pipelines](https://registry.terraform.io/providers/kbst/kustomization/latest/docs/data-sources/build) | data source | -| [kustomization_build.kubeflow_ray_rbac](https://registry.terraform.io/providers/kbst/kustomization/latest/docs/data-sources/build) | data source | -| [kustomization_build.kubeflow_roles](https://registry.terraform.io/providers/kbst/kustomization/latest/docs/data-sources/build) | data source | -| [kustomization_build.mlflow_istio](https://registry.terraform.io/providers/kbst/kustomization/latest/docs/data-sources/build) | data source | -| [kustomization_build.models_web_app](https://registry.terraform.io/providers/kbst/kustomization/latest/docs/data-sources/build) | data source | -| [kustomization_build.notebook_controller](https://registry.terraform.io/providers/kbst/kustomization/latest/docs/data-sources/build) | data source | -| [kustomization_build.profiles_kfam](https://registry.terraform.io/providers/kbst/kustomization/latest/docs/data-sources/build) | data source | -| [kustomization_build.pvc_viewer_controller](https://registry.terraform.io/providers/kbst/kustomization/latest/docs/data-sources/build) | data source | -| [kustomization_build.tensorboard_web_app](https://registry.terraform.io/providers/kbst/kustomization/latest/docs/data-sources/build) | data source | -| [kustomization_build.volumes_web_app](https://registry.terraform.io/providers/kbst/kustomization/latest/docs/data-sources/build) | data source | -| [kustomization_overlay.dex](https://registry.terraform.io/providers/kbst/kustomization/latest/docs/data-sources/overlay) | data source | -| [kustomization_overlay.istio_install](https://registry.terraform.io/providers/kbst/kustomization/latest/docs/data-sources/overlay) | data source | -| [kustomization_overlay.knative_serving](https://registry.terraform.io/providers/kbst/kustomization/latest/docs/data-sources/overlay) | data source | -| [kustomization_overlay.kubeflow_istio_resources](https://registry.terraform.io/providers/kbst/kustomization/latest/docs/data-sources/overlay) | data source | -| [kustomization_overlay.kubeflow_profile](https://registry.terraform.io/providers/kbst/kustomization/latest/docs/data-sources/overlay) | data source | -| [kustomization_overlay.oidc_authservice](https://registry.terraform.io/providers/kbst/kustomization/latest/docs/data-sources/overlay) | data source | +| [helm_release.admission_webhook](https://registry.terraform.io/providers/hashicorp/helm/latest/docs/resources/release) | resource | +| [helm_release.argo_cd](https://registry.terraform.io/providers/hashicorp/helm/latest/docs/resources/release) | resource | +| [helm_release.central_dashboard](https://registry.terraform.io/providers/hashicorp/helm/latest/docs/resources/release) | resource | +| [helm_release.cert_manager](https://registry.terraform.io/providers/hashicorp/helm/latest/docs/resources/release) | resource | +| [helm_release.cluster_issuer](https://registry.terraform.io/providers/hashicorp/helm/latest/docs/resources/release) | resource | +| [helm_release.dex](https://registry.terraform.io/providers/hashicorp/helm/latest/docs/resources/release) | resource | +| [helm_release.istio_base](https://registry.terraform.io/providers/hashicorp/helm/latest/docs/resources/release) | resource | +| [helm_release.istio_ingressgateway](https://registry.terraform.io/providers/hashicorp/helm/latest/docs/resources/release) | resource | +| [helm_release.istiod](https://registry.terraform.io/providers/hashicorp/helm/latest/docs/resources/release) | resource | +| [helm_release.jupyter_web_app](https://registry.terraform.io/providers/hashicorp/helm/latest/docs/resources/release) | resource | +| [helm_release.kubeflow_istio_resources](https://registry.terraform.io/providers/hashicorp/helm/latest/docs/resources/release) | resource | +| [helm_release.kubeflow_namespace](https://registry.terraform.io/providers/hashicorp/helm/latest/docs/resources/release) | resource | +| [helm_release.kubeflow_roles](https://registry.terraform.io/providers/hashicorp/helm/latest/docs/resources/release) | resource | +| [helm_release.notebook_controller](https://registry.terraform.io/providers/hashicorp/helm/latest/docs/resources/release) | resource | +| [helm_release.oidc_authservice](https://registry.terraform.io/providers/hashicorp/helm/latest/docs/resources/release) | resource | +| [helm_release.profile](https://registry.terraform.io/providers/hashicorp/helm/latest/docs/resources/release) | resource | +| [helm_release.profiles_kfam](https://registry.terraform.io/providers/hashicorp/helm/latest/docs/resources/release) | resource | +| [helm_release.pvc_viewer_controller](https://registry.terraform.io/providers/hashicorp/helm/latest/docs/resources/release) | resource | +| [helm_release.volumes_web_app](https://registry.terraform.io/providers/hashicorp/helm/latest/docs/resources/release) | resource | +| [null_resource.kf_apps_end](https://registry.terraform.io/providers/hashicorp/null/latest/docs/resources/resource) | resource | +| [null_resource.kf_apps_start](https://registry.terraform.io/providers/hashicorp/null/latest/docs/resources/resource) | resource | +| [null_resource.kf_core_end](https://registry.terraform.io/providers/hashicorp/null/latest/docs/resources/resource) | resource | +| [null_resource.kf_core_start](https://registry.terraform.io/providers/hashicorp/null/latest/docs/resources/resource) | resource | +| [null_resource.kf_dependencies_end](https://registry.terraform.io/providers/hashicorp/null/latest/docs/resources/resource) | resource | +| [null_resource.kf_dependencies_start](https://registry.terraform.io/providers/hashicorp/null/latest/docs/resources/resource) | resource | +| [time_sleep.wait](https://registry.terraform.io/providers/hashicorp/time/latest/docs/resources/sleep) | resource | ## Inputs | Name | Description | Type | Default | Required | |------|-------------|------|---------|:--------:| -| [completed](#input\_completed) | n/a | `string` | `false` | no | +| [enable\_argocd](#input\_enable\_argocd) | n/a | `bool` | `true` | no | | [enable\_cert\_manager](#input\_enable\_cert\_manager) | n/a | `bool` | `true` | no | -| [enable\_external\_dns](#input\_enable\_external\_dns) | n/a | `bool` | `true` | no | -| [enable\_external\_secrets](#input\_enable\_external\_secrets) | n/a | `bool` | `false` | no | -| [enable\_gatekeeper](#input\_enable\_gatekeeper) | n/a | `bool` | `false` | no | -| [enable\_gpu\_operator](#input\_enable\_gpu\_operator) | n/a | `bool` | `false` | no | +| [enable\_example\_profile](#input\_enable\_example\_profile) | n/a | `bool` | `true` | no | +| [enable\_https](#input\_enable\_https) | n/a | `bool` | `false` | no | | [enable\_istio\_base](#input\_enable\_istio\_base) | n/a | `bool` | `true` | no | | [enable\_istio\_resources](#input\_enable\_istio\_resources) | Enable istio resources for clusters with pre-existing istio | `bool` | `true` | no | | [enable\_istiod](#input\_enable\_istiod) | n/a | `bool` | `true` | no | -| [enable\_kserve](#input\_enable\_kserve) | n/a | `bool` | `false` | no | -| [enable\_kubeflow\_pipelines](#input\_enable\_kubeflow\_pipelines) | n/a | `bool` | `false` | no | -| [enable\_kuberay](#input\_enable\_kuberay) | n/a | `bool` | `false` | no | -| [enable\_mlflow](#input\_enable\_mlflow) | n/a | `bool` | `false` | no | -| [enable\_tensorboard](#input\_enable\_tensorboard) | n/a | `bool` | `false` | no | -| [hostname](#input\_hostname) | n/a | `string` | `"localhost"` | no | -| [port](#input\_port) | n/a | `string` | `"8080"` | no | -| [protocol](#input\_protocol) | n/a | `string` | `"http://"` | no | +| [hostname](#input\_hostname) | n/a | `string` | `"*"` | no | +| [issuer\_name](#input\_issuer\_name) | Required if enable\_https is true | `string` | `"null"` | no | +| [user\_password](#input\_user\_password) | The password for the user | `string` | `"12341234"` | no | ## Outputs diff --git a/examples/aks/README.md b/examples/aks/README.md index 90c0723..63f0d2d 100644 --- a/examples/aks/README.md +++ b/examples/aks/README.md @@ -4,13 +4,12 @@ |------|---------| | [helm](#requirement\_helm) | ~> 2.12.1 | | [kubernetes](#requirement\_kubernetes) | ~> 2.25.2 | -| [kustomization](#requirement\_kustomization) | ~> 0.9.5 | ## Providers | Name | Version | |------|---------| -| [azurerm](#provider\_azurerm) | 3.92.0 | +| [azurerm](#provider\_azurerm) | 3.93.0 | ## Modules @@ -27,7 +26,9 @@ ## Inputs -No inputs. +| Name | Description | Type | Default | Required | +|------|-------------|------|---------|:--------:| +| [enable\_treebeardkf](#input\_enable\_treebeardkf) | Enable Treebeard | `bool` | `false` | no | ## Outputs diff --git a/examples/aks/main.tf b/examples/aks/main.tf index 5a0ac4d..80feac1 100644 --- a/examples/aks/main.tf +++ b/examples/aks/main.tf @@ -1,9 +1,5 @@ terraform { required_providers { - kustomization = { - source = "kbst/kustomization" - version = "~> 0.9.5" - } helm = { source = "hashicorp/helm" version = "~> 2.12.1" @@ -57,10 +53,6 @@ provider "helm" { } } -provider "kustomization" { - kubeconfig_raw = azurerm_kubernetes_cluster.aks.kube_config_raw -} - variable "enable_treebeardkf" { description = "Enable Treebeard" type = bool diff --git a/examples/eks-https-loadbalancer/README.md b/examples/eks-https-loadbalancer/README.md index e69de29..0d98594 100644 --- a/examples/eks-https-loadbalancer/README.md +++ b/examples/eks-https-loadbalancer/README.md @@ -0,0 +1,58 @@ +## Requirements + +| Name | Version | +|------|---------| +| [helm](#requirement\_helm) | ~> 2.12.1 | +| [kubernetes](#requirement\_kubernetes) | ~> 2.25.2 | + +## Providers + +| Name | Version | +|------|---------| +| [aws](#provider\_aws) | 5.38.0 | +| [helm](#provider\_helm) | 2.12.1 | +| [kubernetes](#provider\_kubernetes) | 2.25.2 | +| [null](#provider\_null) | 3.2.2 | + +## Modules + +| Name | Source | Version | +|------|--------|---------| +| [cert\_manager\_role](#module\_cert\_manager\_role) | terraform-aws-modules/iam/aws//modules/iam-role-for-service-accounts-eks | 5.34.0 | +| [ebs\_csi\_role](#module\_ebs\_csi\_role) | terraform-aws-modules/iam/aws//modules/iam-role-for-service-accounts-eks | 5.34.0 | +| [eks](#module\_eks) | terraform-aws-modules/eks/aws | ~> 19.20 | +| [external\_dns\_role](#module\_external\_dns\_role) | terraform-aws-modules/iam/aws//modules/iam-role-for-service-accounts-eks | 5.34.0 | +| [treebeardkf](#module\_treebeardkf) | ../.. | n/a | +| [vpc](#module\_vpc) | terraform-aws-modules/vpc/aws | 5.1.2 | + +## Resources + +| Name | Type | +|------|------| +| [helm_release.cert_manager](https://registry.terraform.io/providers/hashicorp/helm/latest/docs/resources/release) | resource | +| [helm_release.ebs_csi_driver](https://registry.terraform.io/providers/hashicorp/helm/latest/docs/resources/release) | resource | +| [helm_release.external_dns](https://registry.terraform.io/providers/hashicorp/helm/latest/docs/resources/release) | resource | +| [helm_release.issuer](https://registry.terraform.io/providers/hashicorp/helm/latest/docs/resources/release) | resource | +| [helm_release.istio_base](https://registry.terraform.io/providers/hashicorp/helm/latest/docs/resources/release) | resource | +| [helm_release.istio_ingressgateway](https://registry.terraform.io/providers/hashicorp/helm/latest/docs/resources/release) | resource | +| [helm_release.istiod](https://registry.terraform.io/providers/hashicorp/helm/latest/docs/resources/release) | resource | +| [kubernetes_annotations.default-storageclass](https://registry.terraform.io/providers/hashicorp/kubernetes/latest/docs/resources/annotations) | resource | +| [null_resource.cluster_ready](https://registry.terraform.io/providers/hashicorp/null/latest/docs/resources/resource) | resource | +| [null_resource.core_addons](https://registry.terraform.io/providers/hashicorp/null/latest/docs/resources/resource) | resource | +| [null_resource.istio](https://registry.terraform.io/providers/hashicorp/null/latest/docs/resources/resource) | resource | +| [aws_availability_zones.available](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/data-sources/availability_zones) | data source | + +## Inputs + +| Name | Description | Type | Default | Required | +|------|-------------|------|---------|:--------:| +| [aws\_profile](#input\_aws\_profile) | AWS profile to use for authentication. | `any` | n/a | yes | +| [aws\_region](#input\_aws\_region) | AWS region to launch servers. | `any` | n/a | yes | +| [cert\_email\_owner](#input\_cert\_email\_owner) | n/a | `any` | n/a | yes | +| [enable\_treebeardkf](#input\_enable\_treebeardkf) | Enable Treebeard KF | `bool` | `false` | no | +| [host](#input\_host) | n/a | `any` | n/a | yes | +| [hosted\_zone\_id](#input\_hosted\_zone\_id) | n/a | `any` | n/a | yes | + +## Outputs + +No outputs. diff --git a/examples/k3s-existing-istio/README.md b/examples/k3s-existing-istio/README.md index 8090121..5996225 100644 --- a/examples/k3s-existing-istio/README.md +++ b/examples/k3s-existing-istio/README.md @@ -4,7 +4,6 @@ |------|---------| | [helm](#requirement\_helm) | ~> 2.12.1 | | [kubernetes](#requirement\_kubernetes) | ~> 2.25.2 | -| [kustomization](#requirement\_kustomization) | ~> 0.9.5 | ## Providers diff --git a/examples/k3s/README.md b/examples/k3s/README.md index 7b3df83..564f920 100644 --- a/examples/k3s/README.md +++ b/examples/k3s/README.md @@ -108,7 +108,6 @@ k3d cluster delete demo |------|---------| | [helm](#requirement\_helm) | ~> 2.12.1 | | [kubernetes](#requirement\_kubernetes) | ~> 2.25.2 | -| [kustomization](#requirement\_kustomization) | ~> 0.9.5 | ## Providers From 36378bf22c8dfa15e566f39251daafdde58bb2e6 Mon Sep 17 00:00:00 2001 From: alex-treebeard Date: Thu, 29 Feb 2024 20:01:22 +0000 Subject: [PATCH 47/56] rm submodule --- submodules/manifests | 1 - 1 file changed, 1 deletion(-) delete mode 160000 submodules/manifests diff --git a/submodules/manifests b/submodules/manifests deleted file mode 160000 index 776d4f4..0000000 --- a/submodules/manifests +++ /dev/null @@ -1 +0,0 @@ -Subproject commit 776d4f48e8e9d84912a57ac1408a4a7866fd446e From 08ae5a94ecf17f766b31254ac5dba0ff84eaf16e Mon Sep 17 00:00:00 2001 From: alex-treebeard Date: Thu, 29 Feb 2024 20:01:44 +0000 Subject: [PATCH 48/56] rm other --- .gitmodules | 3 --- .terraform.lock.hcl | 62 --------------------------------------------- 2 files changed, 65 deletions(-) delete mode 100644 .gitmodules delete mode 100644 .terraform.lock.hcl diff --git a/.gitmodules b/.gitmodules deleted file mode 100644 index 962835d..0000000 --- a/.gitmodules +++ /dev/null @@ -1,3 +0,0 @@ -[submodule "Manifests"] - path = submodules/manifests - url = https://github.com/kubeflow/manifests \ No newline at end of file diff --git a/.terraform.lock.hcl b/.terraform.lock.hcl deleted file mode 100644 index ec45621..0000000 --- a/.terraform.lock.hcl +++ /dev/null @@ -1,62 +0,0 @@ -# This file is maintained automatically by "terraform init". -# Manual edits may be lost in future updates. - -provider "registry.terraform.io/hashicorp/helm" { - version = "2.12.1" - hashes = [ - "h1:sgYI7lwGqJqPopY3NGmhb1eQ0YbH8PIXaAZAmnJrAvw=", - "zh:1d623fb1662703f2feb7860e3c795d849c77640eecbc5a776784d08807b15004", - "zh:253a5bc62ba2c4314875139e3fbd2feaad5ef6b0fb420302a474ab49e8e51a38", - "zh:282358f4ad4f20d0ccaab670b8645228bfad1c03ac0d0df5889f0aea8aeac01a", - "zh:4fd06af3091a382b3f0d8f0a60880f59640d2b6d9d6a31f9a873c6f1bde1ec50", - "zh:6816976b1830f5629ae279569175e88b497abbbac30ee809948a1f923c67a80d", - "zh:7d82c4150cdbf48cfeec867be94c7b9bd7682474d4df0ebb7e24e148f964844f", - "zh:83f062049eea2513118a4c6054fb06c8600bac96196f25aed2cc21898ec86e93", - "zh:a79eec0cf4c08fca79e44033ec6e470f25ff23c3e2c7f9bc707ed7771c1072c0", - "zh:b2b2d904b2821a6e579910320605bc478bbef063579a23fbfdd6fcb5871b81f8", - "zh:e91177ca06a15487fc570cb81ecef6359aa399459ea2aa7c4f7367ba86f6fcad", - "zh:e976bcb82996fc4968f8382bbcb6673efb1f586bf92074058a232028d97825b1", - "zh:f569b65999264a9416862bca5cd2a6177d94ccb0424f3a4ef424428912b9cb3c", - ] -} - -provider "registry.terraform.io/hashicorp/kubernetes" { - version = "2.25.2" - hashes = [ - "h1:QlTKoO0efmkzgX/9y0DQCEkg7VeidOSQW8epF6B4cEQ=", - "zh:044788ac936e0e8ece8f78a2e4e366ecd435ea8235388eaf2cbc8e7975d9d970", - "zh:24f5ff01df91f51f00ee7ff39430adeb63bb2ca4ea0042e68f06d6b65808c02f", - "zh:49984aa0aa1faa8c4f01e8faa039322f1e6fdaeab0b7e32f5c6e96edfde36a38", - "zh:4eeceaff56bac9fc782e7e33f157fa2c7e9a47b2c3c3d12da2642c312ace73f6", - "zh:4f49b6419345960d5af475e0200c243af4c9c140b0ee64799fe1fc9b023c49ea", - "zh:7958414d516867a2263a978792a24843f80023fb233cf051ff4095adc9803d85", - "zh:c633a755fc95e9ff0cd73656f052947afd85883a0987dde5198113aa48474156", - "zh:cbfe958d119795004ce1e8001449d01c056fa2a062b51d07843d98be216337d7", - "zh:cfb85392e18768578d4c943438897083895719be678227fd90efbe3500702a56", - "zh:d705a661ed5da425dd236a48645bec39fe78a67d2e70e8460b720417cbf260ac", - "zh:ddd7a01263da3793df4f3b5af65f166307eed5acf525e51e058cda59009cc856", - "zh:f569b65999264a9416862bca5cd2a6177d94ccb0424f3a4ef424428912b9cb3c", - ] -} - -provider "registry.terraform.io/kbst/kustomization" { - version = "0.9.5" - constraints = "0.9.5" - hashes = [ - "h1:9/9gxUmvFUs2RHLWfBRzYGKnTRukAiFjyyniqOyl+Gc=", - "zh:068c9d534be0c3e74e89b0701612b44a602705a6919a4160cefe4cbd158f3c63", - "zh:1f961bbef509ef5d9140e0d114b9515ebdbc149d82d2b6406f10cdf5bf4992c5", - "zh:28746a91af26d781fd5a9a16adb2dd9402a6f6c9ac01bba37d73274cb5383d71", - "zh:3cfb8b5d6fcbe536880887606f7421c3da085b13ec970f80e68ba84e12a4e10b", - "zh:5099cee2a50cc8360b2f6afcea379f1b1467ad36f7b2745ef4f408ff9e70ea65", - "zh:5bef3a2ba4b4097a7ced63277f99df31ab0a413d04cfde20e7d8b8c4ac6b2b35", - "zh:732a1005ce23bb1f08c0e0fa28e3d93773d21d340300295190c2162c579d5441", - "zh:9456bd8ef6b81afcc86403b84b515126acf8b272f55598cb7867d2e4b2ac5b36", - "zh:9b81177031139fa1af2c66f7bf2369712d26c02b5273ac80bd66488d6b0e1ebc", - "zh:b7337a5551eb369f66b478c918e0b4c88f1a315e77ce5848f35965661aea86d7", - "zh:bc8cd42e7e35aab7b18200132d73ef33388de13e600103d1b002c0fde4b1aa04", - "zh:bc8f1366a833f11891a3e969c67d919e4822d3a3bfc217acf2e482947c949cd8", - "zh:c295af7193e5b00c2fea37d3201b42dc6466db043199eb972bab71f39a422c3b", - "zh:cdc288d29b9cb3c103252f1880a2d3d7018829fdb8428db2b55b144abd172bea", - ] -} From 0b49034cd7053a77f55c9efc1ff3f6c86441de26 Mon Sep 17 00:00:00 2001 From: alex-treebeard Date: Fri, 1 Mar 2024 09:33:24 +0000 Subject: [PATCH 49/56] devcon --- .devcontainer/{ => codespaces}/Dockerfile | 0 .devcontainer/codespaces/devcontainer.json | 67 ++++++++++++++++++++++ .devcontainer/vm/Dockerfile | 20 +++++++ .devcontainer/{ => vm}/devcontainer.json | 0 4 files changed, 87 insertions(+) rename .devcontainer/{ => codespaces}/Dockerfile (100%) create mode 100644 .devcontainer/codespaces/devcontainer.json create mode 100644 .devcontainer/vm/Dockerfile rename .devcontainer/{ => vm}/devcontainer.json (100%) diff --git a/.devcontainer/Dockerfile b/.devcontainer/codespaces/Dockerfile similarity index 100% rename from .devcontainer/Dockerfile rename to .devcontainer/codespaces/Dockerfile diff --git a/.devcontainer/codespaces/devcontainer.json b/.devcontainer/codespaces/devcontainer.json new file mode 100644 index 0000000..67d9da7 --- /dev/null +++ b/.devcontainer/codespaces/devcontainer.json @@ -0,0 +1,67 @@ +{ + "postCreateCommand": "make post-create", + "postStartCommand": "make post-start", + "remoteUser": "vscode", + "customizations": { + "vscode": { + "settings": { + "terminal.integrated.defaultProfile.linux": "zsh", + "python.pythonPath": "/usr/local/bin/python" + }, + "extensions": [ + "ms-python.python", + "mutantdino.resourcemonitor", + "github.copilot", + "github.copilot-chat" + ] + } + }, + "build": { + "dockerfile": "Dockerfile", + "target": "devcontainer", + "context": "..", + "args": { + "VARIANT": "3.11", + "INSTALL_NODE": "true", + "NODE_VERSION": "18.7" + } + }, + "features": { + // "ghcr.io/devcontainers/features/terraform:1": { + // "version": "1.6.2" + // }, + // "ghcr.io/devcontainers/features/docker-outside-of-docker:1": {}, + // "ghcr.io/devcontainers-contrib/features/kind:1": {}, + // "ghcr.io/devcontainers/features/kubectl-helm-minikube:1": { + // "version": "1.25.6" + // }, + // "ghcr.io/rio/features/k3d:1": {}, + // // "ghcr.io/devcontainers/features/nix:1": {}, + // "ghcr.io/devcontainers-contrib/features/terraform-docs:1": { + // }, + // "ghcr.io/devcontainers/features/github-cli:1": { + // }, + // "ghcr.io/rio/features/k9s:1": { + // }, + // "ghcr.io/devcontainers-contrib/features/istioctl:1": { + // }, + // "ghcr.io/devcontainers-contrib/features/argo-cd:1": { + // }, + }, + "mounts": [ + // "source=${localEnv:HOME}${localEnv:USERPROFILE}/.aws,target=/home/vscode/.aws,type=bind,consistency=cached", + // "source=${localEnv:HOME}${localEnv:USERPROFILE}/.aws,target=/home/vscode/.aws,type=bind,consistency=cached", + // "source=${localEnv:HOME}${localEnv:USERPROFILE}/.ssh,target=/home/vscode/.ssh,type=bind,consistency=cached", + // "source=/var/run/docker.sock,target=/var/run/docker.sock,type=bind,consistency=cached", + // "source=${localEnv:HOME}${localEnv:USERPROFILE}/.zsh_history,target=/home/vscode/.zsh_history,type=bind,consistency=cached", + // "source=${localEnv:HOME}${localEnv:USERPROFILE}/.zshrc,target=/home/vscode/.zshrc,type=bind,consistency=cached", + // "source=${localEnv:HOME}${localEnv:USERPROFILE}/.zshenv,target=/home/vscode/.zshenv,type=bind,consistency=cached", + // "source=${localEnv:HOME}${localEnv:USERPROFILE}/.terraform.d,target=/home/vscode/.terraform.d,type=bind,consistency=cached", + // "source=${localEnv:HOME}${localEnv:USERPROFILE}/.azure,target=/home/vscode/.azure,type=bind,consistency=cached", + // "source=${localEnv:HOME}${localEnv:USERPROFILE}/.config/infracost,target=/home/vscode/.config/infracost,type=bind,consistency=cached", + // "source=${localEnv:HOME}${localEnv:USERPROFILE}/.config/gcloud,target=/home/vscode/.config/gcloud,type=bind,consistency=cached", + // "source=${localEnv:HOME}${localEnv:USERPROFILE}/.config/starship.toml,target=/home/vscode/.config/starship.toml,type=bind,consistency=cached", + // "source=${localEnv:HOME}${localEnv:USERPROFILE}/.kube,target=/home/vscode/.kube,type=bind,consistency=cached", + // "source=${localEnv:HOME}${localEnv:USERPROFILE}/.gitconfig,target=/home/vscode/.gitconfig,type=bind,consistency=cached", + ] +} diff --git a/.devcontainer/vm/Dockerfile b/.devcontainer/vm/Dockerfile new file mode 100644 index 0000000..6124d0a --- /dev/null +++ b/.devcontainer/vm/Dockerfile @@ -0,0 +1,20 @@ +# See here for image contents: https://github.com/microsoft/vscode-dev-containers/tree/v0.148.1/containers/python-3/.devcontainer/base.Dockerfile +ARG VARIANT="3.11" +FROM mcr.microsoft.com/vscode/devcontainers/python:0-${VARIANT} as devcontainer +ARG INSTALL_NODE="true" +ARG NODE_VERSION="lts/*" +RUN if [ "${INSTALL_NODE}" = "true" ]; then su vscode -c "source /usr/local/share/nvm/nvm.sh && nvm install ${NODE_VERSION} 2>&1"; fi +USER vscode +RUN pip3 --disable-pip-version-check --no-cache-dir install -U \ + 'poetry==1.6.1' \ + 'pre-commit==3.6.0' \ + 'pip==23.2.1' +RUN npm install -g @devcontainers/cli@v0.30.0 +ENV DEBIAN_FRONTEND=noninteractive +RUN sudo apt-get update -y && sudo apt-get install -y \ + iputils-ping \ + traceroute \ + kmod \ + vim + ENV PATH="/home/vscode/.local/bin/:$PATH" +ENV EDITOR=vim \ No newline at end of file diff --git a/.devcontainer/devcontainer.json b/.devcontainer/vm/devcontainer.json similarity index 100% rename from .devcontainer/devcontainer.json rename to .devcontainer/vm/devcontainer.json From dd425e9c4290b6cd041063052d579aae691529dc Mon Sep 17 00:00:00 2001 From: alex-treebeard Date: Fri, 1 Mar 2024 09:36:11 +0000 Subject: [PATCH 50/56] dev --- .devcontainer/{codespaces => }/Dockerfile | 0 .devcontainer/{codespaces => }/devcontainer.json | 0 2 files changed, 0 insertions(+), 0 deletions(-) rename .devcontainer/{codespaces => }/Dockerfile (100%) rename .devcontainer/{codespaces => }/devcontainer.json (100%) diff --git a/.devcontainer/codespaces/Dockerfile b/.devcontainer/Dockerfile similarity index 100% rename from .devcontainer/codespaces/Dockerfile rename to .devcontainer/Dockerfile diff --git a/.devcontainer/codespaces/devcontainer.json b/.devcontainer/devcontainer.json similarity index 100% rename from .devcontainer/codespaces/devcontainer.json rename to .devcontainer/devcontainer.json From 82ac69b2769c194dd6232492f30169cf1467de07 Mon Sep 17 00:00:00 2001 From: alex-treebeard Date: Fri, 1 Mar 2024 09:59:20 +0000 Subject: [PATCH 51/56] increase argo job timeout --- .devcontainer/devcontainer.json | 39 +++++++++---------- Makefile | 2 +- README.md | 3 +- .../argo_app/templates/wait-for-sync/job.yaml | 2 +- examples/k3s/main.tf | 8 ---- main.tf | 4 -- 6 files changed, 22 insertions(+), 36 deletions(-) diff --git a/.devcontainer/devcontainer.json b/.devcontainer/devcontainer.json index 67d9da7..252dab4 100644 --- a/.devcontainer/devcontainer.json +++ b/.devcontainer/devcontainer.json @@ -27,26 +27,25 @@ } }, "features": { - // "ghcr.io/devcontainers/features/terraform:1": { - // "version": "1.6.2" - // }, - // "ghcr.io/devcontainers/features/docker-outside-of-docker:1": {}, - // "ghcr.io/devcontainers-contrib/features/kind:1": {}, - // "ghcr.io/devcontainers/features/kubectl-helm-minikube:1": { - // "version": "1.25.6" - // }, - // "ghcr.io/rio/features/k3d:1": {}, - // // "ghcr.io/devcontainers/features/nix:1": {}, - // "ghcr.io/devcontainers-contrib/features/terraform-docs:1": { - // }, - // "ghcr.io/devcontainers/features/github-cli:1": { - // }, - // "ghcr.io/rio/features/k9s:1": { - // }, - // "ghcr.io/devcontainers-contrib/features/istioctl:1": { - // }, - // "ghcr.io/devcontainers-contrib/features/argo-cd:1": { - // }, + "ghcr.io/devcontainers/features/terraform:1": { + "version": "1.6.2" + }, + "ghcr.io/devcontainers/features/docker-outside-of-docker:1": {}, + "ghcr.io/devcontainers-contrib/features/kind:1": {}, + "ghcr.io/devcontainers/features/kubectl-helm-minikube:1": { + "version": "1.25.6" + }, + "ghcr.io/rio/features/k3d:1": {}, + "ghcr.io/devcontainers-contrib/features/terraform-docs:1": { + }, + "ghcr.io/devcontainers/features/github-cli:1": { + }, + "ghcr.io/rio/features/k9s:1": { + }, + "ghcr.io/devcontainers-contrib/features/istioctl:1": { + }, + "ghcr.io/devcontainers-contrib/features/argo-cd:1": { + }, }, "mounts": [ // "source=${localEnv:HOME}${localEnv:USERPROFILE}/.aws,target=/home/vscode/.aws,type=bind,consistency=cached", diff --git a/Makefile b/Makefile index 248f23c..00309ed 100644 --- a/Makefile +++ b/Makefile @@ -23,7 +23,7 @@ docs-rm-new: rm -f examples/k3s/README.md.new rm -f examples/k3s-existing-istio/README.md.new rm -f examples/aks/README.md.new - terraform-docs markdown examples/eks-https-loadbalancer > examples/eks-https-loadbalancer/README.md.new + rm -f examples/eks-https-loadbalancer/README.md.new fmt: terraform fmt . examples/* diff --git a/README.md b/README.md index 9c14d4d..fdde2a2 100644 --- a/README.md +++ b/README.md @@ -114,7 +114,6 @@ This module is built on top of the official [Kubeflow Manifests repo](https://gi |------|---------| | [terraform](#requirement\_terraform) | >= 1.3 | | [helm](#requirement\_helm) | >= 2.12 | -| [kubernetes](#requirement\_kubernetes) | >= 2.25.2 | | [null](#requirement\_null) | >= 3.0 | | [time](#requirement\_time) | >= 0.9 | @@ -122,7 +121,7 @@ This module is built on top of the official [Kubeflow Manifests repo](https://gi | Name | Version | |------|---------| -| [helm](#provider\_helm) | 2.12.1 | +| [helm](#provider\_helm) | >= 2.12 | | [null](#provider\_null) | >= 3.0 | | [time](#provider\_time) | >= 0.9 | diff --git a/charts/argo_app/templates/wait-for-sync/job.yaml b/charts/argo_app/templates/wait-for-sync/job.yaml index 069a367..10ffbb7 100644 --- a/charts/argo_app/templates/wait-for-sync/job.yaml +++ b/charts/argo_app/templates/wait-for-sync/job.yaml @@ -25,5 +25,5 @@ spec: kubectl get -n $NAMESPACE app $APP_NAME -o json | jq '.status.operationState' restartPolicy: Never backoffLimit: 1 - activeDeadlineSeconds: 30 + activeDeadlineSeconds: 45 ttlSecondsAfterFinished: 30 \ No newline at end of file diff --git a/examples/k3s/main.tf b/examples/k3s/main.tf index 1ebfab2..19dcb96 100644 --- a/examples/k3s/main.tf +++ b/examples/k3s/main.tf @@ -5,10 +5,6 @@ terraform { source = "hashicorp/helm" version = "~> 2.12.1" } - kubernetes = { - source = "hashicorp/kubernetes" - version = "~> 2.25.2" - } } backend "local" { } @@ -24,10 +20,6 @@ provider "helm" { } } -provider "kubernetes" { - config_path = var.kubeconfig -} - module "treebeardkf" { source = "../.." } \ No newline at end of file diff --git a/main.tf b/main.tf index d940da3..c2be3b5 100644 --- a/main.tf +++ b/main.tf @@ -6,10 +6,6 @@ terraform { source = "hashicorp/helm" version = ">= 2.12" } - kubernetes = { - source = "hashicorp/kubernetes" - version = ">= 2.25.2" - } null = { source = "hashicorp/null" version = ">= 3.0" From f0ae315d5f7e4be56eb2bc77e7a233d2f0d19774 Mon Sep 17 00:00:00 2001 From: alex-treebeard Date: Fri, 1 Mar 2024 10:04:37 +0000 Subject: [PATCH 52/56] docs --- README.md | 22 ++++++++-------------- 1 file changed, 8 insertions(+), 14 deletions(-) diff --git a/README.md b/README.md index fdde2a2..897dbd1 100644 --- a/README.md +++ b/README.md @@ -40,27 +40,15 @@ may want to make: # resource or data for your k8s cluster ... -# Initialize provisioners -provider "kubernetes" { - ... -} +# Initialize provider provider "helm" { ... } -provider "kustomization" { - ... -} - # Call Kubeflow module module "treebeardkf" { source = "../.." - hostname = "kf.example.com" - protocol = "https://" - port = "" - enable_kuberay = false - enable_mlflow = false } ``` @@ -74,11 +62,13 @@ Some considerations: ### Set a new password +The `user_password` variable allows you to set a non-default password. This is +essential for deploying Kubeflow. ### Make Kubeflow available securely on a network using HTTPS - +Follow the [eks-https-loadbalancer](examples/eks-https-loadbalancer) example to see how you can setup an https loadbalancer for you Kubeflow deployment ### Host your Kubeflow on a domain name using DNS @@ -87,8 +77,12 @@ This is best done by using the external DNS operator. If you are new to external DNS, follow the [docs](https://kubernetes-sigs.github.io/external-dns/v0.14.0/) for setting up a deployment, then use [this guide](https://kubernetes-sigs.github.io/external-dns/v0.14.0/tutorials/istio/) to connect external DNS to the istio gateway *service* for your Kubeflow deployment. +The [eks-https-loadbalancer](examples/eks-https-loadbalancer) example also shows this. + ### Create Profiles for your users +Profiles are a Kubeflow abstraction that lets you securely isolate users from each other. See the [Kubeflow docs on profiles](https://www.kubeflow.org/docs/components/central-dash/profiles/) + ### Teardown 1. Manually remove any manually created Kubeflow resources, e.g. Notebook Servers and Volumes From a54382b37cb624b38f1d4b9e6164c23d82d150e4 Mon Sep 17 00:00:00 2001 From: alex-treebeard Date: Fri, 1 Mar 2024 10:35:00 +0000 Subject: [PATCH 53/56] profile name --- charts/argo_app/templates/wait-for-sync/job.yaml | 2 +- charts/profile/templates/all.yaml | 2 +- kf_dependencies.tf | 6 ++++-- 3 files changed, 6 insertions(+), 4 deletions(-) diff --git a/charts/argo_app/templates/wait-for-sync/job.yaml b/charts/argo_app/templates/wait-for-sync/job.yaml index 10ffbb7..8e3345a 100644 --- a/charts/argo_app/templates/wait-for-sync/job.yaml +++ b/charts/argo_app/templates/wait-for-sync/job.yaml @@ -25,5 +25,5 @@ spec: kubectl get -n $NAMESPACE app $APP_NAME -o json | jq '.status.operationState' restartPolicy: Never backoffLimit: 1 - activeDeadlineSeconds: 45 + activeDeadlineSeconds: 60 ttlSecondsAfterFinished: 30 \ No newline at end of file diff --git a/charts/profile/templates/all.yaml b/charts/profile/templates/all.yaml index a704fcc..33df2b5 100644 --- a/charts/profile/templates/all.yaml +++ b/charts/profile/templates/all.yaml @@ -1,7 +1,7 @@ apiVersion: kubeflow.org/v1 kind: Profile metadata: - name: prod + name: example spec: owner: kind: User diff --git a/kf_dependencies.tf b/kf_dependencies.tf index 2720d54..46f1d26 100644 --- a/kf_dependencies.tf +++ b/kf_dependencies.tf @@ -122,8 +122,10 @@ resource "helm_release" "argo_cd" { null_resource.kf_dependencies_start ] values = [ - < Date: Fri, 1 Mar 2024 11:03:52 +0000 Subject: [PATCH 54/56] doc --- examples/k3s/README.md | 27 ++++++++++++++++----------- 1 file changed, 16 insertions(+), 11 deletions(-) diff --git a/examples/k3s/README.md b/examples/k3s/README.md index 564f920..a595d97 100644 --- a/examples/k3s/README.md +++ b/examples/k3s/README.md @@ -4,8 +4,12 @@ This example will help you try Treebeard Kubeflow in your development environmen ## Pre-reqs: -* 2 cpus and 16G memory in your dev environment (local or via SSH) -* Higher limits for file handlers (this gets reset on system restart) +> [!Note] +> The simplest environment setup may be to fork this repo and open it using github codespaces. + + +* 2 cpus and 8G memory in your dev environment (local or via SSH) +* _If you are running on a laptop/vm with docker_: Higher limits for file handlers (this gets reset on system restart) ```sh # Note that these must be run outside of a container @@ -38,12 +42,14 @@ Some tools can make this process easier: ### 1. Setup Kubernetes -First, let's setup a single node k3d cluster: +Clone this repo (if not in codespaces) and use the makefile to setup kubernetes: ```sh +git clone -b v0.1 https://github.com/treebeardtech/terraform-kubernetes-kubeflow.git +cd terraform-kubernetes-kubeflow export KUBECONFIG=~/.kube/demo.yaml rm $KUBECONFIG # remove if exists from previous attempts -k3d cluster create demo +make k3d-create ``` Once complete, verify the API server has started: @@ -54,16 +60,10 @@ kubectl get nodes ### 2. Install Treebeard Kubeflow -Clone this example into your development environment - -```sh -git clone -b v0.0.3 --recurse-submodules https://github.com/treebeardtech/terraform-kubernetes-kubeflow.git -cd terraform-kubernetes-kubeflow/examples/k3s -``` - Initialise the terraform environment ```sh +cd examples/k3s terraform init ``` @@ -83,6 +83,11 @@ kubectl port-forward --namespace "istio-system" svc/istio-ingressgateway 8080:h then go to http://localhost:8080/ to see the login page +### 4. Uninstall Kubeflow + +```sh +terraform destroy -var kubeconfig=$KUBECONFIG +``` ## Usage From 55af0a0c78613200be2d9c213948a27c1c6c9622 Mon Sep 17 00:00:00 2001 From: alex-treebeard Date: Fri, 1 Mar 2024 11:04:59 +0000 Subject: [PATCH 55/56] doc --- README.md | 2 +- examples/k3s/README.md | 4 ++-- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/README.md b/README.md index 897dbd1..59876fd 100644 --- a/README.md +++ b/README.md @@ -27,7 +27,7 @@ This module is primarily focussed on the notebook environment initially such tha We recommend trying out this module in a development environment first. -To do so, follow the [k3s tutorial](https://github.com/treebeardtech/terraform-kubernetes-kubeflow/tree/main/examples). +To do so, follow the [k3s tutorial](https://github.com/treebeardtech/terraform-helm-kubeflow/tree/main/examples). ## Guides diff --git a/examples/k3s/README.md b/examples/k3s/README.md index a595d97..c092b88 100644 --- a/examples/k3s/README.md +++ b/examples/k3s/README.md @@ -45,8 +45,8 @@ Some tools can make this process easier: Clone this repo (if not in codespaces) and use the makefile to setup kubernetes: ```sh -git clone -b v0.1 https://github.com/treebeardtech/terraform-kubernetes-kubeflow.git -cd terraform-kubernetes-kubeflow +git clone https://github.com/treebeardtech/terraform-helm-kubeflow.git +cd terraform-helm-kubeflow export KUBECONFIG=~/.kube/demo.yaml rm $KUBECONFIG # remove if exists from previous attempts make k3d-create From a97424c2e891f294fe0d1b82c137fe219f5686e0 Mon Sep 17 00:00:00 2001 From: alex-treebeard Date: Fri, 1 Mar 2024 11:11:44 +0000 Subject: [PATCH 56/56] doc --- README.md | 11 +++++------ 1 file changed, 5 insertions(+), 6 deletions(-) diff --git a/README.md b/README.md index 59876fd..b527f74 100644 --- a/README.md +++ b/README.md @@ -9,16 +9,16 @@ Terraform module which creates a Kubeflow instance in a Kubernetes cluster > [!Note] -> This repository is currently suitable for development environments only. Please report any problem you might have by opening a GitHub issue, feature requests welcome. +> This project is currently in beta, we recommend forking/cloning it and using from source. Bug reports and feature requests are welcome. ## About this project -This project simplifies the tasks of managing a Kubeflow instance in Terraform projects. +This project simplifies the task of managing a Kubeflow instance with Terraform. Kubeflow provides a cloud-native AI platform which can be used to deploy applications in scientific computing, traditional machine learning, and generative AI. -This module is primarily focussed on the notebook environment initially such that +This module is primarily focussed on the Jupyter notebook environment initially such that * Developers can deploy and access notebook instances * Notebook instances can use GPUs necessary for deep learning * This can be done across different cloud providers (ranging from individual VMs to managed services like Amazon's EKS) @@ -27,7 +27,7 @@ This module is primarily focussed on the notebook environment initially such tha We recommend trying out this module in a development environment first. -To do so, follow the [k3s tutorial](https://github.com/treebeardtech/terraform-helm-kubeflow/tree/main/examples). +To do so, follow the [k3s tutorial](examples/k3s). ## Guides @@ -57,7 +57,7 @@ module "treebeardkf" { You can incrementally add Kubeflow to your K8s cluster by installing the terraform module. Some considerations: -1. If you are calling this Terraform module from your own module, pass in a string to the `completed` variable in order to manage Kubeflow *after* changes to your other resources. (Note that `depends_on` does not work with this module) +1. If you are calling this Terraform module from your own module, ensure you pass in resources to the `depends_on` field so that Kubeflow installs after they finish setup. 2. If you already have Istio and Cert Manager installed, you will need to ensure Kubeflow works with them. See [examples/k3s-existing-istio](examples/k3s-existing-istio) for a configuration that we have tested like this. ### Set a new password @@ -65,7 +65,6 @@ Some considerations: The `user_password` variable allows you to set a non-default password. This is essential for deploying Kubeflow. - ### Make Kubeflow available securely on a network using HTTPS Follow the [eks-https-loadbalancer](examples/eks-https-loadbalancer) example to see how you can setup an https loadbalancer for you Kubeflow deployment