Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[bitnami/airflow] Consistency on Airflow configuration & init improvements #30349

Merged
merged 8 commits into from
Nov 11, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 6 additions & 2 deletions bitnami/airflow/CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -1,8 +1,12 @@
# Changelog

## 21.5.1 (2024-11-07)
## 21.6.0 (2024-11-11)

* [bitnami/airflow] Release 21.5.1 ([#30306](https://github.com/bitnami/charts/pull/30306))
* [bitnami/airflow] Consistency on Airflow configuration & init improvements ([#30349](https://github.com/bitnami/charts/pull/30349))

## <small>21.5.1 (2024-11-07)</small>

* [bitnami/airflow] Release 21.5.1 (#30306) ([ac89917](https://github.com/bitnami/charts/commit/ac89917b636f342db76e10496306ce03fe6a19ae)), closes [#30306](https://github.com/bitnami/charts/issues/30306)

## 21.5.0 (2024-11-07)

Expand Down
4 changes: 2 additions & 2 deletions bitnami/airflow/Chart.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ annotations:
licenses: Apache-2.0
images: |
- name: airflow
image: docker.io/bitnami/airflow:2.10.3-debian-12-r1
image: docker.io/bitnami/airflow:2.10.3-debian-12-r3
- name: airflow-exporter
image: docker.io/bitnami/airflow-exporter:0.20220314.0-debian-12-r51
apiVersion: v2
Expand Down Expand Up @@ -39,4 +39,4 @@ maintainers:
name: airflow
sources:
- https://github.com/bitnami/charts/tree/main/bitnami/airflow
version: 21.5.1
version: 21.6.0
203 changes: 102 additions & 101 deletions bitnami/airflow/README.md

Large diffs are not rendered by default.

158 changes: 38 additions & 120 deletions bitnami/airflow/templates/_helpers.tpl
Original file line number Diff line number Diff line change
Expand Up @@ -103,45 +103,34 @@ Get the Redis&reg; port
Get the Redis&reg; credentials secret.
*/}}
{{- define "airflow.redis.secretName" -}}
{{- if and (.Values.redis.enabled) (not .Values.redis.auth.existingSecret) -}}
{{/* Create a include for the redis secret
We truncate at 63 chars because some Kubernetes name fields are limited to this (by the DNS naming spec).
*/}}
{{- $name := default "redis" .Values.redis.nameOverride -}}
{{- printf "%s-%s" .Release.Name $name | trunc 63 | trimSuffix "-" -}}
{{- else if and (.Values.redis.enabled) ( .Values.redis.auth.existingSecret) -}}
{{- printf "%s" .Values.redis.auth.existingSecret -}}
{{- else }}
{{- if .Values.externalRedis.existingSecret -}}
{{- printf "%s" .Values.externalRedis.existingSecret -}}
{{- if .Values.redis.enabled -}}
{{- if .Values.redis.auth.existingSecret -}}
{{- print (tpl .Values.redis.auth.existingSecret .) -}}
{{- else -}}
{{ printf "%s-%s" .Release.Name "externalredis" }}
{{- print (include "airflow.redis.fullname" .) -}}
{{- end -}}
{{- else if .Values.externalRedis.existingSecret -}}
{{- print (tpl .Values.externalRedis.existingSecret .) -}}
{{- else -}}
{{- printf "%s-externalredis" (include "common.names.fullname" .) -}}
{{- end -}}
{{- end -}}

{{/*
Get the Postgresql credentials secret.
*/}}
{{- define "airflow.postgresql.secretName" -}}
{{- if .Values.postgresql.enabled }}
{{- if .Values.global.postgresql }}
{{- if .Values.global.postgresql.auth }}
{{- if .Values.global.postgresql.auth.existingSecret }}
{{- tpl .Values.global.postgresql.auth.existingSecret $ -}}
{{- else -}}
{{- default (include "airflow.postgresql.fullname" .) (tpl .Values.postgresql.auth.existingSecret $) -}}
{{- end -}}
{{- else -}}
{{- if and ( .Values.postgresql.auth.existingSecret ) ( .Values.postgresql.auth.enablePostgresUser ) }}
{{- default (include "airflow.postgresql.fullname" .) (tpl .Values.postgresql.auth.existingSecret $) -}}
{{- end -}}
{{- end -}}
{{- define "airflow.database.secretName" -}}
{{- if .Values.postgresql.enabled -}}
{{- $existingSecret := coalesce (((.Values.global.postgresql).auth).existingSecret) .Values.postgresql.auth.existingSecret -}}
juan131 marked this conversation as resolved.
Show resolved Hide resolved
{{- if $existingSecret -}}
{{- print (tpl $existingSecret .) -}}
{{- else -}}
{{- default (include "airflow.postgresql.fullname" .) (tpl .Values.postgresql.auth.existingSecret $) -}}
{{- print (include "airflow.postgresql.fullname" .) -}}
{{- end -}}
{{- else if .Values.externalDatabase.existingSecret -}}
{{- print (tpl .Values.externalDatabase.existingSecret .) -}}
{{- else -}}
{{- default (printf "%s-externaldb" .Release.Name) (tpl .Values.externalDatabase.existingSecret $) -}}
{{- printf "%s-externaldb" (include "common.names.fullname" .) -}}
{{- end -}}
{{- end -}}

Expand All @@ -163,7 +152,18 @@ Get the configmap name
{{- if .Values.existingConfigmap -}}
{{- print (tpl .Values.existingConfigmap .) -}}
{{- else -}}
{{- printf "%s-configuration" (include "common.names.fullname" .) -}}
{{- print (include "common.names.fullname" .) -}}
{{- end -}}
{{- end -}}

{{/*
Get the configmap name for Airflow Webserver
*/}}
{{- define "airflow.web.configMapName" -}}
{{- if .Values.web.existingConfigmap -}}
{{- print (tpl .Values.web.existingConfigmap .) -}}
{{- else -}}
{{- print (include "airflow.web.fullname" .) -}}
{{- end -}}
{{- end -}}

Expand Down Expand Up @@ -305,105 +305,22 @@ Add environment variables to configure database values
{{- end -}}
{{- end -}}

{{/*
Add environment variables to configure database values
*/}}
{{- define "airflow.configure.database" -}}
- name: AIRFLOW_DATABASE_NAME
value: {{ include "airflow.database.name" . }}
- name: AIRFLOW_DATABASE_USERNAME
value: {{ include "airflow.database.user" . }}
{{- if or (not .Values.postgresql.enabled) .Values.postgresql.auth.enablePostgresUser }}
- name: AIRFLOW_DATABASE_PASSWORD
valueFrom:
secretKeyRef:
name: {{ include "airflow.postgresql.secretName" . }}
key: {{ include "airflow.database.existingsecret.key" . }}
{{- else }}
- name: ALLOW_EMPTY_PASSWORD
value: "true"
{{- end }}
- name: AIRFLOW_DATABASE_HOST
value: {{ include "airflow.database.host" . }}
- name: AIRFLOW_DATABASE_PORT_NUMBER
value: {{ include "airflow.database.port" . }}
{{- end -}}

{{/*
Add environment variables to configure redis values
*/}}
{{- define "airflow.configure.redis" -}}
{{- if (not (or (eq .Values.executor "KubernetesExecutor" ) (eq .Values.executor "LocalKubernetesExecutor" ))) }}
- name: REDIS_HOST
value: {{ include "airflow.redis.host" . | quote }}
- name: REDIS_PORT_NUMBER
value: {{ include "airflow.redis.port" . | quote }}
{{- if and (not .Values.redis.enabled) .Values.externalRedis.username }}
- name: REDIS_USER
value: {{ .Values.externalRedis.username | quote }}
{{- end }}
- name: REDIS_PASSWORD
valueFrom:
secretKeyRef:
name: {{ include "airflow.redis.secretName" . }}
key: redis-password
{{- end }}
{{- end -}}

{{/*
Add environment variables to configure airflow common values
*/}}
{{- define "airflow.configure.airflow.common" -}}
- name: AIRFLOW_FERNET_KEY
- name: BITNAMI_DEBUG
value: {{ ternary "true" "false" (or .Values.image.debug .Values.diagnosticMode.enabled) | quote }}
- name: AIRFLOW__CORE__FERNET_KEY
valueFrom:
secretKeyRef:
name: {{ include "airflow.secretName" . }}
key: airflow-fernet-key
- name: AIRFLOW_SECRET_KEY
- name: AIRFLOW__WEBSERVER__SECRET_KEY
valueFrom:
secretKeyRef:
name: {{ include "airflow.secretName" . }}
key: airflow-secret-key
- name: AIRFLOW_LOAD_EXAMPLES
value: {{ ternary "yes" "no" .Values.loadExamples | quote }}
- name: AIRFLOW_STANDALONE_DAG_PROCESSOR
value: {{ ternary "yes" "no" .Values.dagProcessor.enabled | quote }}
{{- if not (or .Values.configuration .Values.existingConfigmap) }}
- name: AIRFLOW_FORCE_OVERWRITE_CONF_FILE
value: "yes"
{{- end }}
{{- if .Values.image.debug }}
- name: BASH_DEBUG
value: "1"
- name: BITNAMI_DEBUG
value: "true"
{{- end }}
{{- end -}}

{{/*
Add environment variables to configure airflow kubernetes executor
*/}}
{{- define "airflow.configure.airflow.kubernetesExecutor" -}}
{{- if (contains "KubernetesExecutor" .Values.executor) }}
- name: AIRFLOW__KUBERNETES__NAMESPACE
value: {{ .Release.Namespace }}
- name: AIRFLOW__KUBERNETES__WORKER_CONTAINER_REPOSITORY
value: {{ printf "%s/%s" .Values.image.registry .Values.image.repository }}
- name: AIRFLOW__KUBERNETES__WORKER_CONTAINER_TAG
value: {{ .Values.image.tag }}
- name: AIRFLOW__KUBERNETES__IMAGE_PULL_POLICY
value: {{ .Values.image.pullPolicy }}
- name: AIRFLOW__KUBERNETES__DAGS_IN_IMAGE
value: "True"
- name: AIRFLOW__KUBERNETES__DELETE_WORKER_PODS
value: "True"
- name: AIRFLOW__KUBERNETES__DELETE_WORKER_PODS_ON_FAILURE
value: "False"
- name: AIRFLOW__KUBERNETES__WORKER_SERVICE_ACCOUNT_NAME
value: {{ include "airflow.serviceAccountName" . }}
- name: AIRFLOW__KUBERNETES__POD_TEMPLATE_FILE
value: "/opt/bitnami/airflow/pod_template.yaml"
{{- end }}
{{- end -}}

{{/*
Expand All @@ -414,7 +331,7 @@ Note, returns 127.0.0.1 if using ClusterIP.
{{- if eq .Values.service.type "ClusterIP" -}}
127.0.0.1
{{- else -}}
{{- .Values.service.loadBalancerIP | default "" -}}
{{- .Values.service.loadBalancerIP | default "127.0.0.1" -}}
{{- end -}}
{{- end -}}

Expand All @@ -424,8 +341,11 @@ If not using ClusterIP, or if a host or LoadBalancerIP is not defined, the value
*/}}
{{- define "airflow.baseUrl" -}}
{{- $host := default (include "airflow.serviceIP" .) .Values.web.baseUrl -}}
{{- $port := printf ":%v" .Values.service.ports.http -}}
{{- $port := printf ":%d" (int .Values.service.ports.http) -}}
{{- $schema := "http://" -}}
{{- if and .Values.ingress.enabled .Values.ingress.tls -}}
{{- $schema := "https://" -}}
{{- end -}}
{{- if regexMatch "^https?://" .Values.web.baseUrl -}}
{{- $schema = "" -}}
{{- end -}}
Expand All @@ -435,9 +355,7 @@ If not using ClusterIP, or if a host or LoadBalancerIP is not defined, the value
{{- if and .Values.ingress.enabled .Values.ingress.hostname -}}
{{- $host = .Values.ingress.hostname -}}
{{- end -}}
{{- if $host -}}
{{- printf "%s%s%s" $schema $host $port -}}
{{- end }}
{{- end -}}

{{/*
Expand Down
Loading
Loading