repository·main·Indexed 20 days ago
https://github.com/airflow-helm/chartsUser-Community Helm charts for deploying Apache Airflow on Kubernetes. Supports Airflow versions 1.10 through 2.10 and executors including CeleryExecutor, KubernetesExecutor, and CeleryKubernetesExecutor. Features include declarative management of users and connections, DAG loading via Git-Sync or PVCs, and production-ready deployment examples for Google Kubernetes Engine (GKE) and local environments like Minikube, Kind, or K3D.
The User-Community Airflow Helm Chart is a standard tool for deploying Apache Airflow on Kubernetes using Helm. It is designed to be production-ready and supports various Airflow versions, executors, and GitOps workflows (such as ArgoCD).
Note: This chart is independent of the official chart found in the apache/airflow repository.
The Apache Airflow helm chart located at charts/airflow utilizes this specific PgBouncer Docker image to provide PgBouncer support. This allows for efficient connection pooling for the Airflow metadata database.
To use this image in your own custom deployments or to inspect its construction, you can refer to the official Dockerfile and changelog.
The Helm chart provides detailed documentation for common configuration tasks. Key categories include:
To avoid storing sensitive data in plain-text, use airflow.connectionsTemplates to inject values from Kubernetes Secrets or ConfigMaps into your connection fields (host, schema, login, password, and extra).
Keys defined in airflow.connectionsTemplates are used as $-based templates (Python string templates) within the connection fields.
airflow:
connections:
- id: my_aws
type: aws
login: ${ACCESS_KEY_ID}
password: ${SECRET_ACCESS_KEY}
connectionsTemplates:
ACCESS_KEY_ID:
kind: secret
name: my-aws-token
key: AWS_ACCESS_KEY_ID
SECRET_ACCESS_KEY:
kind: secret
name: my-aws-token
key: AWS_SECRET_ACCESS_KEYThe chart provides comprehensive support for managing an Airflow deployment on Kubernetes, including:
1.10, 2.0, 2.1, 2.2, 2.3, 2.4, 2.5, 2.6, 2.7, 2.8, 2.9, and 2.10.CeleryExecutor, KubernetesExecutor, and CeleryKubernetesExecutor.Postgres or MySQL databases, and support for PgBouncer configuration.Users, Connections, Variables, and Pools declaratively.Git-Sync sidecars, Persistent Volume Claims (Volumes), or by embedding them directly into the container image.Persistent Volume Claims or remote providers, with support for automatic log cleanup.Init-Containers or by embedding them into the container image.Heartbeat Check or Task Creation Check liveness probes.If you have an existing private key file (e.g., at $HOME/.ssh/id_rsa), you can create a Kubernetes Secret from it using the --from-file flag.
kubectl create secret generic \
my-ssh-keyfile \
--from-file=id_rsa=$HOME/.ssh/id_rsa \
--namespace my-airflow-namespaceYou can use Airflow's _CMD or _SECRET environment variable patterns to set the secret key. This involves mounting a secret as a volume and then using a command to read that value.
Important Requirements:
airflow.webserverSecretKey to an empty string (""), otherwise the direct value will take precedence and ignore your command/secret config.extraEnv (Option 2) is generally preferred over this method.This approach uses AIRFLOW__WEBSERVER__SECRET_KEY_CMD to execute a command (like cat) to retrieve the key from a mounted volume.
airflow:
## WARNING: you must set `webserverSecretKey` to "", otherwise it will take precedence
webserverSecretKey: ""
config:
AIRFLOW__WEBSERVER__SECRET_KEY_CMD: "cat /opt/airflow/webserver-secret-key/value"
extraVolumeMounts:
- name: webserver-secret-key
mountPath: /opt/airflow/webserver-secret-key
readOnly: true
extraVolumes:
- name: webserver-secret-key
secret:
secretName: airflow-webserver-secret-keyDepending on your security requirements and environment, you can configure AWS connections in three ways:
login (Access Key) and password (Secret Key) in the values. (Not recommended for production).airflow.connectionsTemplates.login and password and instead annotate the Airflow serviceAccount with the required IAM role ARN. This allows Airflow to use the pod's identity to access AWS services.Once the installation is complete and resources are ready, you can access the Airflow webserver UI by port-forwarding the service to your local machine.
http://localhost:8080admin / admin## port-forward the airflow webserver
kubectl port-forward svc/${AIRFLOW_NAME}-web 8080:8080 --namespace $AIRFLOW_NAMESPACETo mount Kubernetes Secrets or ConfigMaps as files on every Airflow pod (Webserver, Scheduler, etc.), use the airflow.extraVolumeMounts and airflow.extraVolumes configuration keys.
Follow the Kubernetes VolumeMount and Volume specifications. You must define a name in both extraVolumes and extraVolumeMounts to link them together.
airflow:
extraVolumeMounts:
- name: redshift-creds
mountPath: /opt/airflow/secrets/redshift-creds
readOnly: true
extraVolumes:
- name: redshift-creds
secret:
secretName: redshift-credsIf you want to manage the PVC yourself, create the PersistentVolumeClaim in your namespace and tell the chart to use it via the existingClaim key.
Requirements:
ReadWriteMany access mode.logCleanup for both the scheduler and workers.Configuration:
Set logs.persistence.enabled: true and provide the name of your claim in logs.persistence.existingClaim.
scheduler:
logCleanup:
enabled: false
workers:
logCleanup:
enabled: false
logs:
path: /opt/airflow/logs
persistence:
enabled: true
existingClaim: my-logs-pvc
accessMode: ReadWriteMany