Stackable Operator for Apache Airflow

Purpose#

Custom resource#

The operator defines the AirflowCluster resource with these roles: webserver, worker, scheduler, dagProcessor and triggerer. The worker role appears within spec.celeryExecutors, while dagProcessor and triggerer are optional.

Executor types#

• celeryExecutors — routes jobs through Redis to worker pods, with configurable replicas and role group settings.
• kubernetesExecutors — creates pods dynamically without Redis, managed implicitly by a single role.

Kubernetes resources created#

For every role group you define, the operator creates a StatefulSet with the number of replicas set on that role group. Every pod in the StatefulSet runs two containers: the main Airflow container and a sidecar that gathers metrics.

External dependencies#

Airflow requires an SQL database to store its metadata, plus Redis for job execution — Redis is only needed when using celeryExecutors.

Supported Airflow versions#

See also#

• Core Concepts for what "role" and "role group" mean across every operator.
• Getting Started for the stackablectl install flow (used for Kafka/ZooKeeper/NiFi in that walkthrough, same pattern applies to installing the Airflow operator).