Computational Thinking + Doing

ETL Pipeline Orchestration With Apache Airflow

Managing DAG workflows—using Airflow on AWS, Azure, and GCP.

import sys
print(sys.version)
## 3.9.6 (v3.9.6:db3ff76da1, Jun 28 2021, 11:49:53) 
## [Clang 6.0 (clang-600.0.57)]
!pip install apache-airflow==2.2.3

Airflow Scheduler

Airflow Workers

Airflow Webserver

Airflow in Production

Airflow on AWS

Amazon Managed Workflows for Apache Airflow (MWAA)

Airflow on Azure

Airflow on GCP

Applied Computing