Data Pipelines & ETL Orchestration: Apache Airflow & dbt

A data science model cannot run on stale data. Automated Data Pipelines ensure that raw records are extracted, validated, transformed, and loaded into feature stores on a scheduled, reliable cadence. In this class, we master pipeline orchestration with Apache Airflow and dbt.

Anatomy of an Apache Airflow DAG in Python

from datetime import datetime, timedelta
from airflow import DAG
from airflow.operators.python import PythonOperator

# Default task arguments
default_args = {
    'owner': 'veewom_datascience',
    'depends_on_past': False,
    'start_date': datetime(2026, 1, 1),
    'email_on_failure': True,
    'email': ['alerts@veewom.com'],
    'retries': 2,
    'retry_delay': timedelta(minutes=5)
}

# Define DAG (runs every morning at 02:00 AM UTC)
with DAG(
    dag_id='daily_customer_churn_pipeline',
    default_args=default_args,
    schedule_interval='0 2 * * *',
    catchup=False
) as dag:

    def extract_transactions():
        print("Extracting raw transactions from production Postgres...")

    def run_feature_engineering():
        print("Transforming features & calculating rolling 30d spend...")

    def retrain_and_score():
        print("Running batch churn inference and saving to warehouse...")

    # Define tasks using PythonOperator
    task_extract = PythonOperator(task_id='extract_raw_data', python_callable=extract_transactions)
    task_transform = PythonOperator(task_id='engineer_features', python_callable=run_feature_engineering)
    task_score = PythonOperator(task_id='score_churn_predictions', python_callable=retrain_and_score)

    # Set dependency DAG execution order
    task_extract >> task_transform >> task_score