Modern Data Engineering with Apache Spark & Airflow

Course Overview

Learn how to design, build, and maintain enterprise data pipelines that handle large volumes of incoming data reliably. This course covers batch and stream data processing using PySpark, dynamic workflow scheduling using Apache Airflow, and lakehouse storage practices.

What You Will Learn

• Build automated data ingestion pipelines handling both structured and unstructured inputs.
• Process distributed data volumes efficiently using Apache Spark DataFrames and SQL.
• Schedule and monitor complex pipeline task dependencies using Apache Airflow DAGs.
• Implement data quality validation checks to catch corrupt incoming pipeline records quickly.

Syllabus Breakdown

• Module 1: Distributed Data Processing: PySpark environment setup, transformations, and memory tuning rules.
• Module 2: Pipeline Scheduling with Airflow: Authoring DAGs, configuring operators, setup alerts, and retries.
• Module 3: Data Warehousing & Lakehouses: Schema design, partitioned tables, and storage format selection.
• Module 4: Data Quality & Operations: Automated schema validations, pipeline monitoring, and fault management routines.

Who This Is For

Data analysts, software engineers, and database administrators transitioning into dedicated data engineering roles.