Data Engineering II

Course description

In this course, students deepen their understanding of how data is modelled, stored, and moved through the pipelines that feed production machine learning systems. The focus is on scalable storage paradigms and well-designed, orchestrated data pipelines.

Students learn to model and store data using relational (SQL), non-relational (NoSQL), and data-lake approaches, design ingestion and preprocessing pipelines, and orchestrate them with workflow tools such as Apache Airflow, while versioning datasets in the cloud.

Course content

Data Modelling and Storage: SQL, NoSQL, and Data Lakes
Data Pipeline Design (Ingestion, Preprocessing, ETL/ELT)
Pipeline Orchestration with Apache Airflow (DAGs, Scheduling, Triggers)
Storing and Versioning Datasets in the Cloud (Azure ML Data Assets)
Secure Access and Environment Management
Data Handling for Real-Time and Batch Scenarios

Prerequisites

This course builds on earlier work in which students developed machine learning models for natural language processing (Y2A1) or computer vision (Y2B1). These models will now be prepared for production deployment.
Completion of all Year 1 courses in the Applied Data Science & AI programme.