Curriculum

01

Data Engineering Fundamentals

Foundations8 sections · 20 topics

01Part 1: Introduction to Data Engineering2

1.1 What is Data Engineering?
- Definition and scope.
1.2 Role of Data Engineer in Data Science and Analytics
- Collaboration with data scientists and analysts.

02Part 2: Data Systems and Storage Solutions4

2.1 Overview of RDBMS
- Key characteristics and use cases.
2.2 Overview of Data Warehouse and Data Lake
- Definitions, differences, and practical applications.
2.3 What is a Data Warehouse?
- Deep dive into Data Warehouse architecture and benefits.
2.4 Why Data Warehouse?
- Strategic importance in data analytics.

03Part 3: Data Modeling Fundamentals3

3.1 Data Modeling Overview
- Purpose and principles of data modeling.
3.2 Data Modeling Techniques
- Various techniques used in the field.
3.3 Data Modeling Types
- Conceptual, logical, and physical models.

04Part 4: Advanced Data Modeling Concepts4

4.1 Data Modeling – Facts
- Understanding fact tables and their role in data warehousing.
4.2 Data Modeling – SCD (Slowly Changing Dimensions)
- Types of SCDs and handling dimension changes over time.
4.3 Data Modeling – Different Keys
- Exploring primary keys, foreign keys, and surrogate keys.
4.4 Data Modeling – OLTP vs OLAP
- Contrasting transactional and analytical processing systems.

05Part 5: Practical Data Engineering2

5.1 Stages of Data Engineering
- From data collection to data governance.
5.2 Data Modeling – Real World Example
- Applying data modeling concepts to a practical scenario.

06Real World E2E Projects1

Towards the end of the module, students will undertake a capstone project that encompasses the key learnings from the course. This project could involve designing a data warehouse schema based on given requirements, including the use of SCDs, and demonstrating the ETL process on a dataset to populate the warehouse.

07Supplementary Materials1

Additional readings, case studies, and resources will be provided to deepen students’ understanding of each topic.

08Evaluation3

Mock Interview at the end of each week to test comprehension.
Peer-reviewed assignments for practical sections.
E2E project presentation and report submission.

02

Data Modelling

Foundations9 sections · 27 topics

01Part 1: Basic Concepts of Data Modelling3

Introduction to data modeling
Conceptual, logical, and physical models
Best practices in data modeling

02Part 2: Business Data Requirements – Entities and Classes3

Identifying business data requirements
Entities and classes in data modeling
Entity-relationship modeling (ER modeling)

03Part 3: Business Data Requirements – Attributes3

Types of attributes in data modeling
Attribute domains and data types
Constraints and naming conventions for attributes

04Part 4: How To Link Things Together – Relationships3

Types of relationships in data modeling
Cardinality and optionality in relationships
Role names and associative entities

05Part 5: Requirements Analysis3

Gathering and analyzing data requirements
Functional and non-functional requirements
Documentation of data requirements

06Part 6: Conceptual Data Modeling3

Creating a conceptual data model
Entity-relationship diagrams (ERDs)
Representing business processes and data flows

07Part 7: Logical Data Modeling3

Transforming conceptual model to logical model
Tables, columns, normalization
Primary and foreign keys in data modeling

08Part 8: Physical Data Modelling3

Converting logical model to physical model
Database schema, tables, denormalization
Indexing and partitioning strategies

09Part 9: Data Modelling Tools and Techniques3

Overview of data modeling tools (e.g., ERwin, PowerDesigner)
Reverse engineering and forward engineering
Techniques for data model manipulation

03

SQL Tutorial

Core Programming5 sections · 22 topics

01Part 1: SQL Beginner Lessons6

1.1 Introduction to SQL
- History and importance of SQL in data management.
- Overview of SQL syntax and structure.
1.2 Basic Data Retrieval
- Using SELECT statements to query data.
- Understanding FROM, WHERE, and ORDER BY clauses.
1.3 Working with Functions
- Introduction to common SQL functions for data manipulation (e.g., COUNT, SUM, AVG, MIN, MAX).
1.4 Data Filtering and Sorting
- Advanced use of WHERE clause.
- Sorting results using ORDER BY.
1.5 Basic Data Manipulation
- Inserting data with INSERT.
- Updating data with UPDATE.
- Deleting data with DELETE.
Practice Questions: A set of 200+ exercises focused on querying and manipulating data in a simple database scenario.

02Part 2: SQL Intermediate Lessons6

2.1 Joins and Subqueries
- Understanding different types of joins (INNER JOIN, LEFT JOIN, RIGHT JOIN, FULL JOIN).
- Utilizing subqueries for complex data retrieval.
2.2 Grouping Data
- Using GROUP BY to aggregate data.
- Filtering aggregated data with HAVING.
2.3 Set Operations
- Combining results using UNION, INTERSECT, and EXCEPT.
2.4 Working with Indexes
- Introduction to indexes for performance optimization.
2.5 Data Definition Language (DDL)
- Creating tables with CREATE TABLE.
- Altering tables with ALTER TABLE.
- Dropping tables with DROP TABLE.
Practice Questions: Intermediate-level 200+ exercises that challenge learners to create more complex queries and understand the performance implications of various operations.

03Part 3: SQL Advanced Lessons6

3.1 Advanced Query Techniques
- Window functions and their applications.
- Common Table Expressions (CTEs) and recursive queries.
3.2 Database Design and Normalization
- Basic concepts of database design and normalization to reduce redundancy and improve data integrity.
3.3 Transaction Control and Concurrency
- Understanding transactions, COMMIT, and ROLLBACK.
- Basics of concurrency control and locking mechanisms.
3.4 Performance Tuning and Optimization
- Techniques for optimizing SQL queries for better performance.
3.5 Security and Permissions
- Managing user access and roles with GRANT and REVOKE.
Practice Questions: 200+ Advanced exercises that include optimizing query performance, implementing security measures, and designing efficient database schemas.

04Part 4: Practice Questions1

Comprehensive Practice Set
- A mixed set of 500+ practice questions covering beginner, intermediate, and advanced topics to solidify learners’ understanding and prepare them for real-world SQL tasks.

05Supplementary Materials3

Additional Resources
- Recommended readings, online resources, and tools for practicing SQL beyond the classroom.
SQL Best Practices
- Guidelines for writing clean, efficient, and maintainable SQL code.
Mock Interviews
- Unlimited mock interviews with industry experts.

04

Python Tutorial

Core Programming8 sections · 24 topics

01Part 1: Python Basics2

Introduction to Python programming language
Variables, data types, and operators

02Part 2: OOPs Concept2

Object-Oriented Programming (OOP)
Classes, objects, inheritance, encapsulation, abstraction, polymorphism

03Part 3: NumPy4

NumPy library
Arrays and array operations
Indexing, slicing, and reshaping arrays
Mathematical functions in NumPy

04Part 4: Pandas3

Pandas library for data analysis
Series and DataFrame data structures
Data cleaning and preprocessing with Pandas

05Part 5: Data Visualization4

Introduction to data visualization
Matplotlib library for basic plotting
Seaborn library for statistical data visualization
Plotly library for interactive visualizations

06Part 6: File Handling3

Reading and writing files in Python
Text file manipulation and processing
File manipulation and file formats

07Part 7: Exception Handling3

Introduction to exception handling
Errors, exceptions, try-except blocks
Handling specific exceptions

08Part 8: Regular Expressions Fundamentals3

Introduction to regular expressions
Pattern matching and searching in text
Matching and replacing patterns

05

Explore Cloud Technologies (AWS)

Cloud Platforms15 sections · 58 topics

01AWS Basics4

Introduction to AWS (Amazon Web Services)
Overview of cloud computing and its benefits
Understanding AWS services and solutions
Basics of AWS account setup and management

02Amazon S3 (Simple Storage Service)4

Introduction to Amazon S3
S3 bucket creation and management
Uploading, downloading, and managing objects in S3
Integrating S3 with other AWS services

03Amazon Lambda4

Introduction to Amazon Lambda
Explain serverless computing
Creating Lambda Function
Collecting, processing, and analyzing data with lambda

04Amazon Kinesis4

Introduction to Amazon Kinesis
Real-time streaming data processing
Creating Kinesis data streams
Collecting, processing, and analyzing streaming data

05Amazon Firehose4

Introduction to Amazon Firehose
Creating firehose
Data processing with Firehose
Firehose integration with AWS services

06Amazon MSK (Managed Streaming for Apache Kafka)4

Introduction to Amazon MSK
Apache Kafka basics
Setting up and managing MSK clusters
Streaming data ingestion and processing with MSK

07AWS Glue4

Introduction to AWS Glue
Data cataloging and metadata management
Extract, Transform, Load (ETL) with Glue
Building and managing ETL pipelines using Glue

08DynamoDB4

Introduction to Amazon DynamoDB
NoSQL database fundamentals
Creating and managing DynamoDB tables
Querying and scanning data in DynamoDB

09AWS Redshift4

Introduction to AWS Redshift
Columnar data warehousing with Redshift
Provisioning and managing Redshift clusters
Loading, querying, and optimizing data in Redshift

10Amazon Athena4

Introduction to Amazon Athena
Serverless querying and analysis of data
Creating tables and querying data with Athena
Optimizing performance and cost in Athena

11Amazon Secrets Manager4

Introduction to AWS Secrets Manager
Create and manage secrets
Retrieve and rotate secrets
Security in Secrets Manager

12Amazon CloudWatch4

Introduction to Amazon Cloudwatch
Setting up new Cloudwatch
Check logs in Cloudwatch
Setup alerts in Cloudwatch

13Amazon SNS3

Introduction to Amazon SNS
Setting up new SNS Topic
Subscription to SNS Topics

14Amazon SQS3

Introduction to Amazon SQS
Setting up new SQS Queue
Introduction to Dead Letter Queue (DLQ)

15Amazon QuickSight4

Introduction to Amazon QuickSight
Business intelligence and data visualization
Creating visualizations and dashboards in QuickSight
Sharing and presenting insights from QuickSight

06

Explore Cloud Technologies (Azure)

Cloud Platforms5 sections · 33 topics

01Introduction to Microsoft Azure5

Introduction to Microsoft Azure
Introduction to ARM & Azure Storage
Azure Virtual Machines
Azure Networking – I
Azure Networking – II

02Authentication, Authorization, and Monitoring3

Authentication and Authorization in Azure using RBAC
Microsoft Azure Active Directory
Azure Monitoring

03Data Storage and Integration10

Data Storage in Microsoft Azure
Non-Relational Data Stores and Azure Data Lake Storage
Data Lake and Azure Cosmos DB
Relational Data Stores
Why Azure SQL?
Azure Data Lake Storage Gen2 and Data Streaming Solution
Data Integration with Microsoft Azure Data Factory
Designing Data Flows in Azure
Using Azure Data Factory Pipelines to Copy Data
Monitor Azure Data Factory using Azure Monitor

04Azure Synapse Analytics and Databricks11

Introduction to Microsoft Azure Synapse Analytics
Using Azure Synapse Analytics to Query Data Lake
Optimizing Dedicated SQL Pools in Azure Synapse Analytics
Data Warehousing with Microsoft Azure Synapse Analytics
Data Engineering with MS Azure Synapse Apache Spark Pools
Operational Analytics with Microsoft Azure Synapse Analytics
Handling Slowly Changing Dimensions With Azure Synapse Analytics Pipelines
Microsoft Azure Databricks for Data Engineering
Running Spark on Azure Databricks
Using Azure Databricks to Import and Analyze Data
Introduction to Delta Lake on Azure Databricks

05Azure Stream Analytics, and Azure Service Bus4

Introduction to Azure Stream Analytics
Monitoring & Security
Azure Functions
Azure Service Bus

07

Explore Cloud Technologies (Snowflake)

Cloud Platforms8 sections · 16 topics

01Introduction to Snowflake1

Overview of Snowflake Cloud Data Platform
- Introduction to Snowflake and its architecture.
- Key features: Multi-cluster shared data architecture, separation of storage and compute.
- Snowflake’s position in the cloud data ecosystem.

02Snowflake Data Warehousing2

Setting Up Snowflake
- Creating a Snowflake account.
- Overview of Snowflake editions (Standard, Enterprise, Business Critical).
- Navigating the Snowflake user interface.
Virtual Warehouses
- Understanding virtual warehouses and scaling.
- Best practices for performance and cost management.

03Data Loading and Unloading2

Loading Data into Snowflake
- Introduction to Snowflake’s loading methods: Bulk loading with COPY, loading semi-structured data.
- Working with external stages (AWS S3, Azure Blob, Google Cloud Storage).
Unloading Data
- Best practices for unloading data from Snowflake.
- Using the COPY command to export data.

04Snowflake SQL and Querying2

Querying Data
- Snowflake SQL basics: SELECT statements, filtering, and sorting.
- Working with semi-structured data (JSON, Parquet) using Snowflake.
Advanced Query Techniques
- Time travel and data retention in Snowflake.
- Cloning and zero-copy cloning for efficient data management.

05Security and Data Sharing2

Snowflake Security Features
- Role-based access control (RBAC) and user management.
- Data encryption and compliance features.
- Managing security with Virtual Private Snowflake (VPS).
Data Sharing
- Introduction to Snowflake’s secure data sharing capabilities.
- Setting up and managing data shares with partners and customers.

06Performance Tuning and Optimization2

Best Practices for Query Performance
- Understanding query profiling in Snowflake.
- Optimizing performance with clustering keys.
Resource Monitoring and Management
- Monitoring virtual warehouse performance and utilization.
- Auto-suspend and auto-resume features for cost efficiency.

07Snowflake Ecosystem and Integrations3

Integrations with ETL Tools
- Connecting Snowflake with ETL tools like dbt, Apache Airflow.
Snowflake and BI Tools
- Integrating Snowflake with BI tools like Tableau, Power BI, and Looker.
Snowflake Marketplace
- Exploring the Snowflake Data Marketplace for third-party data.

08Real-World Applications and Case Studies2

Case Study: Building a Data Warehouse in Snowflake
- Designing and implementing a data warehouse solution.
- Data ingestion, transformation, and reporting with Snowflake.
Real World Project: End-to-End Data Pipeline
- Developing a full data pipeline using Snowflake from data ingestion to analytics.

08

Spark

Big Data & Streaming4 sections · 16 topics

01Introduction to Apache Spark4

Overview of big data processing and Apache Spark
Spark architecture and components
Introduction to Resilient Distributed Datasets (RDDs)
Understanding Spark’s distributed computing model

02Spark SQL and Data Frames4

Introduction to Spark SQL module
Working with structured and semi-structured data
Data exploration and analysis using DataFrames
Querying and manipulating data with SQL-like syntax

03Apache Kafka and Flume4

Introduction to Apache Kafka and Apache Flume
Streaming data ingestion using Kafka and Flume
Integration of Kafka and Flume with Spark
Real-time data processing and analysis

04Spark Streaming4

Introduction to Spark Streaming
Processing live data streams with Spark
Windowed operations and aggregations
Real-time analytics using Spark Streaming

09

DevOps

Tooling & BI5 sections · 19 topics

01Introduction to DevOps3

Understanding the DevOps culture and principles
Benefits of DevOps in data engineering
Overview of DevOps tools and practices

02Git4

Introduction to version control systems
Git fundamentals: repositories, branches, commits
Collaborative development with Git
Git workflows: branching strategies, pull requests, merging

03Docker4

Introduction to containerization and Docker
Docker architecture and components
Building Docker images for data engineering applications
Container orchestration with Docker Compose

04Kubernetes4

Introduction to Kubernetes for container orchestration
Kubernetes architecture and components
Deploying and managing applications with Kubernetes
Scaling, monitoring, and updating applications in Kubernetes

05Jenkins4

Introduction to Jenkins for continuous integration and continuous delivery
Jenkins installation and configuration
Building and automating data engineering pipelines with Jenkins
Integration with Git, Docker, and Kubernetes

10

Power BI

Tooling & BI5 sections · 15 topics

01Introduction to Power BI3

Overview of Power BI and its role in data engineering
Introduction to self-service business intelligence
Understanding Power BI components: Power BI Desktop, Power BI Service, Power BI Mobile

02Data Extraction3

Connecting to various data sources in Power BI
Importing data from databases, files, web services, and other sources
Configuring data refresh options and scheduling data updates

03Data Transformation – Shaping & Combining Data3

Understanding data transformation concepts in Power BI
Applying data shaping techniques: filtering, sorting, and removing duplicates
Combining multiple data sources using merging and appending operations

04Data Modeling & DAX (Data Analysis Expressions)3

Introduction to data modeling in Power BI
Creating relationships between tables
Implementing calculations and measures using DAX formulas

05Data Visualization with Analytics3

Creating interactive visualizations using Power BI visuals
Formatting and customizing visual elements
Applying data analytics techniques: forecasting, clustering, and trend analysis

11

dbt (Data Build Tool)

Tooling & BI10 sections · 45 topics

01dbt Cloud Overview5

Overview of dbt
dbt, data platforms, and version control
Setting up dbt Cloud and your data platform
dbt Cloud IDE Overview
Overview of dbt Cloud UI

02Models7

What are models?
Building your first model
What is modularity?
Modularity and the ref functions
Quick history of data modeling
Naming conventions
Reorganize your project

03Sources3

What are sources?
Configure and select from sources
Source freshness

04Tests6

Why testing?
What is testing?
Generic tests
Singular tests
Testing sources
The dbt Build command

05Jinjas3

What is Jinja?
Jinja Basics
Jinja Applications

06Macros4

What are macros?
cents_to_dollars macro
limit_data_in_dev macro
DRY code vs. readability

07Packages4

What are packages?
Installing packages
Packages with macros
Packages with models

08Materializations5

What are materializations?
Tables, views, and ephemeral models
Incremental models
What are snapshots?
Implementing snapshots

09Documentation5

Why is documentation important?
What is documentation?
Writing documentation and doc blocks
Documenting sources
Generate and view documentation

10Deployment3

What is deployment?
Setting up a dbt Cloud job
Reviewing a dbt Cloud job

12

PySpark

Core Programming10 sections · 21 topics

01Part 1: Introduction to PySpark2

1.1 Introduction to PySpark
- Overview of PySpark and its role in big data processing.
- Installation and setup of PySpark in different environments.
- Understanding Spark architecture and cluster computing.
1.2 PySpark Core Components
- SparkContext, SparkSession, and their roles.
- Introduction to RDD (Resilient Distributed Datasets) and DataFrames.
- Key PySpark libraries and functions.

02Part 2: Data Processing with RDDs2

2.1 Introduction to RDDs
- Characteristics and structure of RDDs.
- Creating and manipulating RDDs.
- Transformations and actions in RDDs.
2.2 Advanced RDD Operations
- Aggregating data with reduce, aggregate, and fold.
- Working with key-value RDDs (pair RDDs).
- Using mapPartitions, flatMap, and coalesce for data partitioning.

03Part 3: DataFrames and SQL in PySpark2

3.1 Working with DataFrames
- Creating DataFrames from various data sources (CSV, JSON, Parquet).
- Data exploration and filtering with DataFrame API.
- Common DataFrame transformations (filter, select, groupBy).
3.2 SQL Queries on DataFrames
- Registering DataFrames as temporary tables.
- Writing SQL queries in PySpark.
- Combining SQL queries with DataFrame operations.

04Part 4: Advanced DataFrame Operations2

4.1 Data Cleaning and Preprocessing
- Handling missing values and duplicates.
- Data type casting and string operations.
- Applying user-defined functions (UDFs) in PySpark.
4.2 Joins and Aggregations
- Different types of joins (inner, outer, left, right).
- Grouping, aggregating, and summarizing data.
- Working with complex data structures (arrays, maps).

05Part 5: Working with PySpark MLlib (Machine Learning Library)2

5.1 Introduction to MLlib
- Overview of PySpark MLlib and its capabilities.
- Building machine learning pipelines in PySpark.
5.2 Classification, Regression, and Clustering
- Implementing basic machine learning models (logistic regression, decision trees).
- Exploratory data analysis and feature engineering.
- Unsupervised learning with clustering algorithms (K-means).

06Part 6: Structured Streaming in PySpark2

6.1 Introduction to PySpark Structured Streaming
- Overview of streaming data and its importance.
- Setting up and managing structured streams in PySpark.
6.2 Working with Real-time Data
- Reading and writing streaming data.
- Aggregating, filtering, and transforming streaming data.
- Applying window operations and handling late data.

07Part 7: Performance Tuning and Optimization2

7.1 Optimizing PySpark Jobs
- Partitioning strategies and best practices.
- Caching and persisting RDDs and DataFrames.
- Using broadcast variables and accumulators.
7.2 Advanced Optimizations
- Understanding the Catalyst Optimizer.
- Working with Spark’s Tungsten execution engine.
- Avoiding shuffle operations and minimizing data movement.

08Part 8: Deployment and Best Practices2

8.1 Deploying PySpark Applications
- Deploying PySpark on various environments (local, cluster, cloud).
- Setting up Spark configurations for optimal performance.
- Managing resource allocation and monitoring PySpark jobs.
8.2 Best Practices in PySpark Development
- Writing efficient and maintainable PySpark code.
- Debugging and troubleshooting common PySpark errors.
- Version control and code modularity.

09Practice and Project Work2

Practice Exercises
- A set of 300+ exercises across beginner, intermediate, and advanced levels covering key PySpark operations, transformations, and machine learning tasks.
Capstone Project
- End-to-End Data Pipeline: Students will build an end-to-end data pipeline involving data ingestion, transformation, and analysis using PySpark.
- Real-Time Data Processing Project: Setting up a structured streaming pipeline to process real-time data.

10Evaluation3

Mock Interviews
- Weekly mock interviews to assess comprehension and problem-solving in PySpark.
Peer-reviewed Assignments
- Practical assessments where peers review PySpark code for efficiency, scalability, and clarity.
Final Project Presentation
- A presentation and report on the capstone project, demonstrating the implementation of learned PySpark concepts in a real-world scenario.

13

SQL Interview Preparation

Interview Prep3 sections · 10 topics

01SQL FAANG Questions for Practice3

Duration: 2 Weeks
Week 1: Introduction to SQL interview expectations at top tech companies. Practice with basic to intermediate SQL problems, focusing on data retrieval, aggregation, and filtering.
Week 2: Advanced SQL problem-solving involving joins, subqueries, window functions, and query optimization.

02SQL Data Engineering Interview Prep3

Duration: 2 Weeks
Week 1: Real-world SQL scenarios in data engineering, covering data modeling and ETL processes.
Week 2: Mock interviews simulating data engineering SQL interviews, including data warehouse and data lake querying.

03Learning Outcomes4

Students will familiarize themselves with the types of SQL questions asked in FAANG and tech interviews.
Gain practical experience with SQL through real-world scenarios relevant to data engineering.
Develop strategies for solving complex SQL problems and optimizing queries.
Experience mock interviews to improve interviewing skills with real-time feedback.

Data Engineering Curriculum

What you’ll learn