Azure Data Engineering Certification Curriculum

Introduction To Azure Data Engineering

• Understand the evolving world of data
• Data abundance
• Understanding the Data Engineering Problem
• Understand job responsibilities
• Understanding Data Engineering Processing - Extract Transform and Load
• Overview of Azure Data Engineering Services
• Understand data storage in Azure Storage
• Understand data storage in Azure Data Lake Storage
• Understand Azure Cosmos DB
• Understand Azure SQL Database
• Understand Azure Synapse Analytics
• Understand Azure Stream Analytics
• Understand Azure HDInsight
• Understand other Azure data services

Storing Data in Azure

• How to choose an Azure Storage Service in Azure
• Create an Azure Storage Account
• Connect an app to Azure Storage API
• Connect to your Azure storage account
• Explore Azure Storage security features
• Understand storage account keys
• Understand shared access signatures
• Control network access to your storage account
• Understand Advanced Threat Protection for Azure Storage
• Explore Azure Data Lake Storage security features
• Introduction to Blob storage
• What are blobs?
• Design a storage organization strategy

Hands-On:

• Add the storage client library to your app
• Add Azure Storage configuration to your app
• Connect your application to your Azure Storage account
• Create Azure storage resources
• Configure and initialize the client library
• Get blob references
• Blob uploads and downloads

Azure Data Factory Part - I

• Integrate data with Azure Data Factory or Azure Synapse Pipeline
• Understand Azure Data Factory
• Describe data integration patterns
• Explain the data factory process
• Understand Azure Data Factory components
• Azure Data Factory security
• Set-up Azure Data Factory
• Create linked services
• Create datasets
• Create data factory activities and pipelines
• Manage integration runtimes
• Petabyte-scale ingestion with Azure Data Factory or Azure Synapse Pipeline
• List the data factory ingestion methods
• Describe data factory connectors
• Understand data ingestion security considerations

Hands-On:

• Use the data factory copy activity
• Manage the self hosted integration runtime
• Setup the Azure integration runtime

Azure Data Factory Part - II

• Explain Data Factory transformation methods
• Describe Data Factory transformation types
• Exercise - Author an Azure Data Factory mapping data flow
• Debug mapping data flow
• Exercise - Use Data Factory wrangling data
• Exercise - Use compute transformations within Data Factory
• Exercise - Integrate SQL Server Integration Services packages within Data Factory
• Describe slowly changing dimensions
• Choose between slowly changing dimension types
• Exercise - Design and implement a Type 1 slowly changing dimension with mapping data flows
• Understand data factory control flow
• Work with data factory pipelines
• Debug data factory pipelines
• Add parameters to data factory components
• Exercise - Integrate a Notebook within Azure Synapse Pipelines
• Execute data factory packages
• Describe SQL Server Integration Services
• Understand the Azure-SIS integration runtime
• Set-up Azure-SIS integration runtime
• Run SSIS packages in Azure Data Factory
• Migrate SSIS packages to Azure Data Factory
• Configure a git repository with a development factory
• Create and merge a feature branch
• Deploy a release pipeline
• Visually monitor pipeline runs
• Integrate with Azure Monitor
• Set up alerts
• Rerun pipeline runs

Azure Synapse Analytics Part - I

• What is Azure Synapse Analytics5 min
• How Azure Synapse Analytics works8 min
• When to use Azure Synapse Analytics4 min
• Exercise - Explore Azure Synapse Analytics
• Create Azure Synapse Analytics workspace4 min
• Exercise - Create and manage Azure Synapse Analytics workspace4 min
• Describe Azure Synapse Analytics SQL5 min
• Explain Apache Spark in Azure Synapse Analytics7 min
• Exercise - Create pools in Azure Synapse Analytics9 min
• Orchestrate data integration with Azure Synapse pipelines7 min
• Exercise-Identifying Azure Synapse pipeline components8 min
• Visualize your analytics with Power BI7 min
• Understand hybrid transactional analytical processing with Azure Synapse Link
• Use Azure Synapse Studio

• Understand the Azure Synapse Analytical processes
• Explore the Data hub
• Explore the Develop hub
• Explore the Integrate hub
• Explore the Monitor hub
• Explore the Manage hub
• Describe a Modern Data Warehouse
• Define a Modern Data Warehouse Architecture
• Design ingestion patterns for a Modern Data Warehouse
• Understand data storage for a Modern Data Warehouse
• Understand file formats and structure for a modern data warehouse
• Prepare and transform data with Azure Synapse Analytics
• Serve data for analysis with Azure Synapse Analytics

Work with Data Warehouses using Azure Synapse Analytics - Part I

• Describe a modern data warehouse
• Define a modern data warehouse architecture
• Exercise - Identify modern data warehouse architecture components
• Design ingestion patterns for a modern data warehouse
• Understand data storage for a modern data warehouse
• Understand file formats and structure for a modern data warehouse
• Prepare and transform data with Azure Synapse Analytics

Hands-On:

• Serve data for analysis with Azure Synapse Analytics
• Design a data warehouse schema
• Create data warehouse tables
• Load data warehouse tables
• Query a data warehouse

Work with Data Warehouses using Azure Synapse Analytics - Part II

• Understand data load design goals
• Explain load methods into Azure Synapse Analytics
• Manage source data files
• Manage singleton updates
• Set-up dedicated data load accounts
• Implement workload management
• Simplify ingestion with the Copy Activity
• Understand performance issues related to tables

Hands-On:

• Data Loading in Azure Synapse Analytics
• Data Ingestion using Copy Activity
• implement workload management
• Understand performance issues related to tables

Optimizing Data Queries in Azure

• Understand table distribution design
• Use indexes to improve query performance
• Understand query plans
• Create statistics to improve query performance
• Improve query performance with materialized views
• Use read committed snapshot for data consistency
• How does statistics affect a query plan?
• Describe the integration methods between SQL and spark pools in Azure Synapse Analytics
• Understand the use-cases for SQL and spark pools integration
• Exercise: Integrate SQL and spark pools in Azure Synapse Analytics
• Externalize the use of spark pools within Azure Synapse Workspace
• Transfer data outside the synapse workspace using the PySpark connector
• Explore the development tools for Azure Synapse Analytics
• Understand transact-SQL language capabilities for Azure Synapse Analytics

Hands-On:

• Use table distribution and indexes to improve performance
• Optimize common queries with result-set caching
• Work with windowing functions
• Work with approximate execution
• Work with JSON data in SQL pools
• Encapsulate transact-SQL logic with stored procedures
• Authenticate in Azure Synapse Analytics
• Transfer data between SQL and spark pool in Azure Synapse Analytics
• Authenticate between spark and SQL pool in Azure Synapse Analytics

Managing Workloads in Azure Synapse Analytics

• Scale compute resources in Azure Synapse Analytics
• Pause compute in Azure Synapse Analytics
• Manage workloads in Azure Synapse Analytics
• Use Azure Advisor to review recommendations
• Use dynamic management views to identify and troubleshoot query performance
• Understand skewed data and space usage
• Understand network security options for Azure Synapse Analytics
• Configure Conditional Access
• Configure authentication
• Manage authorization through column and row level security
• Exercise - Manage authorization through column and row level security
• Manage sensitive data with Dynamic Data Masking
• Implement encryption in Azure Synapse Analytics

Hands-On:

• Check for skewed data and space usage
• Understand column store storage details
• View column store storage details
• Understand the impact of wrong choices for column data types
• Compare storage requirements between optimal and sub-optimal column data types
• Describe the impact of materialized views
• Improve the execution plan of a query with a materialized view
• Understand rules for minimally logged operations
• Optimize a delete operation

Deep Dive into Azure Databricks

• Get started with Azure Databricks
• Identify Azure Databricks workloads
• Understand key concepts
• Use Apache Spark in Azure Databricks
• Create a Spark cluster
• Use Spark in notebooks
• Use Spark to work with data files
• Visualize data
• Get Started with Delta Lake
• Create Delta Lake tables
• Create and query catalog tables
• Use Delta Lake for streaming data
• Get started with SQL Warehouses
• Create databases and tables
• Create queries and dashboards
• Understand Azure Databricks notebooks and pipelines
• Create a linked service for Azure Databricks
• Use a Notebook activity in a pipeline
• Use parameters in a notebook

Hands-On:

• Explore Azure Databricks
• Run an Azure Databricks Notebook with Azure Data Factory
• Use a SQL Warehouse in Azure Databricks
• Use Delta Lake in Azure Databricks
• Use Spark in Azure Databricks