Introduction

In the fast-evolving world of data, a unified, end-to-end platform for data integration, warehousing, and analytics is essential for unlocking business value at scale, making Azure Data Factory and Azure Synapse a powerful combination for any modern data engineer. This integrated cloud solution from Microsoft brings together the best of data movement, big data processing, and enterprise data warehousing into a single, cohesive service, simplifying the complexities of building and managing advanced data pipelines. This comprehensive training course is meticulously designed to equip data engineers, data architects, and BI developers with the knowledge and hands-on experience to master Azure Data Factory's code-free ETL/ELT capabilities and leverage the full power of Azure Synapse Analytics for everything from interactive data exploration with serverless SQL pools to large-scale data transformation with Apache Spark. Participants will learn how to design, build, and orchestrate robust, scalable, and secure data solutions that are crucial for driving a data-driven culture within any organization.

Duration

10 days

Target Audience

• Data Engineers
• Data Architects
• ETL/ELT Developers
• Business Intelligence (BI) Developers
• IT Professionals
• Data Analysts
• Students in data-related fields
• Cloud Engineers
• Professionals migrating to Azure
• Anyone building data pipelines on Azure

Objectives

• Understand the core concepts of Azure Data Factory and Azure Synapse Analytics.
• Master the creation and orchestration of data pipelines in Azure Data Factory.
• Learn to build serverless and dedicated SQL pools in Azure Synapse.
• Develop proficiency in using Mapping Data Flows for code-free data transformation.
• Understand the role of Apache Spark pools for big data processing within Synapse.
• Learn to integrate Azure Data Factory with Azure Synapse and other Azure services.
• Explore data governance, security, and monitoring best practices on the platform.
• Develop skills in building a complete, end-to-end data lakehouse architecture.
• Understand the importance of performance tuning and cost management.
• Formulate a strategic approach to designing a unified data analytics solution on Azure.

Course Content

Module 1. Introduction to Azure Data Engineering

• The Modern Data Platform on Azure: Overview and components
• Azure Data Factory (ADF): The cloud-based ETL service
• Azure Synapse Analytics: The unified analytics platform
• The relationship between ADF and Synapse Pipelines
• Setting up an Azure account and resource group

Module 2. Azure Data Factory Fundamentals

• ADF Components: Pipelines, Activities, Datasets, Linked Services
• ADF Studio: The UI for designing and managing pipelines
• Creating Linked Services to connect to data sources
• Defining Datasets to represent data in storage
• Building your first data copy activity pipeline

Module 3. Data Transformation with Mapping Data Flows

• What are Mapping Data Flows?: Code-free data transformation at scale
• The visual designer and transformation types
• Creating a simple data flow with joins and aggregations
• Handling schema drift and data validation
• Debugging and monitoring data flows

Module 4. Pipeline Orchestration & Control

• Control Activities: For Each, If Condition, Wait, Web
• Parameters and Variables: Dynamically managing pipelines
• Triggering pipelines: Schedule, Event, Tumbling Window triggers
• Using the Get Metadata and Lookup activities
• Implementing error handling and logging

Module 5. Introduction to Azure Synapse Analytics

• Synapse Studio: The unified workspace for analytics
• Synapse Architecture: SQL Pools, Spark Pools, Data Explorer Pools
• Data Integration: The role of Synapse Pipelines (powered by ADF)
• Connecting to Azure Data Lake Storage Gen2 (ADLS Gen2)
• The importance of a data lakehouse architecture

Module 6. Synapse SQL Pools (Dedicated and Serverless)

• Dedicated SQL Pools: The enterprise data warehouse
• Serverless SQL Pools: Querying data lakes on-demand
• Querying data in ADLS Gen2 with OPENROWSET
• Creating external tables and views with serverless SQL
• Performance optimization with data distribution and indexing

Module 7. Big Data Processing with Synapse Spark Pools

• Apache Spark Pools: A managed Spark service in Synapse
• Synapse Notebooks: Writing code in Python, Scala, and SQL
• Loading and transforming data from the data lake
• Using Spark DataFrames for big data manipulation
• Integrating Spark notebooks into Synapse Pipelines

Module 8. Integrating ADF and Synapse

• Synapse Pipelines: How they are a part of the Synapse workspace
• Calling Synapse Notebooks from an ADF pipeline
• Using Copy activity to load data into a dedicated SQL pool
• Leveraging serverless SQL as a source in ADF
• Best practices for cross-service integration

Module 9. Ingestion and Loading Strategies

• ELT (Extract, Load, Transform): The modern approach with Synapse
• Incremental Loading: Processing only new or changed data
• PolyBase vs. COPY command: Loading into dedicated SQL pools
• Using COPY activity for bulk data transfer
• Designing a medallion architecture (Bronze, Silver, Gold layers)

Module 10. Data Security & Governance

• IAM: Role-based access control (RBAC) for data resources
• Azure Key Vault: Securely storing credentials and secrets
• Dynamic Data Masking: Protecting sensitive data
• Network Security: Private endpoints and VNet integration
• Data governance with Azure Purview (conceptual overview)

Module 11. Monitoring and Troubleshooting

• Azure Monitor: Collecting metrics and logs
• ADF Monitoring: Tracking pipeline runs, activity details, and errors
• Synapse Monitoring: Monitoring SQL and Spark pool usage
• Setting up alerts and notifications for pipeline failures
• Troubleshooting common data pipeline issues

Module 12. Cost Management

• Understanding Azure costs: Compute, storage, and data movement
• ADF pricing: Pay-per-activity model
• Synapse pricing: Dedicated vs. serverless compute models
• Optimizing data flows and pipelines for cost efficiency
• Strategies for managing and forecasting cloud spending

Module 13. Data Exploration & Analysis

• Exploring data in ADLS Gen2 with Synapse Studio
• Using serverless SQL pools for ad-hoc analysis
• Connecting Power BI to Azure Synapse for reporting
• Creating analytical reports and dashboards
• The role of the data engineer in a BI workflow

Module 14. Real-World Case Study

• Project Overview: Building a complete data pipeline
• Ingestion: From a sample data source into ADLS Gen2
• Transformation: Cleaning and enriching data with Data Flows or Spark
• Storage: Loading the data into a dedicated SQL pool
• Analysis: Querying the data with Synapse SQL
• Final presentation and review of the complete project

Module 15. The Azure Data Engineer Certification Path

• DP-203: Data Engineering on Microsoft Azure: Overview of the exam
• Study plan and key topics for the certification
• Official Microsoft Learn modules and resources
• Sample questions and practice exams
• The value of certification in the job market.

Training Approach

This course will be delivered by our skilled trainers who have vast knowledge and experience as expert professionals in the fields. The course is taught in English and through a mix of theory, practical activities, group discussion and case studies. Course manuals and additional training materials will be provided to the participants upon completion of the training.

Tailor-Made Course

This course can also be tailor-made to meet organization requirement. For further inquiries, please contact us on: Email: info@skillsforafrica.org, training@skillsforafrica.org Tel: +254 702 249 449

Training Venue

The training will be held at our Skills for Africa Training Institute Training Centre. We also offer training for a group at requested location all over the world. The course fee covers the course tuition, training materials, two break refreshments, and buffet lunch.

Visa application, travel expenses, airport transfers, dinners, accommodation, insurance, and other personal expenses are catered by the participant

Certification

Participants will be issued with Skills for Africa Training Institute certificate upon completion of this course.

Airport Pickup and Accommodation

Airport pickup and accommodation is arranged upon request. For booking contact our Training Coordinator through Email: info@skillsforafrica.org, training@skillsforafrica.org Tel: +254 702 249 449

Terms of Payment: Unless otherwise agreed between the two parties’ payment of the course fee should be done 7 working days before commencement of the training.