Introduction

In the modern data landscape, the ability to build and manage scalable, reliable, and cost-effective data pipelines is a fundamental strategic asset, making Amazon Web Services (AWS) for Data Engineers a critical skill for all data professionals. AWS provides an unparalleled suite of services for data ingestion, storage, processing, and analytics, offering the flexibility to architect and implement robust data solutions that can handle massive scale and diverse data types. This comprehensive training course is meticulously designed to equip aspiring data engineers, data architects, and IT professionals with cutting-edge knowledge and practical skills in leveraging AWS’s core data services, including Amazon S3, AWS Glue, Amazon Redshift, and Amazon EMR, to design and build powerful ETL and ELT pipelines, ensuring data quality, security, and governance from source to insight. Participants will gain a deep understanding of how to architect and implement the data infrastructure that powers a data-driven organization on the world's leading cloud platform.

Duration

10 days

Target Audience

• Data Engineers
• Data Architects
• Business Intelligence (BI) Developers
• Data Analysts
• DevOps Engineers
• IT Professionals and Cloud Administrators
• Students in a data-related field
• Professionals transitioning to cloud data platforms
• Anyone responsible for building and managing data pipelines
• ETL/ELT Developers

Objectives

• Understand the core concepts of cloud data engineering and AWS's data services.
• Master the use of Amazon S3 as the foundation for a data lake.
• Learn to build and manage a modern data warehouse with Amazon Redshift.
• Develop proficiency in building serverless ETL pipelines with AWS Glue.
• Understand big data processing with Amazon EMR and its ecosystem.
• Learn to manage real-time data streams using Amazon Kinesis.
• Develop skills in data governance, security, and cost management on AWS.
• Understand the role of AWS Step Functions for workflow orchestration.
• Formulate a strategic approach to designing and building a cloud-native data platform.
• Learn to integrate AWS data services with machine learning tools.

Course Content

Module 1. Introduction to AWS for Data Engineering

• What is Cloud Data Engineering?: Its role and responsibilities
• Why AWS?: The AWS data ecosystem and its breadth of services
• Overview of the AWS data stack
• The AWS Shared Responsibility Model
• Setting up an AWS account and a data-centric environment

Module 2. Foundational AWS Services

• The AWS Management Console: Navigating the UI
• IAM (Identity and Access Management): Users, roles, policies
• VPCs (Virtual Private Clouds): Networking fundamentals for data
• The AWS Command Line Interface (CLI)
• Understanding AWS regions and availability zones

Module 3. Data Storage with Amazon S3

• Amazon S3 (Simple Storage Service): The foundation of a data lake on AWS
• Buckets and Objects: Storage hierarchy
• Storage Classes: S3 Standard, S3 Intelligent-Tiering, S3 Glacier
• Data lakes on S3: Best practices for data organization and partitioning
• Securing data on S3

Module 4. AWS Databases (RDS, Aurora, DynamoDB)

• Amazon RDS: Managed relational databases (PostgreSQL, MySQL, etc.)
• Amazon Aurora: A high-performance, MySQL and PostgreSQL-compatible database
• Amazon DynamoDB: A key-value and document NoSQL database
• When to use each database for different data needs
• Basic database administration and connectivity

Module 5. Data Warehousing with Amazon Redshift

• Amazon Redshift: A petabyte-scale cloud data warehouse
• Redshift Architecture: Leader node and compute nodes
• Data loading techniques with the COPY command
• Data distribution styles and sort keys for performance optimization
• Redshift Spectrum: Querying data directly in S3

Module 6. Serverless ETL with AWS Glue

• What is AWS Glue?: A serverless data integration service
• Glue Data Catalog: A centralized metadata repository
• Glue ETL Jobs: Writing transformation scripts with Spark
• Glue Crawlers for schema discovery
• Building and scheduling an ETL pipeline with AWS Glue

Module 7. Big Data Processing with Amazon EMR

• Amazon EMR (Elastic MapReduce): A managed Spark and Hadoop service
• EMR cluster creation and configuration
• Running Spark, Hive, and Presto jobs on EMR
• EMR notebooks for interactive analysis
• When to use EMR vs. Glue

Module 8. Streaming Data with Amazon Kinesis

• Amazon Kinesis: A platform for real-time data streams
• Kinesis Data Streams: A scalable, durable stream
• Kinesis Firehose: Loading data to S3, Redshift, etc.
• Kinesis Data Analytics: Real-time stream processing
• Building a simple streaming data pipeline

Module 9. Data Orchestration with AWS Step Functions and Airflow

• AWS Step Functions: Serverless workflow orchestration
• Building a state machine for complex ETL workflows
• Apache Airflow on AWS: Managed Workflows for Apache Airflow (MWAA)
• Creating and scheduling DAGs on MWAA
• Choosing between Step Functions and MWAA

Module 10. Serverless Analytics with Amazon Athena

• Amazon Athena: An interactive query service for S3 data
• How Athena works: SQL on S3
• Optimizing Athena queries with data formats and partitioning
• Use cases for ad-hoc analysis and reporting
• Athena vs. Redshift Spectrum

Module 11. Data Governance and Security

• AWS Lake Formation: A managed data lake governance service
• Creating and managing data lake permissions
• Data encryption in AWS
• Auditing and logging with CloudTrail and CloudWatch
• Best practices for securing your data on AWS

Module 12. Machine Learning on AWS (SageMaker)

• Amazon SageMaker: A fully managed ML service
• SageMaker Data Wrangler for data preparation
• Using SageMaker for model training and deployment
• Integrating ML with your data pipelines
• The data engineer's role in MLOps

Module 13. Building an End-to-End Pipeline

• Project Overview: From a data stream to a BI dashboard
• Ingestion: Using Kinesis to collect real-time data
• Processing: Using AWS Glue to transform the data
• Storage: Loading the data into Amazon S3 and Redshift
• Visualization: Connecting a BI tool (e.g., QuickSight)
• Orchestrating the entire workflow with AWS Step Functions

Module 14. Monitoring and Cost Management

• CloudWatch: Metrics, logs, and alarms
• Cost Explorer: Analyzing and controlling spending
• Optimizing resources for cost efficiency
• The benefits of AWS's serverless and managed services for cost
• Budgeting and forecasting

Module 15. The AWS Data Engineering Certification Path

• AWS Certified Data Engineer – Associate: Overview of the exam
• Study plan and key topics to focus on
• Sample questions and practice exams
• The value of certification in the job market
• The future of data engineering on AWS.

Training Approach

This course will be delivered by our skilled trainers who have vast knowledge and experience as expert professionals in the fields. The course is taught in English and through a mix of theory, practical activities, group discussion and case studies. Course manuals and additional training materials will be provided to the participants upon completion of the training.

Tailor-Made Course

This course can also be tailor-made to meet organization requirement. For further inquiries, please contact us on: Email: info@skillsforafrica.org, training@skillsforafrica.org Tel: +254 702 249 449

Training Venue

The training will be held at our Skills for Africa Training Institute Training Centre. We also offer training for a group at requested location all over the world. The course fee covers the course tuition, training materials, two break refreshments, and buffet lunch.

Visa application, travel expenses, airport transfers, dinners, accommodation, insurance, and other personal expenses are catered by the participant

Certification

Participants will be issued with Skills for Africa Training Institute certificate upon completion of this course.

Airport Pickup and Accommodation

Airport pickup and accommodation is arranged upon request. For booking contact our Training Coordinator through Email: info@skillsforafrica.org, training@skillsforafrica.org Tel: +254 702 249 449

Terms of Payment: Unless otherwise agreed between the two parties’ payment of the course fee should be done 7 working days before commencement of the training.