Introduction

In today's data-driven world, the ability to build and maintain the robust, reliable, and scalable infrastructure that powers all data analytics and machine learning is a fundamental strategic asset, making Foundations of Data Engineering an indispensable skill for professionals who want to be at the heart of the modern data ecosystem. Data engineering is the critical discipline that transforms raw, messy data into clean, accessible, and trustworthy information, ensuring that data pipelines are efficient, storage systems are optimized, and data governance is upheld, thereby enabling data scientists, BI analysts, and business leaders to derive meaningful insights. This comprehensive training course is meticulously designed to equip aspiring data engineers, data analysts, BI developers, and IT professionals with cutting-edge knowledge and practical skills in understanding the full data lifecycle, mastering the core principles of ETL/ELT, exploring various data storage technologies (data warehouses and data lakes), and developing proficiency with essential tools and programming concepts to build the foundational data architecture that drives a data-centric organization. Participants will gain a holistic understanding of how to engineer the data solutions that are the backbone of all successful analytics initiatives.

Duration

10 days

Target Audience

• Aspiring Data Engineers
• Data Analysts and BI Developers
• Database Administrators (DBAs)
• IT Professionals and System Administrators
• Data Scientists (seeking to strengthen engineering skills)
• Software Developers
• Students in Computer Science or Data Science
• Solution Architects
• Professionals looking to transition into a data engineering role
• Anyone interested in the infrastructure behind data analytics

Objectives

• Understand the core concepts and responsibilities of a data engineer.
• Master the principles of the data lifecycle and building robust data pipelines.
• Learn about different data storage technologies, including data warehouses and data lakes.
• Develop proficiency in the Extract, Transform, Load (ETL) and ELT processes.
• Understand the fundamentals of data processing frameworks and tools.
• Learn about data quality, governance, and security in a data engineering context.
• Develop skills in using Python for data manipulation and scripting.
• Explore the key components of a modern cloud-based data engineering stack.
• Understand the role of data orchestration and workflow management.
• Formulate a strategic approach to designing and building data infrastructure.

Course Content

Module 1. Introduction to Data Engineering

• Defining Data Engineering: Its purpose, scope, and role in an organization
• The relationship between data engineering, data science, and business intelligence
• The modern data ecosystem: Data sources, pipelines, storage, and consumption
• The data engineer's responsibilities and skill set
• The journey of data: From raw source to actionable insight

Module 2. The Data Engineering Ecosystem

• Data Sources: APIs, databases, files (CSV, JSON), streaming data
• Data Ingestion: Tools and methods for moving data
• Data Storage: Databases, data warehouses, data lakes
• Data Processing: Batch vs. streaming processing
• Data Orchestration: Managing data workflows

Module 3. Fundamentals of Databases

• Relational Databases (SQL): Key concepts, schemas, normalization
• NoSQL Databases: Key-value, document, column-family, graph databases
• Choosing the right database for a specific use case
• Basic SQL for data engineering: SELECT, INSERT, UPDATE, DELETE
• Database connectivity and drivers

Module 4. The ETL/ELT Process

• Defining ETL (Extract, Transform, Load): The traditional approach
• Defining ELT (Extract, Load, Transform): The modern, cloud-native approach
• When to use ETL vs. ELT
• Key challenges in data transformation
• Introduction to common ETL/ELT tools (e.g., Apache Nifi, Stitch, Fivetran)

Module 5. Data Warehousing Concepts

• What is a Data Warehouse?: Its purpose and architecture
• Data Modeling: Star schema and snowflake schema
• ETL in a Data Warehouse Context: Staging, loading, and reporting layers
• OLAP vs. OLTP systems
• Introduction to popular data warehouses (e.g., Snowflake, Amazon Redshift, Google BigQuery)

Module 6. Introduction to Data Lakes

• What is a Data Lake?: Purpose, characteristics, and vs. a data warehouse
• Data Lake Architecture: Storage, catalog, processing
• Schema-on-Read vs. Schema-on-Write: Flexibility vs. structure
• Data lake layers: Raw, cleansed, and curated data
• Introduction to Data Lake file formats (e.g., Parquet, Avro)

Module 7. Core Data Processing with Python

• Python for Data Engineering: Why it's the standard
• Introduction to Pandas: DataFrames for data manipulation
• File I/O: Reading and writing CSV, JSON, Parquet files
• Scripting for simple ETL tasks
• Using Python to interact with databases and APIs

Module 8. Introduction to Cloud Data Engineering

• Why the Cloud?: Scalability, cost-effectiveness, managed services
• Cloud Infrastructure as a Service (IaaS): EC2, virtual machines
• Platform as a Service (PaaS): Managed databases, data warehouses
• Serverless Computing: Lambda, Cloud Functions
• The cloud-native data stack

Module 9. Data Storage on the Cloud

• Object Storage: AWS S3, Azure Blob Storage, Google Cloud Storage
• Cloud-native Databases: Amazon RDS, Azure SQL Database, Google Cloud SQL
• Cloud Data Warehouses: Amazon Redshift, Azure Synapse, Google BigQuery
• Cloud Data Lake Storage: AWS S3, ADLS, GCS
• Understanding storage tiers and cost optimization

Module 10. Batch Processing

• Batch Processing Fundamentals: Processing data in large chunks
• Introduction to Apache Spark: RDDs, DataFrames, Spark SQL
• Executing Spark Jobs: Local vs. distributed mode
• MapReduce Concepts: The foundation of distributed processing
• Use cases for batch processing: Data transformations, reporting, ML training

Module 11. Introduction to Data Orchestration

• What is Data Orchestration?: Managing complex data workflows
• Apache Airflow: Directed Acyclic Graphs (DAGs), operators
• Workflow Scheduling and Monitoring
• The importance of idempotent tasks
• Other orchestration tools (e.g., AWS Step Functions, Prefect)

Module 12. Data Quality and Governance

• The Importance of Data Quality: Trust in data
• Data Validation and Monitoring: Ensuring data integrity
• Introduction to Data Governance: Policies, roles, responsibilities
• Data Catalogs and Metadata Management
• Best practices for maintaining data health

Module 13. Fundamentals of Data Security

• Data Encryption: At rest and in transit
• Access Control: IAM, RBAC, least privilege principle
• Data Masking and Anonymization: Protecting sensitive data
• Auditing and Monitoring data access
• Security considerations across the data pipeline

Module 14. Real-World Case Study: Building a Simple Data Pipeline

• Project Overview: From a data source to a dashboard
• Ingesting Data: Using Python to pull from an API or file
• Transforming Data: Cleaning and preparing data with Pandas
• Loading Data: Writing transformed data to a cloud database
• Orchestration: Building a simple workflow with a tool like Airflow (conceptual)

Module 15. The Future of Data Engineering

• Data Mesh and Data Fabric: Decentralized data architectures
• Real-time and Streaming Data: Apache Kafka, Flink, Spark Streaming
• MLOps: Operationalizing machine learning models
• The rise of the Data Lakehouse
• Continuous learning and evolving data engineering tools.

Training Approach

This course will be delivered by our skilled trainers who have vast knowledge and experience as expert professionals in the fields. The course is taught in English and through a mix of theory, practical activities, group discussion and case studies. Course manuals and additional training materials will be provided to the participants upon completion of the training.

Tailor-Made Course

This course can also be tailor-made to meet organization requirement. For further inquiries, please contact us on: Email: info@skillsforafrica.org, training@skillsforafrica.org Tel: +254 702 249 449

Training Venue

The training will be held at our Skills for Africa Training Institute Training Centre. We also offer training for a group at requested location all over the world. The course fee covers the course tuition, training materials, two break refreshments, and buffet lunch.

Visa application, travel expenses, airport transfers, dinners, accommodation, insurance, and other personal expenses are catered by the participant

Certification

Participants will be issued with Skills for Africa Training Institute certificate upon completion of this course.

Airport Pickup and Accommodation

Airport pickup and accommodation is arranged upon request. For booking contact our Training Coordinator through Email: info@skillsforafrica.org, training@skillsforafrica.org Tel: +254 702 249 449

Terms of Payment: Unless otherwise agreed between the two parties’ payment of the course fee should be done 7 working days before commencement of the training.