Introduction

In the modern digital landscape, the ability to build and manage scalable, reliable, and cost-effective data pipelines is a fundamental strategic asset, making Cloud Data Engineering on Google Cloud Platform (GCP) a critical skill for all data professionals. GCP provides a comprehensive suite of managed, serverless, and highly integrated services that simplify the process of ingesting, storing, processing, and analyzing data at massive scale, freeing data engineers from the complexities of infrastructure management. This comprehensive training course is meticulously designed to equip aspiring data engineers, data analysts, and IT professionals with cutting-edge knowledge and practical skills in leveraging GCP’s core data services, including Cloud Storage, BigQuery, Dataflow, and Pub/Sub, to design and build robust ETL and ELT data pipelines, ensuring data quality, security, and governance from source to insight. Participants will gain a deep understanding of how to architect and implement the data infrastructure that powers a data-driven organization on one of the world's leading cloud platforms.

Duration

10 days

Target Audience

• Data Engineers
• Data Analysts and BI Developers
• IT Professionals and Cloud Administrators
• Data Scientists
• Solution Architects
• Software Developers
• Students in a data-related field
• ETL/ELT Developers
• Professionals transitioning to cloud data platforms
• Anyone responsible for building and managing data pipelines

Objectives

• Understand the core concepts of cloud data engineering and GCP's data services.
• Master the use of Cloud Storage as the foundation for data lakes.
• Learn to build and manage a modern data warehouse with BigQuery.
• Develop proficiency in using Pub/Sub for real-time data ingestion.
• Understand serverless data processing with Dataflow and batch processing with Dataproc.
• Learn to orchestrate data pipelines using Cloud Composer.
• Explore data governance, security, and cost management on GCP.
• Develop skills in building a complete end-to-end data pipeline.
• Understand the integration between GCP data services and machine learning tools.
• Formulate a strategic approach to designing and building a cloud-native data platform.

Course Content

Module 1. Introduction to Cloud Data Engineering & GCP

• What is Cloud Data Engineering?: Its role and responsibilities
• Why GCP?: GCP's unique value proposition for data
• Overview of the GCP Data Engineering ecosystem
• Key GCP services for data: Storage, Processing, Analytics
• Setting up a GCP account and project

Module 2. Core GCP Services for Data

• The GCP Console: Navigating the UI
• IAM (Identity and Access Management): Roles, permissions, service accounts
• Networking Fundamentals: VPCs, subnets, firewall rules
• Using the GCP Command Line Interface (gcloud)
• The importance of project and resource hierarchy

Module 3. Data Storage on GCP (Cloud Storage)

• Cloud Storage: The foundation of a data lake on GCP
• Buckets and Objects: Storage hierarchy
• Storage Classes: Standard, Nearline, Coldline, Archive
• The gsutil tool for command-line access
• Best practices for data organization and security

Module 4. Relational Databases on GCP

• Cloud SQL: Managed relational databases (PostgreSQL, MySQL, SQL Server)
• Cloud Spanner: Horizontally scalable relational database
• When to use Cloud SQL vs. Cloud Spanner
• Basic database administration: backups, replication
• Connecting to databases from a local machine or other GCP services

Module 5. Data Warehousing with BigQuery

• BigQuery Architecture: Storage and compute separation
• Datasets, Tables, and Views: The logical structure
• BigQuery SQL: A powerful, ANSI-compliant SQL dialect
• Loading data into BigQuery from Cloud Storage
• Cost management and slot reservations

Module 6. Data Ingestion with Pub/Sub

• What is Pub/Sub?: A real-time messaging service
• Topics and Subscriptions: The core components
• Use cases for real-time data ingestion
• Publishing and consuming messages with the client libraries
• Integrating Pub/Sub with other GCP services

Module 7. Serverless Data Processing with Dataflow

• What is Dataflow?: A serverless, unified stream and batch processing service
• Apache Beam: The open-source programming model
• Creating a simple Dataflow pipeline with Python
• Understanding the Dataflow job lifecycle
• Dataflow templates and their use

Module 8. Batch Processing with Dataproc

• What is Dataproc?: A managed Spark and Hadoop service
• Creating and managing Dataproc clusters
• Running Spark jobs on Dataproc
• When to use Dataproc vs. Dataflow
• Scaling and cost management for Dataproc

Module 9. Data Transformation with BigQuery

• ETL vs. ELT: The modern approach with BigQuery
• Transforming data within BigQuery with SQL
• Using CTEs and Views for complex transformations
• Materialized Views for performance
• BigQuery Data Transfer Service: Automating data loads

Module 10. Data Orchestration with Cloud Composer

• What is Cloud Composer?: A managed Apache Airflow service
• DAGs (Directed Acyclic Graphs): Defining workflows
• Writing a simple Cloud Composer DAG with Python
• The Cloud Composer web interface
• Monitoring and managing your data pipelines

Module 11. Data Governance & Security

• Data Catalog: Discovering and managing metadata
• Data Loss Prevention (DLP): Protecting sensitive data
• VPC Service Controls: Creating a security perimeter
• Auditing and monitoring data access
• Best practices for securing your data on GCP

Module 12. Machine Learning Integration

• BigQuery ML: Training ML models directly in BigQuery
• Vertex AI: A unified platform for ML
• Using Dataflow for ML data preprocessing
• Building a simple ML pipeline on GCP
• Integrating ML predictions back into your BI reports

Module 13. Building an End-to-End Pipeline

• Project Overview: From an event stream to a BI dashboard
• Ingestion: Using Pub/Sub for real-time events
• Processing: Using Dataflow to transform the data
• Storage: Loading the data into BigQuery
• Visualization: Connecting a BI tool (e.g., Looker) to BigQuery
• Orchestrating the entire workflow with Cloud Composer

Module 14. Monitoring and Cost Management

• Cloud Monitoring: Dashboards and alerts
• Cloud Logging: Centralized log management
• Cost Explorer and Billing Reports: Analyzing and controlling spending
• Optimizing resources for cost efficiency
• The benefits of GCP's serverless and managed services for cost

Module 15. The GCP Data Engineering Certification Path

• Professional Data Engineer Certification: Overview of the exam
• Study plan and key topics to focus on
• Sample questions and practice exams
• The value of certification in the job market
• The future of data engineering on GCP.

Training Approach

This course will be delivered by our skilled trainers who have vast knowledge and experience as expert professionals in the fields. The course is taught in English and through a mix of theory, practical activities, group discussion and case studies. Course manuals and additional training materials will be provided to the participants upon completion of the training.

Tailor-Made Course

This course can also be tailor-made to meet organization requirement. For further inquiries, please contact us on: Email: info@skillsforafrica.org, training@skillsforafrica.org Tel: +254 702 249 449

Training Venue

The training will be held at our Skills for Africa Training Institute Training Centre. We also offer training for a group at requested location all over the world. The course fee covers the course tuition, training materials, two break refreshments, and buffet lunch.

Visa application, travel expenses, airport transfers, dinners, accommodation, insurance, and other personal expenses are catered by the participant

Certification

Participants will be issued with Skills for Africa Training Institute certificate upon completion of this course.

Airport Pickup and Accommodation

Airport pickup and accommodation is arranged upon request. For booking contact our Training Coordinator through Email: info@skillsforafrica.org, training@skillsforafrica.org Tel: +254 702 249 449

Terms of Payment: Unless otherwise agreed between the two parties’ payment of the course fee should be done 7 working days before commencement of the training.