In today’s cloud-native era, Serverless Data Engineering is revolutionizing how organizations build and manage data workflows. By eliminating the need to provision and manage infrastructure, serverless architectures empower data engineers to focus on designing highly scalable, resilient, and cost-efficient pipelines. This course equips participants with the skills to build event-driven data processing systems using serverless technologies such as AWS Lambda, Azure Functions, Google Cloud Functions, and more. Learners will explore the best practices for orchestration, data ingestion, real-time processing, monitoring, and governance in a serverless environment, enabling them to accelerate innovation and reduce operational complexity.

Duration: 10 Days

Target Audience

• Cloud Data Engineers
• DevOps Engineers working with data infrastructure
• Big Data Developers
• Cloud Architects
• DataOps and MLOps Professionals
• Technical Project Managers
• Enterprise Data Platform Engineers
• Analytics Engineers

Course Objectives

• Understand the fundamentals of serverless computing and its benefits for data engineering
• Learn how to design serverless data pipelines for batch and streaming data
• Explore integration with cloud-native services for data ingestion, transformation, and output
• Build event-driven architectures that respond to data triggers efficiently
• Master monitoring, logging, and alerting in serverless environments
• Apply cost optimization strategies for serverless workflows
• Ensure scalability, fault-tolerance, and reliability in serverless pipelines
• Implement real-time analytics and data lake ingestion
• Enforce security and governance across serverless components
• Use infrastructure as code to manage and automate deployments
• Gain hands-on experience through practical labs and a capstone project

Module 1: Introduction to Serverless Data Engineering

• Overview of serverless computing
• Benefits and challenges for data processing
• Key cloud provider offerings: AWS, GCP, Azure
• Serverless vs. container-based architectures
• Use cases in modern data engineering

Module 2: Serverless Functions and Frameworks

• AWS Lambda, Azure Functions, Google Cloud Functions
• Function lifecycle and execution model
• Writing and deploying serverless functions
• Managing concurrency and limits
• Using frameworks like Serverless Framework and SAM

Module 3: Event-Driven Architecture Patterns

• Sources of events: file drops, API calls, queues
• Triggering pipelines on data events
• Designing event producers and consumers
• Ensuring idempotency and retry strategies
• Chaining and fan-out patterns

Module 4: Data Ingestion in Serverless Pipelines

• Using Kinesis, Pub/Sub, and Event Hubs
• Handling structured and unstructured data
• Real-time vs. batch ingestion
• Validating and transforming incoming data
• Integrating with APIs and third-party data sources

Module 5: Storage Services for Serverless Data

• Using Amazon S3, Azure Blob, and GCS
• Object lifecycle management and versioning
• Event notifications on file uploads
• Data partitioning and organization
• Storage security and access policies

Module 6: Real-Time Data Processing with Serverless

• Stream processing with Lambda and Kinesis
• Aggregation and windowing techniques
• Handling late-arriving and out-of-order data
• Delivering processed results to sinks
• Combining stream and batch processing

Module 7: Serverless Orchestration and Workflow Management

• AWS Step Functions, Azure Durable Functions
• Defining state machines and workflows
• Error handling and retries in orchestration
• Chaining multi-step pipelines
• Visualizing and monitoring execution

Module 8: Serverless SQL and Data Transformation

• Querying with Athena, BigQuery, Synapse Serverless
• ETL and ELT design in serverless context
• Leveraging Glue and Dataflow for transformation
• Schema inference and metadata cataloging
• Using PySpark and SQL for data prep

Module 9: Monitoring and Observability

• Logging with CloudWatch, Stackdriver, Azure Monitor
• Tracing and profiling functions
• Creating metrics and dashboards
• Handling cold starts and latency issues
• Alerting and anomaly detection

Module 10: Cost Optimization and Budgeting

• Understanding billing for serverless workloads
• Reducing invocations and execution time
• Managing data transfer costs
• Setting budgets and usage alerts
• Comparing serverless vs. managed alternatives

Module 11: Security and Access Management

• IAM policies for least privilege
• Securing secrets and API keys
• Encrypting data at rest and in transit
• Managing authentication and authorization
• Reviewing serverless security best practices

Module 12: Infrastructure as Code for Serverless

• Using Terraform and CloudFormation
• Creating reproducible deployments
• Versioning infrastructure and code
• Managing environments and secrets
• Automated testing and validation

Module 13: Building Data APIs with Serverless

• Creating APIs using API Gateway and Lambda
• Designing REST and GraphQL endpoints
• Rate limiting and throttling
• Integrating with data stores
• Securing APIs with OAuth and tokens

Module 14: Serverless Machine Learning Inference

• Deploying lightweight models with serverless
• Triggering predictions on data events
• Integrating with SageMaker, Vertex AI, ML.NET
• Streaming inference vs. batch inference
• Scaling ML workloads with autoscaling functions

Module 15: Capstone Project and Real-World Architectures

• Building a full serverless data pipeline
• Real-world case studies from e-commerce and finance
• End-to-end implementation and demo
• Troubleshooting and final optimization
• Presentation and feedback session

Training Approach

This course will be delivered by our skilled trainers who have vast knowledge and experience as expert professionals in the fields. The course is taught in English and through a mix of theory, practical activities, group discussion and case studies. Course manuals and additional training materials will be provided to the participants upon completion of the training.

Tailor-Made Course

This course can also be tailor-made to meet organization requirement. For further inquiries, please contact us on: Email: info@skillsforafrica.org, training@skillsforafrica.org Tel: +254 702 249 449

Training Venue

The training will be held at our Skills for Africa Training Institute Training Centre. We also offer training for a group at requested location all over the world. The course fee covers the course tuition, training materials, two break refreshments, and buffet lunch.

Visa application, travel expenses, airport transfers, dinners, accommodation, insurance, and other personal expenses are catered by the participant

Certification

Participants will be issued with Skills for Africa Training Institute certificate upon completion of this course.

Airport Pickup and Accommodation

Airport pickup and accommodation is arranged upon request. For booking contact our Training Coordinator through Email: info@skillsforafrica.org, training@skillsforafrica.org Tel: +254 702 249 449

Terms of Payment: Unless otherwise agreed between the two parties’ payment of the course fee should be done 7 working days before commencement of the training.