Introduction

In an era defined by an exponential growth in data volume, velocity, and variety, the ability to build and manage scalable infrastructure to process and store massive datasets is a critical skill, making Big Data Engineering with Hadoop Ecosystem an indispensable discipline for all data-driven organizations. This comprehensive training course is designed to provide a hands-on, deep dive into the core components of the Hadoop ecosystem, the open-source framework that revolutionized big data processing and continues to be the foundation for many modern data platforms. By mastering distributed storage with HDFS, resource management with YARN, and parallel processing with both MapReduce and Apache Spark, participants will be empowered to design, build, and maintain the robust and efficient data pipelines that are essential for powering analytics, machine learning, and strategic decision-making in the age of big data.

Duration

10 days

Target Audience

• Data Engineers & Architects
• Data Analysts & BI Developers
• ETL/ELT Developers
• DevOps & Cloud Engineers
• Data Scientists
• Database Administrators (DBAs)
• IT Professionals managing data infrastructure
• Students and career changers in big data
• Professionals looking to build scalable data platforms
• Anyone responsible for processing large datasets

Objectives

• Understand the core concepts of big data and the Hadoop ecosystem.
• Master the architecture and functionality of HDFS (Hadoop Distributed File System).
• Learn about resource management with YARN (Yet Another Resource Negotiator).
• Develop proficiency in the MapReduce programming model.
• Master Apache Spark for high-performance, in-memory data processing.
• Explore key components for data ingestion, warehousing, and streaming.
• Understand the role of NoSQL databases in the Hadoop ecosystem.
• Develop skills in building end-to-end big data pipelines.
• Learn about data governance and security in a big data environment.
• Understand the evolution of the Hadoop ecosystem and future trends.

Course Content

Module 1. Introduction to Big Data & Hadoop

• What is Big Data?: The 3 Vs (Volume, Velocity, Variety)
• The limitations of traditional systems for big data
• What is Hadoop?: Its history and core components
• The Hadoop ecosystem overview: HDFS, YARN, MapReduce, Spark
• Setting up a single-node Hadoop cluster

Module 2. Hadoop Distributed File System (HDFS)

• HDFS Architecture: Namenode, Datanode, Secondary Namenode
• The concepts of blocks, replication factor, and fault tolerance
• Basic HDFS commands: put, get, ls, mkdir
• Reading and writing data to HDFS
• HDFS Federation and High Availability

Module 3. YARN (Yet Another Resource Negotiator)

• YARN's Role: Resource management and job scheduling
• YARN Architecture: ResourceManager, NodeManager, ApplicationMaster
• The lifecycle of a YARN application
• YARN vs. the traditional MapReduce framework
• Resource configuration and tuning

Module 4. MapReduce Framework

• The MapReduce Paradigm: Map and Reduce functions
• The MapReduce Job Lifecycle: Input, Map, Shuffle & Sort, Reduce, Output
• Writing a simple MapReduce program (conceptual)
• The drawbacks of MapReduce and the rise of Spark
• Use cases for MapReduce

Module 5. Introduction to Apache Spark

• What is Spark?: The unified analytics engine
• Spark vs. MapReduce: Speed, versatility, and ease of use
• Spark Architecture: Driver, Executors, Cluster Manager
• Spark's key components: Spark Core, Spark SQL, Spark Streaming, MLlib, GraphX
• Setting up a local Spark environment

Module 6. Spark Core & RDDs

• Resilient Distributed Datasets (RDDs): The foundation of Spark
• RDD Transformations: map, filter, flatMap
• RDD Actions: collect, count, saveAsTextFile
• Lazy Evaluation and directed acyclic graphs (DAGs)
• Caching and persisting RDDs

Module 7. Spark SQL & DataFrames

• Spark SQL: Structured data processing
• DataFrames: A distributed collection of data organized into named columns
• Creating DataFrames from RDDs and various data sources
• Using SQL queries on DataFrames
• Performance optimization with Catalyst Optimizer and Tungsten Engine

Module 8. Data Ingestion with Apache Sqoop & Flume

• Apache Sqoop: Ingesting structured data from RDBMS
• Sqoop commands: import, export
• Apache Flume: Ingesting unstructured data from various sources
• Building a simple Flume agent for log data
• Differentiating between Sqoop and Flume

Module 9. NoSQL Databases: HBase & Cassandra

• HBase: A distributed, scalable, big data store on HDFS
• HBase Architecture: Master, Region Servers
• Apache Cassandra: A distributed, wide-column store
• The CAP theorem and how Cassandra fits
• Use cases for HBase and Cassandra in the big data stack

Module 10. Data Warehousing with Apache Hive

• Apache Hive: Data warehouse infrastructure on Hadoop
• HiveQL: A SQL-like query language
• Hive Architecture: Metastore, Driver, Execution Engine
• Creating tables and loading data in Hive
• Hive vs. Spark SQL

Module 11. Workflow Orchestration with Apache Oozie

• Apache Oozie: A workflow scheduler system for Hadoop jobs
• Oozie Workflow, Coordinator, and Bundle jobs
• Defining a workflow in XML
• Scheduling and monitoring jobs with Oozie
• The limitations of Oozie and the rise of Airflow (conceptual)

Module 12. Data Streaming with Apache Kafka

• Apache Kafka: A distributed event streaming platform
• Kafka Concepts: Topics, Producers, Consumers, Brokers
• Setting up a basic Kafka cluster
• Using Kafka for real-time data ingestion
• Introduction to Spark Streaming

Module 13. Data Governance & Security

• Data Governance: The importance of data quality and lineage
• Hadoop Security: Authentication (Kerberos), Authorization (Sentry/Ranger)
• Data encryption in HDFS
• Best practices for securing a big data platform
• Auditing and monitoring

Module 14. Real-World Case Study & Capstone Project

• Project Overview: An end-to-end big data pipeline
• Ingestion: Using a tool to ingest data
• Processing: Using Spark to clean and transform the data
• Storage: Loading the processed data into HDFS or a data warehouse
• Analysis: Running Hive or Spark SQL queries
• Building a final dashboard or report

Module 15. Big Data Ecosystem Trends

• The evolution of the Hadoop ecosystem
• Cloud-native big data services (E.g., AWS EMR, GCP Dataproc)
• The rise of Data Lakehouses
• MLOps and big data
• Continuous learning and staying updated.

Training Approach

This course will be delivered by our skilled trainers who have vast knowledge and experience as expert professionals in the fields. The course is taught in English and through a mix of theory, practical activities, group discussion and case studies. Course manuals and additional training materials will be provided to the participants upon completion of the training.

Tailor-Made Course

This course can also be tailor-made to meet organization requirement. For further inquiries, please contact us on: Email: info@skillsforafrica.org, training@skillsforafrica.org Tel: +254 702 249 449

Training Venue

The training will be held at our Skills for Africa Training Institute Training Centre. We also offer training for a group at requested location all over the world. The course fee covers the course tuition, training materials, two break refreshments, and buffet lunch.

Visa application, travel expenses, airport transfers, dinners, accommodation, insurance, and other personal expenses are catered by the participant

Certification

Participants will be issued with Skills for Africa Training Institute certificate upon completion of this course.

Airport Pickup and Accommodation

Airport pickup and accommodation is arranged upon request. For booking contact our Training Coordinator through Email: info@skillsforafrica.org, training@skillsforafrica.org Tel: +254 702 249 449

Terms of Payment: Unless otherwise agreed between the two parties’ payment of the course fee should be done 7 working days before commencement of the training.