Data Engineer- Professional Training Program

Apache Spark:

• Spark architecture and cluster modes

• RDDs vs DataFrames vs Datasets

• Spark SQL: working with structured data

• Transformations and actions

• Spark Streaming: real-time data processing

• Machine learning pipelines with MLlib (intro level)

HDFS (Hadoop Distributed File System):

• Blocks and replication

• NameNode and DataNode

• File storage commands and usage

• Integration with Spark, Hive, Sqoop

YARN:

• YARN architecture

• ResourceManager and NodeManager

• Application lifecycle and scheduling

Apache Mesos:

• Mesos vs YARN

• Resource allocation and scheduling

• Deploying Spark/Flink on Mesos

Relational Databases:

• Database design concepts

• SQL queries (SELECT, JOIN, GROUP BY)

• Indexing, constraints, normalization

• MySQL, PostgreSQL / PostGIS, Oracle

NoSQL Databases:

• Concepts: CAP Theorem, BASE vs ACID

• Key-Value, Document, Columnar types

Cassandra:

• Data model, keyspaces, replication, CQL

• Write-heavy optimizations

MongoDB:

• BSON, collections, indexing, aggregation framework

• CRUD operations and performance tuning

HBase:

• Column-family model, integration with Hadoop

• Use cases for wide-column storage

Apache Hive:

• Hive architecture: Metastore, Driver, Executor

• HiveQL syntax: DDL, DML, Joins, UDFs

• Partitions and Bucketing

• Integration with HDFS and Spark

Snowflake:

• Cloud-based architecture: compute vs storage

• Warehouses, databases, and schemas

• Time Travel, cloning, and zero-copy restore

• Working with semi-structured data (JSON, XML)

• Performance tuning and caching

Apache Kafka:

• Producer-consumer architecture

• Topics, partitions, brokers

• Kafka Streams basics

• Kafka + Spark/Flink integration

Apache Flink:

• Stream vs batch processing

• Windows, time semantics (event vs processing time)

• Stateful operators

• Fault-tolerant streaming with checkpoints

Apache Storm:

• Storm architecture: Spouts and Bolts

• Building topologies

• Real-time analytics pipelines