Apache Spark and Scala Training

Running Spark on a cluster

• What is a Cluster

• Cluster management in Hadoop

• Introducing Amazing Elastic MapReduce

• Partitioning Concepts

• Troubleshooting and managing dependencies

Machine Learning with Spark ML

• Introduction MLLib

• Using MLLib

• Linear Regression with MLLib

Spark Streaming

• Spark Streaming

• The DStream API for Spark Streaming

Graph X

• What is Graph X

• About Pregel

• Breadth-First-Search with Pregel

• Using Pregel API with Spark API

Overview

• Introduction to Big Data

• Big Data Analytics

• Need for Kafka

• What is Kafka

• Kafka Features

• Kafka Concepts

• Kafka Architecture

• Kafka Components

• Zookeeper

• Where is Kafka Used

• Kafka Installation

• Kafka Cluster

• Type of Kafka Clusters

• Configuring Single Node Single Broker Cluster

Kafka Producer

• Configuring Single Node Multi Broker Cluster

• Sending a Message to Kafka

• Producing Keyed and Non-Keyed Messages

• Sending a Message Synchronously & Asynchronously

• Configuring Producers

• Serializers

• Serializing Using Apache Avro

• Partitions

Kafka Consumers

• Consumers and Consumer Groups

• Standalone Consumer

• Consumer Groups and Partition Rebalance

• Creating a Kafka Consumer

• Subscribing to Topics

• The Poll Loop

• Configuring Consumers

• Commits and Offsets

• Rebalance Listeners

• Consuming Records With Specific Offsets

• De-serializers

Kafka Internals

• Cluster Membership

• The Controller

• Replication

• Request Processing

• Physical Storage

• Reliability

• Broker Configuration

• Using Producers in a Reliable System

• Using Consumers in a Reliable System

• Validating System Reliability

• Performance Tuning in Kafka

Cluster Architecture and Administering Kafka

• Use Cases - Cross-Cluster Mirroring

• Multi-Cluster Architectures

• Apache Kafka’s Mirror Maker

• Other Cross-Cluster Mirroring Solutions

• Topic Operations

• Consumer Groups

• Dynamic Configuration Changes

• Partition Management

• Consuming and Producing

• Unsafe Operations

Kafka Monitoring & Kafka Connect

• Considerations When Building Data Pipelines

• Metric Basics

• Kafka Broker Metrics

• Client Monitoring

• Lag Monitoring

• End-to-End Monitoring

• Kafka Connect

• When to Use Kafka Connect

• Kafka Connect Properties

Kafka Stream Processing

• Stream Processing

• Stream-Processing Concepts

• Stream-Processing Design Patterns

• Kafka Streams by Example

• Kafka Streams: Architecture Overview

Kafka Integration with Hadoop, Storm, and Spark

• Apache Hadoop Basics

• Hadoop Configuration

• Kafka Integration with Hadoop

• Apache Storm Basics

• Configuration of Storm

• Integration of Kafka with Storm

• Apache Spark Basics

• Spark Configuration

• Kafka Integration with Spark

Kafka Integration with Flume, Talend and Cassandra

• Flume Basics

• Integration of Kafka with Flume

• Cassandra Basics Such as Key Space and Table Creation

• Integration of Kafka with Cassandra

• Talend Basics

• Integration of Kafka with Talend

Career Guidance and Roadmap

• Apache Hadoop Overview

• Hadoop Distribution File System

• Hadoop MapReduce Overview

• Introduction to PIG

• Prerequisites for Apache PIG

• Exploring use cases for PIG

• History of Apache PIG

• Why you need PIG

• Significance od PIG

• PIG over MapReduce

• When PIG suits the most

• When to avoid PIG

PIG Architecture

• PIG Latin Language

• Running PIG in Different Modes

• PIG Architecture

• GRUNT Shell

• PIG Latin Statements

• Running Pig Scripts

• Utility Commands

Data Models, Operators, and Streaming in PIG

• PIG Data Model - Scarlar Data Type

• PIG Data Model - Complex Data Type

• Arithmetic Operators

• Comparison Operators

• Cast Operators

• Type Construction Operators

• Relation Operators

• Loading and Stroing Operators

• Filtering Operators

• Filtering Operators- Pig Streaming with Python

• Grouping and Joining Operators-

• Sorting Operator

• Combining and Splitting Operators

• Diagnostic Operators

Functions in PIG

• Eval Functions

• Load and Store Funtions

• Tuple and Bag Functions

• String Functions

• Math Function

Advanced Concepts in PIG

• File compression in PIG

• Intermediate Compression

• Pig Unit Testing

• Embedded PIG in JAVA

• Pig Macros

• Import Macros

• Parameter Substitutions

Hadoop Overview

• Course overview

• Big Data Overview

• Hadoop Overview

• HDFS

• Hadoop Ecosystem

• What is a Hadoop Framework

• Type of Hadoop Frameworks

No SQL Databases Hbase

• NoSQL Databases Hbase

• NoSQL Introduction

• HBase Overview

• HBase Architecture

• Data Model

• Connecting to HBase

• HBase Shell

Administration in HBASE

• Introduction

• Learn and Understand Hbase Fault Tolerance

• Hardware Recommendations

• Software Recommendations

• Hbase Deployment at scale

• Installation with Cloudera Manager

• Basic Static Configuration

• Rolling Restarts and Upgrades

• Interacting with HBase

Troubleshooting in Hbase

• Introduction

• Troubleshooting Distributed Clusters

• Learn How To Use the Hbase UI

• Learn How To Use the Metrics

• Learn How To Use the Logs

Troubleshooting in Hbase

• Introduction

• Generating Load & Load Test Tool

• Generating With YCSB

• Region Tuning

• Table Storage Tuning

• Memory Tuning

• Tuning with Failures

• Tuning for Modern Hardware

Apache HBase Operations Continuity

• Introduction

• Corruption: hbck

• Corruption: Other Tools

• Security

• Security Demo

• Snapshots

• Import Export and copy Paste

• Cluster Replication

Apache HBASE Ecosystem

• Introduction

• Hue

• HBase With Apache Phoenix’