Big Data Hadoop Training

Apache Sqoop

• Apache Sqoop

• Sqoop and Its Uses

• Sqoop Processing

• Sqoop Import Process

• Sqoop Connectors

• Importing and Exporting Data from MySQL to HDFS

Apache Sqoop

• Apache Flume

• Flume Model

• Scalability in Flume

• Components in Flume’s Architecture

• Configuring Flume Components

• Ingest Twitter Data

Getting Data into HDFS

• Data Ingestion Overview

• Ingesting Data from External Sources with Flume

• Ingesting Data from Relational Databases with Sqoop

• REST Interfaces

• Best Practices for Importing Data

Apache Kafka

• Apache Kafka

• Aggregating User Activity Using Kafka

• Kafka Data Model

• Partitions

• Apache Kafka Architecture

• Setup Kafka Cluster

• Producer Side API Example

• Consumer Side API

• Consumer Side API Example

• Kafka Connect

Hadoop Clients

• What is a Hadoop Client

• Installing and Configuring Hadoop Clients

• Installing and Configuring Hue

• Hue Authentication and Authorization

Cluster Maintenance

• Checking HDFS Status

• Copying Data between Clusters

• Adding and Removing Cluster Nodes

• Rebalancing the Cluster

• Cluster Upgrading

Cloudera Manager

• The Motivation for Cloudera Manager

• Cloudera Manager Features

• Express and Enterprise Versions

• Cloudera Manager Topology

• Installing Cloudera Manager

• Installing Hadoop Using Cloudera Manager

• Performing Basic Administration Tasks using Cloudera Manager

Cluster Monitoring and Troubleshooting

• General System Monitoring

• Monitoring Hadoop Clusters

• Common Troubleshooting Hadoop Clusters

• Common Misconfigurations

Planning Your Hadoop Cluster

• General Planning Considerations

• Choosing the Right Hardware

• Network Considerations

• Configuring Nodes

• Planning for Cluster Management

Advanced Cluster Configuration

• Advanced Configuration Parameters

• Configuring Hadoop Ports

• Explicitly Including and Excluding Hosts

• Configuring HDFS for Rack Awareness

• Configuring HDFS High Availability

MapReduce Framework

• What is MapReduce

• Basic MapReduce Concepts

• Distributed Processing in MapReduce

• Word Count Example

• Map Execution Phases

• Map Execution Distributed Two Node Environment

• MapReduce Jobs

• Hadoop MapReduce Job Work Interaction

• Setting Up the Environment for MapReduce Development

• Set of Classes

• Creating a New Project

• Advanced MapReduce

• Data Types in Hadoop

• Output formats in MapReduce

• Using Distributed Cache

• Joins in MapReduce

• Replicated Join

Apache PIG

• Introduction to Pig

• Components of Pig

• Pig Data Model

• Pig Interactive Modes

• Pig Operations

• Various Relations Performed by Developers

Apache HIVE

• Introduction to Apache Hive

• Hive SQL over Hadoop MapReduce

• Hive Architecture

• Interfaces to Run Hive Queries

• Running Beeline from Command Line

• Hive Meta Store

• Hive DDL and DML

• Creating New Table

• Data Types

• Validation of Data

• File Format Types

• Data Serialization

• Hive Table and Avro Schema

• Hive Optimization Partitioning Bucketing and Sampling

• Non-Partitioned Table

• Data Insertion

• Dynamic Partitioning in Hive

• Bucketing

• What Do Buckets Do

• Hive Analytics UDF and UDAF

• Other Functions of Hive

No SQL Databases HBase

• NoSQL Databases HBase

• NoSQL Introduction

• HBase Overview

• HBase Architecture

• Data Model

• Connecting to HBase

• HBase Shell

Functional Programming using Scala

• Basics of Functional Programming and Scala

• Introduction to Scala

• Scala Installation

• Functional Programming

• Programming with Scala

• Basic Literals and Arithmetic Programming

• Logical Operators

• Type Inference Classes Objects and Functions in Scala

• Type Inference Functions Anonymous Function and Class

• Collections

• Types of Collections

• Operations on List

• Scala REPL

• Features of Scala REPL

Apache Spark

• Apache Spark Next-Generation Big Data Framework

• History of Spark

• Limitations of MapReduce in Hadoop

• Introduction to Apache Spark

• Components of Spark

• Application of In-memory Processing

• Hadoop Ecosystem vs Spark

• Advantages of Spark

• Spark Architecture

• Spark Cluster in Real World

Data warehouse in Hadoop

• Hadoop and the Data Warehouse

• Hadoop Differentiators

• Data Warehouse Differentiators

• When and Where to Use Which

Augmenting Enterprise Data Warehouse

• Introduction

• RDBMS Strengths

• RDBMS Weaknesses

• Typical RDBMS Scenario

• OLAP Database Limitations

• Using Hadoop to Augment Existing Databases

• Benefits of Hadoop

• Hadoop Trade-offs

Advance Programming in Hadoop

• Advance Programming in Hadoop

Writing MapReduce Program

• A Sample MapReduce Program: Introduction

• Map Reduce: List Processing

• MapReduce Data Flow

• The MapReduce Flow: Introduction

• Basic MapReduce API Concepts

• Putting Mapper & Reducer together in MapReduce

• Our MapReduce Program: Word Count

• Getting Data to the Mapper

• Keys and Values are Objects

• What is Writable Comparable

• Writing MapReduce application in Java

• The Driver

• The Driver: Complete Code

• The Driver: Import Statements

• The Driver: Main Code

• The Driver Class: Main Method

• Sanity Checking the Job’s Invocation

• Configuring the Job with Job Conf

• Creating a New Job Conf Object

• Naming the Job

• Specifying Input and Output Directories

• Specifying the Input Format

• Determining Which Files to Read

• Specifying Final Output with Output Format

• Specify the Classes for Mapper and Reducer

• Specify the Intermediate Data Types

• Specify the Final Output Data Types

• Running the Job

• Reprise: Driver Code

• The Mapper

• The Mapper: Complete Code

• The Mapper: import Statements

• The Mapper: Main Code

• The Map Method

• The map Method: Processing the Line

• Reprise: The Map Method

• The Reducer

• The Reducer: Complete Code

• The Reducer: Import Statements

• The Reducer: Main Code

• The reduce Method

• Processing the Values

• Writing the Final Output

• Reprise: The Reduce Method

• Speeding up Hadoop development by using Eclipse

• Integrated Development Environments

• Using Eclipse

• Writing a MapReduce program

Introduction to Combiner

• The Combiner

• MapReduce Example: Word Count

• Word Count with Combiner

• Specifying a Combiner

• Demonstration: Writing and Implementing a Combiner

Sorting & Searching large data sets

• Introduction

• Sorting

• Sorting as a Speed Test of Hadoop

• Shuffle and Sort in MapReduce

• Searching

Performing a secondary sort

• Secondary Sort: Motivation

• Implementing the Secondary Sort

• Secondary Sort: Example

Indexing data and inverted Index

• Indexing

• Inverted Index Algorithm

• Inverted Index: Data Flow

• Aside: Word Count

Term Frequency - Inverse Document Frequency (TF- IDF)

• Term Frequency Inverse Document Frequency (TF-IDF)

• TF-IDF: Motivation

• TF-IDF: Data Mining Example

• TF-IDF Formally Defined

• Computing TF-IDF

Calculating Word co- occurrences

• Word Co-Occurrence: Motivation

• Word Co-Occurrence: Algorithm