MapReduce Certification Training

Functions in PIG

• Eval Functions

• Load and Store Functions

• Tuple and Bag Functions

• String Functions

• Math Function

Advanced Concepts in PIG

• File compression in PIG

• Intermediate Compression

• Pig Unit Testing

• Embedded PIG in JAVA

• Pig Macros

• Import Macros

• Parameter Substitutions

Things you will learn:

• Introduction

• Scala

• Using Resilient Distributed Datasets

• Spark SQL, Data Frames, and Data Sets

• Running Spark on a cluster

• Machine Learning with Spark ML

• Spark Streaming

• Graph X

Introduction

• Big Data Overview

• Apache Hadoop Overview

• Hadoop Distribution File System

• Hadoop MapReduce Overview

• Introduction to IntelliJ and Scala

• Installing IntelliJ and Scala

• Apache Spark Overview

• What’s new in Apache Spark 3

Using Resilient Distributed Datasets

• Scala Basics

• Flow control in Scala

• Functions in Scala

• Data Structures in Scala

Using Resilient Distributed Datasets

• The Resilient Distributed Dataset

• Ratings Histogram Example

• Key / Value RDD's, and the Average Friends by Age example

• Filtering RDD's, and the Minimum Temperature by Location Example

• Check Your Results and Implementation Against Mine

Spark SQL, Data Frames, and Data Sets

• Introduction to Spark SQL

• What are Data Frames

• What are Data Sets

• Item-Based Collaborative Filtering in Spark, cache (), and persist ()

Running Spark on a cluster

• What is a Cluster

• Cluster management in Hadoop

• Introducing Amazing Elastic MapReduce

• Partitioning Concepts

• Troubleshooting and managing dependencies

Machine Learning with Spark ML

• Introducing MLLib

• Using MLLib

• Linear Regression with MLLib

Spark Streaming

• Spark Streaming

• The DStream API for Spark Streaming

• Structured Streaming

Graph X

• What is Graph X

• About Pregel

• Breadth-First-Search with Pregel

• Using Pregel API with Spark API

Things you will learn:

• Introduction

• Installing and Configuring HIVE

• Working on HIVE

• HIVE Implementation

Introduction

• Big Data Overview

• Hadoop Overview

• What is a Hadoop Framework

• Types of Hadoop Frameworks

• What is Hive

• Motivation Behind the Tool

• Hive use cases

• Hive Architecture

• Different Modes of HIVE

Installing and Configuring HIVE

• Downloading, installing, and configuring HIVE

• Hive Shell Commands

• Different configuration properties in HIVE

• Beeswax

• Installing and configuring MySQL Database

• Installing Hive Server

Working on HIVE

• Databases in Hive

• Datatypes in Hive

• Schema on Read

• Schema on Write

• Download Datasets

• Internal Tables

• External Tables

• Partition in HIVE

• Bucketing in HIVE

HIVE Implementation

• Hive in Real Time Projects

• Auditing in Hive

• Troubleshooting Infra issues in Hive

• Troubleshooting User issues in Hive

Things you will learn:

• Hadoop Overview

• Sqoop Overview

• Sqoop Import

• Sqoop Export

• Career Guidance and Roadmap

Hadoop Overview

• Course overview

• Big Data Overview

• Hadoop Overview

• HDFS

• YARN Cluster Overview

• Cluster Setup on Google Cloud

• Environment Update

Sqoop Overvi

• Sqoop Introduction

• Why Sqoop

• Sqoop Features

• Flume vs Sqoop

• Sqoop Architecture & Working

• Sqoop Commands.

Sqoop Import

• Managing Target Directories

• Working with Parquet File Format

• Working with Avro File Format

• Working with Different Compressions

• Conditional Imports

• Split-by and Boundary Queries

• Field delimiters

• Incremental Appends

• Sqoop Hive Import

• Sqoop List Tables/Database

Sqoop Export

• Export from HDFS to MySQL

• Export from Hive to MySQL

• Export Avro Compressed to MySQL

• Sqoop with Airflow

Career Guidance and Roadmap

Things you will learn:

• Hadoop Overview

• No SQL Databases HBase

• Administration in HBASE

• Troubleshooting in HBASE

• Tuning in HBASE

• Apache HBase Operations Continuity

• Apache HBASE Ecosystem

• Career Guidance and Roadmap

Hadoop Overview

• Course overview

• Big Data Overview

• Hadoop Overview

• HDFS

• Hadoop Ecosystem

• What is a Hadoop Framework

• Types of Hadoop frameworks

No SQL Databases HBase

• NoSQL Databases HBase

• NoSQL Introduction

• HBase Overview

• HBase Architecture

• Data Model

• Connecting to HBase

• HBase Shell

Administration in HBASE

• Introduction

• Learn and Understand HBase Fault Tolerance

• Hardware Recommendations

• Software Recommendations

• HBase Deployment at Scale

• Installation with Cloudera Manager

• Basic Static Configuration

• Rolling Restarts and Upgrades

• Interacting with HBase

Troubleshooting in HBASE

• Introduction

• Troubleshooting Distributed Clusters

• Learn How to Use the HBase UI

• Learn How to Use the Metrics

• Learn How to Use the Logs

Tuning in HBASE

• Introduction

• Generating Load & Load Test Tool

• Generating With YCSB

• Region Tuning

• Table Storage Tuning

• Memory Tuning

• Tuning with Failures

• Tuning for Modern Hardware

Apache HBase Operations Continuity

• Introduction

• Corruption: hbck

• Corruption: Other Tools

• Security

• Security Demo

• Snapshots

• Import, Export and Copy Table

• Cluster Replication

Apache HBASE Ecosystem

• Introduction

• Hue

• HBase With Apache Phoenix’

Career Guidance and Roadmap

Things you will learn:

• Apache Flume Overview

• Setting up Agents

• Configuring A Multi Agent Flow

• Flume Sinks

• Flume Channels

• Flume Sink Processors

• Flume Interceptors

• Security

• Career Guidance and Roadmap

Overview

• Course overview

• Big Data Overview

• Hadoop Overview

• HDFS

• Hadoop Ecosystem

• What is a Hadoop Framework

• Types of Hadoop frameworks

• Flume Overview

• Architecture

• Data flow mode

• Reliability and Recoverability

Setting up Agents

• Setting up an individual agent

• Configuring individual components

• Wiring the pieces together

• Data ingestion

• Executing Commands

• Network streams

• Setting Multi-Agent Flow

• Consolidation

• Multiplexing the flow

• Configuration

• Defining the flow

• Configuring individual components

• Adding multiple flows in an agent

Configuring A Multi Agent Flow

• Fan out flow

• Flume Sources

• Avro Source, Exec Source

• NetCat Source

• Sequence Generator Source

• Syslog Sources

• Syslog TCP Source

• Syslog UDP Source

• Legacy Sources

• Avro Legacy Source

• Thrift Legacy Source

• Custom Source

Flume Sinks

• HDFS Sink

• Logger Sink

• Avro Sink

• IRC Sink

• File Roll Sink

• Null Sink

• HbaseSinks

• HbaseSink

• AsyncHBaseSink

• Custom Sink

Flume Channels

• Memory Channel

• JDBC Channel

• Recoverable Memory Channel

• File Channel, Pseudo Transaction Channel

• Custom Channel

• Flume Channel Selectors

• Replicating Channel Selector

• Multiplexing Channel Selector

• Custom Channel Selector

Flume Sink Processors

• Default Sink Processor

• Failover Sink Processor

• Load balancing Sink Processor

• Custom Sink Processor

Flume Interceptors

• Timestamp Interceptor

• Host Interceptor

• Flume Properties

• Property

Security

• Monitoring

• Troubleshooting

• Handling agent failures

• Compatibility

• HDFS

• AVRO

Career Guidance and Roadmap

Things you will learn:

• Apache Drill Overview

• Installing & Configuring Drill

• Querying Simple Delimited Data

• Configuration Options

• Understanding Data Types and Functions in Drill

• Working with Dates and Times in Drill

• Analyzing Nested Data with Drill

• Other Data Types

• Connecting Multiple Data Sources and programming languages

• Career Guidance and Roadmap

Overview

• Course overview

• Big Data Overview

• Hadoop Overview

• HDFS

• Hadoop Ecosystem

• What is a Hadoop Framework

• Types of Hadoop frameworks

• Drill Overview

• What does Drill do

• How does Drill work

• Kinds of data which can be queried with Drill

Installing & Configuring Drill

• Comparison of embedded and distributed modes

• Introducing and configuring workspaces

• Demonstrate Drill’s various interfaces

Querying Simple Delimited Data

• SQL fundamentals

• Querying a simple CSV file

• Arrays in Drill

• Accessing columns in Arrays

Configuration Options

• Extracting headers from csv files

• Changing delimiter characters

• Specifying options in a query

Understanding Data Types and Functions in Drill

• Overview of Drill Data Types

• Converting Strings to Numeric Data Types

• Complex Conversions

• Windowing functions

Working with Dates and Times in Drill

• Understanding dates and times in Drill

• Converting strings to dates

• Reformatting dates

• Intervals and date/time arithmetic in Drill

Analyzing Nested Data with Drill

• Issues querying nested data with Drill

• Maps and Arrays in Drill

• Querying deeply nested data in Drill

Other Data Types

• Log files

• HTTPD