Hadoop clustering components

Hadoop clustering is composed with following daemons on one server or across multiple servers

• NameNode -- keeps track of the file metadata, which files are in the system and how each file is broken down into blocks
• DataNode -- provides backup store of data blocks and constantly report to NameNode to keep track of metadata update
• Secondary NameNode -- assistant daemon for monitoring the state of a cluster HDFS. It communicates with the NameNode to take snapshots of the HDFS metadata at intervals defined by the cluster configuration
• JobTracker -- liaison between application and Hadoop.  It determines the execution plan by determining which files to process, assign nodes to different tasks, and monitors all tasks as they are running.
• one per Hadoop cluster
• automatic relaunch failed task
• oversees the overall execution of a MapReduce job

• TaskTracker -- slave to the JobTracker
• executes individual tasks that the JobTracker assigns
• one per a slave node
• able to spawn multiple map or reduce tasks in parallel
• send heartbeat to JobTracker

Hadoop commands

Usage: hadoop [--config confdir] COMMAND
where COMMAND is one of:

• namenode                     format format the DFS filesystem
• secondarynamenode     run the DFS secondary namenode
• namenode                     run the DFS namenode
• datanode                      run a DFS datanode
• dfsadmin                      run a DFS admin client
• fsck                             run a DFS filesystem checking utility
• fs                                 run a generic filesystem user client
• balancer                       run a cluster balancing utility
• jobtracker                    run the MapReduce job Tracker node
• pipes                            run a Pipes job
• tasktracker                   run a MapReduce task Tracker node
• job                               manipulate MapReduce jobs
• version                         print the version
• jar <jar>                       run a jar file
• distcp <srcurl> <desturl> copy file or directories recursively
• archive -archiveName NAME <src>* <dest> create a hadoop archive
• daemonlog                     get/set the log level for each daemon
• CLASSNAME              run the class named CLASSNAME

Hadoop (software stack)

Currently there are nine sub-projects in Hadop

• Common - common code
• Avro - serialization and RPC
• MapReduce - computation
• HDFS - storage
• Pig - data flow language
• Hive - data warehousing and query language
• HBase - column-oriented database
• ZooKeeper - coordination service
• Chukwa - data collection and analysis

hadoop useful CLI commands

$ hadoop fs -ls /                           (list all files in HDFS root directory)
$ hadoop job -list                         (find all running MapReduce jobs)
$ for svc in /etc/init.d/hadoop-0.20-*; do sudo $svc start; done    (Start up hadoop cluster)
$ for svc in /etc/init.d/hadoop-0.20-*; do sudo $svc stop; done    (Stop cluster)

(HDFS commands)
http://hadoop.apache.org/docs/r1.0.0/file_system_shell.html

(MapReduce commands)
http://hadoop.apache.org/docs/r1.0.0/commands_manual.html#job

hadoop configuration

Configuration
The Hadroop configs are contained under /etc/hadoop/conf.

Log
/var/log/hadoop (all hadoop daemon log files are resided)
hadoop-hadoop-namenode-<HOSTNAME>.log (NameNode logs)

CentOS add user to the sudoers list

go to /etc/sudoer

put your cursor on the "root    ALL=(ALL)       ALL" and add the following on the next line

replace <username> with your username
"<username>    ALL=(ALL)       ALL"

Hadoop stack (Installation) - redhat

Download hadroop from http://www.cloudera.com/hadoop

Prerequisites
1) JDK1.6 update 8 or newer

Download and install the “bootstrap” RPM
$ sudo -s
$ wget http://archive.cloudera.com/redhat/cdh/cdh3-repository-1.0-1.noarch.rpm
$ rpm -ivh cdh3-repository-1.0-1.noarch.rpm

Import Cloudera's RPM signing key
$ rpm --import \
http://archive.cloudera.com/redhat/cdh/RPM-GPG-KEY-cloudera

Install the pseudo-distributed RPM package and it dependencies: Pig, Hive, and Snappy
$ yum install hadoop-0.20-conf-pseudo hadoop-0.20-native \
hadoop-pig hadoop-hive