About Me

Love JAVA related technologies. Recently researching on Enterprise Integration (SOA and Messaging), Mobility and Big Data. I have working in JAVA related technologies as Software Architect, Enterprise Architect and Software Developer/Engineer for over 11 years. Currently, I am working as Senior Consultant of VMWare Inc.

Wednesday, November 14, 2012

Hadoop (setup in standalone)

1) install Hadoop
2) setup environment variable
setup SSH for a Hadoop cluster

1) Define a common account
create a user level account with no Hadoop management privilegesAssume it is "hadoopUser"

2) Generate SSH key pair
execute this command and following the prompts for additional inputs "ssh-keygen -t rsa"
the public keys are stored in location you have specified

3) Distribute the public key to all nodes (master and slaves)
scp <the location of your public key> hadoopUser@<hostname>:<new location>/master_key

on the target host, execute the following commands
$ mkdir ~/.ssh
$ chmod 700 ~/.ssh
$ mv ~/master_key ~/.ssh/authorized_keys
$ chmod 600 ~/.ssh/authorized_keys

4) Hadoop configuration
In "hadoop-env.sh" add "export JAVA_HOME=/usr/share/jdk"

5) the 3 main configuration files should be empty
  1. core-site.xml
  2. hdfs-site.xml
  3. mapred-site.xml
Hadoop runs completely on local machine and it doesn't launch any of the Hadoop daemons.

No comments: