List of frequently used hadoop commands

1. Print the Hadoop version 
   hadoop version
2. 
   
3. List the contents of the root directory in HDFS
   hadoop fs -ls /hadoop fs -ls hdfs:/
4. 
   
5. Create a new directory named “hadoop” below the /user/training directory in HDFS.
   hadoop fs -mkdir /user/training/hadoop
6. Delete a file ‘customers’ from the “retail” directory.
   hadoop fs -rm hadoop/retail/customers
7. Delete all files from the “retail” directory using a wildcard.
   hadoop fs -rm hadoop/retail/*
8. Remove the entire retail directory and all of its contents in HDFS.
   hadoop fs -rm -r hadoop/retail
9. To empty the trash
   hadoop fs -expunge
10. 
    
11. Add the purchases.txt file from the local directory named “/home/training/” to the hadoop directory you created in HDFS
    hadoop fs -copyFromLocal c:/purchases.txt /hadoop/
12. Add the purchases.txt file from “hadoop” directory which is present in HDFS directory to the directory “data” which is present in your local directory
    hadoop fs -copyToLocal /hadoop/purchases.txt  D:/home/training/data
13. ‘-get’ command can be used alternaively to ‘-copyToLocal’ command
    hadoop fs -get hadoop/sample.txt /home/training/
14. Add a sample text file from the local directory named “data” to the new directory
    hadoop fs -put c:/sample.txt /user/training/hadoop
15. Add the entire local directory called “retail” to the  /user/training directory in HDFS.
    hadoop fs -put c:/data/retail /user/training/hadoop
16.  cp is used to copy files between directories present in HDFS
    hadoop fs -cp /user/training/*.txt /user/training/hadoop
17.  Move a directory from one location to other present in HDFS
    hadoop fs -mv hadoop apache_hadoop
18. 
    
19. To view the contents of your text file purchases.txt which is present in your hadoop directory.
    hadoop fs -cat /hadoop/purchases.txt
20. Display last kilobyte of the file “purchases.txt” to stdout.
    hadoop fs -tail hadoop/purchases.txt
21.  
22. Default file permissions are 666 in HDFS Use ‘-chmod’ command to change permissions of a file
    hadoop fs -ls hadoop/purchases.txt
    sudo -u hdfs hadoop fs -chmod 600 hadoop/purchases.txt
23. Default names of owner and group are training,training
    # Use ‘-chown’ to change owner name and group name simultaneously
    hadoop fs -ls hadoop/purchases.txt
    sudo -u hdfs hadoop fs -chown root:root hadoop/purchases.txt
24. Default name of group is training
    # Use ‘-chgrp’ command to change group name
    hadoop fs -ls hadoop/purchases.txt
    sudo -u hdfs hadoop fs -chgrp training hadoop/purchases.txt
25.  
26. Default replication factor to a file is 3.
    # Use ‘-setrep’ command to change replication factor of a file
    hadoop fs -setrep -w 2 apache_hadoop/sample.txt
27.  
28. Copy a directory from one node in the cluster to another node
    # Use -distcp command to copy,
    # -overwrite option to overwrite in an existing files
    # -update command to synchronize both directories
    hadoop fs -distcp hdfs://namenodeA/apache_hadoop hdfs://namenodeB/hadoop
29.  
30. Command to make the name node leave safe mode
    hadoop fs -expunge
    sudo -u hdfs dfsadmin -safemode leave
31.  
32. See how much space this directory occupies in HDFS.
    hadoop fs -du -s -h hadoop/retail
33. Report the amount of space used and available on currently mounted filesystem
    hadoop fs -df hdfs:/
34.  
35. Count the number of directories,files and bytes under the paths that match the specified file pattern
    hadoop fs -count hdfs:/
36.  
37. Run a DFS filesystem checking utility
    hadoop fsck /
38.  
39. Format the namenode:
    hadoop namenode -format
40. Starting Secondary namenode:
    hadoop secondarynamenode
41. Run namenode:
    hadoop namenode
42. Run data node:
    hadoop datanode
43. Cluster Balancing:
    hadoop balancer
44.  
45. Last but not least, always ask for help!
    hadoop fs -help 
46.  
47. List all the hadoop file system shell commands
    hadoop fs

Hadoop / HDFS Commands

HDFS Command Syntax Overview:
hadoop fs : Ex.: hadoop fs -ls
hadoop version : check hadoop installed properly
HELP:
help [cmd]: hopefully this is self-describing

Inspect files:
-ls/lsr : list all files in (hadoop fs -ls /)
-cat : print on stdout
-tail [-f] : output the last part of the
-test : return attributes of file and directory
-touchz : create new emty file size 0
-du/dus : show space utilization
-count : no. of directories, files, and bytes
-setrep : (-r) change the replication factor of file/directory
-stat : info about the specified path

Create/remove files:
-mkdir : create a directory
-mv : move (rename) files
-cp : copy files
-rm/rmr : remove files

Copy/Put files from remote m/c into the HADOOP cluster:
-copyFromLocal : copy a local file to the HDFS
-copyToLocal : copy a file on the HDFS to the local disk
-cp : copies one or more files
-get : copies files to the local file system
-put : copies files from the local file system
-mv : moves one or more files

Hadoop Namenode Commands:
hadoop namenode -format: Format HDFS filesystem from Namenode
hadoop namenode -upgrade: Upgrade the NameNode
start-dfs.sh Start: HDFS Daemons
stop-dfs.sh Stop: HDFS Daemons
start-mapred.sh: Start: MapReduce Daemons
stop-mapred.sh Stop: MapReduce Daemons
hadoop namenode -recover -force: Recover namenode metadata after a cluster failure (may lose data) 

Hadoop Configuration Files:
core-site.xml : Parameters for entire Hadoop cluster
hdfs-site.xml : Parameters for HDFS and its clients
mapred-site.xml : Parameters for MapReduce and its clients
yarn-site.xml : Parameters for nodemanager and resource manager
masters : Host machines for secondary Namenode
slaves : List of slave hosts
hadoop-env.sh : Sets ENV variables for Hadoop 
set JAVA_HOME=%JAVA_HOME%
set HADOOP_PREFIX=D:\Hadoop

Hadoop Job Commands
hadoop job -submit : Submit the job
hadoop job -status : Print job status completion percentage
hadoop job -list all : List all jobs
hadoop job -list-active-trackers : List all available TaskTrackers
hadoop job -set-priority : Set priority for a job. Valid priorities : VERY_HIGH, HIGH, NORMAL, LOW, VERY_LOW
hadoop job -kill-task : Kill a task
hadoop job -history : Display job history including job details, failed and killed jobs

Hadoop mradmin Commands
hadoop mradmin -safemode get : Check Job tracker status
hadoop mradmin -refreshQueues : Reload mapreduce configuration
hadoop mradmin -refreshNodes : Reload active TaskTrackers
hadoop mradmin -refreshServiceAcl : Force Jobtracker to reload service ACL
hadoop mradmin -refreshUserToGroupsMappings : Force jobtracker to reload user group mappings

Hadoop fsck Commands
hadoop fsck / : Filesystem check on HDFS
hadoop fsck / -files : Display files during check
hadoop fsck / -files -blocks : Display files and blocks during check
hadoop fsck / -files -blocks -locations : Display files, blocks and its location hadoop fsck / -files -blocks -locations -racks : Display network topology for data-node locations
hadoop fsck -delete : Delete corrupted files
hadoop fsck -move : Move corrupted files to /lost+found directory

Hadoop Balancer Commands
start-balancer.sh : Balance the cluster
hadoop dfsadmin -setBalancerBandwidth : Adjust bandwidth used by the balancer
hadoop balancer -threshold 20 : Limit balancing to only 20% resources in the cluster

Hadoop Safe Mode (Maintenance Mode) Commands
The following dfsadmin commands helps the cluster to enter or leave safe mode, which is also called as maintenance mode.
In this mode, Namenode does not accept any changes to the name space, it does not replicate or delete blocks.
hadoop dfsadmin -safemode enter : Enter safe mode
hadoop dfsadmin -safemode leave : Leave safe mode
hadoop dfsadmin -safemode get : Get the status of mode
hadoop dfsadmin -safemode wait : Wait until HDFS finishes data block replication
hadoop dfsadmin -report : total usage on the cluster

Launching Hadoop Jobs:
hadoop jar [mainClass] args... : Launch job via jar file
hadoop jar com.twitter.scalding.Tool [mainClass] args : A Scalding job is launched using 
mapred job -kill : If you need to kill a map-reduce job  

Commonly Used Administration Commands:
Format the namenode: hadoop namenode -format
Starting Secondary namenode: hadoop secondrynamenode
Run namenode : hadoop namenode
Run data node: hadoop datanode
Cluster Balancing: hadoop balancer
Run MapReduce job tracker node: hadoop jobtracker
Run MapReduce task tracker node: hadoop tasktracker

Start/Stop Yarn (starts resourcemanager and nodemanager)and DFS (Starts namenode and data node) from sbin directory:
start-yarn, stop-yarn
start-dfs, stop-dfs

Start and Stop ALL daemon from sbin directory:
start-all, stop-all

Check All 5 daemons (Namenode,Secoundary Node,Job Tracker, DataNode, Task Tracker ) are up using:
jps

Hadoop2x--Eclipse-plugin :
Download => https://github.com/winghc/hadoop2x-eclipse-plugin/tree/master/release

UBUNTU

Installing Java 8 on Ubuntu
$ sudo add-apt-repository ppa:webupd8team/java
$ sudo apt-get update
$ sudo apt-get install oracle-java8-installer
Verify Installed Java Version : 
$ java -version
Configuring Java Environment
Install this package using following command.
$ sudo apt-get install oracle-java8-set-default

Install "Guest Additions" on Oracle VirtualBox
- Select from the top menu: VirtualBox -> Devices -> Insert Guest Additions CD image
- In Ubuntu open a terminal, navigate to cd folder (usually /media/VBOXADDITIONS*) and run
$ sudo sh ./VBoxLinuxAdditions.run
$ sudo usermod -a -G vboxsf darren
(and replace ‘darren’ with your username)
* NOTE:  the change will only be visible once you logout and login again!
https://www.virtualbox.org/manual/ch04.html

Expanding virtual harddisk to 20GB
VBoxManage.exe modifyhd "C:\Users\jini\VirtualBox VMs\Ubuntu14\Ubuntu14.vdi" --resize 20480

Set Environment Variable @ home directory
gedit .bashrc (at home directory) or gedit /etc/profile
export JAVA_HOME=/usr/lib/jvm/java-8-oracle
export HADOOP_HOME=/usr/local/hadoop
export HADOOP_PREFIX=/usr/local/hadoop
export PATH=$PATH:$HOME/bin:$JAVA_HOME/bin:$HADOOP_HOME/bin
export CLASSPATH=.:$JAVA_HOME/jre/lib:$JAVA_HOME/lib:$JAVA_HOME/lib/tools.jar

SSH Configuration
$ sudo apt-get install openssh-server
$ ssh-keygen -t rsa -P ""
$ cat $HOME/.ssh/id_rsa.pub >> $HOME/.ssh/authorized_keys
$ Verify => ssh localhost /ssh -v localhost /sudo restart ssh


Open CL Terminal =>> Use Ctrl+Alt+t
Clear CL Screen => $ clear
View Env variable => $ echo $JAVA_HOME
Remove directory recursively as  administrator => $ sudo rm -R [Directory name]
List contents => $ ls -al
Copy file to current directory => $ sudo cp /media/sf_mywork/jdk-8u71-linux-i586.tar.gz .
Untar commpressed file => $ sudo tar -zxf jdk-8u71-linux-i586.tar.gz
Rename file => $ sudo mv jdk1.8.0_71 java 
Change permission recursively => $ sudo chmod -R 777 /usr/local/hadoop