Hadoop Command-line

Hadoop Command-line

Create a file on hadoop file1.txt
- localfile path : /home/cloudera/Python-3.5.3/python_code/1st.py
- hdfs path : /user/cloudera/umesh/file1.txt
1. appendToFile
[cloudera@quickstart Python]$ hadoop fs -appendToFile /home/cloudera/Python-3.5.3/python_code/1st.py /user/cloudera/umesh/file1.txt
2. cat
[cloudera@quickstart Python]$ hadoop fs -cat /user/cloudera/umesh/file1.txt
abcd
verma
tania
umesh
print("Hello world")
3. checksum
hadoop fs -checksum URI
[cloudera@quickstart Python]$ hadoop fs -checksum /user/cloudera/umesh/file1.txt 
/user/cloudera/umesh/file1.txt MD5-of-0MD5-of-512CRC32C 00000200000000000000000040ae2cbb8e2213e5724356c838dfb6b1
4. chgrp
Example1: Change group name:sales of a file to other group name:hrgroup. 
chgrp hrgroup file1
Example2: Give access permissions to a command so that the command can be executed by all users belonging to apache-admins
chgrp apache-admins /etc/init.d/httpd
Example3: Change group ownership all the files located in /var/apache to group:apache
chgrp -R apache /var/apache
Example4:Change group ownership forcefully
chgrp -f apache /var/apache
5. chmod
Following are the symbolic representation of three different roles:
u is for user,
g is for group,
and o is for others.
Following are the symbolic representation of three different permissions:
r is for read permission,
w is for write permission,
x is for execute permission.
1. Add single permission to a file/directory
Changing permission to a single set. + symbol means adding permission. For example, do the following to give execute permission for the user irrespective of anything else:
$ chmod u+x filename
2. Add multiple permission to a file/directory
Use comma to separate the multiple permission sets as shown below.
$ chmod u+r,g+x filename
3. Remove permission from a file/directory
Following example removes read and write permission for the user.
$ chmod u-rx filename
4. Change permission for all roles on a file/directory
Following example assigns execute privilege to user, group and others (basically anybody can execute this file).
$ chmod a+x filename
5. Make permission for a file same as another file (using reference)
If you want to change a file permission same as another file, use the reference option as shown below. In this example, file2’s permission will be set exactly same as file1’s permission.
$ chmod --reference=file1 file2
6. Apply the permission to all the files under a directory recursively
Use option -R to change the permission recursively as shown below.
$ chmod -R 755 directory-name/
7. Change execute permission only on the directories (files are not affected)
On a particular directory if you have multiple sub-directories and files, the following command will assign execute permission only to all the sub-directories in the current directory (not the files in the current directory).
$ chmod u+X *
6. chown
chown root file1
chown :[group-name] [file-name]
To change both user and group owner 
chown [new-owner]:[new-group] [file-name]
There might be situations where-in you may want to first cross-check existing owner/group of a file before making any change. So for those cases, you can use the --from command line option. This option requires you to mention the owner/group name that you want to verify.
chown --from=[curr-own]:[curr-group] [new-owner]:[new-group] [filename]
For example:
chown --from=root:himanshu himanshu:root file1
To make the chown command recursively operate on files and directories, use the -R command line option.
chown -R [new-owner]:[new-group] [directory-name-or-path]
7. copyFromLocal
[cloudera@quickstart Python]$ hadoop fs -copyFromLocal vermadir techaltum
8. copyToLocal
[cloudera@quickstart ~]$ hadoop fs -copyToLocal verma/file.txt /home/cloudera/Python-3.5.3/pfile2.txt
9. count
[cloudera@quickstart Python]$ hadoop fs -count -h /user/cloudera/umesh     
10. copy
• hadoop fs -cp /user/hadoop/file1 /user/hadoop/file2
• hadoop fs -cp /user/hadoop/file1 /user/hadoop/file2 /user/hadoop/dir
11. df
Usage: hadoop fs -df [-h] URI [URI …]
Displays free space.
Options:
• The -h option will format file sizes in a “human-readable” fashion (e.g 64.0m instead of 67108864)
Example:
• hadoop dfs -df /user/hadoop/dir1
12. du
Usage: hadoop fs -du [-s] [-h] URI [URI …]
Displays sizes of files and directories contained in the given directory or the length of a file in case it’s just a file.
Options:
• The -s option will result in an aggregate summary of file lengths being displayed, rather than the individual files.
• The -h option will format file sizes in a “human-readable” fashion (e.g 64.0m instead of 67108864)
Example:
• hadoop fs -du /user/hadoop/dir1 /user/hadoop/file1 hdfs://nn.example.com/user/hadoop/dir1
Exit Code: Returns 0 on success and -1 on error.
13. expunge
Usage: hadoop fs -expunge
Empty the Trash. Refer to the HDFS Architecture Guide for more information on the Trash feature.
14. get
Usage: hadoop fs -get [-ignorecrc] [-crc]
Copy files to the local file system. Files that fail the CRC check may be copied with the -ignorecrc option. Files and CRCs may be copied using the -crc option.
Example:
• hadoop fs -get /user/hadoop/file localfile
• hadoop fs -get hdfs://nn.example.com/user/hadoop/file localfile
15. getfacl
Usage: hadoop fs -getfacl [-R] Displays the Access Control Lists (ACLs) of files and directories. If a directory has a default ACL, then getfacl also displays the default ACL.
Options:
• -R: List the ACLs of all files and directories recursively.
• path: File or directory to list.
Examples:
• hadoop fs -getfacl /file
• hadoop fs -getfacl -R /dir
16. getmerge
Usage: hadoop fs -getmerge [-nl]
Takes a source directory and a destination file as input and concatenates files in src into the destination local file. Optionally -nl can be set to enable adding a newline character (LF) at the end of each file.
Examples:
• hadoop fs -getmerge -nl /src /opt/output.txt
• hadoop fs -getmerge -nl /src/file1.txt /src/file2.txt /output.txt
17. ls
Example : hadoop fs -ls /
18. help
Usage: hadoop fs -help
19. mkdir
Example: hadoop fs -mkdir /user/hadoop/dir1 /user/hadoop/dir2
20. mv
Usage: hadoop fs -mv URI [URI …]
Moves files from source to destination.
Example:hadoop fs -mv /user/hadoop/file1 /user/hadoop/file2
21. put
Usage: hadoop fs -put …
Copy single src, or multiple srcs from local file system to the destination file system. 
Example : hadoop fs -put localfile /user/hadoop/hadoopfile
22. rm
Usage: hadoop fs -rm [-f] [-r |-R] [-skipTrash] URI [URI …]
Delete files specified as args.
Options:
• The -f option will not display a diagnostic message or modify the exit status to reflect an error if the file does not exist.
• The -R option deletes the directory and any content under it recursively.
• The -r option is equivalent to -R.
• The -skipTrash option will bypass trash if enabled, and delete the specified file(s) immediately. This can be useful when it is necessary to delete files from an over-quota directory.
23. rmdir
Usage: hadoop fs -rmdir [–ignore-fail-on-non-empty] URI [URI …]
Delete a directory.
Options:
• –ignore-fail-on-non-empty: When using wildcards, do not fail if a directory still contains files.
Example:
• hadoop fs -rmdir /user/hadoop/emptydir
24. tail
Usage: hadoop fs -tail [-f] URI