Hadoop: Perintah Shell

Sumber: http://www.bogotobogo.com/Hadoop/BigData_Hadoop_fs_commands_list.php

HDFS shell commands

Hadoop Distributed File System (HDFS) is designed to reliably store very large files across machines in a large cluster.

Hadoop file system (fs) shell commands are used to perform various file operations such as copying a file, viewing the contents of the file, changing ownership of files, changing permissions, creating directories etc.

In this tutorial, we'll see basic Hadoop HDFS operations managed through shell commands which are useful to manage files on HDFS clusters.

hadoop fs -ls

To list the contents of a directory:

$ hadoop fs -ls /user

Found 1 items
drwxr-xr-x   - hduser supergroup          0 2015-11-09 16:35 /user/hduser

upload: hadoop fs -cat

To see contents of a file:

$ hadoop fs -cat /user/hdfs/dir1/myfile This is myfile

hadoop fs -mkdir

To create a directory in HDFS:

$  hadoop fs -mkdir /user/hdfs/dir1
mkdir: Permission denied: user=cloudera, access=WRITE, inode="/user/hdfs":hdfs:supergroup:drwxr-xr-x

$ sudo su hdfs
bash-4.1$ hadoop fs -mkdir /user/hdfs/dir1

upload: hadoop fs -put

To copy src files from local file system to the Hadoop data file system:

$ hadoop fs -put <local_src> <HDFS_dest_Path>

bash-4.1$ whoami
hdfs

bash-4.1$ hadoop fs -put myfile /user/hdfs/dir1

download: hadoop fs -get

To copy files from the Hadoop data file system to the local file system:

$ hadoop fs -get /user/hdfs/dir1/myfile .

copy: hadoop fs -cp

To copy a file from source to destination:

bash-4.1$ hadoop fs -cp /user/hdfs/dir1/myfile /user/hdfs/dir2

bash-4.1$ hadoop fs -ls /user/hdfs/dir2/myfile
-rw-r--r--   1 hdfs supergroup          7 2015-03-29 11:43 /user/hdfs/dir2/myfile

copyFromLocal: hadoop fs -copyFromLocal

To copy a file from Local file system to HDFS

hadoop fs -copyFromLocal <localsrc> URI

bash-4.1$ pwd
/home/cloudera/workspace/temp

bash-4.1$ ls
myfile	myfile2

bash-4.1$ hadoop fs -copyFromLocal myfile2 /user/hdfs/dir2/

bash-4.1$ hadoop fs -cat /user/hdfs/dir2/myfile2
This is myfile2

copyToLocal: hadoop fs -copyToLocal

To copy a file to Local file system from HDFS

$ pwd
/home/cloudera/workspace/temp

$ ls
myfile  myfile2

$ hadoop fs -copyToLocal /user/hdfs/dir1/myfile myfile3

$ ls
myfile  myfile2  myfile3

hadoop fs -rm

To remove a file from HDFS

bash-4.1$ hadoop fs -ls /user/hdfs/dir2 Found 2 items -rw-r--r-- 1 hdfs supergroup 7 2015-03-29 11:43 /user/hdfs/dir2/myfile -rw-r--r-- 1 hdfs supergroup 16 2015-03-29 11:52 /user/hdfs/dir2/myfile2

bash-4.1$ hadoop fs -rm /user/hdfs/dir2/myfile2

bash-4.1$ hadoop fs -ls /user/hdfs/dir2 -rw-r--r-- 1 hdfs supergroup 7 2015-03-29 11:43 /user/hdfs/dir2/myfile

directory remove: hadoop fs -rm -r

To remove a directory from HDFS

bash-4.1$ hadoop fs -ls /user/hdfs/ Found 3 items drwxr-xr-x - hdfs supergroup 0 2015-03-28 14:08 /user/hdfs/.Trash drwxr-xr-x - hdfs supergroup 0 2015-03-29 10:55 /user/hdfs/dir1 drwxr-xr-x - hdfs supergroup 0 2015-03-29 12:12 /user/hdfs/dir2

bash-4.1$ hadoop fs -rm -r /user/hdfs/dir2

bash-4.1$ hadoop fs -ls /user/hdfs/ drwxr-xr-x - hdfs supergroup 0 2015-03-28 14:08 /user/hdfs/.Trash drwxr-xr-x - hdfs supergroup 0 2015-03-29 10:55 /user/hdfs/dir1

Referensi

• http://www.bogotobogo.com/Hadoop/BigData_Hadoop_fs_commands_list.php