Hadoop: Menjalankan MapReduce Job - Revision history

Onnowpurbo: /* Menjalankan MapReduce Job */

2015-11-10T01:37:14Z

Menjalankan MapReduce Job

← Older revision		Revision as of 01:37, 10 November 2015
Line 49:		Line 49:

	start-all.sh		start-all.sh
			cd /usr/local/hadoop
	hadoop jar ./share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.1.jar pi 2 5		hadoop jar ./share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.1.jar pi 2 5

Onnowpurbo: /* Perintah HDFS */

2015-11-09T23:06:00Z

Perintah HDFS

← Older revision		Revision as of 23:06, 9 November 2015
Line 154:		Line 154:


			==Monitor Job & Task==


			* NameNode daemon: http://hdnode01:50070
			* JobTracker daemon: http://hdnode01:50030
			* TaskTracker daemon: http://hdnode01:50060

	==Referensi==		==Referensi==

	* http://www.bogotobogo.com/Hadoop/BigData_hadoop_Running_MapReduce_Job.php		* http://www.bogotobogo.com/Hadoop/BigData_hadoop_Running_MapReduce_Job.php

Onnowpurbo at 09:37, 9 November 2015

2015-11-09T09:37:40Z

← Older revision		Revision as of 09:37, 9 November 2015
Line 1:		Line 1:
	Sumber: http://www.bogotobogo.com/Hadoop/BigData_hadoop_Running_MapReduce_Job.php		Sumber: http://www.bogotobogo.com/Hadoop/BigData_hadoop_Running_MapReduce_Job.php



			==Jalankan Hadoop==

			Sebelum melakukan job MapReduce, jangan lupa jalankan Hadoop

			start-all.sh

Onnowpurbo: /* Menjalankan MapReduce Job */

2015-11-09T09:36:33Z

Menjalankan MapReduce Job

@@ Line 79: / Line 79: @@
   Job Finished in 5.541 seconds
   Estimated value of Pi is 3.14280000000000000000
 ==Hadoop FileSystem (HDFS)==

Onnowpurbo: /* Menjalankan MapReduce Job */

2015-11-09T09:34:23Z

Menjalankan MapReduce Job

← Older revision		Revision as of 09:34, 9 November 2015
Line 42:		Line 42:
	start-all.sh		start-all.sh
	hadoop jar ./share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.1.jar pi 2 5		hadoop jar ./share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.1.jar pi 2 5


	Number of Maps = 2		Number of Maps = 2
Line 70:		Line 69:
	Job Finished in 3.729 seconds		Job Finished in 3.729 seconds
	Estimated value of Pi is 3.60000000000000000000		Estimated value of Pi is 3.60000000000000000000


			Kalau mau iseng coba naikan jumlah map & sample per map, misalnya

			hadoop jar ./share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.1.jar pi 20 1000

			Hasilnya akan lebih presisi

			Job Finished in 5.541 seconds
			Estimated value of Pi is 3.14280000000000000000

	==Hadoop FileSystem (HDFS)==		==Hadoop FileSystem (HDFS)==

Onnowpurbo: /* Menjalankan MapReduce Job */

2015-11-09T09:30:46Z

Menjalankan MapReduce Job

← Older revision		Revision as of 09:30, 9 November 2015
Line 42:		Line 42:
	start-all.sh		start-all.sh
	hadoop jar ./share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.1.jar pi 2 5		hadoop jar ./share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.1.jar pi 2 5


	Number of Maps = 2		Number of Maps = 2
	Samples per Map = 5		Samples per Map = 5
	14/07/~~14 01~~:28:02 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable		15/11/09 16:28:55 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
	Wrote input for Map #0		Wrote input for Map #0
	Wrote input for Map #1		Wrote input for Map #1
	Starting Job		Starting Job
	14/07/~~14 01~~:28:07 INFO Configuration.deprecation: session.id is deprecated. Instead, use dfs.metrics.session-id		15/11/09 16:28:59 INFO Configuration.deprecation: session.id is deprecated. Instead, use dfs.metrics.session-id
	14/07/~~14 01~~:28:07 INFO jvm.JvmMetrics: Initializing JVM Metrics with processName=JobTracker, sessionId=		15/11/09 16:28:59 INFO jvm.JvmMetrics: Initializing JVM Metrics with processName=JobTracker, sessionId=
	14/07/~~14 01~~:28:07 INFO input.FileInputFormat: Total input paths to process : 2		15/11/09 16:29:00 INFO input.FileInputFormat: Total input paths to process : 2
	14/07/~~14 01~~:28:07 INFO mapreduce.JobSubmitter: number of splits:2		15/11/09 16:29:00 INFO mapreduce.JobSubmitter: number of splits:2
	~~14/07/14 01:28:09 INFO mapreduce~~.~~JobSubmitter: Submitting tokens for job:~~ ~~job_local1228885165_0001~~		..
	...		..
	File Input Format Counters		..
	Bytes Read=236		Shuffle Errors
			BAD_ID=0
			CONNECTION=0
			IO_ERROR=0
			WRONG_LENGTH=0
			WRONG_MAP=0
			WRONG_REDUCE=0
			File Input Format Counters
			Bytes Read=236
	File Output Format Counters		File Output Format Counters
	Bytes Written=97		Bytes Written=97
	Job Finished in 6.~~072~~ seconds		Job Finished in 3.729 seconds
	Estimated value of Pi is 3.60000000000000000000		Estimated value of Pi is 3.60000000000000000000

Onnowpurbo at 09:28, 9 November 2015

2015-11-09T09:28:38Z

← Older revision		Revision as of 09:28, 9 November 2015
Line 40:		Line 40:
	Jalankan		Jalankan

			start-all.sh
	hadoop jar ./share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.1.jar pi 2 5		hadoop jar ./share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.1.jar pi 2 5

Onnowpurbo: /* Menjalankan MapReduce Job */

2015-11-09T09:05:23Z

Menjalankan MapReduce Job

← Older revision		Revision as of 09:05, 9 November 2015
Line 36:		Line 36:
	ls		ls

	bin include libexec ~~logs README~~.txt ~~share~~		bin etc include lib libexec LICENSE.txt logs NOTICE.txt README.txt sbin share
	~~etc lib LICENSE~~.txt ~~NOTICE~~.txt sbin

	Jalankan		Jalankan

	hadoop jar ./share/hadoop/mapreduce/hadoop-mapreduce-examples-2.4.1.jar pi 2 5		hadoop jar ./share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.1.jar pi 2 5

	Number of Maps = 2		Number of Maps = 2
Line 61:		Line 60:
	Job Finished in 6.072 seconds		Job Finished in 6.072 seconds
	Estimated value of Pi is 3.60000000000000000000		Estimated value of Pi is 3.60000000000000000000




	==Hadoop FileSystem (HDFS)==		==Hadoop FileSystem (HDFS)==

Onnowpurbo: /* Hadoop Setup for Development */

2015-11-09T08:00:45Z

Hadoop Setup for Development

← Older revision		Revision as of 08:00, 9 November 2015
Line 125:		Line 125:


	~~==Hadoop Setup for Development==~~

	~~HadoopSetup.png~~

	~~Picture source : Hadoop MapReduce Fundamentals.~~

	~~Throughout my tutorials on Hadoop Echo Systems, I used:~~

	~~Hadoop Binaries - Local (Linux), Cloudera's Demo VM, and AWS for Cloud.~~
	~~Data Storage - Local (HDFS Pseudo-distributed, single-node) and Cloud.~~
	~~MapReduce - Both Local and Cloud.~~





	~~Ways to MapReduce~~

	~~Java is the most common language to use, but other languages can be used:~~
	~~WayToMapReduces.png~~

	~~Picture source : Hadoop MapReduce Fundamentals.~~

	==Referensi==		==Referensi==

	* http://www.bogotobogo.com/Hadoop/BigData_hadoop_Running_MapReduce_Job.php		* http://www.bogotobogo.com/Hadoop/BigData_hadoop_Running_MapReduce_Job.php

Onnowpurbo: /* Hadoop FileSystem (HDFS) */

2015-11-09T07:59:35Z

Hadoop FileSystem (HDFS)

← Older revision		Revision as of 07:59, 9 November 2015
Line 67:		Line 67:
	==Hadoop FileSystem (HDFS)==		==Hadoop FileSystem (HDFS)==

	~~Files are stored in the~~ Hadoop Distributed File System (HDFS). ~~Suppose we're going to store a~~ file ~~called~~ data.txt in HDFS.		File disimpan dalam Hadoop Distributed File System (HDFS). Misalkan kita akan menyimpan file bernama data.txt di HDFS.

	~~This file is~~ 160 ~~megabytes~~. ~~When a~~ file ~~is loaded into~~ HDFS, ~~it's split into chunks which are called blocks~~. ~~The~~ default ~~size of each block is~~ 64 ~~megabytes~~. ~~Each block is given a unique name~~, ~~which is~~ blk, ~~an underscore~~, ~~and a large number~~. ~~In our case~~, ~~the first block is~~ 64 ~~megabytes~~. ~~The second block is~~ 64 ~~megabytes~~. ~~The third block is the remaining~~ 32 ~~megabytes~~, ~~to make up our~~ 160 megabyte ~~file~~.		File ini adalah 160 megabyte. Ketika sebuah file dimuat ke HDFS, itu dibagi menjadi potongan yang disebut blok. Ukuran default setiap blok adalah 64 megabyte. Setiap blok diberi nama yang unik, yang merupakan blk, garis bawah, dan sejumlah besar. Dalam kasus kami, blok pertama adalah 64 megabyte. Blok kedua adalah 64 megabyte. Blok ketiga adalah sisa 32 megabyte, untuk membuat file yang 160 megabyte.

	[[Image:HDFS Cloud.png\|center\|200px\|thumb]]		[[Image:HDFS Cloud.png\|center\|200px\|thumb]]

			Saat file diupload ke HDFS, setiap blok akan bisa disimpan di salah satu node di cluster. Ada Daemon berjalan pada masing-masing mesin dalam cluster, yang disebut DataNode. Sekarang, kita perlu tahu mana blok membuat file asli. Dan itu ditangani oleh mesin terpisah, menjalankan Daemon yang disebut NameNode. Informasi yang disimpan pada NameNode dikenal sebagai Metadata.

	As the file is uploaded to HDFS, each block will get stored on one node in the cluster. There's a Daemon running on each of the machines in the cluster, and it is called the DataNode. Now, we need to know which blocks make up the original file. And that's handled by a separate machine, running the Daemon called the NameNode. The information stored on the NameNode is known as the Metadata.

			==Perintah HDFS==

	~~HDFS Commands~~		Saat Hadoop jalan, mari membuat hdfsTest.txt di home directory kita:

	~~While Hadoop is running, let's create~~ hdfsTest.txt ~~in our home directory:~~		echo "hdfs test" > hdfsTest.txt

	~~hduser@k~~:~~~$ echo "hdfs test" > hdfsTest.txt~~		Kemudian, kita ingin membuat Home Directory di HDFS :

	~~Then, we want to create Home Directory in HDFS :~~		hadoop fs -mkdir -p /user/hduser

	~~hduser@ubuntu~~:~~~$ hadoop fs -mkdir -p /user/hduser~~		kita dapat mengcopy file hdfsTest.txt dari local disk ke user directory di HDFS:

	~~We can copy file~~ hdfsTest.txt ~~from local disk to the user's directory in HDFS:~~
			hadoop fs -copyFromLocal hdfsTest.txt hdfsTest.txt

			Kita juga dapat menggunakan put selain copyFromLocal:
	~~hduser@ubuntu~~:~~~$ hadoop fs -copyFromLocal hdfsTest.txt hdfsTest.txt~~

	~~We could have used~~ put ~~instead of copyFromLocal:~~		hadoop fs -put hdfsTest.txt

	~~hduser@ubuntu~~:~~~$ hadoop fs -put hdfsTest.txt~~		Lihat isi directory dari user home directory di HDFS:

	~~Get a directory listing of the user's home directory in HDFS:~~		hadoop fs -ls

	~~hduser@k:~$ hadoop fs -ls~~		Found 1 items
	Found 1 items		-rw-r--r-- 1 hduser supergroup 5 2014-07-14 01:49 hdfsTest.txt
	-rw-r--r-- 1 hduser supergroup 5 2014-07-14 01:49 hdfsTest.txt

	~~If we want to display the contents of the~~ HDFS ~~file~~ /user/hduser/hdfsTest.txt:		Jika kita ingin melihat isi file HDFS /user/hduser/hdfsTest.txt:


	~~hduser@ubuntu:~$~~ hadoop fs -cat /user/hduser/hdfsTest.txt		hadoop fs -cat /user/hduser/hdfsTest.txt

			Kita juga dapat mengcopy file ke local disk dari HDFS, dimakan sebagai hdfsTest2.txt :

	~~copy that file to the local disk from HDFS, named as~~ hdfsTest2.txt :		hadoop fs -copyToLocal /user/hduser/hdfsTest.txt hdfsTest2.txt

	~~hduser@k:~$ hadoop fs -copyToLocal /user/hduser/hdfsTest~~.txt ~~hdfsTest2~~.txt		ls
			hdfsTest2.txt hdfsTest.txt

	~~hduser@k~~:~~~$ ls~~		Untuk men-delete file dari Hadoop HDFS:
	~~hdfsTest2.txt hdfsTest.txt~~

	~~To delete the file from Hadoop HDFS:~~		hadoop fs -rm hdfsTest.txt

	~~hduser@k:~$~~ hadoop fs -~~rm hdfsTest.txt~~		hadoop fs -ls

	~~hduser@k:~$ hadoop fs -ls~~
	~~hduser@k:~$~~


Line 125:		Line 125:


	Hadoop Setup for Development		==Hadoop Setup for Development==

	HadoopSetup.png		HadoopSetup.png