Hadoop: Contoh Program Sederhana: Difference between revisions

From OnnoCenterWiki
Jump to navigationJump to search
Onnowpurbo (talk | contribs)
69,477 edits
Onnowpurbo (talk | contribs)
69,477 edits
New page: Sumber: http://www.drdobbs.com/database/hadoop-writing-and-running-your-first-pr/240153197 Hadoop: Writing and Running Your First Project MapReduce on small datasets can be run e...
 
Onnowpurbo (talk | contribs)
69,477 edits
 
(5 intermediate revisions by the same user not shown)
Line 1: Line 1:
Sumber: http://www.drdobbs.com/database/hadoop-writing-and-running-your-first-pr/240153197
Sumber:
 
* http://www.drdobbs.com/database/hadoop-writing-and-running-your-first-pr/240153197
 
* https://hadoop.apache.org/docs/r1.2.1/mapred_tutorial.html
 
 
 
 
 
Hadoop: Writing and Running Your First Project
 
 
 
MapReduce on small datasets can be run easily and without much coding or fiddling — provided you know what to do. Here's how.
 
In the first part of this series on Apache Hadoop, I explained how MapReduce works at a conceptual level. In this installment, the second of three, I show how to write code that runs on Hadoop — starting with a MapReduce program in Java.
Development Environment
 
To get started, we need Java (Oracle JDK 6 is required), Git, Maven, and Hadoop itself. Download the latest stable release of Apache Hadoop (1.0.4) from the releases page, then extract it to a suitable place. On my laptop:
 
% tar zxf hadoop-1.0.4.tar.gz
% export HADOOP_HOME=$(pwd)/hadoop-1.0.4
% $HADOOP_HOME/bin/hadoop version
 
Hadoop 1.0.4
 
In another directory, checkout the Git repository that accompanies this article:
 
% git clone git://github.com/tomwhite/hadoop-drdobbs.git
% cd hadoop-drdobbs
% mvn install
 
The repository contains a small amount of sample data for testing:
 
% cat data/*.tsv
dobbs 2007 20 18 15
dobbs 2008 22 20 12
doctor 2007 545525 366136 57313
doctor 2008 668666 446034 72694
 
The file contains a few lines from the Google Books Ngram Dataset, which I mentioned  in the first part of the series. To recap, the first line says that the word "dobbs" in books from 2007 occurred 20 times overall, and these occurrences were found on 18 pages in 15 books.
 
==Java MapReduce==
 
Let's write the MapReduce job to find the total count for each word. We start with the map function, which is represented in Java by an instance of org.apache.hadoop.mapreduce.Mapper. The first thing we need to decide about our mapper is the types of the input key-value pairs and the output key-value pairs. The declaration of the Mapper class is:


public class Mapper<KEYIN, VALUEIN, KEYOUT, VALUEOUT>
==Source Code==


Because we are processing text, we use TextInputFormat, which determines the input types for us as LongWritable and Text (both found in the org.apache.hadoop.io package). These Writable types are wrappers around standard Java types (long and String, in this case)  optimized for efficiency of serialization. Authors of MapReduce programs can use the Writable types without worrying about serialization. The only time that you might be exposed to serialization is when writing a custom Writable type. In such cases, it is usually better to use a serialization library, such as Avro.
Contoh source code WordCount.java untuk menghitung jumlah masing-masing kata dari sebuah input set.


Going back to the input type, TextInputFormat presents the input to our mapper as (LongWritable, Text) pairs, like this:
cd ~
vi WordCount.java


(0,  "dobbs 2007 20 18 15")
(20, "dobbs 2008 22 20 12")
(40, "doctor 2007 545525 366136 57313")
(72, "doctor 2008 668666 446034 72694")


The key is the offset within the file, and the value is the content of the line. It is the job of the mapper to extract the word and the number of occurrences, and ignore everything else. Therefore, its output is (word, count) pairs, of type (Text, LongWritable). The signature of the mapper looks like this:


public class ProjectionMapper extends Mapper<LongWritable, Text, Text, LongWritable>
  package org.myorg;
 
   
All that remains is for us to write the implementation of the map() method. The source for the whole mapper class appears in Listing One (ProjectionMapper.java).
 
Listing One: ProjectionMapper.java.
 
 
  package com.tom_e_white.drdobbs.mapreduce;
 
import org.apache.hadoop.io.LongWritable;
  import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Mapper;
 
  import java.io.IOException;
  import java.io.IOException;
 
import java.util.*;
  public class ProjectionMapper extends Mapper<LongWritable, Text, Text, LongWritable> {
  private Text word = new Text();
import org.apache.hadoop.fs.Path;
  private LongWritable count = new LongWritable();
import org.apache.hadoop.conf.*;
 
import org.apache.hadoop.io.*;
  @Override
import org.apache.hadoop.mapred.*;
  protected void map(LongWritable key, Text value, Context context)
import org.apache.hadoop.util.*;
      throws IOException, InterruptedException {
    // value is tab separated values: word, year, occurrences, #books, #pages
  public class WordCount {
    // we project out (word, occurrences) so we can sum over all years
    String[] split = value.toString().split("\t+");
    public static class Map extends MapReduceBase implements Mapper<LongWritable, Text, Text, IntWritable> {
    word.set(split[0]);
      private final static IntWritable one = new IntWritable(1);
    if (split.length > 2) {
      private Text word = new Text();
      try {
        count.set(Long.parseLong(split[2]));
      public void map(LongWritable key, Text value, OutputCollector<Text, IntWritable> output, Reporter reporter) throws IOException {
        context.write(word, count);
        String line = value.toString();
      } catch (NumberFormatException e) {
        StringTokenizer tokenizer = new StringTokenizer(line);
        // cannot parse - ignore
        while (tokenizer.hasMoreTokens()) {
      }
          word.set(tokenizer.nextToken());
    }
          output.collect(word, one);
  }
        }
      }
    }
    public static class Reduce extends MapReduceBase implements Reducer<Text, IntWritable, Text, IntWritable> {
      public void reduce(Text key, Iterator<IntWritable> values, OutputCollector<Text, IntWritable> output, Reporter reporter) throws IOException {
        int sum = 0;
        while (values.hasNext()) {
          sum += values.next().get();
        }
        output.collect(key, new IntWritable(sum));
      }
    }
    public static void main(String[] args) throws Exception {
      JobConf conf = new JobConf(WordCount.class);
      conf.setJobName("wordcount");  
      conf.setOutputKeyClass(Text.class);
      conf.setOutputValueClass(IntWritable.class);
      conf.setMapperClass(Map.class);
      conf.setCombinerClass(Reduce.class);
      conf.setReducerClass(Reduce.class);
      conf.setInputFormat(TextInputFormat.class);
      conf.setOutputFormat(TextOutputFormat.class);
      FileInputFormat.setInputPaths(conf, new Path(args[0]));
      FileOutputFormat.setOutputPath(conf, new Path(args[1]));
      JobClient.runJob(conf);
    }
  }
  }


There are a few things to note about this code. First, there are two instance variables, word and count, which are used to store the map output key and value. The map() method is called once per input record, so it pays to avoid unnecessary object creation. The body of map() is straightforward: It splits the tab-separated input line into fields, and uses the first field as the word, and the third as the count. The map output is written using the write method in Context. For simplicity, this code ignores lines with an occurrence field that is not a number, but there are other actions you could take, such as incrementing a MapReduce counter to track how many lines it affects (see the getCounter() method on Context for details).
==Compile==


Running through our tiny dataset, the map output looks like this:
Asumsinya HADOOP_HOME adalah root instalasi dan HADOOP_VERSION adalah versi Hadoop yang di install, compile WordCount.java dan buat jar:


  ("dobbs", 20)
  export HADOOP_HOME=/usr/local/hadoop/share/hadoop/common
  ("dobbs", 22)
  export HADOOP_VERSION=2.7.1
("doctor", 545525)
("doctor", 668666)


As I discussed in the first part of the series, Hadoop transforms the map output so that the values are brought together for a given key, in a process called the shuffle. In our abstract representation, the input to the reduce step looks like this:
mkdir wordcount_classes
javac -classpath ${HADOOP_HOME}/hadoop-common-${HADOOP_VERSION}.jar -d wordcount_classes WordCount.java
jar -cvf /usr/joe/wordcount.jar -C wordcount_classes/ .


("dobbs", [20, 22])
==Penggunaan==
("doctor", [545525, 668666])


All our reduce implementation has to do is sum the counts. We need an implementation of org.apache.hadoop.mapreduce.Reducer with the following signature:
Asumsi
public class LongSumReducer extends Reducer<
        Text, LongWritable, Text, LongWritable>
 
We could write the class ourselves, but we don't need to because Hadoop comes with an implementation, shown in Listing Two (LongSumReducer.java).
Listing Two LongSumReducer.java (code from Apache Hadoop project).
 
package org.apache.hadoop.mapreduce.lib.reduce;
 
import java.io.IOException;
import org.apache.hadoop.io.LongWritable;
import org.apache.hadoop.mapreduce.Reducer;
 
public class LongSumReducer<KEY> extends Reducer<KEY, LongWritable,
                                                  KEY,LongWritable> {
 
  private LongWritable result = new LongWritable();
 
  public void reduce(KEY key, Iterable<LongWritable> values,
                      Context context) throws IOException, InterruptedException {
    long sum = 0;
    for (LongWritable val : values) {
      sum += val.get();
    }
    result.set(sum);
    context.write(key, result);
  }
 
}


Notice that the reduce() method signature is different from map() because it has an iterator over the values, rather than a single value. This reflects the grouping that the framework performs on the values for a key. In LongSumReducer, the implementation is very simple: It sums the values, then writes the total out using the same key as the input.
/usr/joe/wordcount/input - input directory di HDFS
/usr/joe/wordcount/output - output directory di HDFS


The output of the reducer will be:
Sample text-files as input:


  ("dobbs", 42)
  bin/hadoop dfs -ls /usr/joe/wordcount/input/
  ("doctor", 1214191)
  /usr/joe/wordcount/input/file01
/usr/joe/wordcount/input/file02


bin/hadoop dfs -cat /usr/joe/wordcount/input/file01
Hello World Bye World


bin/hadoop dfs -cat /usr/joe/wordcount/input/file02
Hello Hadoop Goodbye Hadoop


Jalankan aplikasi


bin/hadoop jar /usr/joe/wordcount.jar org.myorg.WordCount /usr/joe/wordcount/input /usr/joe/wordcount/output


Output:


bin/hadoop dfs -cat /usr/joe/wordcount/output/part-00000
Bye 1
Goodbye 1
Hadoop 2
Hello 2
World 2




Line 157: Line 117:


* http://www.drdobbs.com/database/hadoop-writing-and-running-your-first-pr/240153197
* http://www.drdobbs.com/database/hadoop-writing-and-running-your-first-pr/240153197
* http://www.drdobbs.com/database/hadoop-writing-and-running-your-first-pr/240153197?pgno=2
* https://hadoop.apache.org/docs/r1.2.1/mapred_tutorial.html

Latest revision as of 09:58, 9 November 2015

Sumber:

Source Code

Contoh source code WordCount.java untuk menghitung jumlah masing-masing kata dari sebuah input set. 

cd ~
vi WordCount.java



package org.myorg;

import java.io.IOException;
import java.util.*;

import org.apache.hadoop.fs.Path;
import org.apache.hadoop.conf.*;
import org.apache.hadoop.io.*;
import org.apache.hadoop.mapred.*;
import org.apache.hadoop.util.*;
	
public class WordCount {
	
   public static class Map extends MapReduceBase implements Mapper<LongWritable, Text, Text, IntWritable> {
     private final static IntWritable one = new IntWritable(1);
     private Text word = new Text();

     public void map(LongWritable key, Text value, OutputCollector<Text, IntWritable> output, Reporter reporter) throws IOException {
       String line = value.toString();
       StringTokenizer tokenizer = new StringTokenizer(line);
       while (tokenizer.hasMoreTokens()) {
         word.set(tokenizer.nextToken());
         output.collect(word, one);
       }
     }
   }

   public static class Reduce extends MapReduceBase implements Reducer<Text, IntWritable, Text, IntWritable> {
     public void reduce(Text key, Iterator<IntWritable> values, OutputCollector<Text, IntWritable> output, Reporter reporter) throws IOException {
       int sum = 0;
       while (values.hasNext()) {
         sum += values.next().get();
       }
       output.collect(key, new IntWritable(sum));
     }
   }

   public static void main(String[] args) throws Exception {
     JobConf conf = new JobConf(WordCount.class);
     conf.setJobName("wordcount"); 
	
     conf.setOutputKeyClass(Text.class);
     conf.setOutputValueClass(IntWritable.class);

     conf.setMapperClass(Map.class);
     conf.setCombinerClass(Reduce.class);
     conf.setReducerClass(Reduce.class);

     conf.setInputFormat(TextInputFormat.class);
     conf.setOutputFormat(TextOutputFormat.class);

     FileInputFormat.setInputPaths(conf, new Path(args[0]));
     FileOutputFormat.setOutputPath(conf, new Path(args[1]));

     JobClient.runJob(conf);
   }
}

Compile

Asumsinya HADOOP_HOME adalah root instalasi dan HADOOP_VERSION adalah versi Hadoop yang di install, compile WordCount.java dan buat jar: 

export HADOOP_HOME=/usr/local/hadoop/share/hadoop/common
export HADOOP_VERSION=2.7.1

mkdir wordcount_classes
javac -classpath ${HADOOP_HOME}/hadoop-common-${HADOOP_VERSION}.jar -d wordcount_classes WordCount.java
jar -cvf /usr/joe/wordcount.jar -C wordcount_classes/ .

Penggunaan

Asumsi 

/usr/joe/wordcount/input - input directory di HDFS
/usr/joe/wordcount/output - output directory di HDFS

Sample text-files as input: 

bin/hadoop dfs -ls /usr/joe/wordcount/input/
/usr/joe/wordcount/input/file01
/usr/joe/wordcount/input/file02

bin/hadoop dfs -cat /usr/joe/wordcount/input/file01
Hello World Bye World

bin/hadoop dfs -cat /usr/joe/wordcount/input/file02
Hello Hadoop Goodbye Hadoop

Jalankan aplikasi 

bin/hadoop jar /usr/joe/wordcount.jar org.myorg.WordCount /usr/joe/wordcount/input /usr/joe/wordcount/output

Output: 

bin/hadoop dfs -cat /usr/joe/wordcount/output/part-00000
Bye 1
Goodbye 1
Hadoop 2
Hello 2
World 2


Referensi

Retrieved from "https://lms.onnocenter.or.id/wiki/index.php?title=Hadoop:_Contoh_Program_Sederhana&oldid=44910"