hadoop编程技巧（8）---Unit Testing (单元测试)

所需的环境：

Hadoop相关jar包裹（下载版本的官方网站上可以）；

下载junit包裹（新以及）。

下载mockito包裹；

下载mrunit包裹；

下载powermock-mockito包裹；

相关的包，如下面的截图（：http://download.csdn.net/detail/fansy1990/7690977）：

应用场景：

在进行Hadoop的一般MR编程时，须要验证我们的业务逻辑，或者说是验证数据流的时候能够使用此环境。这个环境不要求真实的云平台，仅仅是针对算法或者代码逻辑进行验证，方便调试代码。

实例：

Mapper：

package fz.mrtest;

import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.LongWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Mapper;

public class SMSCDRMapper extends Mapper<LongWritable, Text, Text, IntWritable> {
	 
	  private Text status = new Text();
	  private final static IntWritable addOne = new IntWritable(1);
	 
	  /**
	   * Returns the SMS status code and its count
	   */
	  protected void map(LongWritable key, Text value, Context context)
	      throws java.io.IOException, InterruptedException {
	 
	    //655209;1;796764372490213;804422938115889;6 is the Sample record format
	    String[] line = value.toString().split(";");
	    // If record is of SMS CDR
	    if (Integer.parseInt(line[1]) == 1) {
	      status.set(line[4]);
	      context.write(status, addOne);
	    }
	  }
	}

Reducer：

package fz.mrtest;

import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Reducer;


public class SMSCDRReducer extends
  Reducer<Text, IntWritable, Text, IntWritable> {
 
  protected void reduce(Text key, Iterable<IntWritable> values, Context context) throws java.io.IOException, InterruptedException {
    int sum = 0;
    for (IntWritable value : values) {
      sum += value.get();
    }
    context.write(key, new IntWritable(sum));
  }
}

測试主程序：

package fz.mrtest;

import java.io.IOException;
import java.util.ArrayList;
import java.util.List;
 
import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.LongWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mrunit.mapreduce.MapDriver;
import org.apache.hadoop.mrunit.mapreduce.MapReduceDriver;
import org.apache.hadoop.mrunit.mapreduce.ReduceDriver;
import org.junit.Before;
import org.junit.Test;
 
public class SMSCDRMapperReducerTest {
 
  MapDriver<LongWritable, Text, Text, IntWritable> mapDriver;
  ReduceDriver<Text, IntWritable, Text, IntWritable> reduceDriver;
  MapReduceDriver<LongWritable, Text, Text, IntWritable, Text, IntWritable> mapReduceDriver;
 
  @Before
  public void setUp() {
    SMSCDRMapper mapper = new SMSCDRMapper();
    SMSCDRReducer reducer = new SMSCDRReducer();
    mapDriver = MapDriver.newMapDriver(mapper);;
    reduceDriver = ReduceDriver.newReduceDriver(reducer);
    mapReduceDriver = MapReduceDriver.newMapReduceDriver(mapper, reducer);
  }
 
  @Test
  public void testMapper() throws IOException {
    mapDriver.withInput(new LongWritable(), new Text(
        "655209;1;796764372490213;804422938115889;6"));
    mapDriver.withOutput(new Text("6"), new IntWritable(1));
    mapDriver.runTest();
  }
 
  @Test
  public void testReducer() throws IOException {
    List<IntWritable> values = new ArrayList<IntWritable>();
    values.add(new IntWritable(1));
    values.add(new IntWritable(1));
    reduceDriver.withInput(new Text("6"), values);
    reduceDriver.withOutput(new Text("6"), new IntWritable(2));
    reduceDriver.runTest();
  }
  @Test
  public void testMR() throws IOException{
	  mapReduceDriver.withInput(new LongWritable(), new Text(
        "655209;1;796764372490213;804422938115889;6"));
	  mapReduceDriver.withInput(new LongWritable(), new Text(
		        "6552092;1;796764372490213;804422938115889;6"));
	  mapReduceDriver.withOutput(new Text("6"), new IntWritable(2));
	  mapReduceDriver.runTest();
  }
}

（代码源于MRUnit的官网。最后的測试主程序加了个对整个的測试）測试主程序一共同拥有三个測试方法，分别測试Mapper、Reducer、以及Mapper和Reducer的联合測试。

总结：使用Hadoop的单元測试能够方便验证编敲代码的正确性。而不须要使用真实环境验证代码的正确性为高效开发提供了可能。可是针对一些特殊的情况还是须要真实环境測试代码，这点须要特殊考虑，只是普通情况下，此单元測试环境对编写的MR都适用。

分享，成长，快乐

转载请注明blog地址：http://blog.csdn.net/fansy1990