读书人

地图reduce排序中java.io.IOException

发布时间: 2013-10-01 12:15:56 作者: rapoo

mapreduce排序中java.io.IOException: Illegal partition for 2 (-1)
public static class Partition extends Partitioner<IntWritable,IntWritable>{@Overridepublic int getPartition(IntWritable key, IntWritable value, int numPartitions) {int Maxnumber = 65223;int bound = Maxnumber/numPartitions+1;int keynumber = key.get();for (int i = 0; i < numPartitions; i++) {if(keynumber<bound*i&&keynumber>=bound*(i-1)){return i-1;}}return -1;}}?原因:

出现错误的原因是在某种情况下getPartition函数返回了-1,Partition是从0开始的,默认返回-1是不正确的。
解决方法:
默认返回0
public static class Partition extends Partitioner<IntWritable,IntWritable>{@Overridepublic int getPartition(IntWritable key, IntWritable value, int numPartitions)  {int Maxnumber = 65223;int bound = Maxnumber/numPartitions+1;int keynumber = key.get();for (int i = 0; i < numPartitions; i++) {if(keynumber<bound*i&&keynumber>=bound*(i-1)){return i-1;}}return 0;}}
?

读书人网 >互联网

热点推荐