Mapreduce Code wrong output (Not skipping two words in wordcount)

vinayps91
Posts: 2
Joined: Tue Nov 24, 2015 10:16 am

Mapreduce Code wrong output (Not skipping two words in wordcount)

Postby vinayps91 » Wed Jan 20, 2016 11:01 am

Mapper-
public class skipwc_mapper extends Mapper<LongWritable,Text,Text,IntWritable> {

protected void map(LongWritable key, Text value, Context context) throws IOException, InterruptedException {

String line = value.toString();
StringTokenizer t = new StringTokenizer(line);
Text word = null;
while(t.hasMoreTokens()){
// if (t.nextToken() != "in" && t.nextToken() != "of"){
word = new Text(t.nextToken());
context.write(word,new IntWritable(1));
// }
}
}
}

Reducer-
public class skipwc_reducer extends Reducer<Text, IntWritable, Text, IntWritable>{
protected void reduce(Text key, Iterable<IntWritable> values, Context context) throws IOException, InterruptedException {
int tot = 0;
if (key.toString() != "in" && key.toString() != "of") {
while (values.iterator().hasNext()) {
tot += values.iterator().next().get();
}
context.write(key, new IntWritable(tot));
}
}
}

Runner-
public class skipwc_runner {
public static void main(String[] args) throws IOException, InterruptedException, ClassNotFoundException{
Configuration conf = new Configuration();
Job job = new Job(conf);
job.setJarByClass(skipwc_runner.class);

job.setOutputKeyClass(Text.class);
job.setOutputValueClass(IntWritable.class);

job.setMapperClass(skipwc_mapper.class);
job.setReducerClass(skipwc_reducer.class);

job.setInputFormatClass(TextInputFormat.class);
job.setOutputFormatClass(TextOutputFormat.class);

FileInputFormat.addInputPath(job, new Path(args[0]));
FileOutputFormat.setOutputPath(job, new Path(args[1]));

System.exit(job.waitForCompletion(true)?0:1);
}
}

edupristine
Finance Junkie
Posts: 722
Joined: Wed Apr 09, 2014 6:28 am

Re: Mapreduce Code wrong output (Not skipping two words in wordcount)

Postby edupristine » Tue Apr 05, 2016 7:07 am

Hi Vinay


There is a problem in the logic that you used. You are using AND condition instead of OR condition. The AND condition will never gets satisfied. Use OR instead of AND it will work. (change && to ||)
if (key.toString() != "in" && key.toString() != "of") {
change this to
if (key.toString() != "in" || key.toString() != "of") {

For just skipping words, you don’t need reducer, this can be achieved using mapper alone program.


Return to “Big Data & Hadoop”



cron

Disclaimer

Global Association of Risk Professionals, Inc. (GARP®) does not endorse, promote, review or warrant the accuracy of the products or services offered by EduPristine for FRM® related information, nor does it endorse any pass rates claimed by the provider. Further, GARP® is not responsible for any fees or costs paid by the user to EduPristine nor is GARP® responsible for any fees or costs of any person or entity providing any services to EduPristine Study Program. FRM®, GARP® and Global Association of Risk Professionals®, are trademarks owned by the Global Association of Risk Professionals, Inc

CFA Institute does not endorse, promote, or warrant the accuracy or quality of the products or services offered by EduPristine. CFA Institute, CFA®, Claritas® and Chartered Financial Analyst® are trademarks owned by CFA Institute.

Utmost care has been taken to ensure that there is no copyright violation or infringement in any of our content. Still, in case you feel that there is any copyright violation of any kind please send a mail to abuse@edupristine.com and we will rectify it.