Hadoop > mapreduce-user > December 2009

Hi, I have a mapreduce program embeded in a java application, and I am trying to load additional jar files as add-on for the execution of my map reduce job. My program works like this: JobConf conf = ...

Can someone explain how to override the "FileInputFormat" and "RecordReader" in order to be able to read multiple lines of text from input files in a single map task? Here the key will be the offset ...

Hi folks, I do have a questions regarding a map-reduce job who gets killed during the copy phase due to a timeout. I don't really understand the output completely so I first look for some help to ...

Hi all, I'd like to configure FairScheduler on hadoop. but seems it can not work. The following is what I did 1. add fairscheduler.jar to lib 2. add the following property to mapred-site.xml ...

I am writing my custom InputFormat to read N number of lines per map task. For this I have extended the FileInputFormat and RecordReader classes. In my RecordReader I am using LineRecordReader object ...

Hello! I'm trying to rewrite an image resizing program in terms of map/reduce. The problem I see is that the job is not broken up in to small enough tasks. If I only have 1 input file with 10,000 ...

Please direct me to a different forum if I am in the wrong place. I am just trying to compile Hadoop out of the box ... I first download hadoop-0.20.1 and untar it. Next, I run $ANT_HOME/bin/ant, ...

I am outputting a message to stdout and stderr from inside my MAP function. but the stdout and stderr files in log folder are empty. I tried looking these logfiles from the web UI as well. They are ...

Hi, Unit testing a mapper with the old mapred API is easy. However, for the new mapreduce API, I struggle creating the Context object in an easy and elegant way. How do I best mock the Context for ...

Hello! I was wondering if there was any convenient way to sort the Reducer output? Specifically, in WordCount is there a way to sort the results by frequency? Thank you very much, I'm sorry if this ...

Hello communities, Now I'm happy to announce that we've developed new computing model called BSP (Bulk Synchronous Parallel) on top of Hadoop. Here are the slides for the topic "Apache HAMA: An ...

As an exercise while learning MapReduce, I developed an algorithm for matrix multiplication and wrote it up on my web site. If you're interested, it's at: ...

Hi, i want to write a files to hdfs, using hadoop pipes. can anyone tell me how to do that? Im using an external library that writes its output to disk, so probably i have to read that output and ...

Hi, i want to write a files to hdfs, using hadoop pipes. can anyone tell me how to do that? Im using an external library that writes its output to disk, so probably i have to read that output and ...

Hello, I am running specific Map/Reduce task on Hadoop cluster 0.19.2, the job was split on 509 maps, 507 maps run quickly enough, 1-2 minutes each; cluster capacity: 9 maps, 3 reduces. The problem ...

I hope y'all can offer me some insight. I've only started looking hard at hadoop in the last couple of days, but I've yet to see a good example that I can hack for a style of problem I tend to ...

Hi all, I am processing a large tab file to format it suitable for loading into a database with a predefined schema. I have a tab file with a column that I need to normalize out to another table and ...