mapreduce-user@hadoop.apache.org discussions for August 2011

9	From a newbie: Questions and will MapReduce fit our needs

Hi We are considering to use MapReduce for a project. I am participating in an "investigation"-phase where we try to reveal if we would benefit from using the MapReduce framework. A little bit about ...

Per Steffensen
Aug 26, 2011 at 11:14 am
Aug 30, 2011 at 6:42 am

9	Can reducer get parameters from mapper besides key and value?

Hi, Can reducer gets parameters from mapper besides key and value? Such as files, lists, string[] etc. Thanks -- Regards! Jun Tan

谭军
Aug 9, 2011 at 6:29 am
Aug 14, 2011 at 1:26 pm

7	Possible to override the context.write() method in ReduceContext?

Using 0.20.2... Is it possible to override the context.write() method in ReduceContext? I have an entire set of Reducers that I would like to all use a specific function just before every ...

Ross Nordeen
Aug 12, 2011 at 9:06 pm
Aug 15, 2011 at 4:03 pm

6	Can I use the cores of each CPU to be the datanodes instead of CPU?

Hi, I have 2 computers each of which is double-core CPU. The 2 computers are datanodes and another computer is the namenode. Now, I want to increase the number of datanode but no more computers ...

谭军
Aug 8, 2011 at 8:47 am
Aug 9, 2011 at 4:12 am

6	Performance of mappers

Hello all, I have a question regarding the mappers. I can see from the logs that the start time of the mapper is different from start time of logging. I am having a problem because that time ...

Iman E
Aug 5, 2011 at 6:02 pm
Aug 5, 2011 at 8:51 pm

5	Compiling programs with Hadoop 0.21.0

Hello, I am trying to learn Hadoop and doing a project on it. I need to update some files in my project and hence wanted to use version 0.21.0 However, I am confused as to how I can compile my ...

Arko Provo Mukherjee
Aug 31, 2011 at 9:35 pm
Sep 2, 2011 at 5:36 am

5	How to access contents of a Map Reduce job's working directory

I have just started to explore Hadoop but I am stuck in a situation now. I want to run a MapReduce job in hadoop which needs to create a "setup" folder in working directory. During the execution the ...

Shrish Bajpai
Aug 2, 2011 at 12:50 am
Aug 2, 2011 at 10:39 am

4	Very slow MapReduce Job

Hi, I wrote a custom InputFormat for parsing through the Enron Email corpus which is attached in the file named EmailInputFormat I have attached the code in a text file with the sample input mail ...

Varad Meru
Aug 29, 2011 at 10:01 am
Aug 30, 2011 at 7:19 am

4	Status: FAILED Error: null, What's this problem?

Hi, I got the error information below. What's wrong? What I/O exception? [email protected]:/home/cx/software/Hadoop/hadoop-0.20.2# bin/hadoop jar Retrieval.jar Retrieval protein/b.txt protein/a.txt ...

谭军
Aug 11, 2011 at 10:41 am
Aug 12, 2011 at 8:26 am

4	How does Hadoop reuse the objects?

Hi all, I'm using Avro as a serialization format and assume I have a generated specific class FOO that I use as a Mapper output format: class FOO { int a; List<BAR barList; } where BAR is another ...

Vyacheslav Zholudev
Aug 3, 2011 at 7:19 am
Aug 4, 2011 at 9:16 pm

3	Error while building Hadoop-Yarn

Hi, I am using apache-maven-3.0.3 and i have set LD_LIBRARY_PATH=/usr/local/lib which has google protocbuf library. I am getting following error while building hadoop-yarn using mvn clean install ...

Rajesh putta
Aug 19, 2011 at 6:26 am
Aug 29, 2011 at 6:41 am

3	Extending ArrayWritable, Using Combiner and Spill Failed error

I am working on a large application where I need to pass a big array of intwritables from mapper to reducer. Obviously I want to use combiner and I am extending ArrayWritable. Here is the signature ...

Vipul sharma
Aug 18, 2011 at 8:44 pm
Aug 19, 2011 at 5:03 am

3	Need Help with HBase

Hi, I'm a newbie to the Hadoop and Map/Reduce applications. I have set-up a cluster and just running the example map/reduce applications which comes with the Hadoop source code. I want to run some ...

M S Vishwanath Bhat
Aug 11, 2011 at 7:00 pm
Aug 17, 2011 at 4:10 am

3	Reducer Run on Which Machine?

I understand that we can decide which task run by which reducer in Hadoop by using custom partitioner, but is there anyway to decide which reducer run on which machine? Suhendry Effendy

Suhendry Effendy
Aug 5, 2011 at 5:25 am
Aug 5, 2011 at 7:15 pm

3	MultipleOutputs support

Hi all, I'm trying to create M/R tasks that will output more than one "type" of data. Ideal thing would be MultipleOutputs feature of Map Reduce, but in our current production version, CDH3 ( 0.20.2 ...

Vanja Komadinovic
Aug 1, 2011 at 10:21 pm
Aug 4, 2011 at 5:32 pm

2	Capacity scheduler uses all slots from the same tasktracker

Hi, So I just started using capacity scheduler for M/R jobs. I have 4 task trackers each with 4 map/reduce slots. Configured a queue so that it uses 25% (4 slots) of the available slots. I was ...

Sulabh Choudhury
Aug 23, 2011 at 10:52 pm
Aug 23, 2011 at 11:12 pm

2	SequenceFileInputFormat doesn't return whole records

Hi all, I am having issues using SequenceFileInputFormat to retrieve whole records I have 1 job that is used to write to a SequenceFile SequenceFileOutputFormat.setOutputPath(job, new ...

Tim Fletcher
Aug 19, 2011 at 12:32 pm
Aug 19, 2011 at 2:20 pm

2	MultipleOutputs is not working properly when dfs.block.size is changed

Hi all, I have been working on hadoop jobs which are writing output into multiple files. In Hadoop API I have found class MultipleOutputs which implement this functionality. My use case is to change ...

Dino Kečo
Aug 18, 2011 at 8:31 am
Aug 18, 2011 at 10:54 am

2	Map output records/reducer input records mismatch

Hi, I'm having multiple hadoop jobs that use the avro mapred API. Only in one of the jobs I have a visible mismatch between a number of map output records and reducer input records. Does anybody ...

Vyacheslav Zholudev
Aug 16, 2011 at 6:38 pm
Aug 16, 2011 at 7:57 pm

2	Excuting a shell script inside the HDFS

Hi, i'm trying without success to execute a shell script that is inside the HDFS. Is this possible? If it is, how can i do this? Since ever, thanks. Carlos.

Kadu canGica Eduardo
Aug 15, 2011 at 9:51 pm
Aug 16, 2011 at 7:00 pm

2	Under-replication warnings for Distributed Cache?

Hi, we're running an 8-node Hadoop cluster with CDH2. Recently, our monitoring tools caught warnings like this one when fsck'ing the HDFS: /tmp/hadoop-tgp/mapred/system/job_201105191458_1857/job.jar: ...

Christoph Schmitz
Aug 15, 2011 at 1:40 pm
Aug 16, 2011 at 6:39 am

2	Can I define a datastructure that all Mappers share?

Hi, I want to define a matrix or list that all mappers share. So that all mappers can do operations on it. Can I make it? Thanks! -- Regards! Jun Tan

谭军
Aug 15, 2011 at 9:15 am
Aug 15, 2011 at 11:29 am

2	Multiple input formats and multiple output formats in Hadoop 0.20.2

Hi, I am working on a project, which requires multiple input formats and multiple output formats. Basically, I store some sales rank data to a Cassandra cluster and I get a sales rank update file ...

Jian Fang
Aug 10, 2011 at 4:09 pm
Aug 10, 2011 at 4:26 pm

2	Can you see the name of the document being loaded?

I want to calculate some statistics on a per document basis, and it seems like the only way to do this would be to emit a compound key of (key,documentname). 1) Is this the case, or is there a better ...

Jonathan Coveney
Aug 9, 2011 at 9:12 pm
Aug 10, 2011 at 12:25 am

2	Need Help in Setting up the NextGen MapReduce.

Hi All, I need help in setting up the Next Gen Mapreduce. Please provide links to documents/Guide if any to start setting up the Next Gen MR. Thanks and Regards, Vinayakumar B ...

Vinayakumar B
Aug 2, 2011 at 10:16 am
Aug 2, 2011 at 3:13 pm

2	Configuring Hadoop daemon heap sizes

Hi. I'm trying to tweak heap sizes for the Hadoop daemons, i.e. namenode/datanode and jobtracker/tasktracker. I've tried setting HADOOP_NAMENODE_HEAPSIZE, HADOOP_DATANODE_HEAPSIZE, and so on in ...

Kai Ju Liu
Aug 1, 2011 at 6:11 pm
Aug 2, 2011 at 4:39 am

1	Computing the number of mappers when CombineFileInputFormat is used

Hello! I would like to know how Hadoop is computing the number of mappers when CombineFileInputFormat is used? I have read the API specification for CombineFileInputFormat ...

Florin P
Aug 12, 2011 at 6:36 am
Sep 21, 2011 at 8:00 pm

1	No of reduce tasks per heartbeat

Hi, I see in the code that while we assign a number of map tasks, we assign only one reduce task per tasktracker during the heartbeat. Is there a brief somewhere on why this design decision is made ? ...

Sudharsan Sampath
Aug 24, 2011 at 10:49 am
Aug 24, 2011 at 1:50 pm

1	Using MultipleTextOutputFormat to control output filename in MapReduce

Hello Folks, I needed some help with using MultipleTextOutputFormat to control the output filename in MapReduce. Currently I am using it as shown below(or at http://pastebin.com/gJxkdwRd). And it ...

Decimus Phostle
Aug 8, 2011 at 4:07 pm
Aug 23, 2011 at 11:20 pm

1	Task JVM reuse

Hi, If the task jvm is set to be re-used with a -1 option, when does the jvms exit? completes. Is that right? Thanks Sudharsan S

Sudharsan Sampath
Aug 23, 2011 at 6:10 am
Aug 23, 2011 at 6:35 am

1	RE: Wonky reduce progress

Please check the defect in MAPREDUCE jira https://issues.apache.org/jira/browse/MAPREDUCE-2264 This is because the compression is enabled for map outputs and statistics are taken on compressed data ...

Vinayakumar B
Aug 19, 2011 at 8:17 am
Aug 19, 2011 at 3:05 pm

1	M/R vs hbase problem in production

Hi, cluster details: hbase 0.90.2. 10 machines. 1GB switch. use-case M/R job that inserts about 10 million rows to hbase in the reducer, followed by M/R that works with hdfs files. When the first job ...

Lior Schachter
Aug 14, 2011 at 4:32 pm
Aug 14, 2011 at 4:37 pm

1	Creating a custom composite key

Hi all, Is anybody familiar with how to define a custom composite key in a format such as (Text, Text) so your context written as a key and value could be ((Text, Text) LongWritable)? Thanks, -- ...

Roger Chen
Aug 12, 2011 at 10:50 pm
Aug 12, 2011 at 11:23 pm

1	unique PATH environment variable for Mapper class

Is it possible to set the MS. Windows PATH environment variable for Mapper classes? I have maps that depend on JNI and I need to set the PATH so that the .dlls can be found. The PATH can't be ...

Curtis Jensen
Aug 8, 2011 at 9:05 pm
Aug 8, 2011 at 9:14 pm

1	How can I know how many mappers created?

Hi, I just want 2 mappers created to do my job for there are only 2 data nodes. I think that is the most efficient. But how can I know how many mappers have been created while doing my job? Can I do ...

谭军
Aug 7, 2011 at 6:46 am
Aug 8, 2011 at 4:33 am

1	fail to compile hadoop branch-0.22/mapreduce

hello,everyone I checkout the branch-0.22 source code from apache, i can compile the common and hdfs codes successfully,but I got a exception when i compiling branch-0.22/mapreduce as follow.I don't ...

周俊清
Aug 2, 2011 at 2:15 am
Aug 2, 2011 at 4:12 am

0	job success while din't finish

hi ,all The hadoop mapreduce version i used is 0.20.0.when i run a job today,some things happen like this: mayb it's normal form this view ,but the job keep this status for our hours.It didn't ...

Zhengrui.m
Aug 30, 2011 at 9:47 am
Aug 30, 2011 at 9:47 am

0	Recommendations on moving to Hadoop/Hive with Cassandra + RDBMS

Hi, I have an already running system where I define a simple data flow (using a simple custom data flow language) and configure jobs to run against stored data. I use quartz to schedule and run these ...

Tharindu Mathew
Aug 29, 2011 at 7:47 pm
Aug 29, 2011 at 7:47 pm

0	unsubscribe

unsubscribe

Yue Wang
Aug 24, 2011 at 1:18 pm
Aug 24, 2011 at 1:18 pm

0	Re: Task re-scheduling in hadoop

Moving to mapreduce-user@, bcc common-user@ Why would you want to do that? Typically, you want the JT to retry the failed tasks as quickly as possible to fail the job rather than try all tasks and ...

Arun C Murthy
Aug 23, 2011 at 2:53 pm
Aug 23, 2011 at 2:53 pm

0	API Conflicts for Join Methods

Varad Meru
Aug 22, 2011 at 2:07 am
Aug 22, 2011 at 2:07 am

0	Re: Question regarding Capacity Scheduler

Moving to mapreduce-user@, bcc common-user@ Not for the same user - the CS tries to get jobs done as quickly as possible and thus it won't share resources for the same user. However, you can submit ...

Arun C Murthy
Aug 17, 2011 at 6:07 pm
Aug 17, 2011 at 6:07 pm

0	Re: "java.lang.Throwable: Child Error " And " Task process exit with nonzero status of 1."

Hi, I have some junit tests for a MapReduce job that run fine from my IDE, but fail with the following error when using Maven (mvn test): 2011-08-17 15:08:49,191 [main] INFO ...

Cristina Ionescu
Aug 17, 2011 at 1:05 pm
Aug 17, 2011 at 1:05 pm

0	Re: reconfigure number of map/reduce slots per machine without restart

Yes, just restart the TaskTracker. There's no need to restart your JobTracker, so a rolling TT reconfigure+restart should get you going. P.s. general@ is for project wide general discussions, not ...

Harsh J
Aug 16, 2011 at 5:26 am
Aug 16, 2011 at 5:26 am

0	error while running gridmix2

Hello, I am running gridmix2, and get an error due to missing GridMixRunner Exception in thread "main" java.lang.ClassNotFoundException: org.apache.hadoop.mapred.GridMixRunner I looked in all jars ...

Keren Ouaknine
Aug 15, 2011 at 1:49 pm
Aug 15, 2011 at 1:49 pm

0	Fwd: Jobtracker looses track of all tasktrackers, needs mapred restart

Hi, We have hadoop(hadoop-0.20.1) cluster of 14 nodes and daily some jobs execute on this cluster. Recently we faced an issue in which jobtracker looses track of all tasktrackers and as a result job ...

Ajit Ratnaparkhi
Aug 15, 2011 at 11:14 am
Aug 15, 2011 at 11:14 am

0	EOFException when using LZO to compress map/reduce output

Hi, I am using LZO to compress my intermediate map outputs. These are the settings: mapred.map.output.compression.codec = com.hadoop.compression.lzo.LzoCodec pig.tmpfilecompression.codec = lzo But I ...

Rakesh kothari
Aug 15, 2011 at 6:49 am
Aug 15, 2011 at 6:49 am

0	Context.getInputSplit() returns null

I am using hadoop 0.20.2 on CDH. I am trying to get the filename of the file currently being processed. I will extract some information from the filename which will determine the data processing to ...

Vegar Hatlevik
Aug 12, 2011 at 9:26 am
Aug 12, 2011 at 9:26 am

0	[[No Subject]]

Gabriel B M Armelin
Aug 12, 2011 at 12:51 am
Aug 12, 2011 at 12:51 am

0	How can I achieve secondary retrieval in mapreduce?

Hi, I want to write a program to achieve secondary retrieval, but don't know how to do it. I don't know how to express myself, so the source code below my help. I don't know whether my first retieval ...

谭军
Aug 8, 2011 at 2:59 pm
Aug 8, 2011 at 2:59 pm

group	mapreduce-user @
categories	hadoop
discussions	55
posts	168
users	72
website	hadoop.apache.org...
irc	#hadoop

summaries	on	Hotkey: s
users list	on	Hotkey: d
user style	avatars	Hotkey: a

Hadoop > mapreduce-user > August 2011

Search Discussions

55 discussions - 168 posts

72 users for August 2011

Content

People

Support

Translate