mapreduce-user@hadoop.apache.org discussions for September 2011

16	map.input.file returns null for MultipleInputs in Hadoop 0.20.2 version

Hi, I found this link https://issues.apache.org/jira/browse/MAPREDUCE-1743 related to the subject of my mail.Has this been resolved as yet or is there any workaround to get the filename while using ...

Sahana Bhat
Sep 9, 2011 at 12:01 pm
Sep 14, 2011 at 6:21 pm

13	How to Create an effective chained MapReduce program.

Hi I'm trying to write a chained mapreduce program. i'm doing so with a simple loop where in each iteration i create a job ,execute it and every time the current job's output is the next job's input. ...

Ilyal levin
Sep 5, 2011 at 3:49 pm
Sep 8, 2011 at 9:25 am

11	No Mapper but Reducer

Hi I'm having a query here. Is it possible to have no mappers but reducers alone? AFAIK If we need to avoid the tyriggering of reducers we can set numReduceTasks to zero but such a setting on mapper ...

Bejoy KS
Sep 7, 2011 at 7:49 am
Sep 8, 2011 at 2:41 pm

10	How do I set the intermediate output path when I use 2 mapreduce jobs?

Hi, I want to use 2 MR jobs sequentially. And the first job produces intermediate result to a temp file. The second job reads the result in temp file but not the FileInputPath. I tried, but ...

谭军
Sep 21, 2011 at 2:54 pm
Sep 24, 2011 at 3:04 pm

9	FairScheduler Local Task Restriction

Hello All, I have recently switched my small Hadoop dev cluster (v0.20.1) to use the FairScheduler. I have a max of 128 map tasks available and recently noticed that my jobs seem to use a maximum of ...

Adam Shook
Sep 22, 2011 at 7:43 pm
Sep 22, 2011 at 9:21 pm

7	The method addCacheFIle(URI) is undefined for the type Job

Hi, I encountered an error that I cannot understand. Configuration conf = new Configuration(); Job job = new Job(conf, "job1"); job.addCacheFile(new URI(args[0])); Why did it report "The method ...

谭军
Sep 23, 2011 at 1:01 pm
Sep 24, 2011 at 10:55 am

7	Hadoop Streaming job Fails - Permission Denied error

Hi I wanted to try out hadoop steaming and got the sample python code for mapper and reducer. I copied both into my lfs and tried running the steaming job as mention in the documentation. Here the ...

Bejoy KS
Sep 12, 2011 at 9:19 am
Sep 13, 2011 at 8:12 pm

6	Any good gui tools for working with hdfs

My dfs is a real mess and I am looking for a good gui fiile manager to allow me to clean it up deleting a lot of directories Anyone written one??? -- Steven M. Lewis PhD 4221 105th Ave NE Kirkland, ...

Steve Lewis
Sep 20, 2011 at 10:50 pm
Dec 21, 2011 at 6:51 pm

6	Using HADOOP for Processing Videos

Dear MapReduce User Groups: We want to process large amount of videos (typically 30 days old storage with size around 1TB) using Hadoop. Can somebody point me to code samples or classes which can ...

Rajen Bhatt (RBEI/EST1)
Sep 20, 2011 at 4:23 am
Sep 21, 2011 at 5:42 pm

6	Multiple Mappers and One Reducer

Hi, Is it possible to have multiple mappers where each mapper is operating on a different input file and whose result (which is a key value pair from different mappers) is processed by a single ...

Sahana Bhat
Sep 7, 2011 at 9:07 am
Sep 7, 2011 at 11:20 am

5	output from one map reduce job as the input to another map reduce job?

Is it possible to connect the output of one map reduce job so that it is the input to another map reduce job. Basically… then reduce() outputs a key, that will be passed to another map() function ...

Kevin Burton
Sep 27, 2011 at 7:10 pm
Sep 28, 2011 at 7:21 am

5	How to pull data in the Map/Reduce functions?

Hi, Normally the Hadoop framework calls the map()/reduce() for each record in the input split. I read in the 'Hadoop : The Definitive Guide' that that data can be pulled using the new MR API. What is ...

Praveen Sripati
Sep 24, 2011 at 1:43 pm
Sep 26, 2011 at 5:26 am

5	Running example application with capacity scheduler ?

Hi ! I have set up single-node cluster using ...

Arun k
Sep 14, 2011 at 10:58 am
Sep 16, 2011 at 5:23 am

5	Reducers without output files

Hello Everyone, I have a small issue with my Reducer that I am trying to figure out and wanted some advice. In the reducer, when writing to the output file as declared in ...

Arko Provo Mukherjee
Sep 15, 2011 at 3:26 am
Sep 15, 2011 at 6:47 am

5	I keep getting multiple values for unique reduce keys

Hi all, I have ensured that my mapper produces a unique key for every value it writes and further more that each map() call only writes one value. I note here that the value is a custom for which I ...

Rick Ross
Sep 5, 2011 at 12:42 am
Sep 6, 2011 at 5:26 am

4	Lost task tracker reschedules all tasktracker's successful map tasks

Hi, I have a very large MapReduce Job and sometimes a TaskTracker doesn't send a heartbeat in the preconfigured amount of time, so it's considered dead. It's ok, but all tasks already finished by ...

Leonardo Gamas
Sep 29, 2011 at 4:15 pm
Sep 29, 2011 at 7:25 pm

4	Type mismatch in key from map when replacing Mapper with MultithreadMapper

I'd like to implement a MultithreadMapper for my MapReduce job. For this I replaced Mapper with MultithreadMapper in a working code. Here's the exeption I'm getting: java.io.IOException: Type ...

Arsen Zahray
Sep 28, 2011 at 6:03 am
Sep 28, 2011 at 7:37 am

4	Execution directory for child process within mapper

As part of my Java mapper I have a command executes some standalone code on a local slave node. When I run a code it executes fine, unless it is trying to access some local files in which case I get ...

Joris Poort
Sep 26, 2011 at 5:51 pm
Sep 26, 2011 at 7:28 pm

4	Regarding FIFO scheduler

Hi, Lets assume that there are two jobs J1 (100 map tasks) and J2 (200 map tasks) and the cluster has a capacity of 150 map tasks (15 nodes with 10 map tasks per node) and Hadoop is using the default ...

Praveen Sripati
Sep 22, 2011 at 1:06 pm
Sep 22, 2011 at 1:52 pm

4	Passing a Global Variable into a Mapper

Hi, Is there a way to pass some data from the driver class to the Mapper class without going through the HDFS? Does the API provide us with some functionality to pass some variables? Thanks a lot in ...

Arko Provo Mukherjee
Sep 15, 2011 at 5:44 pm
Sep 15, 2011 at 6:11 pm

4	Error calling config method in Mapper

Hi I have written a mapper class that has the following structure public class TestMapper extends MapReduceBase implements Mapper<LongWritable, Text, Text, Text { public void configure(JobConf conf) ...

Shreya Pal
Sep 15, 2011 at 11:15 am
Sep 15, 2011 at 1:07 pm

4	Has anyone ever written a file system where the data is held in resources

When writing tests it is useful to keep all data in resources since this makes automatic execution easier. The structure of a set of resources should make it easy to have a schema such as res:// to ...

Steve Lewis
Sep 14, 2011 at 4:25 pm
Sep 14, 2011 at 5:45 pm

4	Does anyone have sample code for forcing a custom InputFormat to use a small split

I have a problem where there is a single, relatively small (10-20 MB) input file. (It happens it is a fasta file which will have meaning if you are a biologist.) I am already using a custom ...

Steve Lewis
Sep 12, 2011 at 2:28 am
Sep 13, 2011 at 5:05 am

3	modify a specific TaskTracker's Map slots on the fly - FairScheduler?

I need to modify the number of Map slots on a TaskTracker dynamically over the course of a job without restarting the TaskTracker process. If the number of allowed slots is lowered, the current Map ...

Ben Clay
Sep 29, 2011 at 12:53 pm
Sep 29, 2011 at 4:52 pm

3	System.out.println in Map / Reduce

Hi, I am writing some Map Reduce programs in pseudo-distributed mode. I am getting some error in my program and would like to debug it. For that I want to embed some print statements in my Map / ...

Arko Provo Mukherjee
Sep 26, 2011 at 11:48 pm
Sep 27, 2011 at 3:58 pm

3	Re: RE: Making Mumak work with capacity scheduler

Yes Devaraj, code snippet: if (!fs.exists(path)) { if (!fs.mkdirs(path, new FsPermission(JOB_STATUS_STORE_DIR_PERMISSION))) { throw new IOException( "CompletedJobStatusStore mkdirs failed to create " ...

Uma Maheswara Rao G 72686
Sep 22, 2011 at 1:49 pm
Sep 23, 2011 at 6:30 am

3	Re: Running Jobs with capacity scheduler on hadoop in eclipse

Hi ! I have setup hadoop0.20.2 on eclipse Helios and able to run the Example wordcount using ExampleDriver class as mentioned by Faraz in ...

Arun k
Sep 16, 2011 at 8:40 am
Sep 17, 2011 at 4:33 am

3	Debugging Mapreduce programs

Hi Can we use sysouts in java to debug MR programs. Where would those sysouts be printed? What would be the best way to debug a remote system where I have access to the machine only through putty. ...

Shreya Pal
Sep 15, 2011 at 4:12 am
Sep 15, 2011 at 11:52 am

3	How to sort key,value pair by value(In ascending)

Hi, I have the content like *10103*,1042279,*4* *10070*,1001089,*5* *10102*,1015504,*7* *10080*,1024369,*7* *10050*,1025671,*1* ... from which i separated the key,value pairs and got the output after ...

Ksgupta misc
Sep 14, 2011 at 12:44 am
Sep 14, 2011 at 12:39 pm

3	Problem when using MultipleOutputs with many files

Hello guys, I am using hadoop-0.20.2-cdh3u0 and I use MultipleOutputs to divide the HFiles (which are the output of my MR job) so that each file can fit into one region of the table where I am going ...

Panagiotis Antonopoulos
Sep 2, 2011 at 1:15 pm
Sep 5, 2011 at 2:41 pm

3	How does FileInputFormat sub-classes handle many small files

Hi FileInputFormat sub-classes (TextInputFormat and SequenceFileInputFormat) are able to take all files in a folder and split the work of handling them into several sub-jobs (map-jobs). I know it can ...

Per Steffensen
Sep 1, 2011 at 8:59 am
Sep 2, 2011 at 4:58 am

3	FileOutputFormat (getWorkOutputPath)

Hi, what is the right way to use the method getWorkOutputPath from the FileOutputFormat? Here is a sample of my code: public void map(LongWritable key, Text value, OutputCollector<Text, Text output, ...

Kadu canGica Eduardo
Sep 1, 2011 at 12:27 am
Sep 2, 2011 at 4:46 am

2	Output of MAP Class only

Dear Experts: We are scaling down an image to few partitions and processing each partition in the MAP class. We want to save the output of MAP class only to demonstrate and present how this output ...

Rajen Bhatt (RBEI/EST1)
Sep 30, 2011 at 10:49 am
Sep 30, 2011 at 12:25 pm

2	Execution Timing Comparison

Guys: I and my student are new to the MapReduce an Hadoop world, so my question may be very basic one. We have taken one HD resolution image and performed some basic image processing operations by ...

Rajen Bhatt (RBEI/EST1)
Sep 29, 2011 at 9:14 am
Sep 29, 2011 at 9:36 am

2	Hadoop java mapper -copyFromLocal heap size error

As part of my Java mapper I have a command executes some code on the local node and copies a local output file to the hadoop fs. Unfortunately I'm getting the following output: "Error occurred during ...

Joris Poort
Sep 24, 2011 at 12:20 am
Sep 26, 2011 at 5:49 pm

2	Running Hadoop in different modes

Hi, What are the features available in the Fully-Distributed Mode and the Pseudo-Distributed Mode that are not available in the Local (Standalone) Mode? Local (Stanndalone) Mode is very fast and I am ...

Praveen Sripati
Sep 23, 2011 at 6:46 am
Sep 23, 2011 at 12:30 pm

2	quotas for size of intermediate map/reduce output?

Hi All, Is it possible to enforce a maximum to the disk space consumed by a map/reduce job's intermediate output? It looks like you can impose limits on hdfs consumption, or, via the capacity ...

Matt Steele
Sep 21, 2011 at 10:45 pm
Sep 22, 2011 at 12:01 am

2	Error in configuring object

Hi, Main method looks like below: public static void main(String[] args) { JobConf conf = new JobConf(Sentiment_Analysis_Twitter_Driver.class); sLogger.setLevel(Level.ALL); ...

Shreya Pal
Sep 15, 2011 at 9:09 am
Sep 17, 2011 at 6:25 pm

2	Unblacklist a blacklisted tracker (at job level)?

I'm currently running a long-running job. One of our nodes was misconfigured, however, and so got blacklisted in the job. I've since corrected the problem, though, and restarted the tasktracker. But ...

David Rosenstrauch
Sep 14, 2011 at 5:07 pm
Sep 15, 2011 at 4:03 pm

2	Question regarding the MapReduce tutorial

So I was working my way through the tutorial, and when I got to the first example - WordCountV1.0 - I discovered that there are a LOT of deprecated classes/methods used in the tutorial. I could take ...

Harry lippy
Sep 11, 2011 at 1:17 am
Sep 11, 2011 at 6:33 pm

2	Config setting for staging.dir?

Is there a way to change the staging directory in the config files? Mine currently goes to "/<tmp-folder /staging". I read someone else set mapreduce.jobtracker.staging.root.dir, but I have had no ...

Dolan Antenucci
Sep 9, 2011 at 6:38 pm
Sep 9, 2011 at 7:46 pm

2	Setting permissions for slave nodes running mapper

Hi, I'm trying to set permissions for the tasktracker and/or mapred user. Basically I'm trying to execute and modify files from within the mapper, but the code errors out stating that the mapred user ...

Joris Poort
Sep 8, 2011 at 11:59 pm
Sep 9, 2011 at 6:00 pm

2	How to pass a parameter across the cluster.

Hi What is the right way to pass a parameter for all mapper and reducers to see? Thanks

Ilyal levin
Sep 7, 2011 at 11:40 am
Sep 7, 2011 at 12:29 pm

1	Avro 1.5.x version compatibility

Hi, We are looking to upgrade avro 1..4.1 to avro 1.5.x version. Does anyone know if this can cause any incompatibility with hadoop cdh3 distro? Thanks Sudhan S

Sudharsan Sampath
Sep 30, 2011 at 5:36 am
Oct 3, 2011 at 8:19 pm

1	Tasks running out of memory and mapred.child.ulimit

I have a small hadoop task which is running out of memory on a colleague's cluster. I looked at has mapred-site.xml and find <property <name mapred.child.java.opts</name <value -Xmx512M</value ...

Steve Lewis
Sep 30, 2011 at 5:24 pm
Sep 30, 2011 at 6:37 pm

1	How to send objects to map task?

Hi, My application needs to send some objects to map tasks, which specify how to process the input records. I know I can transfer them as string via the configuration file. But I prefer to leverage ...

Zhiwei Xiao
Sep 27, 2011 at 10:43 pm
Sep 28, 2011 at 3:04 pm

1	Reading the value of a counter

Assume I have a class which extends Configured implements Tool Assume I run the job using the run method. Within my job I increment a counter. When I finish the run I want the final value of the ...

Steve Lewis
Sep 26, 2011 at 7:18 pm
Sep 26, 2011 at 7:34 pm

1	Fw:A question about `mvn eclipse:eclipse`

From: "周杰" <zhoujie338@126.com Date: 2011-09-26 09:52:58 To: "mapreduce-user-subscribe@hadoop.apache.org" <mapreduce-user-subscribe@hadoop.apache.org Subject: A question about `mvn eclipse:eclipse` ...

周杰
Sep 26, 2011 at 2:00 am
Sep 26, 2011 at 5:35 am

1	Maintaining map reduce job logs - The best practices

Hi All I do have a query here on maintaining Hadoop map-reduce logs. In default the logs appear in respective task tracker nodes which you can easily drill down from the job tracker web UI at times ...

Bejoy KS
Sep 23, 2011 at 7:53 am
Sep 23, 2011 at 4:36 pm

1	7 of 487 New window Print all Can JobConf object be declared global and used in mapper and reducer class

pranjal shrivastava to mapreduce-user. show details 9:12 AM (22 minutes ago) Hi I declared JobConf object static and globally in the main class and initialized it in the main function.Since it is ...

Pranjal shrivastava
Sep 21, 2011 at 1:36 pm
Sep 21, 2011 at 3:22 pm

group	mapreduce-user @
categories	hadoop
discussions	87
posts	310
users	77
website	hadoop.apache.org...
irc	#hadoop

summaries	on	Hotkey: s
users list	on	Hotkey: d
user style	avatars	Hotkey: a

Hadoop > mapreduce-user > September 2011

Search Discussions

87 discussions - 310 posts

77 users for September 2011

Content

People

Support

Translate