[Hadoop-common-user] sort speeds under java, c++, and streaming

	Owen O'Malley
	Nov 9, 2007 at 7:31 am

On Nov 8, 2007, at 7:45 PM, Milind A Bhandarkar wrote:

Are they repeatable ?

I ran 3 more runs of each and it looks pretty stable:

Java: 4:15, 4:06, 4:11
Pipes: 4:03, 3:51, 4:03
Streaming: 4:58, 4:59, 5:00

-- Owen

Search Discussions

Joydeep Sen Sarma: What about the partitioner or combiner - C++ or Java? Just wondering - there's got to be something that got subtracted. -----Original Message----- From: Owen O'Malley Sent: Thursday, November 08, 2007 7:01 PM To: [email protected] Subject: Re: sort speeds under java, c++, and streaming Yes, that is why it surprised me. -- Owen

Follow ups

Owen O'Malley: I found what was happening. The Java sort application was setting the number of maps, so it was getting 400 maps instead of 320. Forcing it to 320 maps using "-m 320", I get numbers basically the same as pipes. Java: 3:59, 4:03, 4:06 -- Owen
Owen O'Malley: I looked while the Java one was running and it was working a couple of the cpus pretty hard. (I was only running with the default 2tasks/ node, which is really low given these are nice 8 cpu machines.) I should also mention that I was using a 500 node hdfs cluster that is a superset of the 39 node + 1 job tracker map/reduce cluster, so most of the hdfs reads and writes were outside of the map/reduce cluster. What I'm doing is putting together a somewhat representative workload to look at

Discussion Navigation

view	thread \| post
posts	‹ prev \| 12 of 14 \| next ›

Discussion Overview

font	variable	Hotkey: f
user style	avatars	Hotkey: a