[Hadoop-common-user] sort speeds under java, c++, and streaming

On Nov 8, 2007 7:00 PM, Owen O'Malley wrote:

Sure, I guess I should have been more specific to start with. *grin*

The data was generated with:
bin/hadoop jar hadoop-0.15.0-dev-examples.jar randomtextwriter -conf
gridmix-text.xml\
-outFormat org.apache.hadoop.mapred.TextOutputFormat /gridmix/
data/sort/text
contents of gridmix-text.xml:
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="nutch-conf.xsl"?>

<configuration>

<property>
<name>test.randomtextwrite.total_bytes</name>
<value>429496729600</value>
</property>

<property>
<name>test.randomtextwrite.min_words_key</name>
<value>1</value>
</property>

<property>
<name>test.randomtextwrite.max_words_key</name>
<value>10</value>
</property>

<property>
<name>test.randomtextwrite.min_words_value</name>
<value>0</value>
</property>

<property>
<name>test.randomtextwrite.max_words_value</name>
<value>200</value>
</property>

</configuration>

And then ran the sort as:

Java:
bin/hadoop jar hadoop-0.15.0-dev-examples.jar sort \
-inFormat org.apache.hadoop.mapred.KeyValueTextInputFormat \
-outFormat org.apache.hadoop.mapred.TextOutputFormat \
-outKey org.apache.hadoop.io.Text -outValue
org.apache.hadoop.io.Text \
/gridmix/data/sort/text/part-*0 java-out

Pipes:
bin/hadoop pipes -input /gridmix/data/sort/text/part-*0 -output pipe-
out \
-inputformat org.apache.hadoop.mapred.KeyValueTextInputFormat \
-program /gridmix/programs/pipes-sort -reduces 78 \
-jobconf\

mapred.output.key.class=org.apache.hadoop.io.Text,mapred.output.value.cl
ass=org.apache.hadoop.io.Text \
-writer org.apache.hadoop.mapred.TextOutputFormat

Streaming:
bin/hadoop jar contrib/hadoop-0.15.0-dev-streaming.jar \
-input /gridmix/data/sort/text/part-*0 -output stream-out -mapper
cat -reducer cat \
-numReduceTasks 78

Note that these are the commands I used, although they generate 400gb
data and then only sort 10%. Clearly, it is a bit faster to just
generate 40gb and sort all of it. I'm just going to run the bigger
sort in the next couple of days.
It is in the example directory of 0.15 too. The only remaining piece,
is the pipes sort program and I'll upload that to HADOOP-2127.

-- Owen