[Hadoop-common-user] Poor IO performance on a 10 node cluster.

That's because you are assuming that processing time for mappers and reducers to be 0? Counting words is processor intensive and it's likely that lot of those 220 seconds are spent in processing, not just reading the file.

Probably not, at least not out-of-the-box. Things to tune:
1) Number of threads per disk
2) Size of the block size (yours seems relatively small)
3) Network bottlenecks (can be seen using Ganglia)
4) Related to (3), the number of replicas.
5) Selection of the Linux I/O scheduler. Default CFQ scheduler is inappropriate for batch workloads.

Finally, if you don't have enough host-level monitoring to indicate the current bottleneck (CPU, memory, network, or I/O?), you likely won't ever be able to solve this riddle

Brian

Ljddfjfjfififfifjftjiiiiiifjfjjjffkxbznzsjxodiewisshsudddudsjidhddueiweefiuftttoitfiirriifoiffkllddiririiriioerorooiieirrioeekroooeoooirjjfdijdkkduddjudiiehs

Not sure that will help ;)

Sent from my mobile. Please excuse the typos.

Psst. The cats speak in their own language ;-)

Hi Gyuribácsi

I would suggest you divide MapReduce program execution time into 3 parts

a) Map Stage
In this stage, wc splits input data and generates map tasks. Each map task
process one block (in default, you can change it in FileInputFormat.java).
As Brian said, if you have larger blocks size, you may have less number of
map tasks, and then probably less overhead.

b) Reduce Stage
2) shuffle phase
In this phase, reduce task collect intermediate results from every node
that has executed map tasks. Each reduce task can have many current threads
to obtain data(you can configure it in mapred-site.xml, it is
"mapreduce.reduce.shuffle.parallelcopies"). But, be careful to your data
popularity. For example, you have "Hadoop, Hadoop, Hadoop,hello". The
default Hadoop partitioner will assign 3 <Hadoop, 1> key-value pairs to one
node. Thus, if you have two nodes run reduce tasks, one of them will copy 3
times ｍｏｒｅ data than the other. This will cause one node slower than the
other. You may rewrite the partitioner.

3) sort and reduce phase
I think the Hadoop UI ｗｉｌｌ ｇｉｖｅ ｙｏｕ ｓｏｍｅ ｈｉｎｔｓ ａｂｏｕｔ ｈｏｗ ｌｏｎｇ ｔｈｉｓ ｐｈａｓｅ
ｔａｋｅｓ．

Ｂｙ ｄｉｖｉｄｉｎｇ ＭａｐＲｅｄｕｃｅ ａｐｐｌｉｃａｔｉｏｎ ｉｎｔｏ ｔｈｅｓｅ ３ ｐａｒｔｓ， ｙｏｕ ｃａｎ ｅａｓｉｌｙ ｆｉｎｄ
ｗｈｉｃｈ ｏｎｅ ｉｓ ｙｏｕｒ ｂｏｔｔｌｅｎｅｃｋ ａｎｄ ｄｏ ｓｏｍｅ ｐｒｏｆｉｌｉｎｇ． Ａｎｄ Ｉ ｄｏｎ＇ｔ ｋｎｏｗ ｗｈｙ ｍｙ
ｆｏｎｔ ｃｈａｎｇｅ ｔｏ ｔｈｉｓ ｔｙｐｅ．：（

Ｈｏｐｅ ｉｔ ｗｉｌｌ ｂｅ ｈｅｌｐｆｕｌ．
Ｃｈｅｎ

Your Font block size got increased dynamically , check in core-site :) :)

- Jagaran



________________________________
From: He Chen <[email protected]>
To: [email protected]
Sent: Mon, 30 May, 2011 11:39:35 AM
Subject: Re: Poor IO performance on a 10 node cluster.

Hi Gyuribácsi

I would suggest you divide MapReduce program execution time into 3 parts

a) Map Stage
In this stage, wc splits input data and generates map tasks. Each map task
process one block (in default, you can change it in FileInputFormat.java).
As Brian said, if you have larger blocks size, you may have less number of
map tasks, and then probably less overhead.

b) Reduce Stage
2) shuffle phase
In this phase, reduce task collect intermediate results from every node
that has executed map tasks. Each reduce task can have many current threads
to obtain data(you can configure it in mapred-site.xml, it is
"mapreduce.reduce.shuffle.parallelcopies"). But, be careful to your data
popularity. For example, you have "Hadoop, Hadoop, Hadoop,hello". The
default Hadoop partitioner will assign 3 <Hadoop, 1> key-value pairs to one
node. Thus, if you have two nodes run reduce tasks, one of them will copy 3
times ｍｏｒｅ data than the other. This will cause one node slower than the
other. You may rewrite the partitioner.

3) sort and reduce phase
I think the Hadoop UI ｗｉｌｌ ｇｉｖｅ ｙｏｕ ｓｏｍｅ ｈｉｎｔｓ ａｂｏｕｔ ｈｏｗ ｌｏｎｇ ｔｈｉｓ ｐｈａｓｅ
ｔａｋｅｓ．

Ｂｙ ｄｉｖｉｄｉｎｇ ＭａｐＲｅｄｕｃｅ ａｐｐｌｉｃａｔｉｏｎ ｉｎｔｏ ｔｈｅｓｅ ３ ｐａｒｔｓ， ｙｏｕ ｃａｎ ｅａｓｉｌｙ ｆｉｎｄ
ｗｈｉｃｈ ｏｎｅ ｉｓ ｙｏｕｒ ｂｏｔｔｌｅｎｅｃｋ ａｎｄ ｄｏ ｓｏｍｅ ｐｒｏｆｉｌｉｎｇ． Ａｎｄ Ｉ ｄｏｎ＇ｔ ｋｎｏｗ ｗｈｙ ｍｙ
ｆｏｎｔ ｃｈａｎｇｅ ｔｏ ｔｈｉｓ ｔｙｐｅ．：（

Ｈｏｐｅ ｉｔ ｗｉｌｌ ｂｅ ｈｅｌｐｆｕｌ．
Ｃｈｅｎ

That's a small town in Iceland.

I'm sorry, but she's with me now.

On Mon, May 30, 2011 at 8:22 AM, Boris Aleksandrovsky
wrote:

Some things which helped us include setting your vm.swappiness to 0 and
mounting your disks with noatime,nodiratime options.

Also make sure your disks aren't setup with RAID (JBOD is recommended)

You might want to run terasort as you tweak your environment. It's very
helpful when checking if a change helped (or hurt) your cluster.

Hope that helps a bit.

It is also worth using dd to verify your raw disk speeds.

Also, expressing disk transfer rates in bytes per second makes it a bit
easier for most of the disk people I know to figure out what is large or
small.

Each of these disks disk should do about 100MB/s when driven well. Hadoop
does OK, but not nearly full capacity so I would expected more like
40-50MB/s/disk. Also, if one of your disks is doing double duty you may
have some extra cost. 40 x 2 x 10 = 800MB per second. You are doing 100GB
/ 220 seconds = 0.5 GB / s which isn't so terribly bad. It is definitely
less than the theoretically possible 100 x 2 x 10 = 2GB/second and I would
expect you could tune this up a little, but not a massive amount.

In general, blade servers do not make good Hadoop nodes exactly because the
I/O performance tends to be low when you only have a few spindle.

One other reason that this might be a bit below expectations is that your
files may not be well distributed on your cluster. Can you say what you
used to upload the files?

Hi,

Is this the speed you are observing when initial writes of the files
happening (i.e while you are initially putting 10gb files with replication)

Regards,
Raja Nagendra Kumar


Gyuribácsi wrote: