|| at Aug 26, 2010 at 5:50 am
I agree with you on the most part. But I have some other questions. mapper are working on local machine so there's no network transfers during this process, if the original data stored in hdfs is compressed it will only decrease the IO time. One major point is I doubt whether the mapper can deal with only part of the whole data if the data is compressed which seems can't be split ? I've try to do a "select sum()" in hive and trace the job, it seems the .tar.gz data can only worked on one single matchine and stuck there for quite a long time(seems like need to wait other part of data be copied from other machines),while other data not compressed can work on different machines parallelly. Do you know something about this ?
发件人： Harsh J
发送时间： 2010-08-26 12:15:49
主题： Re: data in compression format affect mapreduce speed
Logically it 'should' increase time as its an extra step beyond the
Mapper/Reducer. But while your processing time would slightly (very
very slightly) increase, your IO and Network Transfers time would
decrease by a large margin -- giving you a clear impression that your
total job time has decreased overall. The difference being in writing
out say 10 GB before, and writing out 5-7 GB this time (a crude
With the fast CPUs available these days, compressing and decompressing
should hardly take a noticeable amount of extra time. Its almost
negligible in case of using gzip, lzo or plain deflate.
On Thu, Aug 26, 2010 at 9:13 AM, Ted Yu wrote:
Compressed data would increase processing time in mapper/reducer but
decrease the amount of data transferred between tasktracker nodes.
Normally you should consider applying some form of compression.
On Wed, Aug 25, 2010 at 7:32 PM, shangan wrote:
will data stored in compression format affect mapreduce job speed?
increase or decrease? or more complex relationship between these two ? can
anybody give some explanation in detail?
__________ Information from ESET NOD32 Antivirus, version of virus signature database 5397 (20100825) __________
The message was checked by ESET NOD32 Antivirus.http://www.eset.com