Grokbase Groups Pig user March 2012
FAQ
What is the number of reduce shuffle bytes for this job? Also, is this
job CPU intensive on reducers or is it simple aggregation?

Sent from my iPhone
On Mar 13, 2012, at 5:25 AM, Austin Chungath wrote:

Hi,
I am running a pig query on around 500 GB input data.
The current block size is 128 MB and split size is the default 128 MB.
I have also specified 16 reducers and around 3800 mappers are running.

Now I observe that shuffling is taking a long time to complete execution,
approximately 25 mins per job.

Can anyone suggest how I can bring down the shuffling time? Is there any
property that I can tweak to improve performance?

Thanks & Regards,
Austin

Search Discussions

Discussion Posts

Previous

Follow ups

Related Discussions

Discussion Navigation
viewthread | post
posts ‹ prev | 2 of 5 | next ›
Discussion Overview
groupuser @
categoriespig, hadoop
postedMar 13, '12 at 12:25p
activeMar 14, '12 at 9:12p
posts5
users2
websitepig.apache.org

People

Translate

site design / logo © 2021 Grokbase