Grokbase Groups Pig user March 2009
FAQ
Hello Pig List,

I am now taking my (tested) pig script that will produce distinct counts and trying to apply it to real data. I am finding however, that though the map stage completes (100%), the reduce stage hangs at 97.77% and then fails to produce output.

It appears that the syslog contains notices of "threshold exceeded" before the ultimate failure...

2009-03-19 10:54:10,525 INFO org.apache.pig.impl.util.SpillableMemoryManager: low memory handler called (Usage threshold exceeded) init = 1441792(1408K) used = 131343896(128265K) committed = 186449920(182080K) max = 186449920(182080K)
2009-03-19 10:54:18,150 INFO org.apache.pig.impl.util.SpillableMemoryManager: low memory handler called (Usage threshold exceeded) init = 1441792(1408K) used = 131311248(128233K) committed = 186449920(182080K) max = 186449920(182080K)
2009-03-19 10:54:25,833 INFO org.apache.pig.impl.util.SpillableMemoryManager: low memory handler called (Usage threshold exceeded) init = 1441792(1408K) used = 133580568(130449K) committed = 186449920(182080K) max = 186449920(182080K)

...

Does this mean that the Hadoop cluster requires tuning?

How can I avoid this memory error?



Regards,
Avram

Search Discussions

Discussion Posts

Follow ups

Related Discussions

Discussion Navigation
viewthread | post
posts ‹ prev | 1 of 8 | next ›
Discussion Overview
groupuser @
categoriespig, hadoop
postedMar 19, '09 at 9:14p
activeMar 20, '09 at 10:33p
posts8
users4
websitepig.apache.org

People

Translate

site design / logo © 2021 Grokbase