Grokbase Groups Pig user May 2008
I upped my maximum memory from 1024M to 2048M and the problem went
away. I think the problem was that my sortable memory was already set
to 400M so it was very close to the 50% mark already.

Is there a way to up the spillable threshold to 80%?
On Fri, May 23, 2008 at 10:04 AM, Tanton Gibbs wrote:
It is in a map phase. I don't think I used a custom chunker. My
splits are set to be 128M.
On Fri, May 23, 2008 at 9:07 AM, pi song wrote:
Dear Tanton,

This means the MemoryManager is not successful at reclaiming memory. Did
that happen in Map phase or Reduce phase? Did you use a custom chunker? How
big is your split?


On Fri, May 23, 2008 at 3:52 PM, Tanton Gibbs <>
I upgraded to hadoop 17 and the latest Pig from svn.

I'm now getting a ton of lines in my log files that say:

2008-05-23 00:49:27,832 INFO
org.apache.pig.impl.util.SpillableMemoryManager: low memory handler
called init = 1441792(1408K) used = 483176072(471851K) committed =
641335296(626304K) max = 954466304(932096K)

In addition, jobs on big files are running very slowly.

Does anyone have any ideas as to what I could have screwed up?


Search Discussions

Discussion Posts


Follow ups

Related Discussions

Discussion Navigation
viewthread | post
posts ‹ prev | 4 of 12 | next ›
Discussion Overview
groupuser @
categoriespig, hadoop
postedMay 23, '08 at 5:52a
activeMay 30, '08 at 4:32p



site design / logo © 2021 Grokbase