Grokbase Groups Pig user May 2008
FAQ
Hi Tanton,

I am having the same problem, but I have got an Out of Memory Exception
in the reduce phase. Which Hadoop config parameter did you change? Is it
the io.seqfile.compress.blocksize?

My current value for this parameter is:

<property>
<name>io.seqfile.compress.blocksize</name>
<value>1000000</value>
<description>The minimum block size for compression in block compressed
SequenceFiles.
</description>
</property>

The logs I got:

2008-05-27 11:05:38,087 INFO org.apache.pig.impl.util.SpillableMemoryManager: low memory handler called init = 5439488(5312K) used = 273852568(267434K) committed = 279642112(273088K) max = 279642112(273088K)
2008-05-27 11:05:38,802 INFO org.apache.pig.impl.util.SpillableMemoryManager: low memory handler called init = 5439488(5312K) used = 273860104(267441K) committed = 279642112(273088K) max = 279642112(273088K)
2008-05-27 11:05:44,893 INFO org.apache.pig.impl.util.SpillableMemoryManager: low memory handler called init = 5439488(5312K) used = 279642088(273087K) committed = 279642112(273088K) max = 279642112(273088K)
2008-05-27 11:05:52,704 INFO org.apache.pig.impl.util.SpillableMemoryManager: low memory handler called init = 5439488(5312K) used = 217972656(212863K) committed = 279642112(273088K) max = 279642112(273088K)
2008-05-27 11:05:56,510 INFO org.apache.pig.impl.util.SpillableMemoryManager: low memory handler called init = 5439488(5312K) used = 269271376(262960K) committed = 279642112(273088K) max = 279642112(273088K)
2008-05-27 11:06:03,686 INFO org.apache.pig.impl.util.SpillableMemoryManager: low memory handler called init = 5439488(5312K) used = 244418296(238689K) committed = 279642112(273088K) max = 279642112(273088K)
2008-05-27 11:06:10,610 INFO org.apache.pig.impl.util.SpillableMemoryManager: low memory handler called init = 5439488(5312K) used = 269740120(263418K) committed = 279642112(273088K) max = 279642112(273088K)
2008-05-27 11:06:16,370 INFO org.apache.pig.impl.util.SpillableMemoryManager: low memory handler called init = 5439488(5312K) used = 271831992(265460K) committed = 279642112(273088K) max = 279642112(273088K)
2008-05-27 11:06:19,813 INFO org.apache.pig.impl.util.SpillableMemoryManager: low memory handler called init = 5439488(5312K) used = 258029960(251982K) committed = 279642112(273088K) max = 279642112(273088K)
2008-05-27 11:06:23,948 INFO org.apache.pig.impl.util.SpillableMemoryManager: low memory handler called init = 5439488(5312K) used = 279642104(273087K) committed = 279642112(273088K) max = 279642112(273088K)
2008-05-27 11:06:27,208 INFO org.apache.pig.impl.util.SpillableMemoryManager: low memory handler called init = 5439488(5312K) used = 195321216(190743K) committed = 279642112(273088K) max = 279642112(273088K)
2008-05-27 11:06:30,932 INFO org.apache.pig.impl.util.SpillableMemoryManager: low memory handler called init = 5439488(5312K) used = 266489648(260243K) committed = 279642112(273088K) max = 279642112(273088K)
2008-05-27 11:06:34,463 INFO org.apache.pig.impl.util.SpillableMemoryManager: low memory handler called init = 5439488(5312K) used = 279642088(273087K) committed = 279642112(273088K) max = 279642112(273088K)
2008-05-27 11:06:38,214 INFO org.apache.pig.impl.util.SpillableMemoryManager: low memory handler called init = 5439488(5312K) used = 279642112(273088K) committed = 279642112(273088K) max = 279642112(273088K)
2008-05-27 11:06:44,571 INFO org.apache.pig.impl.util.SpillableMemoryManager: low memory handler called init = 5439488(5312K) used = 268382184(262091K) committed = 279642112(273088K) max = 279642112(273088K)
2008-05-27 11:11:02,570 INFO org.apache.hadoop.mapred.TaskRunner: Communication exception: java.lang.OutOfMemoryError: Java heap space

2008-05-27 11:11:02,571 ERROR org.apache.pig.backend.hadoop.executionengine.mapreduceExec.PigMapReduce: java.lang.OutOfMemoryError: Java heap space
2008-05-27 11:11:03,234 INFO org.apache.hadoop.ipc.Client: java.net.SocketException: Socket closed
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:129)
at java.io.FilterInputStream.read(FilterInputStream.java:116)
at org.apache.hadoop.ipc.Client$Connection$1.read(Client.java:190)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
at java.io.BufferedInputStream.read(BufferedInputStream.java:237)
at java.io.DataInputStream.readInt(DataInputStream.java:370)
at org.apache.hadoop.ipc.Client$Connection.run(Client.java:276)

Thanks,
Iván


El vie, 23-05-2008 a las 10:23 -0500, Tanton Gibbs escribió:
I upped my maximum memory from 1024M to 2048M and the problem went
away. I think the problem was that my sortable memory was already set
to 400M so it was very close to the 50% mark already.

Is there a way to up the spillable threshold to 80%?
On Fri, May 23, 2008 at 10:04 AM, Tanton Gibbs wrote:
It is in a map phase. I don't think I used a custom chunker. My
splits are set to be 128M.
On Fri, May 23, 2008 at 9:07 AM, pi song wrote:
Dear Tanton,

This means the MemoryManager is not successful at reclaiming memory. Did
that happen in Map phase or Reduce phase? Did you use a custom chunker? How
big is your split?

Pi

On Fri, May 23, 2008 at 3:52 PM, Tanton Gibbs <tanton.gibbs@gmail.com>
wrote:
I upgraded to hadoop 17 and the latest Pig from svn.

I'm now getting a ton of lines in my log files that say:

2008-05-23 00:49:27,832 INFO
org.apache.pig.impl.util.SpillableMemoryManager: low memory handler
called init = 1441792(1408K) used = 483176072(471851K) committed =
641335296(626304K) max = 954466304(932096K)

In addition, jobs on big files are running very slowly.

Does anyone have any ideas as to what I could have screwed up?

Thanks!
Tanton

Search Discussions

Discussion Posts

Previous

Follow ups

Related Discussions

Discussion Navigation
viewthread | post
posts ‹ prev | 5 of 12 | next ›
Discussion Overview
groupuser @
categoriespig, hadoop
postedMay 23, '08 at 5:52a
activeMay 30, '08 at 4:32p
posts12
users5
websitepig.apache.org

People

Translate

site design / logo © 2021 Grokbase