Grokbase Groups Pig user April 2009
FAQ
I am getting the following error at Reduce stage in one of my PIG
scripts. Any suggestions of diagnosing/fixing this further?

Java.lang.OutOfMemoryError: Java heap space
at java.util.Arrays.copyOf(Arrays.java:2786)
at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:94)
at java.io.DataOutputStream.write(DataOutputStream.java:90)
at java.io.FilterOutputStream.write(FilterOutputStream.java:80)
at org.apache.pig.data.DataReaderWriter.writeDatum(DataReaderWriter.java:213)
at org.apache.pig.data.DefaultTuple.write(DefaultTuple.java:291)
at org.apache.pig.data.DefaultAbstractBag.write(DefaultAbstractBag.java:233)
at org.apache.pig.data.DataReaderWriter.writeDatum(DataReaderWriter.java:162)
at org.apache.pig.data.DefaultTuple.write(DefaultTuple.java:291)
at org.apache.pig.impl.io.PigNullableWritable.write(PigNullableWritable.java:83)
at org.apache.hadoop.io.serializer.WritableSerialization$WritableSerializer.serialize(WritableSerialization.java:90)
at org.apache.hadoop.io.serializer.WritableSerialization$WritableSerializer.serialize(WritableSerialization.java:77)
at org.apache.hadoop.mapred.IFile$Writer.append(IFile.java:156)
at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.spillSingleRecord(MapTask.java:857)
at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.collect(MapTask.java:467)
at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapReduce$Map.collect(PigMapReduce.java:101)
at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapBase.runPipeline(PigMapBase.java:219)
at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapBase.map(PigMapBase.java:208)
at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapReduce$Map.map(PigMapReduce.java:86)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:47)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:227)
at org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:2198)

Search Discussions

  • Vadim Zaliva at Apr 14, 2009 at 4:17 am
    Forgot to mention, that my tasktrackers are configured with:

    mapred.child.java.opts=-Xmx1024m

    Sincerely,
    Vadim
  • Vadim Zaliva at Apr 14, 2009 at 11:38 pm
    I have opened JIRA report for this issue:

    https://issues.apache.org/jira/browse/PIG-766

    I am stuck between this issue and PIG-761. I guess the only recourse for me is
    to stop using pig and start rewriting my code in Java.

    Just wanted to share my experience with PIG. It has a nice potential though.

    Vadim
  • Avram Aelony at Apr 15, 2009 at 12:54 am
    I am in the same boat in this respect.
    Unfortunately, I have to agree.
    -Avram


    -----Original Message-----
    From: Vadim Zaliva
    Sent: Tuesday, April 14, 2009 4:38 PM
    To: pig-user@hadoop.apache.org
    Subject: Re: OutOfMemoryError

    I have opened JIRA report for this issue:

    https://issues.apache.org/jira/browse/PIG-766

    I am stuck between this issue and PIG-761. I guess the only recourse for me is
    to stop using pig and start rewriting my code in Java.

    Just wanted to share my experience with PIG. It has a nice potential though.

    Vadim

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupuser @
categoriespig, hadoop
postedApr 14, '09 at 4:12a
activeApr 15, '09 at 12:54a
posts4
users2
websitepig.apache.org

2 users in discussion

Vadim Zaliva: 3 posts Avram Aelony: 1 post

People

Translate

site design / logo © 2021 Grokbase