Grokbase Groups Pig user August 2009
FAQ
Hi,

I haven't seen this before but nightly jobs failed over the weekend because
due to memory issues. The weird part is the jobs failed during the map phase
(at about ~98% complete).

The task tracker for the failed map jobs shows the following errors:

Task attempt_200908100026_0065_m_000002_0 failed to report status for
602 seconds. Killing!
Task attempt_200908100026_0065_m_000002_1 failed to report status for
603 seconds. Killing!

The logs indicate memory to be the issue:

2009-08-10 11:53:37.829 INFO [Low Memory Detector]
org.apache.pig.impl.util.SpillableMemoryManager - low memory handler
called (Collection threshold exceeded) init = 5439488(5312K) used =
287290336(280556K) committed = 363593728(355072K) max =
536870912(524288K)

2009-08-10 11:53:43.522 INFO [Low Memory Detector]
org.apache.pig.impl.util.SpillableMemoryManager - low memory handler
called (Collection threshold exceeded) init = 5439488(5312K) used =
350217672(342009K) committed = 422510592(412608K) max =
536870912(524288K)

2009-08-10 11:53:45.290 INFO [Low Memory Detector]
org.apache.pig.impl.util.SpillableMemoryManager - low memory handler
called (Usage threshold exceeded) init = 5439488(5312K) used =
376781240(367950K) committed = 422510592(412608K) max =
536870912(524288K)

2009-08-10 11:53:45.290 INFO [Low Memory Detector]
org.apache.pig.impl.util.SpillableMemoryManager - low memory handler
called (Collection threshold exceeded) init = 5439488(5312K) used =
380504752(371586K) committed = 456720384(446016K) max =
536870912(524288K)

2009-08-10 11:53:46.752 INFO [Low Memory Detector]
org.apache.pig.impl.util.SpillableMemoryManager - low memory handler
called (Collection threshold exceeded) init = 5439488(5312K) used =
401755464(392339K) committed = 482344960(471040K) max =
536870912(524288K)

2009-08-10 11:53:50.599 INFO [Low Memory Detector]
org.apache.pig.impl.util.SpillableMemoryManager - low memory handler
called (Collection threshold exceeded) init = 5439488(5312K) used =
443763584(433362K) committed = 527171584(514816K) max =
536870912(524288K)

2009-08-10 11:53:54.686 INFO [Low Memory Detector]
org.apache.pig.impl.util.SpillableMemoryManager - low memory handler
called (Collection threshold exceeded) init = 5439488(5312K) used =
491575560(480054K) committed = 536870912(524288K) max =
536870912(524288K)

2009-08-10 11:53:56.414 INFO [Low Memory Detector]
org.apache.pig.impl.util.SpillableMemoryManager - low memory handler
called (Collection threshold exceeded) init = 5439488(5312K) used =
514928920(502860K) committed = 536870912(524288K) max =
536870912(524288K)

2009-08-10 11:53:57.553 INFO [Low Memory Detector]
org.apache.pig.impl.util.SpillableMemoryManager - low memory handler
called (Collection threshold exceeded) init = 5439488(5312K) used =
520781832(508576K) committed = 536870912(524288K) max =
536870912(524288K)

2009-08-10 11:53:58.747 INFO [Low Memory Detector]
org.apache.pig.impl.util.SpillableMemoryManager - low memory handler
called (Collection threshold exceeded) init = 5439488(5312K) used =
526636552(514293K) committed = 536870912(524288K) max =
536870912(524288K)

2009-08-10 11:53:59.935 INFO [Low Memory Detector]
org.apache.pig.impl.util.SpillableMemoryManager - low memory handler
called (Collection threshold exceeded) init = 5439488(5312K) used =
532493568(520013K) committed = 536870912(524288K) max =
536870912(524288K)

2009-08-10 11:54:01.158 INFO [Low Memory Detector]
org.apache.pig.impl.util.SpillableMemoryManager - low memory handler
called (Collection threshold exceeded) init = 5439488(5312K) used =
536870904(524287K) committed = 536870912(524288K) max =
536870912(524288K)

2009-08-10 11:54:02.389 INFO [Low Memory Detector]
org.apache.pig.impl.util.SpillableMemoryManager - low memory handler
called (Collection threshold exceeded) init = 5439488(5312K) used =
536870904(524287K) committed = 536870912(524288K) max =
536870912(524288K)

2009-08-10 11:54:03.778 INFO [Low Memory Detector]
org.apache.pig.impl.util.SpillableMemoryManager - low memory handler
called (Collection threshold exceeded) init = 5439488(5312K) used =
489852536(478371K) committed = 536870912(524288K) max =
536870912(524288K)

2009-08-10 12:03:40.298 WARN [Comm thread for
attempt_200908100026_0065_m_000077_1]
org.apache.hadoop.mapred.TaskRunner - Parent died. Exiting
attempt_200908100026_0065_m_000077_1

I have seen this before when jobs fail on the reduce phase but this is the
first time I am noticing jobs failing during the map phase. Surprisingly,
jobs that load and process much more data ran successfully but when I tried
running the ones that failed, they failed again. Some of the jobs that
failed do nothing more than, load, filter and write out the filtered data.
This leads me to believe that the problem is more specific than I had
originally thought. Any pointers on what the issue might be will be
extremely helpful.

Thanks,

Krishna

Search Discussions

Discussion Posts

Follow ups

Related Discussions

Discussion Navigation
viewthread | post
posts ‹ prev | 1 of 5 | next ›
Discussion Overview
groupuser @
categoriespig, hadoop
postedAug 10, '09 at 8:00p
activeAug 11, '09 at 4:04p
posts5
users3
websitepig.apache.org

People

Translate

site design / logo © 2021 Grokbase