FAQ
Hi all

I have a big map task that takes a long time to complete and produces
a lot of information.
The merge part makes the task timeout (you can see this in the end of
this email, the merge part is aborted after ten minutes, the default
time).
I've increased the mapred.tasks.timeout property to 30 min instead of
10, in hadoop-site.xml as follows:
<property> <name>mapred.task.timeout</name> <value>1800000</value> </
property>
But stiil the task fails with:
Task attempt_200811201704_0001_m_000000_0 failed to report status for
603 seconds. Killing!

Is there any other property I should change?.

Regards
David Alves



17:35:40,697 INFO [MapTask] Starting flush of map output
17:35:40,697 INFO [MapTask] bufstart = 17472260; bufend = 51354916;
bufvoid = 99614720
17:35:40,697 INFO [MapTask] kvstart = 39119; kvend = 39468; length =
327680
17:35:40,950 INFO [MapTask] Index: (0, 33884416, 33884416)
17:35:40,950 INFO [MapTask] Finished spill 152
17:35:45,333 INFO [Merger] Merging 153 sorted segments
17:35:46,337 INFO [Merger] Merging 9 intermediate segments out of a
total of 153
17:36:16,849 INFO [Merger] Merging 10 intermediate segments out of a
total of 145
17:36:47,615 INFO [Merger] Merging 10 intermediate segments out of a
total of 136
17:37:21,529 INFO [Merger] Merging 10 intermediate segments out of a
total of 127
17:37:59,883 INFO [Merger] Merging 10 intermediate segments out of a
total of 118
17:38:35,370 INFO [Merger] Merging 10 intermediate segments out of a
total of 109
17:39:14,795 INFO [Merger] Merging 10 intermediate segments out of a
total of 100
17:39:51,787 INFO [Merger] Merging 10 intermediate segments out of a
total of 91
17:40:28,721 INFO [Merger] Merging 10 intermediate segments out of a
total of 82
17:41:05,650 INFO [Merger] Merging 10 intermediate segments out of a
total of 73
17:41:43,285 INFO [Merger] Merging 10 intermediate segments out of a
total of 64
17:42:23,531 INFO [Merger] Merging 10 intermediate segments out of a
total of 55
17:43:01,709 INFO [Merger] Merging 10 intermediate segments out of a
total of 46
17:43:40,209 INFO [Merger] Merging 10 intermediate segments out of a
total of 37
17:44:20,707 INFO [Merger] Merging 10 intermediate segments out of a
total of 28
17:44:57,700 INFO [Merger] Merging 10 intermediate segments out of a
total of 19

Search Discussions

  • David Alves at Nov 21, 2008 at 10:51 am
    Hi again

    Browsing the source code (Merger.class) I see that merger actually
    call reporter.progress() so shouldn't this make the task be reported
    as still working?

    Regards
    David Alves

    On Nov 20, 2008, at 6:29 PM, David Alves wrote:

    Hi all

    I have a big map task that takes a long time to complete and
    produces a lot of information.
    The merge part makes the task timeout (you can see this in the end
    of this email, the merge part is aborted after ten minutes, the
    default time).
    I've increased the mapred.tasks.timeout property to 30 min instead
    of 10, in hadoop-site.xml as follows:
    <property> <name>mapred.task.timeout</name> <value>1800000</value>
    </property>
    But stiil the task fails with:
    Task attempt_200811201704_0001_m_000000_0 failed to report status
    for 603 seconds. Killing!

    Is there any other property I should change?.

    Regards
    David Alves



    17:35:40,697 INFO [MapTask] Starting flush of map output
    17:35:40,697 INFO [MapTask] bufstart = 17472260; bufend = 51354916;
    bufvoid = 99614720
    17:35:40,697 INFO [MapTask] kvstart = 39119; kvend = 39468; length
    = 327680
    17:35:40,950 INFO [MapTask] Index: (0, 33884416, 33884416)
    17:35:40,950 INFO [MapTask] Finished spill 152
    17:35:45,333 INFO [Merger] Merging 153 sorted segments
    17:35:46,337 INFO [Merger] Merging 9 intermediate segments out of a
    total of 153
    17:36:16,849 INFO [Merger] Merging 10 intermediate segments out of
    a total of 145
    17:36:47,615 INFO [Merger] Merging 10 intermediate segments out of
    a total of 136
    17:37:21,529 INFO [Merger] Merging 10 intermediate segments out of
    a total of 127
    17:37:59,883 INFO [Merger] Merging 10 intermediate segments out of
    a total of 118
    17:38:35,370 INFO [Merger] Merging 10 intermediate segments out of
    a total of 109
    17:39:14,795 INFO [Merger] Merging 10 intermediate segments out of
    a total of 100
    17:39:51,787 INFO [Merger] Merging 10 intermediate segments out of
    a total of 91
    17:40:28,721 INFO [Merger] Merging 10 intermediate segments out of
    a total of 82
    17:41:05,650 INFO [Merger] Merging 10 intermediate segments out of
    a total of 73
    17:41:43,285 INFO [Merger] Merging 10 intermediate segments out of
    a total of 64
    17:42:23,531 INFO [Merger] Merging 10 intermediate segments out of
    a total of 55
    17:43:01,709 INFO [Merger] Merging 10 intermediate segments out of
    a total of 46
    17:43:40,209 INFO [Merger] Merging 10 intermediate segments out of
    a total of 37
    17:44:20,707 INFO [Merger] Merging 10 intermediate segments out of
    a total of 28
    17:44:57,700 INFO [Merger] Merging 10 intermediate segments out of
    a total of 19

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupcommon-user @
categorieshadoop
postedNov 20, '08 at 6:33p
activeNov 21, '08 at 10:51a
posts2
users1
websitehadoop.apache.org...
irc#hadoop

1 user in discussion

David Alves: 2 posts

People

Translate

site design / logo © 2022 Grokbase