FAQ
I am running a map reduce ob where a few reduce tasks fail with an out of
memory error -
Increasing the memory is not an option. However if a retry had information
that an earlier attempt
failed out of memory and especially it it had access to a few numbers
describing how far the earlier attempt
managed to get, it could defend against the error
I have seen little information about how a retried task might access the
error logs or other information
from previous attempts - is there such a mechanism???


--
Steven M. Lewis PhD
Institute for Systems Biology
Seattle WA

Search Discussions

  • Ted Yu at Jul 13, 2010 at 11:53 pm
    A general solution for OOME is to reduce the size of input to (reduce) task
    so that each (reduce) task consumes less memory.
    On Tue, Jul 13, 2010 at 10:16 AM, Steve Lewis wrote:

    I am running a map reduce ob where a few reduce tasks fail with an out of
    memory error -
    Increasing the memory is not an option. However if a retry had information
    that an earlier attempt
    failed out of memory and especially it it had access to a few numbers
    describing how far the earlier attempt
    managed to get, it could defend against the error
    I have seen little information about how a retried task might access the
    error logs or other information
    from previous attempts - is there such a mechanism???


    --
    Steven M. Lewis PhD
    Institute for Systems Biology
    Seattle WA
  • Steve Lewis at Jul 14, 2010 at 1:59 am
    Yes - of course but the question is whether there is a way to do it while
    the job is running rather than
    restarting with different parameter
    On Tue, Jul 13, 2010 at 4:51 PM, Ted Yu wrote:

    A general solution for OOME is to reduce the size of input to (reduce) task
    so that each (reduce) task consumes less memory.

    On Tue, Jul 13, 2010 at 10:16 AM, Steve Lewis wrote:

    I am running a map reduce ob where a few reduce tasks fail with an out of
    memory error -
    Increasing the memory is not an option. However if a retry had information
    that an earlier attempt
    failed out of memory and especially it it had access to a few numbers
    describing how far the earlier attempt
    managed to get, it could defend against the error
    I have seen little information about how a retried task might access the
    error logs or other information
    from previous attempts - is there such a mechanism???


    --
    Steven M. Lewis PhD
    Institute for Systems Biology
    Seattle WA

    --
    Steven M. Lewis PhD
    Institute for Systems Biology
    Seattle WA
  • Ted Yu at Jul 14, 2010 at 2:41 am
    Feel free to comment on https://issues.apache.org/jira/browse/MAPREDUCE-1928
    On Tue, Jul 13, 2010 at 6:57 PM, Steve Lewis wrote:

    Yes - of course but the question is whether there is a way to do it while
    the job is running rather than
    restarting with different parameter

    On Tue, Jul 13, 2010 at 4:51 PM, Ted Yu wrote:

    A general solution for OOME is to reduce the size of input to (reduce)
    task so that each (reduce) task consumes less memory.

    On Tue, Jul 13, 2010 at 10:16 AM, Steve Lewis wrote:

    I am running a map reduce ob where a few reduce tasks fail with an out of
    memory error -
    Increasing the memory is not an option. However if a retry had
    information that an earlier attempt
    failed out of memory and especially it it had access to a few numbers
    describing how far the earlier attempt
    managed to get, it could defend against the error
    I have seen little information about how a retried task might access the
    error logs or other information
    from previous attempts - is there such a mechanism???


    --
    Steven M. Lewis PhD
    Institute for Systems Biology
    Seattle WA

    --
    Steven M. Lewis PhD
    Institute for Systems Biology
    Seattle WA

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupmapreduce-user @
categorieshadoop
postedJul 13, '10 at 5:17p
activeJul 14, '10 at 2:41a
posts4
users2
websitehadoop.apache.org...
irc#hadoop

2 users in discussion

Ted Yu: 2 posts Steve Lewis: 2 posts

People

Translate

site design / logo © 2022 Grokbase