FAQ
Hi,

We have a Pipes C++ application where the Reduce task does a lot of
computation. After some time the task gets killed by the Hadoop
framework. The job output shows the following error:

Task task_200803051654_0001_r_000000_0 failed to report status for 604
seconds. Killing!

Is there any way to send a heartbeat to the TaskTracker from a Pipes
application. I believe this is possible in Java using
org.apache.hadoop.util.Progress and we're looking for something
equivalent in the C++ Pipes API.

-Rahul Sood
rsood@yahoo-inc.com

Search Discussions

  • Richard Kasperski at Mar 5, 2008 at 5:43 pm
    I think you just need to write to stderr. My understanding is that
    hadoop is happy as long as input is being consumed, output is being
    generated or status is being generated.

    Rahul Sood wrote:
    Hi,

    We have a Pipes C++ application where the Reduce task does a lot of
    computation. After some time the task gets killed by the Hadoop
    framework. The job output shows the following error:

    Task task_200803051654_0001_r_000000_0 failed to report status for 604
    seconds. Killing!

    Is there any way to send a heartbeat to the TaskTracker from a Pipes
    application. I believe this is possible in Java using
    org.apache.hadoop.util.Progress and we're looking for something
    equivalent in the C++ Pipes API.

    -Rahul Sood
    rsood@yahoo-inc.com

  • Miles Osborne at Mar 5, 2008 at 6:58 pm
    Is this also true for streaming?

    Miles
    On 05/03/2008, Richard Kasperski wrote:

    I think you just need to write to stderr. My understanding is that
    hadoop is happy as long as input is being consumed, output is being
    generated or status is being generated.


    Rahul Sood wrote:
    Hi,

    We have a Pipes C++ application where the Reduce task does a lot of
    computation. After some time the task gets killed by the Hadoop
    framework. The job output shows the following error:

    Task task_200803051654_0001_r_000000_0 failed to report status for 604
    seconds. Killing!

    Is there any way to send a heartbeat to the TaskTracker from a Pipes
    application. I believe this is possible in Java using
    org.apache.hadoop.util.Progress and we're looking for something
    equivalent in the C++ Pipes API.

    -Rahul Sood
    rsood@yahoo-inc.com


    --
    The University of Edinburgh is a charitable body, registered in Scotland,
    with registration number SC005336.
  • Vlad Kudelin at Mar 5, 2008 at 9:20 pm
    There is or was an option you could try to play with, smth. like
    -jobconf mapred.task.timeout=600000
    The number is in milliseconds.

    // I actually needed to use it to *decrease* the default timeout which I
    believe is quite big; at times things hang, and this is a possible
    workaround to get a hung one killed by the f/w.

    PS: I totally am with Richard's know-how: hadoop doesn't care whether
    your app has finished (and how it's finished); all what matters is that
    stdin is consumed...

    Vlad.


    Richard Kasperski wrote:
    I think you just need to write to stderr. My understanding is that
    hadoop is happy as long as input is being consumed, output is being
    generated or status is being generated.

    Rahul Sood wrote:
    Hi,

    We have a Pipes C++ application where the Reduce task does a lot of
    computation. After some time the task gets killed by the Hadoop
    framework. The job output shows the following error:

    Task task_200803051654_0001_r_000000_0 failed to report status for 604
    seconds. Killing!

    Is there any way to send a heartbeat to the TaskTracker from a Pipes
    application. I believe this is possible in Java using
    org.apache.hadoop.util.Progress and we're looking for something
    equivalent in the C++ Pipes API.

    -Rahul Sood
    rsood@yahoo-inc.com

  • Owen O'Malley at Mar 7, 2008 at 8:12 am

    On Mar 5, 2008, at 9:31 AM, Rahul Sood wrote:

    Hi,

    We have a Pipes C++ application where the Reduce task does a lot of
    computation. After some time the task gets killed by the Hadoop
    framework. The job output shows the following error:

    Task task_200803051654_0001_r_000000_0 failed to report status for 604
    seconds. Killing!

    Is there any way to send a heartbeat to the TaskTracker from a Pipes
    application. I believe this is possible in Java using
    org.apache.hadoop.util.Progress and we're looking for something
    equivalent in the C++ Pipes API.
    The context object has a progress method that should be called during
    long computations...
    http://tinyurl.com/yt7hyx
    search for progress...


    -- Owen

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupcommon-user @
categorieshadoop
postedMar 5, '08 at 5:32p
activeMar 7, '08 at 8:12a
posts5
users5
websitehadoop.apache.org...
irc#hadoop

People

Translate

site design / logo © 2022 Grokbase