FAQ
After my Hadoop job has successfully completed I'd like to log the total
amount of time it took. This is the "Finished in" statistic in the web UI.
How do I get this number programmatically? Is there some way I can query
the Job object? I didn't see anything in the API documentation.

Search Discussions

  • Tom Melendez at Dec 2, 2011 at 6:11 pm

    On Fri, Dec 2, 2011 at 9:57 AM, W.P. McNeill wrote:
    After my Hadoop job has successfully completed I'd like to log the total
    amount of time it took. This is the "Finished in" statistic in the web UI.
    How do I get this number programmatically? Is there some way I can query
    the Job object? I didn't see anything in the API documentation.
    This probably *doesn't* help you, but if you're using (or planning on
    using) oozie, it has a restful API that can give you this information.

    Thanks,

    Tom
  • Harsh J at Dec 2, 2011 at 8:54 pm
    I remember hitting this once in 0.20 - seems like an API limitation. The resolution we took back then was to get a list of all tasks, and get the end time with the last ended task's completion time (sort and pick). There may be other ways though - others can comment on that perhaps (metrics? job-history?)
    On 02-Dec-2011, at 11:27 PM, W.P. McNeill wrote:

    After my Hadoop job has successfully completed I'd like to log the total
    amount of time it took. This is the "Finished in" statistic in the web UI.
    How do I get this number programmatically? Is there some way I can query
    the Job object? I didn't see anything in the API documentation.
  • Raj V at Dec 3, 2011 at 12:44 am
    As Harsh said, I don't think there is a simple way to way to find when the job ended, especially after the job is completed.ÂÂ

    But cant you just wait for your job to complete and log the time when the job completed?ÂÂ

    Raj


    ________________________________
    From: Harsh J <harsh@cloudera.com>
    To: common-user@hadoop.apache.org
    Sent: Friday, December 2, 2011 12:53 PM
    Subject: Re: How do I programmatically get total job execution time?

    I remember hitting this once in 0.20 - seems like an API limitation. The resolution we took back then was to get a list of all tasks, and get the end time with the last ended task's completion time (sort and pick). There may be other ways though - others can comment on that perhaps (metrics? job-history?)
    On 02-Dec-2011, at 11:27 PM, W.P. McNeill wrote:

    After my Hadoop job has successfully completed I'd like to log the total
    amount of time it took. This is the "Finished in" statistic in the web UI.
    How do I get this number programmatically? Is there some way I can query
    the Job object? I didn't see anything in the API documentation.
    02-Dec-2011, at 11:27 PM, W.P. McNeill wrote:
    After my Hadoop job has successfully completed I'd like to log the total
    amount of time it took. This is the "Finished in" statistic in the web UI.
    How do I get this number programmatically? Is there some way I can query
    the Job object? I didn't see anything in the API documentation.

  • Praveen Sripati at Dec 3, 2011 at 6:19 am
    Hi,

    Ran a job using new MR API in stand alone mode and 0.21. Both,
    Job#getFinishTime and Job#getStartTime are returning 0. Not sure, if this
    is a bug.

    Thanks,
    Praveen
    On Sat, Dec 3, 2011 at 6:14 AM, Raj V wrote:

    As Harsh said, I don't think there is a simple way to way to find when the
    job ended, especially after the job is completed.

    But cant you just wait for your job to complete and log the time when the
    job completed?

    Raj


    ________________________________
    From: Harsh J <harsh@cloudera.com>
    To: common-user@hadoop.apache.org
    Sent: Friday, December 2, 2011 12:53 PM
    Subject: Re: How do I programmatically get total job execution time?

    I remember hitting this once in 0.20 - seems like an API limitation. The
    resolution we took back then was to get a list of all tasks, and get the
    end time with the last ended task's completion time (sort and pick). There
    may be other ways though - others can comment on that perhaps (metrics?
    job-history?)
    On 02-Dec-2011, at 11:27 PM, W.P. McNeill wrote:

    After my Hadoop job has successfully completed I'd like to log the total
    amount of time it took. This is the "Finished in" statistic in the web
    UI.
    How do I get this number programmatically? Is there some way I can query
    the Job object? I didn't see anything in the API documentation.
    02-Dec-2011, at 11:27 PM, W.P. McNeill wrote:
    After my Hadoop job has successfully completed I'd like to log the total
    amount of time it took. This is the "Finished in" statistic in the web
    UI.
    How do I get this number programmatically? Is there some way I can query
    the Job object? I didn't see anything in the API documentation.

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupcommon-user @
categorieshadoop
postedDec 2, '11 at 5:57p
activeDec 3, '11 at 6:19a
posts5
users5
websitehadoop.apache.org...
irc#hadoop

People

Translate

site design / logo © 2022 Grokbase