FAQ
I'm working a project to collect MapReduce job information on an application
level. For example, a DW ETL process may involves several MapReduce jobs, we
want to have a dashboard to show the progress of those jobs for the specific
ETL process.

JobStatus does not provide all information like JobTracker web
page. JobInProgress is used in JobTracker and JobHistory and it is in
JobTracker memory, and seem not exposed to the client side.

The current method I am using is to check history log files and job conf XML
file to extract those information like jobdetailhistory.jsp and
jobhistory.jsp.

Is there a better way to collect the information like JobInProgress?

Thanks.

Search Discussions

  • Robert Evans at Sep 21, 2011 at 6:48 pm
    Not that I know of. We scrape web pages which is a horrible thing to do. There is a JIRA to add in some web service APIs to expose this type of information, but it is not going to be available for a while.

    --Bobby Evans

    On 9/21/11 1:01 PM, "Benyi Wang" wrote:

    I'm working a project to collect MapReduce job information on an application
    level. For example, a DW ETL process may involves several MapReduce jobs, we
    want to have a dashboard to show the progress of those jobs for the specific
    ETL process.

    JobStatus does not provide all information like JobTracker web
    page. JobInProgress is used in JobTracker and JobHistory and it is in
    JobTracker memory, and seem not exposed to the client side.

    The current method I am using is to check history log files and job conf XML
    file to extract those information like jobdetailhistory.jsp and
    jobhistory.jsp.

    Is there a better way to collect the information like JobInProgress?

    Thanks.

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupcommon-user @
categorieshadoop
postedSep 21, '11 at 6:01p
activeSep 21, '11 at 6:48p
posts2
users2
websitehadoop.apache.org...
irc#hadoop

2 users in discussion

Robert Evans: 1 post Benyi Wang: 1 post

People

Translate

site design / logo © 2022 Grokbase