FAQ
Hi,

I am trying to monitor the time to complete a map phase and reduce
phase in hadoop. Is there any way to measure the time taken to
complete map and reduce phase in a cluster.

Thanks,
Amit

--
View this message in context: http://lucene.472066.n3.nabble.com/measure-the-time-taken-to-complete-map-and-reduce-phase-tp3136991p3136991.html
Sent from the Hadoop lucene-users mailing list archive at Nabble.com.

Search Discussions

  • Madhu phatak at Jul 5, 2011 at 2:58 am
    The console will tell how much time taken by job
    On Jul 5, 2011 8:26 AM, "sangroya" wrote:
    Hi,

    I am trying to monitor the time to complete a map phase and reduce
    phase in hadoop. Is there any way to measure the time taken to
    complete map and reduce phase in a cluster.

    Thanks,
    Amit

    --
    View this message in context:
    http://lucene.472066.n3.nabble.com/measure-the-time-taken-to-complete-map-and-reduce-phase-tp3136991p3136991.html
    Sent from the Hadoop lucene-users mailing list archive at Nabble.com.
  • Bharath Mundlapudi at Jul 5, 2011 at 3:07 am
    Short answer: Job History Logs(JHL).
    Long answer: JHL stores all the meta information related to jobs which were executed like Job start/finish, Task start/finish, TaskAttempt start/finish.

    Job.Map_Phase = Function ( Job.Map(first).StartTime, Job.Map(last).FinishTime)
    Job.Reduce_Phase = Function ( Job.Reduce(first).StartTime, Job.Reduce(last).FinishTime)


    -Bharath



    ________________________________
    From: sangroya <sangroyaamit@gmail.com>
    To: hadoop-user@lucene.apache.org
    Sent: Monday, July 4, 2011 2:12 AM
    Subject: measure the time taken to complete map and reduce phase

    Hi,

    I am trying to monitor the time to complete a map phase and reduce
    phase in hadoop. Is there any way to measure the time taken to
    complete map and reduce phase in a cluster.

    Thanks,
    Amit

    --
    View this message in context: http://lucene.472066.n3.nabble.com/measure-the-time-taken-to-complete-map-and-reduce-phase-tp3136991p3136991.html
    Sent from the Hadoop lucene-users mailing list archive at Nabble.com.
  • Alberto Andreotti at Jul 5, 2011 at 3:08 am
    Hi Amit,

    I think you can just measure how much it takes to complete the job. I mean,
    from submission until it's done. This would require just regular Java calls.

    Alberto.
    On 4 July 2011 06:12, sangroya wrote:

    Hi,

    I am trying to monitor the time to complete a map phase and reduce
    phase in hadoop. Is there any way to measure the time taken to
    complete map and reduce phase in a cluster.

    Thanks,
    Amit

    --
    View this message in context:
    http://lucene.472066.n3.nabble.com/measure-the-time-taken-to-complete-map-and-reduce-phase-tp3136991p3136991.html
    Sent from the Hadoop lucene-users mailing list archive at Nabble.com.


    --
    José Pablo Alberto Andreotti.
    Tel: 54 351 4730292
    Móvil: 54351156526363.
    MSN: albertoandreotti@gmail.com
    Skype: andreottialberto
  • Real great.. at Jul 5, 2011 at 3:24 am
    hey,amit
    it comes in the web interface too..
    On 7/5/11, Alberto Andreotti wrote:
    Hi Amit,

    I think you can just measure how much it takes to complete the job. I mean,
    from submission until it's done. This would require just regular Java calls.

    Alberto.
    On 4 July 2011 06:12, sangroya wrote:

    Hi,

    I am trying to monitor the time to complete a map phase and reduce
    phase in hadoop. Is there any way to measure the time taken to
    complete map and reduce phase in a cluster.

    Thanks,
    Amit

    --
    View this message in context:
    http://lucene.472066.n3.nabble.com/measure-the-time-taken-to-complete-map-and-reduce-phase-tp3136991p3136991.html
    Sent from the Hadoop lucene-users mailing list archive at Nabble.com.


    --
    José Pablo Alberto Andreotti.
    Tel: 54 351 4730292
    Móvil: 54351156526363.
    MSN: albertoandreotti@gmail.com
    Skype: andreottialberto

    --
    Regards,
    R.V.
  • Hailong.yang1115 at Jul 5, 2011 at 4:52 am
    Hi sangroya,

    You can look at the job administration portal at port of 50030 on your JobTracker such as 'http://localhost:50030'. At the bottom of the web page there is an item named 'Job Tracker History', click into it and find you job with the job id. There goes the information you want.


    Cheers!

    Hailong

    2011-07-05



    ***********************************************
    * Hailong Yang, PhD. Candidate
    * Sino-German Joint Software Institute,
    * School of Computer Science&Engineering, Beihang University
    * Phone: (86-010)82315908
    * Email: hailong.yang1115@gmail.com
    * Address: G413, New Main Building in Beihang University,
    * No.37 XueYuan Road,HaiDian District,
    * Beijing,P.R.China,100191
    ***********************************************



    发件人: sangroya
    发送时间: 2011-07-05 10:56:38
    收件人: hadoop-user
    抄送:
    主题: measure the time taken to complete map and reduce phase

    Hi,
    I am trying to monitor the time to complete a map phase and reduce
    phase in hadoop. Is there any way to measure the time taken to
    complete map and reduce phase in a cluster.
    Thanks,
    Amit
    --
    View this message in context: http://lucene.472066.n3.nabble.com/measure-the-time-taken-to-complete-map-and-reduce-phase-tp3136991p3136991.html
    Sent from the Hadoop lucene-users mailing list archive at Nabble.com.
  • Sangroya at Jul 7, 2011 at 7:49 am
    Hi,

    Thanks!

    I am able to parse the Job History Logs(JHL). But, I need to know how
    hadoop assign a name to a file in Job History Logs(JHL).

    I can see that files are named on my local single node cluster as this:

    localhost_1309975809398_job_201107062010_0759_sangroya_word+count.

    But, I am just wondering, what is the exact pattern to name every file
    like this.

    Best Regards,
    Amit

    On Tue, Jul 5, 2011 at 6:53 AM, Hailong [via Lucene]
    wrote:
    Hi sangroya,

    You can look at the job administration portal at port of 50030 on your
    JobTracker such as '<a
    href="http://localhost:50030'">http://localhost:50030'. At the bottom of the
    web page there is an item named 'Job Tracker History', click into it and
    find you job with the job id. There goes the information you want.


    Cheers!

    Hailong

    2011-07-05



    ***********************************************
    * Hailong Yang, PhD. Candidate
    * Sino-German Joint Software Institute,
    * School of Computer Science&Engineering, Beihang University
    * Phone: (86-010)82315908
    * Email: [hidden email]
    * Address: G413, New Main Building in Beihang University,
    * No.37 XueYuan Road,HaiDian District,
    * Beijing,P.R.China,100191
    ***********************************************



    发件人: sangroya
    发送时间: 2011-07-05 10:56:38
    收件人: hadoop-user
    抄送:
    主题: measure the time taken to complete map and reduce phase

    Hi,
    I am trying to monitor the time to complete a map phase and reduce
    phase in hadoop. Is there any way to measure the time taken to
    complete map and reduce phase in a cluster.
    Thanks,
    Amit
    --
    View this message in context:
    http://lucene.472066.n3.nabble.com/measure-the-time-taken-to-complete-map-and-reduce-phase-tp3136991p3136991.html
    Sent from the Hadoop lucene-users mailing list archive at Nabble.com.


    ________________________________
    If you reply to this email, your message will be added to the discussion
    below:
    http://lucene.472066.n3.nabble.com/measure-the-time-taken-to-complete-map-and-reduce-phase-tp3136991p3139665.html
    To unsubscribe from measure the time taken to complete map and reduce phase,
    click here.

    --
    View this message in context: http://lucene.472066.n3.nabble.com/measure-the-time-taken-to-complete-map-and-reduce-phase-tp3136991p3147426.html
    Sent from the Hadoop lucene-users mailing list archive at Nabble.com.
  • Hailong.yang1115 at Jul 7, 2011 at 8:58 am
    Hi sangroya,

    I think you may be interested in reading the following piece of code from JobHistory.java in Hadoop.

    /**
    * Generates the job history filename for a new job
    */
    private static String getNewJobHistoryFileName(JobConf jobConf, JobID id) {
    return JOBTRACKER_UNIQUE_STRING
    + id.toString() + "_" + getUserName(jobConf) + "_"
    + trimJobName(getJobName(jobConf));
    }

    /**
    * Trims the job-name if required
    */
    private static String trimJobName(String jobName) {
    if (jobName.length() > JOB_NAME_TRIM_LENGTH) {
    jobName = jobName.substring(0, JOB_NAME_TRIM_LENGTH);
    }
    return jobName;
    }

    Roughly speaking, the history file name is composed in the following way:

    hostname of JT + "_" + start time of JT + "_" + job id + "_" + user name + "_" + trimed job name

    Cheers!

    Hailong

    2011-07-07



    ***********************************************
    * Hailong Yang, PhD. Candidate
    * Sino-German Joint Software Institute,
    * School of Computer Science&Engineering, Beihang University
    * Phone: (86-010)82315908
    * Email: hailong.yang1115@gmail.com
    * Address: G413, New Main Building in Beihang University,
    * No.37 XueYuan Road,HaiDian District,
    * Beijing,P.R.China,100191
    ***********************************************



    发件人: sangroya
    发送时间: 2011-07-07 15:49:58
    收件人: hadoop-user
    抄送:
    主题: Re: measure the time taken to complete map and reduce phase

    Hi,
    Thanks!
    I am able to parse the Job History Logs(JHL). But, I need to know how
    hadoop assign a name to a file in Job History Logs(JHL).
    I can see that files are named on my local single node cluster as this:
    localhost_1309975809398_job_201107062010_0759_sangroya_word+count.
    But, I am just wondering, what is the exact pattern to name every file
    like this.
    Best Regards,
    Amit
    On Tue, Jul 5, 2011 at 6:53 AM, Hailong [via Lucene]
    wrote:
    Hi sangroya,

    You can look at the job administration portal at port of 50030 on your
    JobTracker such as '<a
    href="http://localhost:50030'">http://localhost:50030'. At the bottom of the
    web page there is an item named 'Job Tracker History', click into it and
    find you job with the job id. There goes the information you want.


    Cheers!

    Hailong

    2011-07-05



    ***********************************************
    * Hailong Yang, PhD. Candidate
    * Sino-German Joint Software Institute,
    * School of Computer Science&Engineering, Beihang University
    * Phone: (86-010)82315908
    * Email: [hidden email]
    * Address: G413, New Main Building in Beihang University,
    * No.37 XueYuan Road,HaiDian District,
    * Beijing,P.R.China,100191
    ***********************************************



    发件人: sangroya
    发送时间: 2011-07-05 10:56:38
    收件人: hadoop-user
    抄送:
    主题: measure the time taken to complete map and reduce phase

    Hi,
    I am trying to monitor the time to complete a map phase and reduce
    phase in hadoop. Is there any way to measure the time taken to
    complete map and reduce phase in a cluster.
    Thanks,
    Amit
    --
    View this message in context:
    http://lucene.472066.n3.nabble.com/measure-the-time-taken-to-complete-map-and-reduce-phase-tp3136991p3136991.html
    Sent from the Hadoop lucene-users mailing list archive at Nabble.com.


    ________________________________
    If you reply to this email, your message will be added to the discussion
    below:
    http://lucene.472066.n3.nabble.com/measure-the-time-taken-to-complete-map-and-reduce-phase-tp3136991p3139665.html
    To unsubscribe from measure the time taken to complete map and reduce phase,
    click here.
    --
    View this message in context: http://lucene.472066.n3.nabble.com/measure-the-time-taken-to-complete-map-and-reduce-phase-tp3136991p3147426.html
    Sent from the Hadoop lucene-users mailing list archive at Nabble.com.
  • Sangroya at Jul 7, 2011 at 3:50 pm
    Hi,

    Thanks for the response!

    I have the following queries regarding the Job History file.

    I want to know if what the TOTAL_MAPS in the Job history represents.

    Also, if FINISHED_MAPS represents the TOTAL_MAPS or the (TOTAL_MAPS -
    FAILED_MAPS).

    Does FINISHED_MAPS represents successfully executed maps.

    I have the same question for REDUCE tasks.

    Thanks,
    Amit



    On Thu, Jul 7, 2011 at 10:58 AM, Hailong [via Lucene]
    wrote:
    Hi sangroya,

    I think you may be interested in reading the following piece of code from
    JobHistory.java in Hadoop.

    /**
    * Generates the job history filename for a new job
    */
    private static String getNewJobHistoryFileName(JobConf jobConf, JobID
    id) {
    return JOBTRACKER_UNIQUE_STRING
    + id.toString() + "_" + getUserName(jobConf) + "_"
    + trimJobName(getJobName(jobConf));
    }

    /**
    * Trims the job-name if required
    */
    private static String trimJobName(String jobName) {
    if (jobName.length() > JOB_NAME_TRIM_LENGTH) {
    jobName = jobName.substring(0, JOB_NAME_TRIM_LENGTH);
    }
    return jobName;
    }

    Roughly speaking, the history file name is composed in the following way:

    hostname of JT + "_" + start time of JT + "_" + job id + "_" + user name +
    "_" + trimed job name

    Cheers!

    Hailong

    2011-07-07



    ***********************************************
    * Hailong Yang, PhD. Candidate
    * Sino-German Joint Software Institute,
    * School of Computer Science&Engineering, Beihang University
    * Phone: (86-010)82315908
    * Email: [hidden email]
    * Address: G413, New Main Building in Beihang University,
    * No.37 XueYuan Road,HaiDian District,
    * Beijing,P.R.China,100191
    ***********************************************



    发件人: sangroya
    发送时间: 2011-07-07 15:49:58
    收件人: hadoop-user
    抄送:
    主题: Re: measure the time taken to complete map and reduce phase

    Hi,
    Thanks!
    I am able to parse the Job History Logs(JHL). But, I need to know how
    hadoop assign a name to a file in Job History Logs(JHL).
    I can see that files are named on my local single node cluster as this:
    localhost_1309975809398_job_201107062010_0759_sangroya_word+count.
    But, I am just wondering, what is the exact pattern to name every file
    like this.
    Best Regards,
    Amit
    On Tue, Jul 5, 2011 at 6:53 AM, Hailong [via Lucene]
    <[hidden email]> wrote:
    Hi sangroya,

    You can look at the job administration portal at port of 50030 on your
    JobTracker such as '<a
    href="<a href="http://localhost:50030'">http://localhost:50030'"><a
    href="http://localhost:50030'">http://localhost:50030'. At the bottom of the
    web page there is an item named 'Job Tracker History', click into it and
    find you job with the job id. There goes the information you want.


    Cheers!

    Hailong

    2011-07-05



    ***********************************************
    * Hailong Yang, PhD. Candidate
    * Sino-German Joint Software Institute,
    * School of Computer Science&Engineering, Beihang University
    * Phone: (86-010)82315908
    * Email: [hidden email]
    * Address: G413, New Main Building in Beihang University,
    * No.37 XueYuan Road,HaiDian District,
    * Beijing,P.R.China,100191
    ***********************************************



    发件人: sangroya
    发送时间: 2011-07-05 10:56:38
    收件人: hadoop-user
    抄送:
    主题: measure the time taken to complete map and reduce phase

    Hi,
    I am trying to monitor the time to complete a map phase and reduce
    phase in hadoop. Is there any way to measure the time taken to
    complete map and reduce phase in a cluster.
    Thanks,
    Amit
    --
    View this message in context:

    http://lucene.472066.n3.nabble.com/measure-the-time-taken-to-complete-map-and-reduce-phase-tp3136991p3136991.html
    Sent from the Hadoop lucene-users mailing list archive at Nabble.com.


    ________________________________
    If you reply to this email, your message will be added to the discussion
    below:

    http://lucene.472066.n3.nabble.com/measure-the-time-taken-to-complete-map-and-reduce-phase-tp3136991p3139665.html
    To unsubscribe from measure the time taken to complete map and reduce
    phase,
    click here.
    --
    View this message in context:
    http://lucene.472066.n3.nabble.com/measure-the-time-taken-to-complete-map-and-reduce-phase-tp3136991p3147426.html
    Sent from the Hadoop lucene-users mailing list archive at Nabble.com.


    ________________________________
    If you reply to this email, your message will be added to the discussion
    below:
    http://lucene.472066.n3.nabble.com/measure-the-time-taken-to-complete-map-and-reduce-phase-tp3136991p3147566.html
    To unsubscribe from measure the time taken to complete map and reduce phase,
    click here.

    --
    View this message in context: http://lucene.472066.n3.nabble.com/measure-the-time-taken-to-complete-map-and-reduce-phase-tp3136991p3148620.html
    Sent from the Hadoop lucene-users mailing list archive at Nabble.com.
  • Hailong.yang1115 at Jul 8, 2011 at 5:35 am
    I think the TOTAL_MAPS has the same meaning with the FINISHED_MAPS, which represents the total number of map tasks successfully executed. Although there is another metric named Launched_map_tasks, which is the sum of FINISHED_MAPS, FAILED_MAPS and KILLED_MAPS.

    And it is the same for Reduce tasks.


    Cheer!

    Hailong

    2011-07-08



    ***********************************************
    * Hailong Yang, PhD. Candidate
    * Sino-German Joint Software Institute,
    * School of Computer Science&Engineering, Beihang University
    * Phone: (86-010)82315908
    * Email: hailong.yang1115@gmail.com
    * Address: G413, New Main Building in Beihang University,
    * No.37 XueYuan Road,HaiDian District,
    * Beijing,P.R.China,100191
    ***********************************************



    发件人: sangroya
    发送时间: 2011-07-07 23:50:45
    收件人: hadoop-user
    抄送:
    主题: Re: Re: measure the time taken to complete map and reduce phase

    Hi,
    Thanks for the response!
    I have the following queries regarding the Job History file.
    I want to know if what the TOTAL_MAPS in the Job history represents.
    Also, if FINISHED_MAPS represents the TOTAL_MAPS or the (TOTAL_MAPS -
    FAILED_MAPS).
    Does FINISHED_MAPS represents successfully executed maps.
    I have the same question for REDUCE tasks.
    Thanks,
    Amit
    On Thu, Jul 7, 2011 at 10:58 AM, Hailong [via Lucene]
    wrote:
    Hi sangroya,

    I think you may be interested in reading the following piece of code from
    JobHistory.java in Hadoop.

    /**
    * Generates the job history filename for a new job
    */
    private static String getNewJobHistoryFileName(JobConf jobConf, JobID
    id) {
    return JOBTRACKER_UNIQUE_STRING
    + id.toString() + "_" + getUserName(jobConf) + "_"
    + trimJobName(getJobName(jobConf));
    }

    /**
    * Trims the job-name if required
    */
    private static String trimJobName(String jobName) {
    if (jobName.length() > JOB_NAME_TRIM_LENGTH) {
    jobName = jobName.substring(0, JOB_NAME_TRIM_LENGTH);
    }
    return jobName;
    }

    Roughly speaking, the history file name is composed in the following way:

    hostname of JT + "_" + start time of JT + "_" + job id + "_" + user name +
    "_" + trimed job name

    Cheers!

    Hailong

    2011-07-07



    ***********************************************
    * Hailong Yang, PhD. Candidate
    * Sino-German Joint Software Institute,
    * School of Computer Science&Engineering, Beihang University
    * Phone: (86-010)82315908
    * Email: [hidden email]
    * Address: G413, New Main Building in Beihang University,
    * No.37 XueYuan Road,HaiDian District,
    * Beijing,P.R.China,100191
    ***********************************************



    发件人: sangroya
    发送时间: 2011-07-07 15:49:58
    收件人: hadoop-user
    抄送:
    主题: Re: measure the time taken to complete map and reduce phase

    Hi,
    Thanks!
    I am able to parse the Job History Logs(JHL). But, I need to know how
    hadoop assign a name to a file in Job History Logs(JHL).
    I can see that files are named on my local single node cluster as this:
    localhost_1309975809398_job_201107062010_0759_sangroya_word+count.
    But, I am just wondering, what is the exact pattern to name every file
    like this.
    Best Regards,
    Amit
    On Tue, Jul 5, 2011 at 6:53 AM, Hailong [via Lucene]
    <[hidden email]> wrote:
    Hi sangroya,

    You can look at the job administration portal at port of 50030 on your
    JobTracker such as '<a
    href="<a href="http://localhost:50030'">http://localhost:50030'"><a
    href="http://localhost:50030'">http://localhost:50030'. At the bottom of the
    web page there is an item named 'Job Tracker History', click into it and
    find you job with the job id. There goes the information you want.


    Cheers!

    Hailong

    2011-07-05



    ***********************************************
    * Hailong Yang, PhD. Candidate
    * Sino-German Joint Software Institute,
    * School of Computer Science&Engineering, Beihang University
    * Phone: (86-010)82315908
    * Email: [hidden email]
    * Address: G413, New Main Building in Beihang University,
    * No.37 XueYuan Road,HaiDian District,
    * Beijing,P.R.China,100191
    ***********************************************



    发件人: sangroya
    发送时间: 2011-07-05 10:56:38
    收件人: hadoop-user
    抄送:
    主题: measure the time taken to complete map and reduce phase

    Hi,
    I am trying to monitor the time to complete a map phase and reduce
    phase in hadoop. Is there any way to measure the time taken to
    complete map and reduce phase in a cluster.
    Thanks,
    Amit
    --
    View this message in context:

    http://lucene.472066.n3.nabble.com/measure-the-time-taken-to-complete-map-and-reduce-phase-tp3136991p3136991.html
    Sent from the Hadoop lucene-users mailing list archive at Nabble.com.


    ________________________________
    If you reply to this email, your message will be added to the discussion
    below:

    http://lucene.472066.n3.nabble.com/measure-the-time-taken-to-complete-map-and-reduce-phase-tp3136991p3139665.html
    To unsubscribe from measure the time taken to complete map and reduce
    phase,
    click here.
    --
    View this message in context:
    http://lucene.472066.n3.nabble.com/measure-the-time-taken-to-complete-map-and-reduce-phase-tp3136991p3147426.html
    Sent from the Hadoop lucene-users mailing list archive at Nabble.com.


    ________________________________
    If you reply to this email, your message will be added to the discussion
    below:
    http://lucene.472066.n3.nabble.com/measure-the-time-taken-to-complete-map-and-reduce-phase-tp3136991p3147566.html
    To unsubscribe from measure the time taken to complete map and reduce phase,
    click here.
    --
    View this message in context: http://lucene.472066.n3.nabble.com/measure-the-time-taken-to-complete-map-and-reduce-phase-tp3136991p3148620.html
    Sent from the Hadoop lucene-users mailing list archive at Nabble.com.

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupcommon-user @
categorieshadoop
postedJul 5, '11 at 2:56a
activeJul 8, '11 at 5:35a
posts10
users6
websitehadoop.apache.org...
irc#hadoop

People

Translate

site design / logo © 2022 Grokbase