FAQ
Hi,
Is it possible to get the process id of each task in a MapReduce job?
When I run a mapreduce job and do a monitoring in linux using ps, i just see
the id of the mapreduce job process but not its constituent map/reduce
tasks.
The use case is to monitor the resource usage of each task by using sar
utility in linux with specific process id of task.

Thanks,
Bikash

Search Discussions

  • Varad Meru at Sep 29, 2011 at 1:36 am
    The process ids of each individual task can be seen using jps and jconsole commands provided by java.

    jconsole command on command-line interface provides a GUI screen for monitoring running tasks within java.

    The tasks are only visible as java virtual machine instance in the os system monitoring tool.


    Regards,
    Varad Meru
    -----------------------------------
    Sent from my iPod
    On 29-Sep-2011, at 0:15, bikash sharma wrote:

    Hi,
    Is it possible to get the process id of each task in a MapReduce job?
    When I run a mapreduce job and do a monitoring in linux using ps, i just see
    the id of the mapreduce job process but not its constituent map/reduce
    tasks.
    The use case is to monitor the resource usage of each task by using sar
    utility in linux with specific process id of task.

    Thanks,
    Bikash
  • Bikash sharma at Sep 30, 2011 at 12:52 pm
    Thanks Varad.
    On Wed, Sep 28, 2011 at 9:35 PM, Varad Meru wrote:

    The process ids of each individual task can be seen using jps and jconsole
    commands provided by java.

    jconsole command on command-line interface provides a GUI screen for
    monitoring running tasks within java.

    The tasks are only visible as java virtual machine instance in the os
    system monitoring tool.


    Regards,
    Varad Meru
    -----------------------------------
    Sent from my iPod
    On 29-Sep-2011, at 0:15, bikash sharma wrote:

    Hi,
    Is it possible to get the process id of each task in a MapReduce job?
    When I run a mapreduce job and do a monitoring in linux using ps, i just see
    the id of the mapreduce job process but not its constituent map/reduce
    tasks.
    The use case is to monitor the resource usage of each task by using sar
    utility in linux with specific process id of task.

    Thanks,
    Bikash
  • Harsh J at Sep 29, 2011 at 4:42 am
    Hello Bikash,

    The tasks run on the tasktracker, so that is where you'll need to look
    for the process ID -- not the JobTracker/client.

    Crudely speaking,
    $ ssh tasktracker01 # or whichever.
    $ jps | grep Child | cut -d " " -f 1
    # And lo, PIDs to play with.
    On Thu, Sep 29, 2011 at 12:15 AM, bikash sharma wrote:
    Hi,
    Is it possible to get the process id of each task in a MapReduce job?
    When I run a mapreduce job and do a monitoring in linux using ps, i just see
    the id of the mapreduce job process but not its constituent map/reduce
    tasks.
    The use case is to monitor the resource usage of each task by using sar
    utility in linux with specific process id of task.

    Thanks,
    Bikash


    --
    Harsh J
  • Bikash sharma at Sep 30, 2011 at 12:52 pm
    Thanks so much Harsh!
    On Thu, Sep 29, 2011 at 12:42 AM, Harsh J wrote:

    Hello Bikash,

    The tasks run on the tasktracker, so that is where you'll need to look
    for the process ID -- not the JobTracker/client.

    Crudely speaking,
    $ ssh tasktracker01 # or whichever.
    $ jps | grep Child | cut -d " " -f 1
    # And lo, PIDs to play with.
    On Thu, Sep 29, 2011 at 12:15 AM, bikash sharma wrote:
    Hi,
    Is it possible to get the process id of each task in a MapReduce job?
    When I run a mapreduce job and do a monitoring in linux using ps, i just see
    the id of the mapreduce job process but not its constituent map/reduce
    tasks.
    The use case is to monitor the resource usage of each task by using sar
    utility in linux with specific process id of task.

    Thanks,
    Bikash


    --
    Harsh J

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupcommon-user @
categorieshadoop
postedSep 28, '11 at 6:46p
activeSep 30, '11 at 12:52p
posts5
users3
websitehadoop.apache.org...
irc#hadoop

People

Translate

site design / logo © 2022 Grokbase