FAQ
Hey all,

Does anyone have any experience trying to measure IO time spent in
their map/reduce jobs? I know how to profile a sample of map and
reduce tasks, but that appears to exclude IO time. Just subtracting
the total cpu time from the total run time of a task seems like too
coarse an approach.

-Bryan

Search Discussions

  • Jdd dhok at Feb 13, 2009 at 9:11 am
    Hi,
    Linux kernel provides delay accounting information through a netlink
    socket to user space. You can read more about it here:
    http://www.mjmwired.net/kernel/Documentation/accounting/taskstats.txt.
    I think there's a python tool called iotop that uses this feature.

    Hope this helps.

    Regards,
    Jaideep

    On Fri, Feb 13, 2009 at 2:40 AM, Bryan Duxbury wrote:
    Hey all,

    Does anyone have any experience trying to measure IO time spent in their
    map/reduce jobs? I know how to profile a sample of map and reduce tasks, but
    that appears to exclude IO time. Just subtracting the total cpu time from
    the total run time of a task seems like too coarse an approach.

    -Bryan


    --
    - JDD

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupcommon-user @
categorieshadoop
postedFeb 12, '09 at 9:10p
activeFeb 13, '09 at 9:11a
posts2
users2
websitehadoop.apache.org...
irc#hadoop

2 users in discussion

Jdd dhok: 1 post Bryan Duxbury: 1 post

People

Translate

site design / logo © 2022 Grokbase