FAQ
Hi,

I recently started developing with Hadoop using the 20.2 API. I'm looking
to profile one of my jobs but I haven't been able to find any documentation
about how to do this. For the earlier (deprecated) API, there's some
documentation on how to profile with the JobConf class (
http://hadoop.apache.org/common/docs/current/mapred_tutorial.html#Profiling).
Is there anything equivalent in 20.2, or is there another process to use for
it? Search engines and javadoc searching haven't turned up anything so far
for me.

Thanks,
David

Search Discussions

  • Hemanth Yamijala at Aug 13, 2010 at 5:17 am
    Hi,
    I recently started developing with Hadoop using the 20.2 API.  I'm looking
    to profile one of my jobs but I haven't been able to find any documentation
    about how to do this.  For the earlier (deprecated) API, there's some
    documentation on how to profile with the JobConf class
    (http://hadoop.apache.org/common/docs/current/mapred_tutorial.html#Profiling).
    Is there anything equivalent in 20.2, or is there another process to use for
    it?  Search engines and javadoc searching haven't turned up anything so far
    for me.
    All APIs in the deprecated JobConf class are wrappers around the
    Configuration API that you probably are using in the 20.2 new API. For
    e.g. :

    setProfileEnabled = configuration.setBoolean("mapred.task.profile", value)
    setProfileTaskRange = configuration.set("mapred.task.profile.maps",
    value) or configuration.set("mapred.task.profile.reduces", value)
    setProfileParams = configuration.set("mapred.task.profile.params", value)

    Could you try setting these parameters in the Configuration you might
    be using for the Job and see if that works ?

    Thanks
    Hemanth
  • David Jurgens at Aug 16, 2010 at 8:36 pm
    Hi Hemanth,

    It looks like calling configuration.setBoolean("mapred.task.profile",
    true) will enable profiling with the 20.2 APIs. I am able to see the
    profiling output when I check the web interface. Thanks for your help! Is
    there a good place to document this setting so others can find this
    information?

    Thanks,
    David

    On Thu, Aug 12, 2010 at 10:16 PM, Hemanth Yamijala wrote:

    Hi,
    I recently started developing with Hadoop using the 20.2 API. I'm looking
    to profile one of my jobs but I haven't been able to find any
    documentation
    about how to do this. For the earlier (deprecated) API, there's some
    documentation on how to profile with the JobConf class
    (
    http://hadoop.apache.org/common/docs/current/mapred_tutorial.html#Profiling
    ).
    Is there anything equivalent in 20.2, or is there another process to use for
    it? Search engines and javadoc searching haven't turned up anything so far
    for me.
    All APIs in the deprecated JobConf class are wrappers around the
    Configuration API that you probably are using in the 20.2 new API. For
    e.g. :

    setProfileEnabled = configuration.setBoolean("mapred.task.profile", value)
    setProfileTaskRange = configuration.set("mapred.task.profile.maps",
    value) or configuration.set("mapred.task.profile.reduces", value)
    setProfileParams = configuration.set("mapred.task.profile.params", value)

    Could you try setting these parameters in the Configuration you might
    be using for the Job and see if that works ?

    Thanks
    Hemanth
  • Hemanth Yamijala at Aug 17, 2010 at 6:20 am
    David,
    It looks like calling configuration.setBoolean("mapred.task.profile",
    true) will enable profiling with the 20.2 APIs.  I am able to see the
    profiling output when I check the web interface.  Thanks for your help!  Is
    there a good place to document this setting so others can find this
    information?
    Forrest documentation for MapReduce
    (http://hadoop.apache.org/common/docs/r0.20.0/mapred_tutorial.html)
    should be updated. I do see there is mention of these configuration
    parameters in the Profiling section, but it doesn't explicitly mention
    setting them using the configuration API. That in turn is because
    JobConf is still a preferred way of setting parameters in the Hadoop
    0.20 major release. Later versions of the documentation will hopefully
    correct this.

    Thanks
    hemanth
    On Thu, Aug 12, 2010 at 10:16 PM, Hemanth Yamijala wrote:

    Hi,
    I recently started developing with Hadoop using the 20.2 API.  I'm
    looking
    to profile one of my jobs but I haven't been able to find any
    documentation
    about how to do this.  For the earlier (deprecated) API, there's some
    documentation on how to profile with the JobConf class

    (http://hadoop.apache.org/common/docs/current/mapred_tutorial.html#Profiling).
    Is there anything equivalent in 20.2, or is there another process to use
    for
    it?  Search engines and javadoc searching haven't turned up anything so
    far
    for me.
    All APIs in the deprecated JobConf class are wrappers around the
    Configuration API that you probably are using in the 20.2 new API. For
    e.g. :

    setProfileEnabled = configuration.setBoolean("mapred.task.profile", value)
    setProfileTaskRange = configuration.set("mapred.task.profile.maps",
    value) or configuration.set("mapred.task.profile.reduces", value)
    setProfileParams = configuration.set("mapred.task.profile.params", value)

    Could you try setting these parameters in the Configuration you might
    be using for the Job and see if that works ?

    Thanks
    Hemanth

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupmapreduce-user @
categorieshadoop
postedAug 12, '10 at 10:34p
activeAug 17, '10 at 6:20a
posts4
users2
websitehadoop.apache.org...
irc#hadoop

2 users in discussion

David Jurgens: 2 posts Hemanth Yamijala: 2 posts

People

Translate

site design / logo © 2022 Grokbase