Hi Hadoop users,

I have been reading about Hadoop metrics framework, and I was wondering is
it possible to create custom metrics for specific job.

In my use case I want to capture some specific information per record which
is processed. So I was thinking to create custom code in my Hadoop job to
send these information using Hadoop metrics. Is this possible to implement
? If so can anyone share any example how things should be done?

Thanks,
dinok

Search Discussions

  • Harsh J at Nov 2, 2011 at 9:42 am
    Jobs provide counters as a way to store some information. Are you
    looking for something beyond plain counters?
    On Wed, Nov 2, 2011 at 1:14 PM, Dino Kečo wrote:
    Hi Hadoop users,
    I have been reading about Hadoop metrics framework, and I was wondering is
    it possible to create custom metrics for specific job.
    In my use case I want to capture some specific information per record which
    is processed. So I was thinking to create custom code in my Hadoop job to
    send these information using Hadoop metrics. Is this possible to implement ?
    If so can anyone share any example how things should be done?
    Thanks,
    dinok


    --
    Harsh J
  • Daniel Iancu at Nov 2, 2011 at 9:56 am
    Hi Dino
    It can't be used effectively to monitor MR mappers since tasks on a node
    run as separate JVMs and each task will overwrite the same metric
    published by other tasks. You cannot aggregate values from running tasks
    before publish them because tasks don't communicate with each other.
    If you publish a metric per task, this will give you tones of graphs in
    Ganglia.

    On other hand I extended it the API and use it to monitor our client
    (webapp) and it works fine and provide valuable information. I have
    started a blog post to document this, will be ready soon, I'll add the
    link here if you need some examples. I wonder why this API is not a
    separate project, it can be reused in apps that don't need Hadoop. I
    could not find any java client API to publish metrics in Ganglia.

    Regards
    Daniel


    On 11/02/2011 09:44 AM, Dino Kečo wrote:
    Hi Hadoop users,

    I have been reading about Hadoop metrics framework, and I was
    wondering is it possible to create custom metrics for specific job.

    In my use case I want to capture some specific information per record
    which is processed. So I was thinking to create custom code in my
    Hadoop job to send these information using Hadoop metrics. Is this
    possible to implement ? If so can anyone share any example how things
    should be done?

    Thanks,
    dinok
  • Dino Kečo at Nov 2, 2011 at 10:30 am
    Hi Harsh,

    Yes I am looking for much detailed information beyond counters. Counters
    are pretty cool but I would like to gather more information related to
    record (like processing time in some part of code, some business logic
    attributes and etc.) So I was hoping to use hadoop metrics to save that
    into file or table and use it for reporting.

    Daniel,

    Thanks for info, please share your blog links so we can see something new :)

    Regards,
    dinok

    On Wed, Nov 2, 2011 at 10:59 AM, Daniel Iancu wrote:

    **
    Hi Dino
    It can't be used effectively to monitor MR mappers since tasks on a node
    run as separate JVMs and each task will overwrite the same metric published
    by other tasks. You cannot aggregate values from running tasks before
    publish them because tasks don't communicate with each other.
    If you publish a metric per task, this will give you tones of graphs in
    Ganglia.

    On other hand I extended it the API and use it to monitor our client
    (webapp) and it works fine and provide valuable information. I have started
    a blog post to document this, will be ready soon, I'll add the link here if
    you need some examples. I wonder why this API is not a separate project, it
    can be reused in apps that don't need Hadoop. I could not find any java
    client API to publish metrics in Ganglia.

    Regards
    Daniel




    On 11/02/2011 09:44 AM, Dino Kečo wrote:

    Hi Hadoop users,

    I have been reading about Hadoop metrics framework, and I was wondering
    is it possible to create custom metrics for specific job.

    In my use case I want to capture some specific information per record
    which is processed. So I was thinking to create custom code in my Hadoop
    job to send these information using Hadoop metrics. Is this possible to
    implement ? If so can anyone share any example how things should be done?

    Thanks,
    dinok

  • David Rosenstrauch at Nov 2, 2011 at 3:36 pm

    On 11/02/2011 03:44 AM, Dino Kečo wrote:
    I have been reading about Hadoop metrics framework, and I was wondering is
    it possible to create custom metrics for specific job.
    You can create your own custom counters, to count particular statistics
    you're interested in. That can take you a long way in this regard.

    HTH,

    DR

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupmapreduce-user @
categorieshadoop
postedNov 2, '11 at 7:45a
activeNov 2, '11 at 3:36p
posts5
users4
websitehadoop.apache.org...
irc#hadoop

People

Translate

site design / logo © 2021 Grokbase