FAQ
Hi,

I have set up ganglia for my cluster, and it works fine. What are
the changes I need to make to make ganglia show hadoop related parameters?
My gmond/gmetad config is default except for one change in gmond.conf for
all nodes:

/* You can specify as many tcp_accept_channels as you like to share
an xml description of the state of the cluster */
tcp_accept_channel {
port = 8650
}

I have changed the hadoop-metrics.properties file as follows:

# Configuration of the "dfs" context for ganglia
dfs.class=org.apache.hadoop.metrics.ganglia.GangliaContext
dfs.period=10
dfs.servers=hadoop1:8649 hadoop2:8649 hadoop3:8649 ejabber:8649


# Configuration of the "mapred" context for null
#mapred.class=org.apache.hadoop.metrics.spi.NullContext

# Configuration of the "mapred" context for file
#mapred.class=org.apache.hadoop.metrics.file.FileContext
#mapred.period=10
#mapred.fileName=/tmp/mrmetrics.log

# Configuration of the "mapred" context for ganglia
mapred.class=org.apache.hadoop.metrics.ganglia.GangliaContext
mapred.period=10
mapred.servers=hadoop1:8649 hadoop2:8649 hadoop3:8649 ejabber:8649


# Configuration of the "jvm" context for null
#jvm.class=org.apache.hadoop.metrics.spi.NullContext

# Configuration of the "jvm" context for file
#jvm.class=org.apache.hadoop.metrics.file.FileContext
#jvm.period=10
#jvm.fileName=/tmp/jvmmetrics.log

# Configuration of the "jvm" context for ganglia
jvm.class=org.apache.hadoop.metrics.ganglia.GangliaContext
jvm.period=10
jvm.servers=hadoop1:8649 hadoop2:8649 hadoop3:8649 ejabber:8649

rpc.class=org.apache.hadoop.metrics.ganglia.GangliaContext
rpc.period=10
rpc.servers=hadoop1:8649 hadoop2:8649 hadoop3:8649 ejabber:8649

hadoop1,2,3 and ejabber are my nodes.

But I am unable to see any new metrics in ganglia's drop down menu. Is there
something else I need to do as well? I have restarted gmond on all nodes.

thanks,
Hari

Search Discussions

  • Sagar Shukla at Nov 22, 2010 at 1:12 pm
    Hi Hari,
    Not sure if you have already followed article - http://wiki.apache.org/hadoop/GangliaMetrics to configure Ganglia for Hadoop Monitoring. Also in the details given by you, I do not see any configuration settings done in gmetad config. You will need to define the hosts from whom metrics should be collected. Once this configuration, you should see the host in gmetad UI interface.

    Thanks,
    Sagar

    -----Original Message-----
    From: Hari Sreekumar
    Sent: Monday, November 22, 2010 6:25 PM
    To: common-user@hadoop.apache.org
    Subject: Using ganglia to monitor hadoop

    Hi,

    I have set up ganglia for my cluster, and it works fine. What are
    the changes I need to make to make ganglia show hadoop related parameters?
    My gmond/gmetad config is default except for one change in gmond.conf for
    all nodes:

    /* You can specify as many tcp_accept_channels as you like to share
    an xml description of the state of the cluster */
    tcp_accept_channel {
    port = 8650
    }

    I have changed the hadoop-metrics.properties file as follows:

    # Configuration of the "dfs" context for ganglia
    dfs.class=org.apache.hadoop.metrics.ganglia.GangliaContext
    dfs.period=10
    dfs.servers=hadoop1:8649 hadoop2:8649 hadoop3:8649 ejabber:8649


    # Configuration of the "mapred" context for null
    #mapred.class=org.apache.hadoop.metrics.spi.NullContext

    # Configuration of the "mapred" context for file
    #mapred.class=org.apache.hadoop.metrics.file.FileContext
    #mapred.period=10
    #mapred.fileName=/tmp/mrmetrics.log

    # Configuration of the "mapred" context for ganglia
    mapred.class=org.apache.hadoop.metrics.ganglia.GangliaContext
    mapred.period=10
    mapred.servers=hadoop1:8649 hadoop2:8649 hadoop3:8649 ejabber:8649


    # Configuration of the "jvm" context for null
    #jvm.class=org.apache.hadoop.metrics.spi.NullContext

    # Configuration of the "jvm" context for file
    #jvm.class=org.apache.hadoop.metrics.file.FileContext
    #jvm.period=10
    #jvm.fileName=/tmp/jvmmetrics.log

    # Configuration of the "jvm" context for ganglia
    jvm.class=org.apache.hadoop.metrics.ganglia.GangliaContext
    jvm.period=10
    jvm.servers=hadoop1:8649 hadoop2:8649 hadoop3:8649 ejabber:8649

    rpc.class=org.apache.hadoop.metrics.ganglia.GangliaContext
    rpc.period=10
    rpc.servers=hadoop1:8649 hadoop2:8649 hadoop3:8649 ejabber:8649

    hadoop1,2,3 and ejabber are my nodes.

    But I am unable to see any new metrics in ganglia's drop down menu. Is there
    something else I need to do as well? I have restarted gmond on all nodes.

    thanks,
    Hari

    DISCLAIMER
    ==========
    This e-mail may contain privileged and confidential information which is the property of Persistent Systems Ltd. It is intended only for the use of the individual or entity to which it is addressed. If you are not the intended recipient, you are not authorized to read, retain, copy, print, distribute or use this message. If you have received this communication in error, please notify the sender and delete all copies of this message. Persistent Systems Ltd. does not accept any liability for virus infected mails.
  • Hari Sreekumar at Nov 22, 2010 at 1:20 pm
    Hi Sagar,

    I followed that article. But I am not clear about some things:
    1. Should I use port 8650 or 8649? I have tried both actually, and 8649
    makes sense, but just to confirm. (I have set tcp_accept_channel as 8650)
    2. In hadoop-metrics.properties, should I mention all my nodes as I have
    done or only the local node? e.g, localhost:8649 everywhere?

    I have configured gmetad.conf, and I am able to see metrics from all my
    nodes in my ganglia UI. I just don't see any hadoop specific metrics.

    Thanks,
    Hari
    On Mon, Nov 22, 2010 at 6:42 PM, Sagar Shukla wrote:

    Hi Hari,
    Not sure if you have already followed article -
    http://wiki.apache.org/hadoop/GangliaMetrics to configure Ganglia for
    Hadoop Monitoring. Also in the details given by you, I do not see any
    configuration settings done in gmetad config. You will need to define the
    hosts from whom metrics should be collected. Once this configuration, you
    should see the host in gmetad UI interface.

    Thanks,
    Sagar

    -----Original Message-----
    From: Hari Sreekumar
    Sent: Monday, November 22, 2010 6:25 PM
    To: common-user@hadoop.apache.org
    Subject: Using ganglia to monitor hadoop

    Hi,

    I have set up ganglia for my cluster, and it works fine. What are
    the changes I need to make to make ganglia show hadoop related parameters?
    My gmond/gmetad config is default except for one change in gmond.conf for
    all nodes:

    /* You can specify as many tcp_accept_channels as you like to share
    an xml description of the state of the cluster */
    tcp_accept_channel {
    port = 8650
    }

    I have changed the hadoop-metrics.properties file as follows:

    # Configuration of the "dfs" context for ganglia
    dfs.class=org.apache.hadoop.metrics.ganglia.GangliaContext
    dfs.period=10
    dfs.servers=hadoop1:8649 hadoop2:8649 hadoop3:8649 ejabber:8649


    # Configuration of the "mapred" context for null
    #mapred.class=org.apache.hadoop.metrics.spi.NullContext

    # Configuration of the "mapred" context for file
    #mapred.class=org.apache.hadoop.metrics.file.FileContext
    #mapred.period=10
    #mapred.fileName=/tmp/mrmetrics.log

    # Configuration of the "mapred" context for ganglia
    mapred.class=org.apache.hadoop.metrics.ganglia.GangliaContext
    mapred.period=10
    mapred.servers=hadoop1:8649 hadoop2:8649 hadoop3:8649 ejabber:8649


    # Configuration of the "jvm" context for null
    #jvm.class=org.apache.hadoop.metrics.spi.NullContext

    # Configuration of the "jvm" context for file
    #jvm.class=org.apache.hadoop.metrics.file.FileContext
    #jvm.period=10
    #jvm.fileName=/tmp/jvmmetrics.log

    # Configuration of the "jvm" context for ganglia
    jvm.class=org.apache.hadoop.metrics.ganglia.GangliaContext
    jvm.period=10
    jvm.servers=hadoop1:8649 hadoop2:8649 hadoop3:8649 ejabber:8649

    rpc.class=org.apache.hadoop.metrics.ganglia.GangliaContext
    rpc.period=10
    rpc.servers=hadoop1:8649 hadoop2:8649 hadoop3:8649 ejabber:8649

    hadoop1,2,3 and ejabber are my nodes.

    But I am unable to see any new metrics in ganglia's drop down menu. Is
    there
    something else I need to do as well? I have restarted gmond on all nodes.

    thanks,
    Hari

    DISCLAIMER
    ==========
    This e-mail may contain privileged and confidential information which is
    the property of Persistent Systems Ltd. It is intended only for the use of
    the individual or entity to which it is addressed. If you are not the
    intended recipient, you are not authorized to read, retain, copy, print,
    distribute or use this message. If you have received this communication in
    error, please notify the sender and delete all copies of this message.
    Persistent Systems Ltd. does not accept any liability for virus infected
    mails.
  • Matthew E. Kennedy at Nov 22, 2010 at 3:38 pm
    Hari,

    I have this set up and working and it looks to me like your *.servers properties are set incorrectly. Point that property to the multicast address that is listed in your gmond.conf file on port 8649.

    You can find the multicast address with:
    cat /etc/gmond.conf | grep mcast

    an example of the config follows:
    dfs.servers=<mcast_join_ip_addr>:8649

    Good luck,
    Matt
    On Nov 22, 2010, at 8:20 AM, Hari Sreekumar wrote:

    Hi Sagar,

    I followed that article. But I am not clear about some things:
    1. Should I use port 8650 or 8649? I have tried both actually, and 8649
    makes sense, but just to confirm. (I have set tcp_accept_channel as 8650)
    2. In hadoop-metrics.properties, should I mention all my nodes as I have
    done or only the local node? e.g, localhost:8649 everywhere?

    I have configured gmetad.conf, and I am able to see metrics from all my
    nodes in my ganglia UI. I just don't see any hadoop specific metrics.

    Thanks,
    Hari

    On Mon, Nov 22, 2010 at 6:42 PM, Sagar Shukla <sagar_shukla@persistent.co.in
    wrote:
    Hi Hari,
    Not sure if you have already followed article -
    http://wiki.apache.org/hadoop/GangliaMetrics to configure Ganglia for
    Hadoop Monitoring. Also in the details given by you, I do not see any
    configuration settings done in gmetad config. You will need to define the
    hosts from whom metrics should be collected. Once this configuration, you
    should see the host in gmetad UI interface.

    Thanks,
    Sagar

    -----Original Message-----
    From: Hari Sreekumar
    Sent: Monday, November 22, 2010 6:25 PM
    To: common-user@hadoop.apache.org
    Subject: Using ganglia to monitor hadoop

    Hi,

    I have set up ganglia for my cluster, and it works fine. What are
    the changes I need to make to make ganglia show hadoop related parameters?
    My gmond/gmetad config is default except for one change in gmond.conf for
    all nodes:

    /* You can specify as many tcp_accept_channels as you like to share
    an xml description of the state of the cluster */
    tcp_accept_channel {
    port = 8650
    }

    I have changed the hadoop-metrics.properties file as follows:

    # Configuration of the "dfs" context for ganglia
    dfs.class=org.apache.hadoop.metrics.ganglia.GangliaContext
    dfs.period=10
    dfs.servers=hadoop1:8649 hadoop2:8649 hadoop3:8649 ejabber:8649


    # Configuration of the "mapred" context for null
    #mapred.class=org.apache.hadoop.metrics.spi.NullContext

    # Configuration of the "mapred" context for file
    #mapred.class=org.apache.hadoop.metrics.file.FileContext
    #mapred.period=10
    #mapred.fileName=/tmp/mrmetrics.log

    # Configuration of the "mapred" context for ganglia
    mapred.class=org.apache.hadoop.metrics.ganglia.GangliaContext
    mapred.period=10
    mapred.servers=hadoop1:8649 hadoop2:8649 hadoop3:8649 ejabber:8649


    # Configuration of the "jvm" context for null
    #jvm.class=org.apache.hadoop.metrics.spi.NullContext

    # Configuration of the "jvm" context for file
    #jvm.class=org.apache.hadoop.metrics.file.FileContext
    #jvm.period=10
    #jvm.fileName=/tmp/jvmmetrics.log

    # Configuration of the "jvm" context for ganglia
    jvm.class=org.apache.hadoop.metrics.ganglia.GangliaContext
    jvm.period=10
    jvm.servers=hadoop1:8649 hadoop2:8649 hadoop3:8649 ejabber:8649

    rpc.class=org.apache.hadoop.metrics.ganglia.GangliaContext
    rpc.period=10
    rpc.servers=hadoop1:8649 hadoop2:8649 hadoop3:8649 ejabber:8649

    hadoop1,2,3 and ejabber are my nodes.

    But I am unable to see any new metrics in ganglia's drop down menu. Is
    there
    something else I need to do as well? I have restarted gmond on all nodes.

    thanks,
    Hari

    DISCLAIMER
    ==========
    This e-mail may contain privileged and confidential information which is
    the property of Persistent Systems Ltd. It is intended only for the use of
    the individual or entity to which it is addressed. If you are not the
    intended recipient, you are not authorized to read, retain, copy, print,
    distribute or use this message. If you have received this communication in
    error, please notify the sender and delete all copies of this message.
    Persistent Systems Ltd. does not accept any liability for virus infected
    mails.
  • Hari Sreekumar at Nov 22, 2010 at 5:21 pm
    Hi Matt,
    Tried using the specified(default) mcast join ip address. Still no
    new metrics. Do I have to restart the hadoop cluster for the changes to take
    effect? The new metrics will appear in the "Metrics" dropdown menu of
    ganglia UI home page right? Or is there some other link for custom metrics?

    thanks,
    Hari
    On Mon, Nov 22, 2010 at 9:08 PM, Matthew E. Kennedy wrote:

    Hari,

    I have this set up and working and it looks to me like your *.servers
    properties are set incorrectly. Point that property to the multicast
    address that is listed in your gmond.conf file on port 8649.

    You can find the multicast address with:
    cat /etc/gmond.conf | grep mcast

    an example of the config follows:
    dfs.servers=<mcast_join_ip_addr>:8649

    Good luck,
    Matt
    On Nov 22, 2010, at 8:20 AM, Hari Sreekumar wrote:

    Hi Sagar,

    I followed that article. But I am not clear about some things:
    1. Should I use port 8650 or 8649? I have tried both actually, and 8649
    makes sense, but just to confirm. (I have set tcp_accept_channel as 8650)
    2. In hadoop-metrics.properties, should I mention all my nodes as I have
    done or only the local node? e.g, localhost:8649 everywhere?

    I have configured gmetad.conf, and I am able to see metrics from all my
    nodes in my ganglia UI. I just don't see any hadoop specific metrics.

    Thanks,
    Hari

    On Mon, Nov 22, 2010 at 6:42 PM, Sagar Shukla <
    sagar_shukla@persistent.co.in
    wrote:
    Hi Hari,
    Not sure if you have already followed article -
    http://wiki.apache.org/hadoop/GangliaMetrics to configure Ganglia for
    Hadoop Monitoring. Also in the details given by you, I do not see any
    configuration settings done in gmetad config. You will need to define
    the
    hosts from whom metrics should be collected. Once this configuration,
    you
    should see the host in gmetad UI interface.

    Thanks,
    Sagar

    -----Original Message-----
    From: Hari Sreekumar
    Sent: Monday, November 22, 2010 6:25 PM
    To: common-user@hadoop.apache.org
    Subject: Using ganglia to monitor hadoop

    Hi,

    I have set up ganglia for my cluster, and it works fine. What are
    the changes I need to make to make ganglia show hadoop related
    parameters?
    My gmond/gmetad config is default except for one change in gmond.conf
    for
    all nodes:

    /* You can specify as many tcp_accept_channels as you like to share
    an xml description of the state of the cluster */
    tcp_accept_channel {
    port = 8650
    }

    I have changed the hadoop-metrics.properties file as follows:

    # Configuration of the "dfs" context for ganglia
    dfs.class=org.apache.hadoop.metrics.ganglia.GangliaContext
    dfs.period=10
    dfs.servers=hadoop1:8649 hadoop2:8649 hadoop3:8649 ejabber:8649


    # Configuration of the "mapred" context for null
    #mapred.class=org.apache.hadoop.metrics.spi.NullContext

    # Configuration of the "mapred" context for file
    #mapred.class=org.apache.hadoop.metrics.file.FileContext
    #mapred.period=10
    #mapred.fileName=/tmp/mrmetrics.log

    # Configuration of the "mapred" context for ganglia
    mapred.class=org.apache.hadoop.metrics.ganglia.GangliaContext
    mapred.period=10
    mapred.servers=hadoop1:8649 hadoop2:8649 hadoop3:8649 ejabber:8649


    # Configuration of the "jvm" context for null
    #jvm.class=org.apache.hadoop.metrics.spi.NullContext

    # Configuration of the "jvm" context for file
    #jvm.class=org.apache.hadoop.metrics.file.FileContext
    #jvm.period=10
    #jvm.fileName=/tmp/jvmmetrics.log

    # Configuration of the "jvm" context for ganglia
    jvm.class=org.apache.hadoop.metrics.ganglia.GangliaContext
    jvm.period=10
    jvm.servers=hadoop1:8649 hadoop2:8649 hadoop3:8649 ejabber:8649

    rpc.class=org.apache.hadoop.metrics.ganglia.GangliaContext
    rpc.period=10
    rpc.servers=hadoop1:8649 hadoop2:8649 hadoop3:8649 ejabber:8649

    hadoop1,2,3 and ejabber are my nodes.

    But I am unable to see any new metrics in ganglia's drop down menu. Is
    there
    something else I need to do as well? I have restarted gmond on all
    nodes.
    thanks,
    Hari

    DISCLAIMER
    ==========
    This e-mail may contain privileged and confidential information which is
    the property of Persistent Systems Ltd. It is intended only for the use
    of
    the individual or entity to which it is addressed. If you are not the
    intended recipient, you are not authorized to read, retain, copy, print,
    distribute or use this message. If you have received this communication
    in
    error, please notify the sender and delete all copies of this message.
    Persistent Systems Ltd. does not accept any liability for virus infected
    mails.
  • Jinsong Hu at Nov 22, 2010 at 5:59 pm
    Hi, There:
    I have been searching good gc setting for hadoop namenode and datanode,
    and I use this setting for
    namenode and data node:

    -XX:NewSize=18m -XX:MaxNewSize=18m -XX:+
    HeapDumpOnOutOfMemoryError -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=70
    -XX:+CMSClassUnloadingEnabled -XX:+DisableExplicitGC -XX:+UseCompressedOops
    -XX:+DoEscapeAnalysis -XX:+AggressiveOpts -verbose:gc -XX:+PrintGCDetails -
    XX:+PrintGCTimeStamps -Xmx3G -Dcom.sun.management.jmxremote.port=8004 -Xloggc:/usr/lib/hadoop/logs/gc-namenode.log
    -XX:ParallelGCThreads=8 -XX:PermSize=256m -XX
    :MaxPermSize=256m

    The problem , however, is that I see frequent CMS gc that happens every 100
    second. and 2/3 of the GC says things like this:

    CMS: abort preclean due to time 795.999:
    [CMS-concurrent-abortable-preclean: 0.
    188/5.082 secs] [Times: user=0.15 sys=0.00, real=5.08 secs]

    I searched internet and some people says that is ok. However, I would like
    to see if anybody else has a good GC setting for hadoop that produces less
    frequent GC and works well.

    The advantage of the above setting is that it controls memory growth well,
    and there is no sudden full GC.
    the default setting without above tunning produces very bad gc profile. If
    anybody else can share their gc tunning experience, that is really
    appreciated.

    Jimmy
  • Matthew E. Kennedy at Nov 22, 2010 at 8:10 pm
    I started to get mapred and jvm metrics without restarting the cluster, but it did require a restart to start getting dfs stats. Yes, you should see the new metrics appear in the dropdown.
    On Nov 22, 2010, at 12:21 PM, Hari Sreekumar wrote:

    Hi Matt,
    Tried using the specified(default) mcast join ip address. Still no
    new metrics. Do I have to restart the hadoop cluster for the changes to take
    effect? The new metrics will appear in the "Metrics" dropdown menu of
    ganglia UI home page right? Or is there some other link for custom metrics?

    thanks,
    Hari

    On Mon, Nov 22, 2010 at 9:08 PM, Matthew E. Kennedy <matt.kennedy@spadac.com
    wrote:
    Hari,

    I have this set up and working and it looks to me like your *.servers
    properties are set incorrectly. Point that property to the multicast
    address that is listed in your gmond.conf file on port 8649.

    You can find the multicast address with:
    cat /etc/gmond.conf | grep mcast

    an example of the config follows:
    dfs.servers=<mcast_join_ip_addr>:8649

    Good luck,
    Matt
    On Nov 22, 2010, at 8:20 AM, Hari Sreekumar wrote:

    Hi Sagar,

    I followed that article. But I am not clear about some things:
    1. Should I use port 8650 or 8649? I have tried both actually, and 8649
    makes sense, but just to confirm. (I have set tcp_accept_channel as 8650)
    2. In hadoop-metrics.properties, should I mention all my nodes as I have
    done or only the local node? e.g, localhost:8649 everywhere?

    I have configured gmetad.conf, and I am able to see metrics from all my
    nodes in my ganglia UI. I just don't see any hadoop specific metrics.

    Thanks,
    Hari

    On Mon, Nov 22, 2010 at 6:42 PM, Sagar Shukla <
    sagar_shukla@persistent.co.in
    wrote:
    Hi Hari,
    Not sure if you have already followed article -
    http://wiki.apache.org/hadoop/GangliaMetrics to configure Ganglia for
    Hadoop Monitoring. Also in the details given by you, I do not see any
    configuration settings done in gmetad config. You will need to define
    the
    hosts from whom metrics should be collected. Once this configuration,
    you
    should see the host in gmetad UI interface.

    Thanks,
    Sagar

    -----Original Message-----
    From: Hari Sreekumar
    Sent: Monday, November 22, 2010 6:25 PM
    To: common-user@hadoop.apache.org
    Subject: Using ganglia to monitor hadoop

    Hi,

    I have set up ganglia for my cluster, and it works fine. What are
    the changes I need to make to make ganglia show hadoop related
    parameters?
    My gmond/gmetad config is default except for one change in gmond.conf
    for
    all nodes:

    /* You can specify as many tcp_accept_channels as you like to share
    an xml description of the state of the cluster */
    tcp_accept_channel {
    port = 8650
    }

    I have changed the hadoop-metrics.properties file as follows:

    # Configuration of the "dfs" context for ganglia
    dfs.class=org.apache.hadoop.metrics.ganglia.GangliaContext
    dfs.period=10
    dfs.servers=hadoop1:8649 hadoop2:8649 hadoop3:8649 ejabber:8649


    # Configuration of the "mapred" context for null
    #mapred.class=org.apache.hadoop.metrics.spi.NullContext

    # Configuration of the "mapred" context for file
    #mapred.class=org.apache.hadoop.metrics.file.FileContext
    #mapred.period=10
    #mapred.fileName=/tmp/mrmetrics.log

    # Configuration of the "mapred" context for ganglia
    mapred.class=org.apache.hadoop.metrics.ganglia.GangliaContext
    mapred.period=10
    mapred.servers=hadoop1:8649 hadoop2:8649 hadoop3:8649 ejabber:8649


    # Configuration of the "jvm" context for null
    #jvm.class=org.apache.hadoop.metrics.spi.NullContext

    # Configuration of the "jvm" context for file
    #jvm.class=org.apache.hadoop.metrics.file.FileContext
    #jvm.period=10
    #jvm.fileName=/tmp/jvmmetrics.log

    # Configuration of the "jvm" context for ganglia
    jvm.class=org.apache.hadoop.metrics.ganglia.GangliaContext
    jvm.period=10
    jvm.servers=hadoop1:8649 hadoop2:8649 hadoop3:8649 ejabber:8649

    rpc.class=org.apache.hadoop.metrics.ganglia.GangliaContext
    rpc.period=10
    rpc.servers=hadoop1:8649 hadoop2:8649 hadoop3:8649 ejabber:8649

    hadoop1,2,3 and ejabber are my nodes.

    But I am unable to see any new metrics in ganglia's drop down menu. Is
    there
    something else I need to do as well? I have restarted gmond on all
    nodes.
    thanks,
    Hari

    DISCLAIMER
    ==========
    This e-mail may contain privileged and confidential information which is
    the property of Persistent Systems Ltd. It is intended only for the use
    of
    the individual or entity to which it is addressed. If you are not the
    intended recipient, you are not authorized to read, retain, copy, print,
    distribute or use this message. If you have received this communication
    in
    error, please notify the sender and delete all copies of this message.
    Persistent Systems Ltd. does not accept any liability for virus infected
    mails.
  • Hari Sreekumar at Nov 23, 2010 at 5:33 am
    Hey Matthew,

    Got it working now.... had to restart the cluster. I am not sure
    if restarting the cluster is what did the trick though. Even after
    restarting, I had no metrics for about half an hour and then they started
    coming. Thanks a lot for helping me set it up!

    Cheers,
    Hari
    On Tue, Nov 23, 2010 at 1:40 AM, Matthew E. Kennedy wrote:

    I started to get mapred and jvm metrics without restarting the cluster, but
    it did require a restart to start getting dfs stats. Yes, you should see
    the new metrics appear in the dropdown.
    On Nov 22, 2010, at 12:21 PM, Hari Sreekumar wrote:

    Hi Matt,
    Tried using the specified(default) mcast join ip address. Still no
    new metrics. Do I have to restart the hadoop cluster for the changes to take
    effect? The new metrics will appear in the "Metrics" dropdown menu of
    ganglia UI home page right? Or is there some other link for custom metrics?
    thanks,
    Hari

    On Mon, Nov 22, 2010 at 9:08 PM, Matthew E. Kennedy <
    matt.kennedy@spadac.com
    wrote:
    Hari,

    I have this set up and working and it looks to me like your *.servers
    properties are set incorrectly. Point that property to the multicast
    address that is listed in your gmond.conf file on port 8649.

    You can find the multicast address with:
    cat /etc/gmond.conf | grep mcast

    an example of the config follows:
    dfs.servers=<mcast_join_ip_addr>:8649

    Good luck,
    Matt
    On Nov 22, 2010, at 8:20 AM, Hari Sreekumar wrote:

    Hi Sagar,

    I followed that article. But I am not clear about some things:
    1. Should I use port 8650 or 8649? I have tried both actually, and 8649
    makes sense, but just to confirm. (I have set tcp_accept_channel as
    8650)
    2. In hadoop-metrics.properties, should I mention all my nodes as I
    have
    done or only the local node? e.g, localhost:8649 everywhere?

    I have configured gmetad.conf, and I am able to see metrics from all my
    nodes in my ganglia UI. I just don't see any hadoop specific metrics.

    Thanks,
    Hari

    On Mon, Nov 22, 2010 at 6:42 PM, Sagar Shukla <
    sagar_shukla@persistent.co.in
    wrote:
    Hi Hari,
    Not sure if you have already followed article -
    http://wiki.apache.org/hadoop/GangliaMetrics to configure Ganglia for
    Hadoop Monitoring. Also in the details given by you, I do not see any
    configuration settings done in gmetad config. You will need to define
    the
    hosts from whom metrics should be collected. Once this configuration,
    you
    should see the host in gmetad UI interface.

    Thanks,
    Sagar

    -----Original Message-----
    From: Hari Sreekumar
    Sent: Monday, November 22, 2010 6:25 PM
    To: common-user@hadoop.apache.org
    Subject: Using ganglia to monitor hadoop

    Hi,

    I have set up ganglia for my cluster, and it works fine. What are
    the changes I need to make to make ganglia show hadoop related
    parameters?
    My gmond/gmetad config is default except for one change in gmond.conf
    for
    all nodes:

    /* You can specify as many tcp_accept_channels as you like to share
    an xml description of the state of the cluster */
    tcp_accept_channel {
    port = 8650
    }

    I have changed the hadoop-metrics.properties file as follows:

    # Configuration of the "dfs" context for ganglia
    dfs.class=org.apache.hadoop.metrics.ganglia.GangliaContext
    dfs.period=10
    dfs.servers=hadoop1:8649 hadoop2:8649 hadoop3:8649 ejabber:8649


    # Configuration of the "mapred" context for null
    #mapred.class=org.apache.hadoop.metrics.spi.NullContext

    # Configuration of the "mapred" context for file
    #mapred.class=org.apache.hadoop.metrics.file.FileContext
    #mapred.period=10
    #mapred.fileName=/tmp/mrmetrics.log

    # Configuration of the "mapred" context for ganglia
    mapred.class=org.apache.hadoop.metrics.ganglia.GangliaContext
    mapred.period=10
    mapred.servers=hadoop1:8649 hadoop2:8649 hadoop3:8649 ejabber:8649


    # Configuration of the "jvm" context for null
    #jvm.class=org.apache.hadoop.metrics.spi.NullContext

    # Configuration of the "jvm" context for file
    #jvm.class=org.apache.hadoop.metrics.file.FileContext
    #jvm.period=10
    #jvm.fileName=/tmp/jvmmetrics.log

    # Configuration of the "jvm" context for ganglia
    jvm.class=org.apache.hadoop.metrics.ganglia.GangliaContext
    jvm.period=10
    jvm.servers=hadoop1:8649 hadoop2:8649 hadoop3:8649 ejabber:8649

    rpc.class=org.apache.hadoop.metrics.ganglia.GangliaContext
    rpc.period=10
    rpc.servers=hadoop1:8649 hadoop2:8649 hadoop3:8649 ejabber:8649

    hadoop1,2,3 and ejabber are my nodes.

    But I am unable to see any new metrics in ganglia's drop down menu. Is
    there
    something else I need to do as well? I have restarted gmond on all
    nodes.
    thanks,
    Hari

    DISCLAIMER
    ==========
    This e-mail may contain privileged and confidential information which
    is
    the property of Persistent Systems Ltd. It is intended only for the
    use
    of
    the individual or entity to which it is addressed. If you are not the
    intended recipient, you are not authorized to read, retain, copy,
    print,
    distribute or use this message. If you have received this
    communication
    in
    error, please notify the sender and delete all copies of this message.
    Persistent Systems Ltd. does not accept any liability for virus
    infected
    mails.

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupcommon-user @
categorieshadoop
postedNov 22, '10 at 12:55p
activeNov 23, '10 at 5:33a
posts8
users4
websitehadoop.apache.org...
irc#hadoop

People

Translate

site design / logo © 2022 Grokbase