FAQ
I was trying to get Ganglia 3.1 to work with the stable hadoop-0.20.2
version from Apache. I patched this release from HADOOP-4675 using
HADOOP-4675-v7.patch as suggested by CDH3 release notes [1] I am unable to
see any hadoop metrics on the Ganglia monitoring UI. The other metrics that
gmond spews (system CPU/Memory etc) seem to work.

When I switch to FileContext the metrics are written properly to the log
file. Once I moved to GangliaContext31 it doesn't show anything. I tried
pointing the "servers" param to localhost:8649 while listening on that port
using netcat on that machine... nothing comes up on netcat. Has anyone faced
this issue?

This is what most of my hadoop-metrics looks like:

dfs.class=org.apache.hadoop.metrics.ganglia.GangliaContext31
dfs.period=10
dfs.fileName=/tmp/dfsmetrics.log
dfs.servers=$Ganglia_Server:9988

# Configuration of the "mapred" context for null
mapred.class=org.apache.hadoop.metrics.ganglia.GangliaContext31
mapred.period=10
mapred.fileName=/tmp/mrmetrics.log
mapred.servers=$Ganglia_Server:9988

# Configuration of the "jvm" context for null
jvm.class=org.apache.hadoop.metrics.ganglia.GangliaContext31
jvm.period=10
jvm.fileName=/tmp/jvmmetrics.log
jvm.servers=$GANGLIA_SERVER:9988

-G.

[1] - http://archive.cloudera.com/cdh/3/hadoop-0.20.2+228.CHANGES.txt

Search Discussions

  • Brian Bockelman at Aug 24, 2010 at 1:43 pm

    On Aug 24, 2010, at 8:27 AM, Gautam wrote:

    I was trying to get Ganglia 3.1 to work with the stable hadoop-0.20.2
    version from Apache. I patched this release from HADOOP-4675 using
    HADOOP-4675-v7.patch as suggested by CDH3 release notes [1] I am unable to
    see any hadoop metrics on the Ganglia monitoring UI. The other metrics that
    gmond spews (system CPU/Memory etc) seem to work.

    When I switch to FileContext the metrics are written properly to the log
    file. Once I moved to GangliaContext31 it doesn't show anything. I tried
    pointing the "servers" param to localhost:8649 while listening on that port
    using netcat on that machine... nothing comes up on netcat. Has anyone faced
    this issue?
    This is possibly misleading - netcat won't work if Hadoop is using UDP.

    My advice is to do:

    telnet $Ganglia_Server 9988

    and see if it spits out a bunch of XML. In the typical Ganglia configuration, it is set up to listen on UDP and write on TCP of the same port.

    A third thing to test is to switch the hadoop-metrics back to the file output, and make sure something gets written to the log file. The issue might be upstream.

    Brian
    This is what most of my hadoop-metrics looks like:

    dfs.class=org.apache.hadoop.metrics.ganglia.GangliaContext31
    dfs.period=10
    dfs.fileName=/tmp/dfsmetrics.log
    dfs.servers=$Ganglia_Server:9988

    # Configuration of the "mapred" context for null
    mapred.class=org.apache.hadoop.metrics.ganglia.GangliaContext31
    mapred.period=10
    mapred.fileName=/tmp/mrmetrics.log
    mapred.servers=$Ganglia_Server:9988

    # Configuration of the "jvm" context for null
    jvm.class=org.apache.hadoop.metrics.ganglia.GangliaContext31
    jvm.period=10
    jvm.fileName=/tmp/jvmmetrics.log
    jvm.servers=$GANGLIA_SERVER:9988

    -G.

    [1] - http://archive.cloudera.com/cdh/3/hadoop-0.20.2+228.CHANGES.txt
  • Gautam at Aug 25, 2010 at 10:53 am
    Brian,

    Works for me now.. one should point the "servers" param to the multicast
    address that gmond writes to and listens on... and not the ganglia server.
    Started working once I did this.

    thanks for you inputs,
    -G.
    On Tue, Aug 24, 2010 at 7:12 PM, Brian Bockelman wrote:

    On Aug 24, 2010, at 8:27 AM, Gautam wrote:

    I was trying to get Ganglia 3.1 to work with the stable hadoop-0.20.2
    version from Apache. I patched this release from HADOOP-4675 using
    HADOOP-4675-v7.patch as suggested by CDH3 release notes [1] I am unable to
    see any hadoop metrics on the Ganglia monitoring UI. The other metrics that
    gmond spews (system CPU/Memory etc) seem to work.

    When I switch to FileContext the metrics are written properly to the log
    file. Once I moved to GangliaContext31 it doesn't show anything. I tried
    pointing the "servers" param to localhost:8649 while listening on that port
    using netcat on that machine... nothing comes up on netcat. Has anyone faced
    this issue?
    This is possibly misleading - netcat won't work if Hadoop is using UDP.

    My advice is to do:

    telnet $Ganglia_Server 9988

    and see if it spits out a bunch of XML. In the typical Ganglia
    configuration, it is set up to listen on UDP and write on TCP of the same
    port.

    A third thing to test is to switch the hadoop-metrics back to the file
    output, and make sure something gets written to the log file. The issue
    might be upstream.

    Brian
    This is what most of my hadoop-metrics looks like:

    dfs.class=org.apache.hadoop.metrics.ganglia.GangliaContext31
    dfs.period=10
    dfs.fileName=/tmp/dfsmetrics.log
    dfs.servers=$Ganglia_Server:9988

    # Configuration of the "mapred" context for null
    mapred.class=org.apache.hadoop.metrics.ganglia.GangliaContext31
    mapred.period=10
    mapred.fileName=/tmp/mrmetrics.log
    mapred.servers=$Ganglia_Server:9988

    # Configuration of the "jvm" context for null
    jvm.class=org.apache.hadoop.metrics.ganglia.GangliaContext31
    jvm.period=10
    jvm.fileName=/tmp/jvmmetrics.log
    jvm.servers=$GANGLIA_SERVER:9988

    -G.

    [1] - http://archive.cloudera.com/cdh/3/hadoop-0.20.2+228.CHANGES.txt

    --
    "If you really want something in this life, you have to work for it. Now,
    quiet! They're about to announce the lottery numbers..."
  • Brian Bockelman at Aug 25, 2010 at 12:59 pm
    Hi Gautam,

    Yup - that's one possible way to configure Ganglia and is common at many sites. That's why I usually recommend the "telnet" trick to determine what IP address your configuration is using.

    Brian
    On Aug 25, 2010, at 5:53 AM, Gautam wrote:

    Brian,

    Works for me now.. one should point the "servers" param to the multicast
    address that gmond writes to and listens on... and not the ganglia server.
    Started working once I did this.

    thanks for you inputs,
    -G.
    On Tue, Aug 24, 2010 at 7:12 PM, Brian Bockelman wrote:

    On Aug 24, 2010, at 8:27 AM, Gautam wrote:

    I was trying to get Ganglia 3.1 to work with the stable hadoop-0.20.2
    version from Apache. I patched this release from HADOOP-4675 using
    HADOOP-4675-v7.patch as suggested by CDH3 release notes [1] I am unable to
    see any hadoop metrics on the Ganglia monitoring UI. The other metrics that
    gmond spews (system CPU/Memory etc) seem to work.

    When I switch to FileContext the metrics are written properly to the log
    file. Once I moved to GangliaContext31 it doesn't show anything. I tried
    pointing the "servers" param to localhost:8649 while listening on that port
    using netcat on that machine... nothing comes up on netcat. Has anyone faced
    this issue?
    This is possibly misleading - netcat won't work if Hadoop is using UDP.

    My advice is to do:

    telnet $Ganglia_Server 9988

    and see if it spits out a bunch of XML. In the typical Ganglia
    configuration, it is set up to listen on UDP and write on TCP of the same
    port.

    A third thing to test is to switch the hadoop-metrics back to the file
    output, and make sure something gets written to the log file. The issue
    might be upstream.

    Brian
    This is what most of my hadoop-metrics looks like:

    dfs.class=org.apache.hadoop.metrics.ganglia.GangliaContext31
    dfs.period=10
    dfs.fileName=/tmp/dfsmetrics.log
    dfs.servers=$Ganglia_Server:9988

    # Configuration of the "mapred" context for null
    mapred.class=org.apache.hadoop.metrics.ganglia.GangliaContext31
    mapred.period=10
    mapred.fileName=/tmp/mrmetrics.log
    mapred.servers=$Ganglia_Server:9988

    # Configuration of the "jvm" context for null
    jvm.class=org.apache.hadoop.metrics.ganglia.GangliaContext31
    jvm.period=10
    jvm.fileName=/tmp/jvmmetrics.log
    jvm.servers=$GANGLIA_SERVER:9988

    -G.

    [1] - http://archive.cloudera.com/cdh/3/hadoop-0.20.2+228.CHANGES.txt

    --
    "If you really want something in this life, you have to work for it. Now,
    quiet! They're about to announce the lottery numbers..."

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupcommon-user @
categorieshadoop
postedAug 24, '10 at 1:28p
activeAug 25, '10 at 12:59p
posts4
users2
websitehadoop.apache.org...
irc#hadoop

2 users in discussion

Brian Bockelman: 2 posts Gautam: 2 posts

People

Translate

site design / logo © 2022 Grokbase