FAQ

[Tomcat-users] [OT] Inspecting JMX

Christopher Schultz
Jan 26, 2012 at 6:32 pm
All,

I'm trying to get into the business of inspecting my JVM using the
mbeans that Tomcat (and the JVM) expose, but I'm having great
difficulty. I'm not even sure if I'm doing the right things.

I have a running Tomcat instance (happens to be 7.0.25 running on Sun
JRE 1.6.something on Mac OS X Lion), and I can happily connect to it
using jconsole (using a process id) with no special startup parameters
for Tomcat itself.

If I understand this correctly, it's because when the JVM starts up,
it allows localhost JMX inspection with no authentication using the
"Attach" API [1].

I have enabled neither Tomcat's
mbeans.GlobalResourcesLifecycleListener not the
JmxRemoveLifecycleListener [2].

I can see a wealth of information about the JVM and Tomcat in this
configuration.

Now I'm trying to get similar information using a command-line tool
that is very simple called check_jmx -- it's a plug-in for Nagios. It
appears that this tool does not support the "attach" API and so it
looks like I'll have to enable "remote JMX", so I've followed the
instructions on Tomcat's monitoring page to enable remote JMX [3]:

$ export CATALINA_OPTS='\
- -Dcom.sun.management.jmxremote \
- -Dcom.sun.management.jmxremote.port=1234 \
- -Dcom.sun.management.jmxremote.ssl=false \
- -Dcom.sun.management.jmxremote.authenticate=false'

When running Tomcat, I can see that those parameters are actually part
of the command line. I can also see that the port is bound:

$ netstat -an | grep 1234
tcp46 0 0 *.1234 *.*
LISTEN
tcp4 0 0 127.0.0.1.12345 *.*
LISTEN
tcp6 0 0 ::1.12345 *.*
LISTEN

I can use this tool with a command line such as the following:

$ ./check_jmx -U service:jmx:rmi:///jndi/rmi://localhost:1234/jmxrmi \
-O java.lang:type=Memory -A HeapMemoryUsage -K used \
-I HeapMemoryUsage -J used -vvvv -w 4248302272 -c 5498760192

JMX OK
HeapMemoryUsage.used=35587776{committed=85000192;init=0;max=129957888;used=35587776}

Without going into too much detail, those options allow check_jmx to
probe a certain value, then use the -w and -c options to determine
whether the value obtained from the JMX server is okay, in the warning
zone, or in the critical zone, and then builds a text response based
upon that.

So, now I want to move to my server which happens to be an Amazon EC2
instance with an elastic IP assigned to it so I can get to it from the
outside. Most ports are blocked. I try the same type of thing: set up
the com.sun.management.etc parameters and launch Tomcat. netstat reports:

$ netstat -plan | grep 1234
tcp6 0 0 :::1234 :::*
LISTEN 2819/java

So I should be good to go. I launch the command the same as on my
local machine, but this time it hangs and then I get an exception:

$ ./check_jmx -U service:jmx:rmi:///jndi/rmi://localhost:1234/jmxrmi
- -O java.lang:type=Memory -A HeapMemoryUsage -K used -I HeapMemoryUsage
- -J used -vvvv -w 4248302272 -c 5498760192
JMX CRITICAL Connection refused to host: [public IP]; nested exception
is:
java.net.ConnectException: Connection timed out connecting to
java.lang:type=Memory by URL
service:jmx:rmi:///jndi/rmi://localhost:1234/jmxrmijava.rmi.ConnectException:
Connection refused to host: 107.21.113.203; nested exception is:
java.net.ConnectException: Connection timed out
at sun.rmi.transport.tcp.TCPEndpoint.newSocket(TCPEndpoint.java:601)
at sun.rmi.transport.tcp.TCPChannel.createConnection(TCPChannel.java:198)
at sun.rmi.transport.tcp.TCPChannel.newConnection(TCPChannel.java:184)
at sun.rmi.server.UnicastRef.invoke(UnicastRef.java:110)
at javax.management.remote.rmi.RMIServerImpl_Stub.newClient(Unknown
Source)
at
javax.management.remote.rmi.RMIConnector.getConnection(RMIConnector.java:2329)
at
javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:279)
at
javax.management.remote.JMXConnectorFactory.connect(JMXConnectorFactory.java:248)
at
javax.management.remote.JMXConnectorFactory.connect(JMXConnectorFactory.java:207)
at org.nagios.JMXQuery.connect(JMXQuery.java:53)
at org.nagios.JMXQuery.main(JMXQuery.java:75)
Caused by: java.net.ConnectException: Connection timed out
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:351)
at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:213)
at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:200)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366)
at java.net.Socket.connect(Socket.java:529)
at java.net.Socket.connect(Socket.java:478)
at java.net.Socket.(Socket.java:189)
at
sun.rmi.transport.proxy.RMIDirectSocketFactory.createSocket(RMIDirectSocketFactory.java:22)
at
sun.rmi.transport.proxy.RMIMasterSocketFactory.createSocket(RMIMasterSocketFactory.java:128)
at sun.rmi.transport.tcp.TCPEndpoint.newSocket(TCPEndpoint.java:595)
... 10 more

So, [public IP] is actually my public, IPv4 address. I figure the
problem might be IPv4-versus-IPv6, but before I go "preferring" the
IPv4 stack, I check netstat during the execution of check_jmx:

$ netstat -plan | grep 1234
tcp6 0 0 :::1234 :::*
LISTEN 2819/java
tcp6 0 0 127.0.0.1:40292 127.0.0.1:1234
ESTABLISHED 2950/java
tcp6 0 0 127.0.0.1:1234 127.0.0.1:40292
ESTABLISHED 2819/java

(pid 2819 is my Tomcat process, and 2950 is check_jmx). So, what else
is check_jmx doing?

$ netstat -plan | grep 2950
(Not all processes could be identified, non-owned process info
will not be shown, you would have to be root to see it all.)
tcp6 0 0 127.0.0.1:40292 127.0.0.1:1234
ESTABLISHED 2950/java
tcp6 0 1 10.192.214.127:44419 107.21.113.203:51087
SYN_SENT 2950/java

So, check_jmx is actually successfully establishing a socket with the
JMX port but it looks like it's doing something weird with the
(random) RMI port. First, it's trying to use the public IP address
instead of just localhost (which seems stupid, but that's probably
what the JMX service told it to do). Second, it's calling-out on the
internet IP address, which is probably a result of using a
non-localhost destination IP address.

So, a couple of questions:

1. Is there a way to tell the JVM which interface to use for the
second, random RMI port that gets opened? I think if everything runs
on :: or 127.0.0.1 then all will probably be well.

2. Should I just give up and use JmxRemoteLifecycleListener? Will that
actually buy me anything? I will always be running check_jmx
locally. I'm not s huge fan of having to deploy with an optional
library tossed-into Tomcat's lib direcory (even though it's actually
from the find Apache folks).

3. Should I just give up and use the manager app's jmxproxy? I don't
currently deploy the manager app, and I'd like to avoid doing that
if possible. But, it may be a slightly cleaner solution.

4. Should I hack the code for check_jmx to use the Attach API and
try to avoid all of this stupid port business? Getting the PID
of the Tomcat process shouldn't be hard as long as I use
CATALINA_PID and get the value from there.

Those playing along at home might remember that I recently posted a
question about Kitty, another JMX CLI program. The documentation for
Kitty has the same type of ugly RMI URLs in it, so I suspect I'll have
the same problem with another tool: I think this is squarely a
server-process configuration issue that I have to deal with.

Any suggestions would be most appreciated.

Thanks,
- -chris

[1]
http://docs.oracle.com/javase/6/docs/technotes/guides/management/agent.html

[2]
http://tomcat.apache.org/tomcat-7.0-doc/config/listeners.html#JMX_Remote_Lifecycle_Listener_-_org.apache.catalina.mbeans.JmxRemoteLifecycleListener


[3]
http://tomcat.apache.org/tomcat-7.0-doc/monitoring.html#Enabling_JMX_Remote


---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
reply

Search Discussions

13 responses

  • Caldarale, Charles R at Jan 26, 2012 at 6:41 pm

    From: Christopher Schultz
    Subject: [OT] Inspecting JMX
    Should I just give up and use JmxRemoteLifecycleListener?
    Yes. You really aren't get to get anywhere on EC2 without it.
    Should I hack the code for check_jmx to use the Attach API.
    Maybe; might be fun. (Always seemed to be a security hole to me :-)

    - Chuck


    THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers.
  • Christopher Schultz at Jan 26, 2012 at 6:46 pm
    Chuck,
    On 1/26/12 1:41 PM, Caldarale, Charles R wrote:
    From: Christopher Schultz
    Subject: [OT] Inspecting JMX
    Should I just give up and use JmxRemoteLifecycleListener?
    Yes. You really aren't get to get anywhere on EC2 without it.
    I figured since I was using localhost everything would stay there
    anyway. I'll give JmxRemoteLifecycleListener a chance.
    Should I hack the code for check_jmx to use the Attach API.
    Maybe; might be fun. (Always seemed to be a security hole to me
    :-)
    I'm not sure you can disable it at the JVM level, can you? Actually, I
    guess I could tell the JMX server to require a valid client cert, and
    then just have an empty trust store :)

    - -chris


    ---------------------------------------------------------------------
    To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
    For additional commands, e-mail: users-help@tomcat.apache.org
  • Kees Jan Koster at Jan 26, 2012 at 6:52 pm
    Dear Chris,
    Should I just give up and use JmxRemoteLifecycleListener?
    Yes. You really aren't get to get anywhere on EC2 without it.
    I figured since I was using localhost everything would stay there
    anyway. I'll give JmxRemoteLifecycleListener a chance.

    Or you could write a small piece of code that lives inside your app and writes the monitoring data into the Nagios queue directly. It is a simple text-based protocol.
    --
    Kees Jan

    http://java-monitor.com/
    kjkoster@kjkoster.org
    +31651838192

    Human beings make life so interesting. Do you know that in a universe so full of wonders,
    they have managed to invent boredom. Quite astonishing... -- Terry Pratchett


    ---------------------------------------------------------------------
    To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
    For additional commands, e-mail: users-help@tomcat.apache.org
  • Christopher Schultz at Jan 26, 2012 at 7:43 pm
    Kees Jan,
    On 1/26/12 1:52 PM, Kees Jan Koster wrote:
    Dear Chris,
    Should I just give up and use JmxRemoteLifecycleListener?
    Yes. You really aren't get to get anywhere on EC2 without it.
    I figured since I was using localhost everything would stay
    there anyway. I'll give JmxRemoteLifecycleListener a chance.

    Or you could write a small piece of code that lives inside your
    app and writes the monitoring data into the Nagios queue directly.
    It is a simple text-based protocol.
    I'm monitoring the server remotely, so writing-out isn't currently in
    the cards.

    - -chris


    ---------------------------------------------------------------------
    To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
    For additional commands, e-mail: users-help@tomcat.apache.org
  • Christopher Schultz at Jan 26, 2012 at 8:33 pm
    Chuck
    On 1/26/12 1:41 PM, Caldarale, Charles R wrote:
    From: Christopher Schultz
    Subject: [OT] Inspecting JMX
    Should I just give up and use JmxRemoteLifecycleListener?
    Yes. You really aren't get to get anywhere on EC2 without it.
    Okay, so I enabled the JmxRemoteLifecycleListener like this:

    <Listener
    className="org.apache.catalina.mbeans.JmxRemoteLifecycleListener"
    rmiRegistryPortPlatform="1234"
    rmiServerPortPlatform="1235"
    />

    ... and had the same problem: the client connects to the registry port
    and then tries to connect to [public IP]:1235.

    So, I tried adding useLocalPorts="true" to the above and not I'm able
    to connect -- using the same URL as before, actually:

    ./check_jmx \
    -U service:jmx:rmi:///jndi/rmi://localhost:1234/jmxrmi \
    -O java.lang:type=Memory -A HeapMemoryUsage -K used \
    -I HeapMemoryUsage -J used -vvvv -w 4248302272 -c 5498760192

    JMX OK
    HeapMemoryUsage.used=39749208{committed=59572224;init=67108864;max=59703296;used=39749208}

    So, that's good. The JmxRemoteLifecycleListener installs a special
    RMIClientSocketFactory that always binds to "localhost" instead of
    whatever the default is that the JVM uses. I'm surprised there's not a
    way to tell the JVM to do this kind of thing. I wasn't able to find
    any canonical source of information about what system properties
    affect the JMX configuration... just lots of "simple" examples of how
    things should "just work". :(

    The nice thing about having the JmxRemoteLifecycleListener installed
    is that the ports are predictable and so I can tunnel them. Now I can
    use jconsole from my computer to observe whats happening on the server.
    Should I hack the code for check_jmx to use the Attach API.
    Maybe; might be fun. (Always seemed to be a security hole to me
    :-)
    It looks simple to code... I'll take a look at it. I'm sure I'll have
    some similar kind of problem.

    Thanks,
    - -chris


    ---------------------------------------------------------------------
    To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
    For additional commands, e-mail: users-help@tomcat.apache.org
  • Pid at Jan 26, 2012 at 10:36 pm

    On 26/01/2012 20:33, Christopher Schultz wrote:
    Chuck
    On 1/26/12 1:41 PM, Caldarale, Charles R wrote:
    From: Christopher Schultz
    Subject: [OT] Inspecting JMX
    Should I just give up and use JmxRemoteLifecycleListener?
    Yes. You really aren't get to get anywhere on EC2 without it.
    Okay, so I enabled the JmxRemoteLifecycleListener like this:

    <Listener
    className="org.apache.catalina.mbeans.JmxRemoteLifecycleListener"
    rmiRegistryPortPlatform="1234"
    rmiServerPortPlatform="1235"
    />

    ... and had the same problem: the client connects to the registry port
    and then tries to connect to [public IP]:1235.

    So, I tried adding useLocalPorts="true" to the above and not I'm able
    to connect -- using the same URL as before, actually:
    In addition to the valve...

    See if you can get the unauthenticated version to work first, you will
    probably need to set the hostname, too:

    -Djava.rmi.server.hostname=www.externalhostname.com
    -Dcom.sun.management.jmxremote=true
    -Dcom.sun.management.jmxremote.ssl=false
    -Dcom.sun.management.jmxremote.authenticate=false


    p
    ./check_jmx \
    -U service:jmx:rmi:///jndi/rmi://localhost:1234/jmxrmi \
    -O java.lang:type=Memory -A HeapMemoryUsage -K used \
    -I HeapMemoryUsage -J used -vvvv -w 4248302272 -c 5498760192

    JMX OK
    HeapMemoryUsage.used=39749208{committed=59572224;init=67108864;max=59703296;used=39749208}

    So, that's good. The JmxRemoteLifecycleListener installs a special
    RMIClientSocketFactory that always binds to "localhost" instead of
    whatever the default is that the JVM uses. I'm surprised there's not a
    way to tell the JVM to do this kind of thing. I wasn't able to find
    any canonical source of information about what system properties
    affect the JMX configuration... just lots of "simple" examples of how
    things should "just work". :(

    The nice thing about having the JmxRemoteLifecycleListener installed
    is that the ports are predictable and so I can tunnel them. Now I can
    use jconsole from my computer to observe whats happening on the server.
    Should I hack the code for check_jmx to use the Attach API.
    Maybe; might be fun. (Always seemed to be a security hole to me
    :-)
    It looks simple to code... I'll take a look at it. I'm sure I'll have
    some similar kind of problem.

    Thanks,
    -chris

    ---------------------------------------------------------------------
    To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
    For additional commands, e-mail: users-help@tomcat.apache.org
    --

    [key:62590808]
  • Christopher Schultz at Jan 31, 2012 at 8:24 pm
    Pid,
    On 1/26/12 5:35 PM, Pid wrote:
    On 26/01/2012 20:33, Christopher Schultz wrote:
    Chuck
    On 1/26/12 1:41 PM, Caldarale, Charles R wrote:
    From: Christopher Schultz
    Subject: [OT]
    Inspecting JMX
    Should I just give up and use JmxRemoteLifecycleListener?
    Yes. You really aren't get to get anywhere on EC2 without it.
    Okay, so I enabled the JmxRemoteLifecycleListener like this:

    <Listener
    className="org.apache.catalina.mbeans.JmxRemoteLifecycleListener"
    rmiRegistryPortPlatform="1234"
    rmiServerPortPlatform="1235" />

    ... and had the same problem: the client connects to the registry
    port and then tries to connect to [public IP]:1235.

    So, I tried adding useLocalPorts="true" to the above and not I'm
    able to connect -- using the same URL as before, actually:
    In addition to the valve...

    See if you can get the unauthenticated version to work first, you
    will probably need to set the hostname, too:

    -Djava.rmi.server.hostname=www.externalhostname.com
    -Dcom.sun.management.jmxremote=true
    -Dcom.sun.management.jmxremote.ssl=false
    -Dcom.sun.management.jmxremote.authenticate=false
    Hmm... I'll have to look at using the hostname: that might solve the
    initial problem. The problem is that the RMI server is using the
    public hostname of the server to redirect connections, and I want it
    to use localhost because I don't intend to expose JMX services to the
    world.

    - -chris


    ---------------------------------------------------------------------
    To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
    For additional commands, e-mail: users-help@tomcat.apache.org
  • André Warnier at Jan 26, 2012 at 8:31 pm
    Chris,

    independently of anything else, you may want to have a look at jmxsh
    http://code.google.com/p/jmxsh/

    It is interactive as well as scriptable.
    (If I, who do not understand much about JMX and mbeans (or even java), managed to use it
    with Tomcat and other stand-alone java applications, it means that it is really simple).

    ---------------------------------------------------------------------
    To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
    For additional commands, e-mail: users-help@tomcat.apache.org
  • Christopher Schultz at Jan 26, 2012 at 8:35 pm
    André,
    On 1/26/12 3:30 PM, André Warnier wrote:
    independently of anything else, you may want to have a look at
    jmxsh http://code.google.com/p/jmxsh/
    Now that I have *something* working, I can experiment to see what
    other things work as well :)
    It is interactive as well as scriptable. (If I, who do not
    understand much about JMX and mbeans (or even java), managed to use
    it with Tomcat and other stand-alone java applications, it means
    that it is really simple).
    If you got lost in my original post and you managed to get JMX
    monitoring up and running, then I think you might be on to something,
    here.

    - -chris


    ---------------------------------------------------------------------
    To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
    For additional commands, e-mail: users-help@tomcat.apache.org
  • André Warnier at Jan 26, 2012 at 9:07 pm

    Christopher Schultz wrote:
    -----BEGIN PGP SIGNED MESSAGE-----
    Hash: SHA1

    André,
    On 1/26/12 3:30 PM, André Warnier wrote:
    independently of anything else, you may want to have a look at
    jmxsh http://code.google.com/p/jmxsh/
    Now that I have *something* working, I can experiment to see what
    other things work as well :)
    It is interactive as well as scriptable. (If I, who do not
    understand much about JMX and mbeans (or even java), managed to use
    it with Tomcat and other stand-alone java applications, it means
    that it is really simple).
    If you got lost in my original post and you managed to get JMX
    monitoring up and running, then I think you might be on to something,
    here.
    I needed a simple scriptable tool at some point in the past, to force Tomcat (and other
    Java programs) to do a major GC from time to time, to get rid of some dangling native
    socket left open in an inaccessible (but not yet collected) java object.
    I believe it was Chuck who pointed me to jmxsh originally.
    I only used it further by curiosity, without really understanding what I was doing.
    But I know one useful feature is that it allows one to browse interactively, to find the
    available JMX?/mbeans?.

    Example : force (the JVM running) Tomcat to do a GC :

    # gc_tomcat.jmxsh
    # force the target JVM to do a GC, via the jmxsh shell.
    # invoke as :
    # java -jar jmxsh.jar gc_tomcat.jmxsh
    #
    # In the following command, replace the port number (-p)
    # by the port which has been specified in the parameter :
    # -Dcom.sun.management.jmxremote.port=xxxxx
    # of the target Java JVM startup line
    # (and so on for -U and -P)
    jmx_connect -h 127.0.0.1 -p 8200 -U controlRole -P mypass
    jmx_invoke -m java.lang:type=Memory gc
    jmx_close

    To get the interactive shell, use the command "java -jar jmxsh.jar".

    ---------------------------------------------------------------------
    To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
    For additional commands, e-mail: users-help@tomcat.apache.org
  • Rainer Jung at Jan 28, 2012 at 1:51 pm

    On 26.01.2012 19:32, Christopher Schultz wrote:

    Now I'm trying to get similar information using a command-line tool
    that is very simple called check_jmx -- it's a plug-in for Nagios. It
    appears that this tool does not support the "attach" API and so it
    looks like I'll have to enable "remote JMX", so I've followed the
    instructions on Tomcat's monitoring page to enable remote JMX [3]:
    3. Should I just give up and use the manager app's jmxproxy? I don't
    currently deploy the manager app, and I'd like to avoid doing that
    if possible. But, it may be a slightly cleaner solution.

    4. Should I hack the code for check_jmx to use the Attach API and
    try to avoid all of this stupid port business? Getting the PID
    of the Tomcat process shouldn't be hard as long as I use
    CATALINA_PID and get the value from there.
    Note that the naive check_jmx attempt will not scale. Monitoring JVMs
    using JMX by starting a new JVM on the polling server for each poll and
    each monitored instance will soon killk your monitoring server.

    You either need to use an agent running in the target JVM and providing
    access via a simpler non-Java protocol, or you need a long running Java
    based gateway, which does the JMX communication with the target JVMs and
    gets itself queried with something simpler.

    In that sense the Tomcat Manager can act as an agent via its jmxproxy
    feature, making JMX data available for each HTTP client that can parse
    simle text output.

    Another a bit more sophisticated approach which can be well integrated
    with Nagios is Jmx4Perl as a client in combination with Jolikia as the
    agent (all Open Source).

    Of course there are many more options available.

    Regards,

    Rainer


    ---------------------------------------------------------------------
    To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
    For additional commands, e-mail: users-help@tomcat.apache.org
  • Christopher Schultz at Jan 31, 2012 at 8:29 pm
    Rainer,
    On 1/28/12 8:51 AM, Rainer Jung wrote:
    On 26.01.2012 19:32, Christopher Schultz wrote:

    Now I'm trying to get similar information using a command-line
    tool that is very simple called check_jmx -- it's a plug-in for
    Nagios. It appears that this tool does not support the "attach"
    API and so it looks like I'll have to enable "remote JMX", so
    I've followed the instructions on Tomcat's monitoring page to
    enable remote JMX [3]:
    3. Should I just give up and use the manager app's jmxproxy? I
    don't currently deploy the manager app, and I'd like to avoid
    doing that if possible. But, it may be a slightly cleaner
    solution.

    4. Should I hack the code for check_jmx to use the Attach API
    and try to avoid all of this stupid port business? Getting the
    PID of the Tomcat process shouldn't be hard as long as I use
    CATALINA_PID and get the value from there.
    Note that the naive check_jmx attempt will not scale. Monitoring
    JVMs using JMX by starting a new JVM on the polling server for each
    poll and each monitored instance will soon killk your monitoring
    server.
    Yeah, we're using check_nrpe to invoke the check_jmx processes on the
    server that is being monitored, but I expect that we will be
    monitoring half a dozen JMX variables on several VMs, and that means
    lots of JVMs being fired up just to connect and ask a simple question.
    You either need to use an agent running in the target JVM and
    providing access via a simpler non-Java protocol, or you need a
    long running Java based gateway, which does the JMX communication
    with the target JVMs and gets itself queried with something
    simpler.
    Are you suggesting a long-running process with a persistent connection
    to the JVM, so that a new JVM for each request can be avoided? Someone
    has to have done that already, right?
    In that sense the Tomcat Manager can act as an agent via its
    jmxproxy feature, making JMX data available for each HTTP client
    that can parse simle text output.
    I may consider doing that, though we aren't using the manager app for
    anything else right now. I'll see if I can use it without allowing
    access to any other services (like deploy, etc.).
    Another a bit more sophisticated approach which can be well
    integrated with Nagios is Jmx4Perl as a client in combination with
    Jolikia as the agent (all Open Source).
    I'll check that out.
    Of course there are many more options available.
    I'll document what I find on the wiki. I'm surprised that there's
    little monitoring information on the Tomcat site. :(

    - -chris


    ---------------------------------------------------------------------
    To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
    For additional commands, e-mail: users-help@tomcat.apache.org
  • Mark Eggers at Jan 31, 2012 at 9:30 pm
    Chris,

    (comments inline)
    ----- Original Message -----

    From: Christopher Schultz <chris@christopherschultz.net>
    To: Tomcat Users List <users@tomcat.apache.org>
    Cc:
    Sent: Tuesday, January 31, 2012 12:29 PM
    Subject: Re: [OT] Inspecting JMX

    -----BEGIN PGP SIGNED MESSAGE-----
    Hash: SHA1

    Rainer,
    On 1/28/12 8:51 AM, Rainer Jung wrote:
    On 26.01.2012 19:32, Christopher Schultz wrote:

    Now I'm trying to get similar information using a command-line
    tool that is very simple called check_jmx -- it's a plug-in for
    Nagios. It appears that this tool does not support the
    "attach"
    API and so it looks like I'll have to enable "remote
    JMX", so
    I've followed the instructions on Tomcat's monitoring page to
    enable remote JMX [3]:
    3. Should I just give up and use the manager app's jmxproxy? I
    don't currently deploy the manager app, and I'd like to avoid
    doing that if possible. But, it may be a slightly cleaner
    solution.

    4. Should I hack the code for check_jmx to use the Attach API
    and try to avoid all of this stupid port business? Getting the
    PID of the Tomcat process shouldn't be hard as long as I use
    CATALINA_PID and get the value from there.
    Note that the naive check_jmx attempt will not scale. Monitoring
    JVMs using JMX by starting a new JVM on the polling server for each
    poll and each monitored instance will soon killk your monitoring
    server.
    Yeah, we're using check_nrpe to invoke the check_jmx processes on the
    server that is being monitored, but I expect that we will be
    monitoring half a dozen JMX variables on several VMs, and that means
    lots of JVMs being fired up just to connect and ask a simple question.
    You either need to use an agent running in the target JVM and
    providing access via a simpler non-Java protocol, or you need a
    long running Java based gateway, which does the JMX communication
    with the target JVMs and gets itself queried with something
    simpler.
    Are you suggesting a long-running process with a persistent connection
    to the JVM, so that a new JVM for each request can be avoided? Someone
    has to have done that already, right?
    You might take a look at collectd (http://collectd.org/ ). There's a Java and JMX plugin available.

    You could run that, connect to your local Tomcat, gather statistics, and then send the information back to a central server with the network plugin. It appears that there is an option to encrypt the information as well.

    On the central server, you could equip the collectd daemon with the collectd-nagios program to inform Nagios about trouble. Unfortunately, that approach looks a little inflexible at the moment.

    One other issue is that collectd just generates information and sends it off (network, RRD, CSV). What you do with it after that is up to you. While there are examples given in the documentation about how to deal with the information (mostly focused on RRD), the actual work is left as an exercise for the implementer.

    Finally, the collectd tool is designed to run on UNIX/Linux. While this fits your clients in the cloud, it may or may not fit your server platform. There is a collectd to JMX program written in Java, but I couldn't get it to work with the later versions of collectd. I suspect that someone will have to write a collectd server that implements the bare necessities (network, RRD, CSV) if your collection / server platform is Windows.
    In that sense the Tomcat Manager can act as an agent via its
    jmxproxy feature, making JMX data available for each HTTP client
    that can parse simle text output.
    I may consider doing that, though we aren't using the manager app for
    anything else right now. I'll see if I can use it without allowing
    access to any other services (like deploy, etc.).
    Another a bit more sophisticated approach which can be well
    integrated with Nagios is Jmx4Perl as a client in combination with
    Jolikia as the agent (all Open Source).
    I'll check that out.
    Of course there are many more options available.
    I'll document what I find on the wiki. I'm surprised that there's
    little monitoring information on the Tomcat site. :(

    - -chris


    ---------------------------------------------------------------------
    To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
    For additional commands, e-mail: users-help@tomcat.apache.org
    . . . . just my two cents.
    /mde/

    ---------------------------------------------------------------------
    To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
    For additional commands, e-mail: users-help@tomcat.apache.org

Related Discussions