Grokbase Groups Cayenne user May 2010
FAQ
This is a follow-up on the "out of connections" error I am periodically receiving from the production MySQL DBMS.

If you will recall, where we left it was that the evidence appears to support the interim conclusion that it is not Cayenne and that it may be the JDBC Driver or even MySQL.

Actions Taken (based on previous input):
I upgraded the connector to "mysql-connector-java-5.1.12" and the MySQL server was reluctantly upgraded by the webhost tech-person to "MySQL/5.0.88" (it appears they do not like MySQL 5.1 for some reason). I also perused my code for any "open" connections (which I am not sure that I have any control over since all DBMS connections are managed by Cayenne).

Results:
It appears that the problem continues. It even occurs with *very* light usage. There does not seem to be any obvious trigger. This causes sever symptoms of a request for a webpage taking 30-60 sec to load. (It appears that the 10 used-connections somehow magically allow the user to access after this 30-60 sec delay. ????????????)

Question:
I am pretty good at tracking down bugs, but this one is really difficult to devise a methodical plan to solve. My only idea right now is to monitor the Cayenne managed connection pool (with the hopes that I can monitor open connections and associate these connections with the cause). I have not been able to figure out how to do this via the Cayenne API. I am convinced that monitoring this connection pool will provide insight into a problem.

Therefore: can you provide some example code of how to monitor open (and closed) connections in the Cayenne managed connection pool (so that I can display the result in a JSP)?

Thanks,
Joe

Search Discussions

  • Andrus Adamchik at May 12, 2010 at 2:45 pm
    Don't remember if you went through that already, but the easiest thing
    you can do to understand what's going on is to take a thread dump of
    the running (stuck) application.

    There are other debugging options available (such as subclassing
    DriverDataSourceFactory to log when connections are opened and
    closed), but try the thread dump first.

    Andrus

    On May 12, 2010, at 5:39 PM, Joe Baldwin wrote:

    This is a follow-up on the "out of connections" error I am
    periodically receiving from the production MySQL DBMS.

    If you will recall, where we left it was that the evidence appears
    to support the interim conclusion that it is not Cayenne and that it
    may be the JDBC Driver or even MySQL.

    Actions Taken (based on previous input):
    I upgraded the connector to "mysql-connector-java-5.1.12" and the
    MySQL server was reluctantly upgraded by the webhost tech-person to
    "MySQL/5.0.88" (it appears they do not like MySQL 5.1 for some
    reason). I also perused my code for any "open" connections (which I
    am not sure that I have any control over since all DBMS connections
    are managed by Cayenne).

    Results:
    It appears that the problem continues. It even occurs with *very*
    light usage. There does not seem to be any obvious trigger. This
    causes sever symptoms of a request for a webpage taking 30-60 sec to
    load. (It appears that the 10 used-connections somehow magically
    allow the user to access after this 30-60 sec delay. ????????????)

    Question:
    I am pretty good at tracking down bugs, but this one is really
    difficult to devise a methodical plan to solve. My only idea right
    now is to monitor the Cayenne managed connection pool (with the
    hopes that I can monitor open connections and associate these
    connections with the cause). I have not been able to figure out how
    to do this via the Cayenne API. I am convinced that monitoring this
    connection pool will provide insight into a problem.

    Therefore: can you provide some example code of how to monitor open
    (and closed) connections in the Cayenne managed connection pool (so
    that I can display the result in a JSP)?

    Thanks,
    Joe
  • Joe Baldwin at May 12, 2010 at 3:18 pm
    Andrus,

    I am not sure I follow you.

    While I am not an expert on stack traces, the "out of connections" exception was discovered by perusing the Catalina logs for errors near the time that the loading-delays occurred. This generated a pretty detailed stack trace of its own, (which revealed nothing more than MySQL was reporting that there were no more open connections).

    1. Will this thread dump report anything that the MySQL exception does not?
    2. DriverDataSourceFactory - I was actually looking for something that was more "data-pull" than "data-push", i.e. I would like to query the connection pool class to return information at the moment it is requested.

    Thanks,
    Joe


    On May 12, 2010, at 10:45 AM, Andrus Adamchik wrote:

    Don't remember if you went through that already, but the easiest thing you can do to understand what's going on is to take a thread dump of the running (stuck) application.

    There are other debugging options available (such as subclassing DriverDataSourceFactory to log when connections are opened and closed), but try the thread dump first.

    Andrus

    On May 12, 2010, at 5:39 PM, Joe Baldwin wrote:

    This is a follow-up on the "out of connections" error I am periodically receiving from the production MySQL DBMS.

    If you will recall, where we left it was that the evidence appears to support the interim conclusion that it is not Cayenne and that it may be the JDBC Driver or even MySQL.

    Actions Taken (based on previous input):
    I upgraded the connector to "mysql-connector-java-5.1.12" and the MySQL server was reluctantly upgraded by the webhost tech-person to "MySQL/5.0.88" (it appears they do not like MySQL 5.1 for some reason). I also perused my code for any "open" connections (which I am not sure that I have any control over since all DBMS connections are managed by Cayenne).

    Results:
    It appears that the problem continues. It even occurs with *very* light usage. There does not seem to be any obvious trigger. This causes sever symptoms of a request for a webpage taking 30-60 sec to load. (It appears that the 10 used-connections somehow magically allow the user to access after this 30-60 sec delay. ????????????)

    Question:
    I am pretty good at tracking down bugs, but this one is really difficult to devise a methodical plan to solve. My only idea right now is to monitor the Cayenne managed connection pool (with the hopes that I can monitor open connections and associate these connections with the cause). I have not been able to figure out how to do this via the Cayenne API. I am convinced that monitoring this connection pool will provide insight into a problem.

    Therefore: can you provide some example code of how to monitor open (and closed) connections in the Cayenne managed connection pool (so that I can display the result in a JSP)?

    Thanks,
    Joe
  • Andrus Adamchik at May 12, 2010 at 3:20 pm

    On May 12, 2010, at 6:17 PM, Joe Baldwin wrote:

    1. Will this thread dump report anything that the MySQL exception
    does not?
    Yes, it will report who is holding to all those connections.

    Andrus
  • Joe Baldwin at May 12, 2010 at 3:29 pm
    OK, I will implement this. Do I need to make the query from the same thread/session or will I get the same results from any session?

    On May 12, 2010, at 11:19 AM, Andrus Adamchik wrote:

    On May 12, 2010, at 6:17 PM, Joe Baldwin wrote:

    1. Will this thread dump report anything that the MySQL exception does not?
    Yes, it will report who is holding to all those connections.

    Andrus
  • Andrus Adamchik at May 12, 2010 at 3:34 pm
    I think we are talking about different things. I am referring to this
    (sorry the URL with lots of ads, just something i googled quickly) :

    http://www.crazysquirrel.com/computing/java/basics/java-thread-dump.jspx

    Andrus
    On May 12, 2010, at 6:28 PM, Joe Baldwin wrote:

    OK, I will implement this. Do I need to make the query from the
    same thread/session or will I get the same results from any session?

    On May 12, 2010, at 11:19 AM, Andrus Adamchik wrote:

    On May 12, 2010, at 6:17 PM, Joe Baldwin wrote:

    1. Will this thread dump report anything that the MySQL exception
    does not?
    Yes, it will report who is holding to all those connections.

    Andrus
  • Joe Baldwin at May 12, 2010 at 3:47 pm
    Andrus,

    I thought you were referring to
    Thread.dumpStack()

    I don't believe I can execute the steps outlined in your reference (1. the problem is not reproducible on my development machine and 2. the webhost prevents command line access like this on the Linux server.)

    Any other ideas? :)

    Joe


    On May 12, 2010, at 11:33 AM, Andrus Adamchik wrote:

    I think we are talking about different things. I am referring to this (sorry the URL with lots of ads, just something i googled quickly) :

    http://www.crazysquirrel.com/computing/java/basics/java-thread-dump.jspx

    Andrus
    On May 12, 2010, at 6:28 PM, Joe Baldwin wrote:

    OK, I will implement this. Do I need to make the query from the same thread/session or will I get the same results from any session?

    On May 12, 2010, at 11:19 AM, Andrus Adamchik wrote:

    On May 12, 2010, at 6:17 PM, Joe Baldwin wrote:

    1. Will this thread dump report anything that the MySQL exception does not?
    Yes, it will report who is holding to all those connections.

    Andrus
  • Aristedes Maniatis at May 12, 2010 at 4:17 pm

    On 13/05/10 1:46 AM, Joe Baldwin wrote:
    I don't believe I can execute the steps outlined in your reference (1. the problem is not reproducible on my development machine and 2. the webhost prevents command line access like this on the Linux server.)

    Any other ideas?:)
    Attach a profiler like YourKit to the running application: that also lets you get a thread dump at any time. And Andrus mentioned subclassing DriverDataSourceFactory earlier.

    Ari

    --
    -------------------------->
    Aristedes Maniatis
    GPG fingerprint CBFB 84B4 738D 4E87 5E5C 5EFA EF6A 7D2E 3E49 102A
  • Joe Baldwin at May 12, 2010 at 4:31 pm
    Ari,

    My original question was how can I query the Cayenne Connection Pool Manger to find out how many connections are currently open and how many are currently closed?

    Is there a way to do this via the Cayenne API? (If there is I have not been able to determine how to access the PoolManager from the DataContext, or whatever means is available.)

    I would like to start with the standard API first if possible.

    Joe



    On May 12, 2010, at 12:16 PM, Aristedes Maniatis wrote:
    On 13/05/10 1:46 AM, Joe Baldwin wrote:
    I don't believe I can execute the steps outlined in your reference (1. the problem is not reproducible on my development machine and 2. the webhost prevents command line access like this on the Linux server.)

    Any other ideas?:)
    Attach a profiler like YourKit to the running application: that also lets you get a thread dump at any time. And Andrus mentioned subclassing DriverDataSourceFactory earlier.

    Ari

    --
    -------------------------->
    Aristedes Maniatis
    GPG fingerprint CBFB 84B4 738D 4E87 5E5C 5EFA EF6A 7D2E 3E49 102A
  • Andrus Adamchik at May 13, 2010 at 6:20 am
    Currently Cayenne has no built in monitoring API. We'll likely be
    building one at some point, based on JMX. But you can use JMX Java
    tools right away. E.g. to take a peek at thread dump remotely:

    http://java.sun.com/developer/technicalArticles/J2SE/jconsole.html

    If that also doesn't work for you, I suggest overriding
    DriverDataSourceFactory and adding logging.

    Andrus
    On May 12, 2010, at 6:46 PM, Joe Baldwin wrote:

    Andrus,

    I thought you were referring to
    Thread.dumpStack()

    I don't believe I can execute the steps outlined in your reference
    (1. the problem is not reproducible on my development machine and 2.
    the webhost prevents command line access like this on the Linux
    server.)

    Any other ideas? :)

    Joe


    On May 12, 2010, at 11:33 AM, Andrus Adamchik wrote:

    I think we are talking about different things. I am referring to
    this (sorry the URL with lots of ads, just something i googled
    quickly) :

    http://www.crazysquirrel.com/computing/java/basics/java-thread-dump.jspx

    Andrus
    On May 12, 2010, at 6:28 PM, Joe Baldwin wrote:

    OK, I will implement this. Do I need to make the query from the
    same thread/session or will I get the same results from any session?

    On May 12, 2010, at 11:19 AM, Andrus Adamchik wrote:

    On May 12, 2010, at 6:17 PM, Joe Baldwin wrote:

    1. Will this thread dump report anything that the MySQL
    exception does not?
    Yes, it will report who is holding to all those connections.

    Andrus
  • Joe Baldwin at May 14, 2010 at 3:35 pm
    Andrus,

    As you suggested, the webhost tech-person sent me two stack traces (attached below) that were executed during delay-events (that I mentioned previously). Those are included below. (These are very difficult to get since the problem is intermittent and lasts for only a couple of minutes at a time, then disappears.

    I am not familiar enough with these traces to discern any problems. (The webhost tech believes there might be some important data at the end of the second file.) Do you see anything that looks out of the ordinary?

    Thanks
    Joe


    On May 12, 2010, at 11:19 AM, Andrus Adamchik wrote:

    On May 12, 2010, at 6:17 PM, Joe Baldwin wrote:

    1. Will this thread dump report anything that the MySQL exception does not?
    Yes, it will report who is holding to all those connections.

    Andrus

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupuser @
categoriescayenne
postedMay 12, '10 at 2:40p
activeMay 14, '10 at 3:35p
posts11
users3
websitecayenne.apache.org

People

Translate

site design / logo © 2022 Grokbase