FAQ
hi,

I am using replicated leveldb with ActiveMQ.5.11.1. We have recently
observed an issue where when client connection removal caused broker to
shutdown. This is very weird. The client is a durable consumer who uses
transactions and calls commit after each 10 messages.

Can someone suggest what should I do here and what caused this exception ?

Logs:


[20150817 18:25:06.016 EDT (ActiveMQ BrokerService[mqbroker] Task-9)
org.apache.activemq.broker.util.LoggingBrokerPlugin#removeConnection 241
INFO] -
Removing Connection: ConnectionInfo {commandId = 1, responseRequired = true,
connectionId = ID:app1.ia55.net-49035-1439207692004-1:3, clientId =
milestoneClient, clientIp = tcp://a.b.c.d:15615, userName = null, password =
*****, brokerPath = null, brokerMasterConnector
= false, manageable = true, clientMaster = true, faultTolerant = true,
failoverReconnect = true}

[20150817 18:25:06.016 EDT (ActiveMQ BrokerService[mqbroker] Task-9)
org.apache.activemq.broker.TransactionBroker#removeConnection 325 WARN] -
ERROR R
olling back disconnected client's transactions:
org.apache.activemq.broker.SuppressReplyException: ShutdownBrokerInitiated
         at
org.apache.activemq.util.DefaultIOExceptionHandler.handle(DefaultIOExceptionHandler.java:151)
         at
org.apache.activemq.broker.BrokerService.handleIOException(BrokerService.java:2618)
         at
org.apache.activemq.leveldb.LevelDBStore.verify_running(LevelDBStore.scala:521)
         at
org.apache.activemq.leveldb.LevelDBStore.rollback(LevelDBStore.scala:566)
         at
org.apache.activemq.leveldb.LevelDBStore$LevelDBTransactionStore.rollback(LevelDBStore.scala:1124)
         at
org.apache.activemq.transaction.LocalTransaction.rollback(LocalTransaction.java:94)
         at
org.apache.activemq.broker.TransactionBroker.removeConnection(TransactionBroker.java:323)
         at
org.apache.activemq.broker.BrokerFilter.removeConnection(BrokerFilter.java:132)
         at
org.apache.activemq.broker.MutableBrokerFilter.removeConnection(MutableBrokerFilter.java:137)
         at
org.apache.activemq.broker.util.LoggingBrokerPlugin.removeConnection(LoggingBrokerPlugin.java:243)
         at
org.apache.activemq.broker.MutableBrokerFilter.removeConnection(MutableBrokerFilter.java:137)
         at
org.apache.activemq.broker.TransportConnection.processRemoveConnection(TransportConnection.java:862)
         at
org.apache.activemq.broker.TransportConnection.doStop(TransportConnection.java:1187)
         at
org.apache.activemq.broker.TransportConnection$4.run(TransportConnection.java:1117)
         at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
         at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
         at java.lang.Thread.run(Thread.java:745)
Caused by: java.io.IOException: Not running
         at
org.apache.activemq.leveldb.LevelDBStore.verify_running(LevelDBStore.scala:517)


Thanks,
Anuj



--
View this message in context: http://activemq.2283324.n4.nabble.com/Durable-client-removal-caused-broker-to-shutdown-in-replicated-leveldb-tp4701209.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.

Search Discussions

  • Khandelwalanuj at Aug 20, 2015 at 4:38 am
    Did anyone get a chance to look at this ?



    --
    View this message in context: http://activemq.2283324.n4.nabble.com/Durable-client-removal-caused-broker-to-shutdown-in-replicated-leveldb-tp4701209p4701232.html
    Sent from the ActiveMQ - User mailing list archive at Nabble.com.
  • Tim Bain at Aug 20, 2015 at 12:55 pm
    It looks like the mailing list ignored or rejected my response below. It
    also didn't accept your email with the second stack trace (which is the
    only relevant one). Weird.

    Looking at the second stack trace, it's even more clear that this error has
    nothing to do with client connection removal. LevelDB had an
    ArrayIndexOutOfBoundsException and shut down. Period, end of story,
    nothing to do with client connection removal.

    So the real question is what caused the ArrayIndexOutOfBoundsException, and
    hopefully someone who knows the LevelDB code (not me) can help look into
    that.

    Tim
    On Aug 19, 2015 7:41 AM, "Tim Bain" wrote:

    It looks like the broker shut down because LevelDB wasn't running (or
    thought it wasn't because its test of whether it was running failed). Were
    there any system-level failures that happened around the same time? In the
    past, you've had NFS trouble; did that happen again here?
    On Aug 19, 2015 1:14 AM, "khandelwalanuj" wrote:

    hi,

    I am using replicated leveldb with ActiveMQ.5.11.1. We have recently
    observed an issue where when client connection removal caused broker to
    shutdown. This is very weird. The client is a durable consumer who uses
    transactions and calls commit after each 10 messages.

    Can someone suggest what should I do here and what caused this exception ?

    Logs:


    [20150817 18:25:06.016 EDT (ActiveMQ BrokerService[mqbroker] Task-9)
    org.apache.activemq.broker.util.LoggingBrokerPlugin#removeConnection 241
    INFO] -
    Removing Connection: ConnectionInfo {commandId = 1, responseRequired =
    true,
    connectionId = ID:app1.ia55.net-49035-1439207692004-1:3, clientId =
    milestoneClient, clientIp = tcp://a.b.c.d:15615, userName = null,
    password =
    *****, brokerPath = null, brokerMasterConnector
    = false, manageable = true, clientMaster = true, faultTolerant = true,
    failoverReconnect = true}

    [20150817 18:25:06.016 EDT (ActiveMQ BrokerService[mqbroker] Task-9)
    org.apache.activemq.broker.TransactionBroker#removeConnection 325 WARN] -
    ERROR R
    olling back disconnected client's transactions:
    org.apache.activemq.broker.SuppressReplyException: ShutdownBrokerInitiated
    at

    org.apache.activemq.util.DefaultIOExceptionHandler.handle(DefaultIOExceptionHandler.java:151)
    at

    org.apache.activemq.broker.BrokerService.handleIOException(BrokerService.java:2618)
    at

    org.apache.activemq.leveldb.LevelDBStore.verify_running(LevelDBStore.scala:521)
    at
    org.apache.activemq.leveldb.LevelDBStore.rollback(LevelDBStore.scala:566)
    at

    org.apache.activemq.leveldb.LevelDBStore$LevelDBTransactionStore.rollback(LevelDBStore.scala:1124)
    at

    org.apache.activemq.transaction.LocalTransaction.rollback(LocalTransaction.java:94)
    at

    org.apache.activemq.broker.TransactionBroker.removeConnection(TransactionBroker.java:323)
    at

    org.apache.activemq.broker.BrokerFilter.removeConnection(BrokerFilter.java:132)
    at

    org.apache.activemq.broker.MutableBrokerFilter.removeConnection(MutableBrokerFilter.java:137)
    at

    org.apache.activemq.broker.util.LoggingBrokerPlugin.removeConnection(LoggingBrokerPlugin.java:243)
    at

    org.apache.activemq.broker.MutableBrokerFilter.removeConnection(MutableBrokerFilter.java:137)
    at

    org.apache.activemq.broker.TransportConnection.processRemoveConnection(TransportConnection.java:862)
    at

    org.apache.activemq.broker.TransportConnection.doStop(TransportConnection.java:1187)
    at

    org.apache.activemq.broker.TransportConnection$4.run(TransportConnection.java:1117)
    at

    java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at

    java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)
    Caused by: java.io.IOException: Not running
    at

    org.apache.activemq.leveldb.LevelDBStore.verify_running(LevelDBStore.scala:517)


    Thanks,
    Anuj



    --
    View this message in context:
    http://activemq.2283324.n4.nabble.com/Durable-client-removal-caused-broker-to-shutdown-in-replicated-leveldb-tp4701209.html
    Sent from the ActiveMQ - User mailing list archive at Nabble.com.
  • Khandelwalanuj at Aug 20, 2015 at 2:58 pm
    As we can see that before 'ArrayIndexOutOfBoundsException' we got below logs:

      20150817 18:24:42.582 EDT (ActiveMQ NIO Worker 6)
    org.apache.activemq.leveldb.util.Log$class#debug 112 DEBUG] - on rollback,
    the transaction TX:ID:app1.net-38906-1439632105449-7:13:42 does not exist

    That's why I am suspecting that could this cause broker to stop ?

    Thanks,
    Anuj



    --
    View this message in context: http://activemq.2283324.n4.nabble.com/Durable-client-removal-caused-broker-to-shutdown-in-replicated-leveldb-tp4701209p4701241.html
    Sent from the ActiveMQ - User mailing list archive at Nabble.com.
  • Tim Bain at Aug 21, 2015 at 1:28 pm
    I don't see an obvious causal link between those two, though I haven't
    looked at that code (my usual trick of going to GrepCode didn't work; maybe
    GC doesn't index Scala code?) so I can't say for sure. Unless you see a
    causal link I'm missing and want to explain it?

    Hopefully someone who knows that code will provide some insight. If not,
    you could always go track down that code and see what you can figure out.
    This list could use an active LevelDB expert, and you're the most active
    poster on the topic, so you've presumably got a very good starting point
    for becoming an expert on the subject.
    On Aug 20, 2015 8:58 AM, "khandelwalanuj" wrote:

    As we can see that before 'ArrayIndexOutOfBoundsException' we got below
    logs:

    20150817 18:24:42.582 EDT (ActiveMQ NIO Worker 6)
    org.apache.activemq.leveldb.util.Log$class#debug 112 DEBUG] - on rollback,
    the transaction TX:ID:app1.net-38906-1439632105449-7:13:42 does not exist

    That's why I am suspecting that could this cause broker to stop ?

    Thanks,
    Anuj



    --
    View this message in context:
    http://activemq.2283324.n4.nabble.com/Durable-client-removal-caused-broker-to-shutdown-in-replicated-leveldb-tp4701209p4701241.html
    Sent from the ActiveMQ - User mailing list archive at Nabble.com.

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupusers @
categoriesactivemq
postedAug 19, '15 at 7:14a
activeAug 21, '15 at 1:28p
posts5
users2
websiteactivemq.apache.org

2 users in discussion

Khandelwalanuj: 3 posts Tim Bain: 2 posts

People

Translate

site design / logo © 2022 Grokbase