FAQ
Hi, I'm testing ActiveMQ 5.10 trying to migrate from 5.6 in hopes to fix a
variety of dispatching issues I've been dealing with for a while. I'm
running the following topology to verify producer/consumer dispatching on
reconnection works as it does in 5.6. This is my topology:

producer ------> brokerA
                          / \ |
\ /
                         brokerB <----- consumers

Broker configuration (it's the same config in both brokerA and brokerB, they
point at each other):

<beans
   xmlns="http://www.springframework.org/schema/beans"
   xmlns:amq="http://activemq.apache.org/schema/core"
   xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
   xsi:schemaLocation="http://www.springframework.org/schema/beans
http://www.springframework.org/schema/beans/spring-beans-2.0.xsd
   http://activemq.apache.org/schema/core
" rel="nofollow">http://activemq.apache.org/schema/core/activemq-core.xsd">


     <bean
class="org.springframework.beans.factory.config.PropertyPlaceholderConfigurer"/>

     <broker xmlns="http://activemq.apache.org/schema/core"
             brokerName="localhost-b2"
             persistent="false"
             useJmx="true">


         <destinationPolicy>
             <policyMap>
                 <policyEntries>

                     <policyEntry queue="test.set1..>">
                         <deadLetterStrategy>

                             <individualDeadLetterStrategy
queuePrefix="DLQ.set1." useQueueForQueueMessages="true" />
                         </deadLetterStrategy>
                     </policyEntry>
                     <policyEntry queue="test.set2..>">
                         <deadLetterStrategy>

                             <individualDeadLetterStrategy
queuePrefix="DLQ.set2." useQueueForQueueMessages="true" />
                         </deadLetterStrategy>
                     </policyEntry>
                 </policyEntries>
             </policyMap>
         </destinationPolicy>


         <managementContext>
             <managementContext createConnector="true" connectorPort="1299"/>
         </managementContext>


         <networkConnectors>

             <networkConnector name="failover-b2"

uri="static:failover:(tcp://ec2-50-17-148-126.compute-1.amazonaws.com:61618)?randomize=false&amp;maxReconnectAttempts=-1"
                                     duplex="true"
                         networkTTL="4"
                                     alwaysSyncSend="false"
                                     conduitSubscriptions="false"/>
         </networkConnectors>



         <persistenceAdapter>
             <memoryPersistenceAdapter/>
         </persistenceAdapter>

         <systemUsage>
             <systemUsage>
                 <memoryUsage>
                     <memoryUsage limit="4300 mb"/>
                 </memoryUsage>
                 <storeUsage>
                     <storeUsage limit="100 gb" name="store"/>
                 </storeUsage>
                 <tempUsage>
                     <tempUsage limit="10 gb"/>
                 </tempUsage>
             </systemUsage>
         </systemUsage>

         <transportConnectors>

             <transportConnector name="openwire-in"
uri="tcp://0.0.0.0:61626"/>

             <transportConnector name="openwire-out"
uri="tcp://0.0.0.0:61627"/>

             <transportConnector name="openwire-networkConnector"
uri="tcp://0.0.0.0:61628"/>
         </transportConnectors>


     </broker>

</beans>


I'm using the swiss army project, I built it locally with no source code
changes. I run it with these parameters:

producer:
ant producer -Durl=tcp://ec2-54-234-168-125.compute-1.amazonaws.com:61626
-Dtopic=false -Dsub=my.test.dest -Dmax=10000 -DparallelThreads=1
-DsleepTime=10 -DmessageSize=400

consumer:
  ant consumer -Durl=tcp://ec2-50-17-148-126.compute-1.amazonaws.com:61617
-Dtopic=false -Dsubje=my.test.dest -Dmax=500000 -DparallelThreads=2
-DsleepTime=10

Brokers are always started up 1st. I see connector started messages on both
brokers connectors and network connection.

Test Case 1:
If consumers are started up 1st and then producers everything works ok. I
can stop the producer and start it up again and everything is ok. If I stop
the consumer threads and bring them back up I see them consume one or two
messages per thread and then they don't get anything else.
On the brokers I only see the EOFException for starting/stopping of
consumers and producers I manually induce.

Test Case 2:
I start the producer thread and then the consumer threads. The producer
enqueues normally but the consumer threads only receive 0 or 1 message. I
try stopping the consumer process and start it again but it never gets
anything.

When I stop my topology I can always see the broker producers connect to is
never able to shutdown gracefully. I end up always killing it manually. I
see the following message appear in the log several times:

The connection to 'vm://localhost-b2#0' is taking a long time to shutdown.

I always end up issuing "kill -9".

I never see anything in the logs that would signal bigger issues at all.

I this a known issue? Do you guys see any obvious explanation in the
configuration of the producer/consumes or brokers (I certainly don't)?

Help would be greatly appreciated.



--
View this message in context: http://activemq.2283324.n4.nabble.com/JMS-to-JMS-bridge-reconnection-dispatching-not-working-in-simple-conditions-tp4684887.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.

Search Discussions

  • Pollotek at Aug 23, 2014 at 4:41 am
    I ran the same topology with everything in the same server, both brokers,
    producer and multi-threaded consumer. I can reproduce this EVERY SINGLE
    TIME.

    I tried with 5.9.1 and see exactly the same behavior. I tried with 5.6
    (which is the version I run in production) where I know this basic topology
    works and confirmed that it does in fact work. I use exactly the same
    configuration on 5.10, 5.9.1 and 5.6 since it seems quite simple.

    I think my topology is pretty basic for a Network of Brokers in a production
    environment, I'm pretty surprised it doesn't work.

    Am I missing something here? Is there a flag I missed in my configuration?
    Is there a migration guide that I didn't see?

    I can supply test cases for this, logs, heap dumps or thread dumps if it
    helps getting this fixed.



    --
    View this message in context: http://activemq.2283324.n4.nabble.com/JMS-to-JMS-bridge-reconnection-dispatching-not-working-in-simple-conditions-tp4684887p4684969.html
    Sent from the ActiveMQ - User mailing list archive at Nabble.com.
  • Artnaseef at Aug 29, 2014 at 2:29 am
    The issue of "connection takes a long time to shutdown" has been around a
    long time. I've seen at least one fix for it recently, but it may still
    have other causes.

    If you can track down the cause and (even better) provide a patch, it would
    be greatly appreciated.

    Once reproduced, the next step that I would try is grabbing a stack trace.

    Is you don't want to do that, no worries. Other than leading to the
    annoyance of needing a kill -9, is there any other impact?



    --
    View this message in context: http://activemq.2283324.n4.nabble.com/JMS-to-JMS-bridge-reconnection-dispatching-not-working-in-simple-conditions-tp4684887p4685122.html
    Sent from the ActiveMQ - User mailing list archive at Nabble.com.
  • Pollotek at Aug 29, 2014 at 6:44 pm
    Actually it has caused problems with my integration test environments
    because is completely unexpected and Spring doesn't now about kill -9, it
    only knows about calling the shutdown hooks.

    I ran JConsole and immediately detected this deadlock:

    Name: ActiveMQ Transport: tcp://localhost/127.0.0.1:61618@55559
    State: WAITING on
    java.util.concurrent.locks.ReentrantLock$NonfairSync@4b2d19f2 owned by:
    ActiveMQ BrokerService[localhost-b2] Task-3
    Total blocked: 1 Total waited: 3

    Stack trace:
    sun.misc.Unsafe.park(Native Method)
    java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
    java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:834)
    java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:867)
    java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1197)
    java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:214)
    java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:290)
    org.apache.activemq.transport.MutexTransport.oneway(MutexTransport.java:66)
    org.apache.activemq.transport.ResponseCorrelator.oneway(ResponseCorrelator.java:60)
    org.apache.activemq.broker.TransportConnection.dispatch(TransportConnection.java:1370)
    org.apache.activemq.broker.TransportConnection.processDispatch(TransportConnection.java:889)
    org.apache.activemq.broker.TransportConnection.dispatchSync(TransportConnection.java:849)
    org.apache.activemq.broker.TransportConnection$1.onCommand(TransportConnection.java:150)
    org.apache.activemq.transport.ResponseCorrelator.onCommand(ResponseCorrelator.java:116)
    org.apache.activemq.transport.MutexTransport.onCommand(MutexTransport.java:50)
    org.apache.activemq.transport.vm.VMTransport.doDispatch(VMTransport.java:138)
    org.apache.activemq.transport.vm.VMTransport.dispatch(VMTransport.java:130)
        - locked java.util.concurrent.atomic.AtomicBoolean@c7d2c2b
    org.apache.activemq.transport.vm.VMTransport.oneway(VMTransport.java:107)
    org.apache.activemq.transport.MutexTransport.oneway(MutexTransport.java:68)
    org.apache.activemq.transport.ResponseCorrelator.asyncRequest(ResponseCorrelator.java:81)
    org.apache.activemq.transport.ResponseCorrelator.request(ResponseCorrelator.java:86)
    org.apache.activemq.network.DemandForwardingBridgeSupport.addSubscription(DemandForwardingBridgeSupport.java:905)
    org.apache.activemq.network.DemandForwardingBridgeSupport.addConsumerInfo(DemandForwardingBridgeSupport.java:1178)
    org.apache.activemq.network.DemandForwardingBridgeSupport.serviceRemoteConsumerAdvisory(DemandForwardingBridgeSupport.java:763)
        - locked java.net.URI@10efd3fa
    org.apache.activemq.network.DemandForwardingBridgeSupport.serviceRemoteCommand(DemandForwardingBridgeSupport.java:614)
    org.apache.activemq.network.DemandForwardingBridgeSupport$3.onCommand(DemandForwardingBridgeSupport.java:224)
    org.apache.activemq.transport.ResponseCorrelator.onCommand(ResponseCorrelator.java:116)
    org.apache.activemq.transport.MutexTransport.onCommand(MutexTransport.java:50)
    org.apache.activemq.transport.failover.FailoverTransport$3.onCommand(FailoverTransport.java:208)
    org.apache.activemq.transport.WireFormatNegotiator.onCommand(WireFormatNegotiator.java:113)
    org.apache.activemq.transport.AbstractInactivityMonitor.onCommand(AbstractInactivityMonitor.java:270)
    org.apache.activemq.transport.TransportSupport.doConsume(TransportSupport.java:83)
    org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:214)
    org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:196)
    java.lang.Thread.run(Thread.java:745)


    Name: ActiveMQ BrokerService[localhost-b2] Task-3
    State: WAITING on
    java.util.concurrent.locks.ReentrantLock$NonfairSync@4d6775bd owned by:
    ActiveMQ Transport: tcp://localhost/127.0.0.1:61618@55559
    Total blocked: 0 Total waited: 127

    Stack trace:
    sun.misc.Unsafe.park(Native Method)
    java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
    java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:834)
    java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:867)
    java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1197)
    java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:214)
    java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:290)
    org.apache.activemq.transport.MutexTransport.oneway(MutexTransport.java:66)
    org.apache.activemq.transport.ResponseCorrelator.oneway(ResponseCorrelator.java:60)
    org.apache.activemq.network.DemandForwardingBridgeSupport.serviceLocalCommand(DemandForwardingBridgeSupport.java:1023)
    org.apache.activemq.network.DemandForwardingBridgeSupport$2.onCommand(DemandForwardingBridgeSupport.java:206)
    org.apache.activemq.transport.ResponseCorrelator.onCommand(ResponseCorrelator.java:116)
    org.apache.activemq.transport.MutexTransport.onCommand(MutexTransport.java:50)
    org.apache.activemq.transport.vm.VMTransport.doDispatch(VMTransport.java:138)
    org.apache.activemq.transport.vm.VMTransport.dispatch(VMTransport.java:130)
        - locked java.util.concurrent.atomic.AtomicBoolean@14dc51c8
    org.apache.activemq.transport.vm.VMTransport.oneway(VMTransport.java:107)
    org.apache.activemq.transport.MutexTransport.oneway(MutexTransport.java:68)
    org.apache.activemq.transport.ResponseCorrelator.oneway(ResponseCorrelator.java:60)
    org.apache.activemq.broker.TransportConnection.dispatch(TransportConnection.java:1370)
    org.apache.activemq.broker.TransportConnection.processDispatch(TransportConnection.java:889)
    org.apache.activemq.broker.TransportConnection.iterate(TransportConnection.java:935)
    org.apache.activemq.thread.PooledTaskRunner.runTask(PooledTaskRunner.java:133)
    org.apache.activemq.thread.PooledTaskRunner$1.run(PooledTaskRunner.java:48)
    java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    java.lang.Thread.run(Thread.java:745)




    On Fri, Aug 22, 2014 at 9:41 PM, pollotek [via ActiveMQ] wrote:

    I ran the same topology with everything in the same server, both brokers,
    producer and multi-threaded consumer. I can reproduce this EVERY SINGLE
    TIME.

    I tried with 5.9.1 and see exactly the same behavior. I tried with 5.6
    (which is the version I run in production) where I know this basic topology
    works and confirmed that it does in fact work. I use exactly the same
    configuration on 5.10, 5.9.1 and 5.6 since it seems quite simple.

    I think my topology is pretty basic for a Network of Brokers in a
    production environment, I'm pretty surprised it doesn't work.

    Am I missing something here? Is there a flag I missed in my configuration?
    Is there a migration guide that I didn't see?

    I can supply test cases for this, logs, heap dumps or thread dumps if it
    helps getting this fixed.

    ------------------------------
    If you reply to this email, your message will be added to the discussion
    below:

    http://activemq.2283324.n4.nabble.com/JMS-to-JMS-bridge-reconnection-dispatching-not-working-in-simple-conditions-tp4684887p4684969.html
    To unsubscribe from JMS to JMS bridge reconnection dispatching not
    working in simple conditions, click here
    <http://activemq.2283324.n4.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=4684887&code=Y2xhdWRpby5zYW50YW5hQGdtYWlsLmNvbXw0Njg0ODg3fC0xNTA0MDM5ODky>
    .
    NAML
    <http://activemq.2283324.n4.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>



    --
    View this message in context: http://activemq.2283324.n4.nabble.com/JMS-to-JMS-bridge-reconnection-dispatching-not-working-in-simple-conditions-tp4684887p4685158.html
    Sent from the ActiveMQ - User mailing list archive at Nabble.com.
  • Artnaseef at Aug 31, 2014 at 3:19 am
    Good catch. Something looks very wrong with this deadlock picture. Both
    threads are blocking on this call:


    org.apache.activemq.transport.MutexTransport.oneway(MutexTransport.java:66)

    Which is this instruction:
         writeLock.lock();

    Which is this method call:
         java.util.concurrent.locks.ReentrantLock.lock();

    Either one or the other should hold the lock if it's truly a deadlock, which
    leads to the question of why they both block attempting to lock it. Can you
    upload the full stacktrace? If you need a place, it would be appropriate to
    create a ticket and attach it there.

    Further, since "ActiveMQ Transport: tcp://localhost/127.0.0.1:61618@55559"
    calls MutexTransport.oneway() twice in the same stack trace, it must be the
    thread already holding the lock - so, why would it block waiting to lock it
    again? The ReentrantLock allows multiple, recursive locks by the same
    thread.




    --
    View this message in context: http://activemq.2283324.n4.nabble.com/JMS-to-JMS-bridge-reconnection-dispatching-not-working-in-simple-conditions-tp4684887p4685180.html
    Sent from the ActiveMQ - User mailing list archive at Nabble.com.
  • Artnaseef at Aug 31, 2014 at 3:26 am
    Hmm, it looks like there are two reentrant locks.

    Are there 2 brokers running in the same JVM connecting over a network
    connector with the VM transport?



    --
    View this message in context: http://activemq.2283324.n4.nabble.com/JMS-to-JMS-bridge-reconnection-dispatching-not-working-in-simple-conditions-tp4684887p4685184.html
    Sent from the ActiveMQ - User mailing list archive at Nabble.com.
  • Pollotek at Sep 2, 2014 at 8:14 pm
    I filed https://issues.apache.org/jira/browse/AMQ-5342 with the configuration
    of the brokers, consumer, producer and full thread dump.

    The two brokers are running in their own JVMs connecting to each other over
    tcp open wire. Internally the brokers create vm transports to bridge among
    their connectors.



    --
    View this message in context: http://activemq.2283324.n4.nabble.com/JMS-to-JMS-bridge-reconnection-dispatching-not-working-in-simple-conditions-tp4684887p4685246.html
    Sent from the ActiveMQ - User mailing list archive at Nabble.com.

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupusers @
categoriesactivemq
postedAug 21, '14 at 5:57p
activeSep 2, '14 at 8:14p
posts7
users2
websiteactivemq.apache.org

2 users in discussion

Pollotek: 4 posts Artnaseef: 3 posts

People

Translate

site design / logo © 2022 Grokbase