[ActiveMQ-users] Creating a Network of Replicated LevelDB broker clusters

My understanding has been that, for replicated LevelDB, you need
a set of three brokers, one master two slave, in order to have proper
redundancy. You can then network those sets of brokers together
using networkConnectors.

That's what we're planning on deploying. It's what I've been testing
for our setup.

Yes, I ran into http://issues.apache.org/jira/browse/AMQ-5082
fairly early. I've not seen any response to my patch, so while I've
got a patched version for myself it's not clear to me whether or
not the fix will be accepted by the developers.

Jim

Hi Jim,

How big is your network? Do you see any significant degradation with a network of replicated levelDB brokers? Please see my comment below in red on my original post on the forum.

Thanks.

Kevin

From: "James A. Robinson-2 [via ActiveMQ]" <ml-node+[email protected]
Date: Saturday, March 28, 2015 at 7:04 AM
To: SNEI <[email protected]
Subject: Re: Creating a Network of Replicated LevelDB broker clusters
My understanding has been that, for replicated LevelDB, you need
a set of three brokers, one master two slave, in order to have proper
redundancy. You can then network those sets of brokers together
using networkConnectors.

That's what we're planning on deploying. It's what I've been testing
for our setup.

[ksu] How are you planning on discover the brokers? Via multi-cast auto-discovery, or by specifying URLs statically? The later gets harder to maintain as the the fabric’s membership grows. However, I am not sure if there is any timing issue associated with the auto-discovery of masters, and the auto-discvoery of brokers for the network.


Yes, I ran into http://issues.apache.org/jira/browse/AMQ-5082
fairly early. I've not seen any response to my patch, so while I've
got a patched version for myself it's not clear to me whether or
not the fix will be accepted by the developers.

[ksu] Could you share the patch? Which version of ActiveMQ are you using?


Jim


________________________________
If you reply to this email, your message will be added to the discussion below:
http://activemq.2283324.n4.nabble.com/Creating-a-Network-of-Replicated-LevelDB-broker-clusters-tp4693979p4694015.html
To unsubscribe from Creating a Network of Replicated LevelDB broker clusters, click here<http://activemq.2283324.n4.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=4693979&code=a2V2aW4uc3VAYW0uc29ueS5jb218NDY5Mzk3OXw5MzA0MDE1ODU=>.
NAML<http://activemq.2283324.n4.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>

I haven't started stress testing the configuration yet, the current plan
is to have network two clusters of three. So six machines in all, only
two active as the master at a time.
I don't see any reason why auto-discovery wouldn't work
if you wanted to go that route. From the rest of the codes
perspective there's one active broker per cluster, so I would
think it would work just like networking stand-alone brokers.

I'm not using auto-discovery myself, I'm planning on following
naming conventions and having my cfengine system maintain
the configuration, building a complete graph of the network.

I've no idea how well, or poorly, that will work. You can see
some of my configuration details by looking at a previous email
I sent to the list:

http://mail-archives.apache.org/mod_mbox/activemq-users/201503.mbox/%3CCAPd0[email protected]%3E
The patch is listed in the ticket. One of the maintainers did
respond to the ticket, indicating he was going to look it over
and incorporate it if it passed muster.

Jim

With static configuration, are you using the ‘masterslave’ transport prefix to specify the brokers in each of the leveDB replication cluster? I am getting errors using that. If I just use the ‘static’ transport prefix, it works, but I do get exceptions in the brokers.
I am using the latest distribution (5.11-1) running on Mac, and I am seeing brokers been shutdown by itself with the replication cluster just sitting idle, doing nothing. This is with a small cluster of 3 brokers talking to a zookeeper running on the same box.

Do you have a sample activemq.xml configuration file you could share for your 3 x 3 cluster? I am trying to do the same, with everything running on my Mac initially.

Thanks.


Kevin


From: "James A. Robinson-2 [via ActiveMQ]" <ml-node+[email protected]
Date: Monday, March 30, 2015 at 2:43 PM
To: SNEI <[email protected]
Subject: Re: Creating a Network of Replicated LevelDB broker clusters
I haven't started stress testing the configuration yet, the current plan
is to have network two clusters of three. So six machines in all, only
two active as the master at a time.
I don't see any reason why auto-discovery wouldn't work
if you wanted to go that route. From the rest of the codes
perspective there's one active broker per cluster, so I would
think it would work just like networking stand-alone brokers.

I'm not using auto-discovery myself, I'm planning on following
naming conventions and having my cfengine system maintain
the configuration, building a complete graph of the network.

I've no idea how well, or poorly, that will work. You can see
some of my configuration details by looking at a previous email
I sent to the list:

http://mail-archives.apache.org/mod_mbox/activemq-users/201503.mbox/%3CCAPd04b7106shHXUR+1VUddBmPUH+=mp414xOxnNpEuOuKwfo5g@...%3E[ksu] Could you share the patch? Which version of ActiveMQ are you using?

The patch is listed in the ticket. One of the maintainers did
respond to the ticket, indicating he was going to look it over
and incorporate it if it passed muster.

Jim


________________________________
If you reply to this email, your message will be added to the discussion below:
http://activemq.2283324.n4.nabble.com/Creating-a-Network-of-Replicated-LevelDB-broker-clusters-tp4693979p4694084.html
To unsubscribe from Creating a Network of Replicated LevelDB broker clusters, click here<http://activemq.2283324.n4.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=4693979&code=a2V2aW4uc3VAYW0uc29ueS5jb218NDY5Mzk3OXw5MzA0MDE1ODU=>.
NAML<http://activemq.2283324.n4.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>

your 3 x 3 cluster? I am trying to do the same, with everything running
on my Mac initially.

On the amq-prod-1 cluster my network connector url is

masterslave:(tcp://amq-prod-2a:61616,tcp://amq-prod-2b:61616,tcp://amq-prod-2c:61616)

On the amq-prod-2 cluster my network connector url is:

masterslave:(tcp://amq-prod-1a:61616,tcp://amq-prod-1b:61616,tcp://amq-prod-1c:61616)

The following are the messages I get in my log:

2015-03-30 17:38:00,236 [ActiveMQ Task-1] INFO DiscoveryNetworkConnector
    - Establishing network connection from
vm://amq-prod-1?async=false&network=true to
failover:(tcp://amq-prod-2a:61616,tcp://amq-prod-2b:61616,tcp://amq-prod-2c:61616)?randomize=false&maxReconnectAttempts=0
2015-03-30 17:38:00,450 [ActiveMQ Task-1] INFO FailoverTransport
    - Successfully connected to tcp://amq-prod-2b:61616
2015-03-30 17:38:00,767 [//amq-prod-1#42] INFO
  DemandForwardingBridgeSupport - Network connection between
vm://amq-prod-1#42 and tcp://amq-prod-2b:61616 (amq-prod-2) has been
established.

Here's a full example of the config:

<?xml version="1.0"?>
<beans xmlns="http://www.springframework.org/schema/beans"
     xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
     xsi:schemaLocation="
http://www.springframework.org/schema/beans
http://www.springframework.org/schema/beans/spring-beans.xsd
http://activemq.apache.org/schema/core
" rel="nofollow">http://activemq.apache.org/schema/core/activemq-core.xsd">

     <!-- Allow the use of system properties as variables in this
configuration file -->
     <bean
class="org.springframework.beans.factory.config.PropertyPlaceholderConfigurer">
         <property name="locations">
             <value>file:${activemq.conf}/credentials.properties</value>
         </property>
     </bean>

     <broker
xmlns="http://activemq.apache.org/schema/core"
useJmx="true"
brokerName="amq-prod-1">

<managementContext>
     <!-- http://activemq.apache.org/jmx.html -->
             <!-- Do not use the legacy JMX Connector. -->
     <managementContext createConnector="false" />
</managementContext>

<persistenceAdapter>
     <!-- http://activemq.apache.org/replicated-leveldb-store.html -->
     <replicatedLevelDB
replicas="3"
zkSessionTimeout="5s"
zkPath="/activemq/amq-prod-1"
zkAddress="zk1.example.org:2181,zk2.example.org:2182,zk3.example.org:2181,
zk4.example.org:2181,zk5.example.org:2181,zk6.example.org:2181" />
</persistenceAdapter>
  <plugins>
     <!-- http://activemq.apache.org/statisticsplugin.html -->
     <statisticsBrokerPlugin/>
</plugins>

         <!-- http://activemq.apache.org/producer-flow-control.html -->
<systemUsage>
     <systemUsage>
<memoryUsage>
     <memoryUsage percentOfJvmHeap="70"/>
</memoryUsage>
<storeUsage>
     <storeUsage limit="6 gb"/>
</storeUsage>
<tempUsage>
     <tempUsage limit="2 gb"/>
</tempUsage>
     </systemUsage>
</systemUsage>

<!-- http://activemq.apache.org/producer-flow-control.html -->
<!-- http://activemq.apache.org/message-cursors.html -->
<destinationPolicy>
     <policyMap>
<policyEntries>
     <policyEntry topic=">" memoryLimit="1mb">
     </policyEntry>
     <policyEntry queue=">" memoryLimit="1mb" optimizedDispatch="true">
     </policyEntry>
</policyEntries>
     </policyMap>
</destinationPolicy>

<!-- http://activemq.apache.org/configuring-transports.html -->
<transportConnectors>
     <transportConnector name="openwire" uri="nio://0.0.0.0:61616"/>
     <transportConnector name="stomp+nio" uri="stomp+nio://0.0.0.0:61613"/>
     <transportConnector name="amqp" uri="amqp+nio://0.0.0.0:5672"/>
</transportConnectors>

<!-- http://activemq.apache.org/networks-of-brokers.html -->
         <!-- networkTTL is based on the number of clusters networked
together -->
<networkConnectors>
<networkConnector
uri="masterslave:(tcp://amq-prod-2a:61616,tcp://amq-prod-2b:61616,tcp://amq-prod-2c:61616)"
consumerTTL="1" messageTTL="-1" decreaseNetworkConsumerPriority="true" />
</networkConnectors>

<shutdownHooks>
     <!-- Shut down the jetty based web console application. -->
     <bean xmlns="http://www.springframework.org/schema/beans"
class="org.apache.activemq.hooks.SpringContextHook"/>
</shutdownHooks>
     </broker>

     <import resource="jetty.xml"/>
</beans>

Thanks. Mine is working now. It looks like the one major difference is the zk session timeout. I was using the default, and you had it set to 5 seconds. However, I am still seeing exceptions such as this one: "java.lang.RuntimeException: An IOException was thrown (should never happen in this method).”

Is your broker running pretty clean when you send messages through the fabric of network of replicated master/slaves?


Kevin


From: "James A. Robinson-2 [via ActiveMQ]" <ml-node+[email protected]
Date: Monday, March 30, 2015 at 5:29 PM
To: SNEI <[email protected]
Subject: Re: Creating a Network of Replicated LevelDB broker clusters
your 3 x 3 cluster? I am trying to do the same, with everything running
on my Mac initially.

On the amq-prod-1 cluster my network connector url is

masterslave:(tcp://amq-prod-2a:61616,tcp://amq-prod-2b:61616,tcp://amq-prod-2c:61616)

On the amq-prod-2 cluster my network connector url is:

masterslave:(tcp://amq-prod-1a:61616,tcp://amq-prod-1b:61616,tcp://amq-prod-1c:61616)

The following are the messages I get in my log:

2015-03-30 17:38:00,236 [ActiveMQ Task-1] INFO DiscoveryNetworkConnector
    - Establishing network connection from
vm://amq-prod-1?async=false&network=true to
failover:(tcp://amq-prod-2a:61616,tcp://amq-prod-2b:61616,tcp://amq-prod-2c:61616)?randomize=false&maxReconnectAttempts=0
2015-03-30 17:38:00,450 [ActiveMQ Task-1] INFO FailoverTransport
    - Successfully connected to tcp://amq-prod-2b:61616
2015-03-30 17:38:00,767 [//amq-prod-1#42] INFO
  DemandForwardingBridgeSupport - Network connection between
vm://amq-prod-1#42 and tcp://amq-prod-2b:61616 (amq-prod-2) has been
established.

Here's a full example of the config:

<?xml version="1.0"?>
<beans xmlns="http://www.springframework.org/schema/beans"
     xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
     xsi:schemaLocation="
http://www.springframework.org/schema/beans
http://www.springframework.org/schema/beans/spring-beans.xsd
http://activemq.apache.org/schema/core
" rel="nofollow">http://activemq.apache.org/schema/core/activemq-core.xsd">

     <!-- Allow the use of system properties as variables in this
configuration file -->
     <bean
class="org.springframework.beans.factory.config.PropertyPlaceholderConfigurer">
         <property name="locations">
             <value>file:${activemq.conf}/credentials.properties</value>
         </property>
     </bean>

     <broker
xmlns="http://activemq.apache.org/schema/core"
useJmx="true"
brokerName="amq-prod-1">

<managementContext>
     <!-- http://activemq.apache.org/jmx.html -->
             <!-- Do not use the legacy JMX Connector. -->
     <managementContext createConnector="false" />
</managementContext>

<persistenceAdapter>
     <!-- http://activemq.apache.org/replicated-leveldb-store.html -->
     <replicatedLevelDB
replicas="3"
zkSessionTimeout="5s"
zkPath="/activemq/amq-prod-1"
zkAddress="zk1.example.org:2181,zk2.example.org:2182,zk3.example.org:2181,
zk4.example.org:2181,zk5.example.org:2181,zk6.example.org:2181" />
</persistenceAdapter>
  <plugins>
     <!-- http://activemq.apache.org/statisticsplugin.html -->
     <statisticsBrokerPlugin/>
</plugins>

         <!-- http://activemq.apache.org/producer-flow-control.html -->
<systemUsage>
     <systemUsage>
<memoryUsage>
     <memoryUsage percentOfJvmHeap="70"/>
</memoryUsage>
<storeUsage>
     <storeUsage limit="6 gb"/>
</storeUsage>
<tempUsage>
     <tempUsage limit="2 gb"/>
</tempUsage>
     </systemUsage>
</systemUsage>

<!-- http://activemq.apache.org/producer-flow-control.html -->
<!-- http://activemq.apache.org/message-cursors.html -->
<destinationPolicy>
     <policyMap>
<policyEntries>
     <policyEntry topic=">" memoryLimit="1mb">
     </policyEntry>
     <policyEntry queue=">" memoryLimit="1mb" optimizedDispatch="true">
     </policyEntry>
</policyEntries>
     </policyMap>
</destinationPolicy>

<!-- http://activemq.apache.org/configuring-transports.html -->
<transportConnectors>
     <transportConnector name="openwire" uri="nio://0.0.0.0:61616"/>
     <transportConnector name="stomp+nio" uri="stomp+nio://0.0.0.0:61613"/>
     <transportConnector name="amqp" uri="amqp+nio://0.0.0.0:5672"/>
</transportConnectors>

<!-- http://activemq.apache.org/networks-of-brokers.html -->
         <!-- networkTTL is based on the number of clusters networked
together -->
<networkConnectors>
<networkConnector
uri="masterslave:(tcp://amq-prod-2a:61616,tcp://amq-prod-2b:61616,tcp://amq-prod-2c:61616)"
consumerTTL="1" messageTTL="-1" decreaseNetworkConsumerPriority="true" />
</networkConnectors>

<shutdownHooks>
     <!-- Shut down the jetty based web console application. -->
     <bean xmlns="http://www.springframework.org/schema/beans"
class="org.apache.activemq.hooks.SpringContextHook"/>
</shutdownHooks>
     </broker>

     <import resource="jetty.xml"/>
</beans>


________________________________
If you reply to this email, your message will be added to the discussion below:
http://activemq.2283324.n4.nabble.com/Creating-a-Network-of-Replicated-LevelDB-broker-clusters-tp4693979p4694097.html
To unsubscribe from Creating a Network of Replicated LevelDB broker clusters, click here<http://activemq.2283324.n4.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=4693979&code=a2V2aW4uc3VAYW0uc29ueS5jb218NDY5Mzk3OXw5MzA0MDE1ODU=>.
NAML<http://activemq.2283324.n4.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>

fabric of network of replicated master/slaves?


Yes, though I'm only running the examples/openwire/swissarmy
producer/consumer example using the defaults. On the 2nd
cluster I set up a consumer to read from the 1st cluster:

[amq-prod-2c swissarmy]$ ant consumer
'-Durl=failover://(tcp://amq-prod-1a:61616,tcp://amq-prod-1b:61616,tcp://amq-prod-1c:61616)'

And on the 1st cluster I run a producer pushing to the 2nd cluster:

[amq-prod-1c swissarmy]$ ant producer
'-Durl=failover://(tcp://amq-prod-2a:61616,tcp://amq-prod-2b:61616,tcp://amq-prod-2c:61616)'

producer:
      [echo] Running producer against server at $url =
failover://(tcp://amq-prod-2a:61616,tcp://amq-prod-2b:61616,tcp://amq-prod-2c:61616)
for subject $subject = TEST.FOO
      [java] Connecting to URL:
failover://(tcp://amq-prod-2a:61616,tcp://amq-prod-2b:61616,tcp://amq-prod-2c:61616)
(:)
      [java] Publishing a Message with size 1000 to queue: TEST.FOO
      [java] Using non-persistent messages
      [java] Sleeping between publish 0 ms
      [java] Running 1 parallel threads
      [java] log4j:WARN No appenders could be found for logger
(org.apache.activemq.thread.TaskRunnerFactory).
      [java] log4j:WARN Please initialize the log4j system properly.
      [java] log4j:WARN See
http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
      [java] [Thread-1] Sending message: 'Message: 0 sent at: Tue Mar 31
12:38:39 PDT 2015 ...'
...
      [java] [Thread-1] Sending message: 'Message: 1999 sent at: Tue Mar 31
12:38:40 PDT 201...'
      [java] [Thread-1] Done.

consumer:
      [echo] Running consumer against server at $url =
failover://(tcp://amq-prod-1a:61616,tcp://amq-prod-1b:61616,tcp://amq-prod-1c:61616)
for subject $subject = TEST.FOO
      [java] Connecting to URL:
failover://(tcp://amq-prod-1a:61616,tcp://amq-prod-1b:61616,tcp://amq-prod-1c:61616)
(:)
      [java] Consuming queue: TEST.FOO
      [java] Using a non-durable subscription
      [java] Running 1 parallel threads
      [java] log4j:WARN No appenders could be found for logger
(org.apache.activemq.thread.TaskRunnerFactory).
      [java] log4j:WARN Please initialize the log4j system properly.
      [java] log4j:WARN See
http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
      [java] [Thread-1] We are about to wait until we consume: 2000
message(s) then we will shutdown
      [java] [Thread-1] Received: 'Message: 0 sent at: Tue Mar 31 12:38:39
PDT 2015 ...' (length 1000)
...
      [java] [Thread-1] Received: 'Message: 1999 sent at: Tue Mar 31
12:38:40 PDT 201...' (length 1000)
      [java] [Thread-1] Closing connection
      [java] All threads completed their work

The only log information I see in /var/log/activemq.log on either master are
details about the master/slave coordination and the establishment of the
network.

I am running the broker using the activemq script, and in console mode, not sure if that produces more verbose information than how you are running it.

What version of ActiveMQ are you running with? I am working with 5.11.1..

Thanks.

Kevin

From: "James A. Robinson-2 [via ActiveMQ]" <ml-node+[email protected]
Date: Tuesday, March 31, 2015 at 12:37 PM
To: SNEI <[email protected]
Subject: Re: Creating a Network of Replicated LevelDB broker clusters
fabric of network of replicated master/slaves?


Yes, though I'm only running the examples/openwire/swissarmy
producer/consumer example using the defaults. On the 2nd
cluster I set up a consumer to read from the 1st cluster:

[amq-prod-2c swissarmy]$ ant consumer
'-Durl=failover://(tcp://amq-prod-1a:61616,tcp://amq-prod-1b:61616,tcp://amq-prod-1c:61616)'

And on the 1st cluster I run a producer pushing to the 2nd cluster:

[amq-prod-1c swissarmy]$ ant producer
'-Durl=failover://(tcp://amq-prod-2a:61616,tcp://amq-prod-2b:61616,tcp://amq-prod-2c:61616)'

producer:
      [echo] Running producer against server at $url =
failover://(tcp://amq-prod-2a:61616,tcp://amq-prod-2b:61616,tcp://amq-prod-2c:61616)
for subject $subject = TEST.FOO
      [java] Connecting to URL:
failover://(tcp://amq-prod-2a:61616,tcp://amq-prod-2b:61616,tcp://amq-prod-2c:61616)
(:)
      [java] Publishing a Message with size 1000 to queue: TEST.FOO
      [java] Using non-persistent messages
      [java] Sleeping between publish 0 ms
      [java] Running 1 parallel threads
      [java] log4j:WARN No appenders could be found for logger
(org.apache.activemq.thread.TaskRunnerFactory).
      [java] log4j:WARN Please initialize the log4j system properly.
      [java] log4j:WARN See
http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
      [java] [Thread-1] Sending message: 'Message: 0 sent at: Tue Mar 31
12:38:39 PDT 2015 ...'
...
      [java] [Thread-1] Sending message: 'Message: 1999 sent at: Tue Mar 31
12:38:40 PDT 201...'
      [java] [Thread-1] Done.

consumer:
      [echo] Running consumer against server at $url =
failover://(tcp://amq-prod-1a:61616,tcp://amq-prod-1b:61616,tcp://amq-prod-1c:61616)
for subject $subject = TEST.FOO
      [java] Connecting to URL:
failover://(tcp://amq-prod-1a:61616,tcp://amq-prod-1b:61616,tcp://amq-prod-1c:61616)
(:)
      [java] Consuming queue: TEST.FOO
      [java] Using a non-durable subscription
      [java] Running 1 parallel threads
      [java] log4j:WARN No appenders could be found for logger
(org.apache.activemq.thread.TaskRunnerFactory).
      [java] log4j:WARN Please initialize the log4j system properly.
      [java] log4j:WARN See
http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
      [java] [Thread-1] We are about to wait until we consume: 2000
message(s) then we will shutdown
      [java] [Thread-1] Received: 'Message: 0 sent at: Tue Mar 31 12:38:39
PDT 2015 ...' (length 1000)
...
      [java] [Thread-1] Received: 'Message: 1999 sent at: Tue Mar 31
12:38:40 PDT 201...' (length 1000)
      [java] [Thread-1] Closing connection
      [java] All threads completed their work

The only log information I see in /var/log/activemq.log on either master are
details about the master/slave coordination and the establishment of the
network.


________________________________
If you reply to this email, your message will be added to the discussion below:
http://activemq.2283324.n4.nabble.com/Creating-a-Network-of-Replicated-LevelDB-broker-clusters-tp4693979p4694156.html
To unsubscribe from Creating a Network of Replicated LevelDB broker clusters, click here<http://activemq.2283324.n4.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=4693979&code=a2V2aW4uc3VAYW0uc29ueS5jb218NDY5Mzk3OXw5MzA0MDE1ODU=>.
NAML<http://activemq.2283324.n4.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>

I'm running 5.11.1 plus the patch in ticket AMQ-5082.

Jim