I have set up 3 cluster nodes namely rabbit at A, rabbit at B, rabbit at C all running
on erlang 16B and rabbitmq 3.1.1. I set net_ticktime to 2 so as to detect
node failure faster.


I have did some test on it and found that there seems to be a weird behavior
when setting policy for mirroring.
For the purpose of testing, I modified the code for the java tutorial 1
hello world to send out 200,000 messages instead of 1 and also the queue is
created using the management console and not created in the code. Each time
it sends a message it will print out the current number starting from 1 til
199,999


Below is what I did:


1)For my 1st test, I only added a queue for the sending of messages which I
call it "test". Throughout this test rabbit at A is the master. Halfway into
sending the messages i will cut off the network connection for rabbit at B. The
observation is that there is no disruption to the master. The result is
consistent for 10 tries.


2)For my 2nd test, it is almost identical to the 1st test except that i am
using mirroring using the following command:


rabbitmqctl set_policy HA '^(?!amq\.).*' '{"ha-mode": "all"}'"


The observation is different from that of the 1st test. Shortly after I cut
off the network connection from rabbit at B, rabbit at A which is the master
handling the client's messages comes to a pause for over 15 seconds and the
pause is consistent for 10 tries. The client gets stuck in basic publish
when rabbit at A comes to a pause.






I am quite puzzled about this behavior and would like to find out if that is
the intended behavior for rabbit's mirroring feature? Does anyone else
encounter such behavior when using mirroring?


I have attached my code for sending messages. Thanks.
send.txt <http://rabbitmq.1065348.n5.nabble.com/file/n27572/send.txt>






--
View this message in context: http://rabbitmq.1065348.n5.nabble.com/RabbitMQ-s-Mirroring-results-in-weird-behavior-when-slave-goes-down-tp27572.html
Sent from the RabbitMQ mailing list archive at Nabble.com.

Search Discussions

  • Tim Watson at Jun 25, 2013 at 10:48 am
    Thomas,


    On 25 Jun 2013, at 07:28, thomas wrote:

    I have set up 3 cluster nodes namely rabbit at A, rabbit at B, rabbit at C all running
    on erlang 16B and rabbitmq 3.1.1. I set net_ticktime to 2 so as to detect
    node failure faster.

    That's a bit excessive I think. Let me quote Erlang's net_kernel man page for a moment:


    <quote>
    net_ticktime = TickTime
    Specifies the net_kernel tick time. TickTime is given in seconds. Once every TickTime/4 second, all connected nodes are ticked (if anything else has been written to a node) and if nothing has been received from another node within the last four (4) tick times that node is considered to be down. This ensures that nodes which are not responding, for reasons such as hardware errors, are considered to be down.


    The time T, in which a node that is not responding is detected, is calculated as: MinT < T < MaxT where:


    MinT = TickTime - TickTime / 4
    MaxT = TickTime + TickTime / 4
    TickTime is by default 60 (seconds). Thus, 45 < T < 75 seconds.


    Note: All communicating nodes should have the same TickTime value specified.


    Note: Normally, a terminating node is detected immediately.


    </quote>


    So, you're increasing the requirement for nodes to ping one another every 2 / 4 seconds, i.e., every 500 milliseconds. You're also sending 200k messages to a node and expecting HA to distribute those messages across all your nodes, which happens over the same distribution channel as that TickTime message. So I suspect you're not doing yourself any favours here. I'd suggest that *if you must* change net_ticktime (and personally I would leave it alone if I were you) then you should set it to maybe 45 seconds, but not to 2 seconds - that's almost guaranteed to end up in weird behaviour.

    2)For my 2nd test, it is almost identical to the 1st test except that i am
    using mirroring using the following command:

    rabbitmqctl set_policy HA '^(?!amq\.).*' '{"ha-mode": "all"}'"

    The observation is different from that of the 1st test. Shortly after I cut
    off the network connection from rabbit at B, rabbit at A which is the master
    handling the client's messages comes to a pause for over 15 seconds and the
    pause is consistent for 10 tries. The client gets stuck in basic publish
    when rabbit at A comes to a pause.



    I am quite puzzled about this behavior and would like to find out if that is
    the intended behavior for rabbit's mirroring feature? Does anyone else
    encounter such behavior when using mirroring?

    What version of rabbit are you using? Do you have a cluster auto-recovery (i.e., autoheal) set up, and if so, which mode are you using? Some delay (which blocks publishers temporarily) is possible during failover, but also if you've got autoheal set up, then node can be restarted and waiting (for node restarts) can occur. Remember that a cluster partition is a serious problem, which rabbitmq clusters are /not/ tolerant of. If you're expecting partitions, you should consider using the federation or shovel plugins instead. Automatic cluster partition recovery is there to help, but isn't a panacea.


    Cheers,
    Tim


    -------------- next part --------------
    An HTML attachment was scrubbed...
    URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20130625/1eb29152/attachment.htm>
  • Thomas at Jun 26, 2013 at 5:24 am
    Hi Tim,


    Thanks for the reply. I am using rabbitmq 3.1.1 with autoheal for
    cluster_partition_handling.


    My test is just to find out the behavior of rabbitmq master should its slave
    goes down. For my case, initially I will have rabbit at A, rabbit at B and
    rabbit at C running. Then I will send out messages from client to rabbit at A
    which is the master throughout the test. Soon after the client started to
    send out messages, I will bring down a slave which in my case is rabbit at B by
    closing its network connection.


    I have changed the net_ticktime to 60 and carry out the same test using the
    same code but this time round i increases the number of messages to 500,000
    so as to better observe the behavior of sending messages for more than a
    minute.


    My observation is still similar to that of my previous test where
    net_ticktime is 2.
    Approximately 10 seconds after I shut down the network connection of
    rabbit at B, the sending of messages to rabbit at A comes to a pause for close to
    a minute and then continues as per normal.


    Trying the same test but with no mirroring does not have any disruption.


    By the way, I have try out the same test for rabbitmq mirroring except that
    there will be a delay of 10ms between sending every message and there was no
    disruption.
  • Tim Watson at Jun 26, 2013 at 10:38 am
    Hi Thomas,


    Here are my thoughts (inline below). I'm not the greatest clustering/HA expert, but from what I can gather, I think what you're seeing might be a result of the combination of a network partition and the `autoheal' mode.


    On 26 Jun 2013, at 06:24, thomas wrote:

    Hi Tim,

    Thanks for the reply. I am using rabbitmq 3.1.1 with autoheal for
    cluster_partition_handling.

    My test is just to find out the behavior of rabbitmq master should its slave
    goes down.

    This is clearly documented on the website - RabbitMQ clusters /do not/ handle network partitions well. Things will probably go wrong. If the only issue you're running into is a busy producer getting blocked then that's great - the worst cases could have involved lost messages and all sorts.


    Now, the intention with HA is *not* to go belly up in the face of disappearing slaves - quite the opposite. Quite why the master is suddenly blocking the producer I'm not sure - I'll discuss this with the rest of the team but it doesn't sound quite right to me.

    Approximately 10 seconds after I shut down the network connection of
    rabbit at B, the sending of messages to rabbit at A comes to a pause for close to
    a minute and then continues as per normal.

    Just to be clear, in a RabbitMQ cluster there is no concept of a master node - only mirror queues introduce the master/slave concept, and then only for queues that are mirrored. Each mirrored queue's master can end up on different nodes, depending on which you were connect to when it was declared.

    Trying the same test but with no mirroring does not have any disruption.

    That is not surprising, since the queue you're connected to isn't trying to replicate to any other nodes.

    By the way, I have try out the same test for rabbitmq mirroring except that
    there will be a delay of 10ms between sending every message and there was no
    disruption.

    Can you share your test code? I'm surprised there is such a big difference, but still... If you've got a decent automated test that consistently replicates this failure mode and you're able to share it, we might be able to use it to make improvements to partition handling in the future. If not code, a test script (and description of how you're disabling the network link) with precise steps would do.

    From my observation it can be seen that RabbitMQ server will exhibit weird
    behavior under stress condition when a slave goes down. I presume this weird
    behavior will only surface under stress condition.

    As I've stated before, clustering is not designed to handle network partitions. In previous versions of rabbit, when a partition occurred it was up to a system administrator to handle it. This usually involved stopping some (or all) of the nodes and restarting them. The 'autoheal' feature simply attempts to replicate what a sysadmin might do, and therefore (whilst cluster nodes are restarting) there can be delays. The intention is not to block the master (or producers), but depending on how the cluster partition is handled, that might be one of the potential side effects.


    We will certainly take a look to see if this is avoidable - I can't comment on that right now - and being able to repeat the tests could help with that too.

    Is there a rough estimate for the maximum load that a RabbitMQ server can
    take per second for mirroring? Thanks.

    I'm not sure whether mirroring is really the problem here. Mirrored queues can handle plenty of load, although there /is/ some performance slow down due to replication (and esp. so if you're using publisher confirms) but there is a not a specific limit - this is hardware/resource and use-case dependent. I suspect the problem /you/ have could be a combination of network partition and cluster partition resolution (i.e., autoheal) mode.


    Based on http://www.rabbitmq.com/partitions.html, what I'd suggest is that you change 'autoheal' to 'ignore' and try your test again. That will either confirm or eliminate my suspicion quickly enough.


    Cheers,
    Tim
  • Thomas at Jun 27, 2013 at 1:18 am
    Hi Tim,


    I have tried using ignore instead of autoheal for cluster_partition_handling
    but the pause still persist, but the pause begins approximately 20 seconds
    after bringing down the mirror node.


    Below is the code for my sender:
    send.txt <http://rabbitmq.1065348.n5.nabble.com/file/n27633/send.txt>


    My rabbitmq server is running on Ubuntu 12.04 with erlang 16B.


    The following steps are what I did:
    1) I begin with all rabbitmq servers down.
    2) I started rabbitmq server in the following sequence: rabbit at A , rabbit at B,
    rabbit at C
    3) I started my client to begin sending messages
    4) I shut down the network connection for rabbti at B using ifconfig
    <interface> down
    5) I observe the output from the client


    Please kindly let me know if you experience the same problem as I can't be
    sure if there could be any problem on my side which I doubt so. Thanks.






    --
    View this message in context: http://rabbitmq.1065348.n5.nabble.com/RabbitMQ-s-Mirroring-results-in-weird-behavior-when-slave-goes-down-tp27572p27633.html
    Sent from the RabbitMQ mailing list archive at Nabble.com.
  • Tim Watson at Jul 3, 2013 at 9:46 am
    Hi Thomas,


    On 27 Jun 2013, at 02:18, thomas wrote:

    I have tried using ignore instead of autoheal for cluster_partition_handling
    but the pause still persist, but the pause begins approximately 20 seconds
    after bringing down the mirror node.

    That won't help you I'm afraid.

    Please kindly let me know if you experience the same problem as I can't be
    sure if there could be any problem on my side which I doubt so. Thanks.

    There's nothing wrong with your sending code. The issue appears to be that the channel is expecting to receive credit (during normal flow control) from not just the master queue process, but from all the slaves too. This can cause a delay of up to (and even exceeding) net_ticktime, basically until the node notices that its peer has gone away. The reason why introducing a delay to the publishing rate stops the pause, is that when publishing slows down, we don't end up hitting flow control and the logic that's waiting for slaves therefore doesn't kick in.


    Lowering the net_ticktime can alleviate this to some extent, but will not solve the problem consistently as there's always going to be a possibility of delay (depending on the current load, network characteristics, etc). You should also bear in mind that lowering the net_ticktime too much can be quite dangerous if the network is already under considerable load, since ticks might get lost behind other traffic, leading to false (unwanted) net-splits occurring.


    I have filed a bug to try and improve this behaviour (in the face of slave deaths) and will try to get the fix into the next patch release. Thanks for reporting this behaviour, and for sticking with the conversation while we worked through the possible causes.


    Cheers,
    Tim

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
grouprabbitmq-discuss @
categoriesrabbitmq
postedJun 25, '13 at 6:28a
activeJul 3, '13 at 9:46a
posts6
users2
websiterabbitmq.com
irc#rabbitmq

2 users in discussion

Thomas: 3 posts Tim Watson: 3 posts

People

Translate

site design / logo © 2017 Grokbase