Hi all,

I was wondering if I could get some advice on RabbitMQ HA. I've read
that RabbitMQ has clustering out of the box but that only helps with
load correct? IE, if one RabbitMQ broker can not handle all of our
traffic then we should look into clustering correct? Since we will be
slowly rolling out our RabbitMQ implementation I don't think load will
be our initial problem. Instead we want to be able to gracefully handle
an outage of one or more RabbitMQ machines. How is this traditionally
handled? Is there any built in replication similar to a master-master
database setup?

Any ideas?

Thanks

Search Discussions

  • Alexis Richardson at Jan 12, 2012 at 5:33 pm
    Mark

    Does this help? http://www.rabbitmq.com/ha.html

    alexis

    On Thu, Jan 12, 2012 at 5:30 PM, Mark wrote:
    Hi all,

    I was wondering if I could get some advice on RabbitMQ HA. I've read that
    RabbitMQ has clustering out of the box but that only helps with load
    correct? IE, if one RabbitMQ broker can not handle all of our traffic then
    we should look into clustering correct? Since we will be slowly rolling out
    our RabbitMQ implementation I don't think load will be our initial problem.
    Instead we want to be able to gracefully handle an outage of one or more
    RabbitMQ machines. How is this traditionally handled? Is there any built in
    replication similar to a master-master database setup?

    Any ideas?

    Thanks
    _______________________________________________
    rabbitmq-discuss mailing list
    rabbitmq-discuss at lists.rabbitmq.com
    https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
  • Jrob at Jan 18, 2012 at 1:04 am
    In order to do this in our setup, we use HAProxy <http://haproxy.1wt.eu/> in
    front of three virtual machines (our systems are all deployed on RackSpace
    Cloud VMs). All of our software only knows about the Virtual IP that
    points at our HAProxy installation; HAProxy knows about the three VMs, each
    of which is running one node of a RabbitMQ cluster. All of our Queues are
    created as HA queues so that messages are replicated across all nodes in
    the event of a single node failure.

    This method allows us to simply add another VM and bring it into the
    cluster if we want to expand capacity, and no updates are required to our
    code base as it continues to interact with HAProxy for connections. The
    trick was to make sure we set everything up properly to automatically
    reconnect in our code in the event of a disconnection, as HAProxy times out
    connections at an interval determined by your settings.

    Having said all that, we have yet to move this into production, so I can't
    speak to whether or not it will behave as anticipated in the wild.
      However, initial testing is very promising, and when we force a node to
    fail, HAProxy automatically detects that it has gone down, and just stops
    handing out connections to that node. Just in case anyone complains about
    the single point of failure that is HAProxy, that has also been setup to
    failover on a Virtual IP; should our main HAProxy machine ever go down, the
    other HAProxy instance will take over and continue handing out node
    connections.

    This has been an interesting experiment, feel free to bug me if you want
    more details on our setup.

    - jrob
    -------------- next part --------------
    An HTML attachment was scrubbed...
    URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20120117/ef7f62c1/attachment.htm>
  • Jrob at Jan 18, 2012 at 1:15 am
    Sorry for the multi-post, but I meant to give props to the book RabbitMQ in
    Action which provides a very detailed example of the setup I described. I
    really only had to plug in the Ruby equivalents in our environment, and
    stomp a few bugs, and we were good to go.
    -------------- next part --------------
    An HTML attachment was scrubbed...
    URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20120117/4e15987c/attachment.htm>
  • Michael Klishin at Jan 18, 2012 at 11:17 am

    jrob:

    I really only had to plug in the Ruby equivalents in our environment, and stomp a few bugs, and we were good to go.
    John,

    Does this include automatic recovery mode that amqp gem supports for publishers in 0.9.0? I am always curious how well it works in
    for other people.

    Thanks.

    MK

    http://github.com/michaelklishin
    http://twitter.com/michaelklishin
  • Jrob at Jan 18, 2012 at 5:16 pm
    Yup, that was it exactly. I just had the diff up, so here are the two
    major changes I had to make:

    - channel.queue( @queue, :durable => true ).subscribe(:ack=>true) do
    header,message|
    + channel.queue( @queue, :durable => true, :arguments => {'x-ha-policy'
    => 'all'} ).subscribe(:ack=>true) do |header,message|

        def self.get_channel(host)
          connection = AMQP.connect( :host=>host )
    + connection.on_connection_interruption do |conn, settings|
    + conn.reconnect(false,1)
    + end
    +
          channel = AMQP::Channel.new( connection )
    + channel.auto_recovery = true
    +
    + channel
        end

    As I said, this hasn't gone into production yet, but it seems to be doing
    well in our staging environment. I don't think our message rate is high
    enough to really exercise this setup in staging, so I can't really comment
    on performance yet. (We're holding off moving this into production as we
    are trying to provide high availability for our applications and services
    across the board, which in some cases means we will be using one VM for
    multiple services. Since the other services aren't ready to go yet, we
    haven't pushed this up.)

    - jrob

    -------------- next part --------------
    An HTML attachment was scrubbed...
    URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20120118/63fd7f23/attachment.htm>
  • Anup Katariya at Jan 17, 2014 at 5:04 am
    Hi jrob,


    Curious to know whether you pushed this setup in production and how has it
    been? Any surprises? We are also testing this in perf lab and plan to push
    to production in couple of weeks.


    Anup

    On Wednesday, January 18, 2012 9:16:22 AM UTC-8, jrob wrote:

    Yup, that was it exactly. I just had the diff up, so here are the two
    major changes I had to make:

    - channel.queue( @queue, :durable => true ).subscribe(:ack=>true) do
    header,message|
    + channel.queue( @queue, :durable => true, :arguments => {'x-ha-policy'
    => 'all'} ).subscribe(:ack=>true) do |header,message|

    def self.get_channel(host)
    connection = AMQP.connect( :host=>host )
    + connection.on_connection_interruption do |conn, settings|
    + conn.reconnect(false,1)
    + end
    +
    channel = AMQP::Channel.new( connection )
    + channel.auto_recovery = true
    +
    + channel
    end

    As I said, this hasn't gone into production yet, but it seems to be doing
    well in our staging environment. I don't think our message rate is high
    enough to really exercise this setup in staging, so I can't really comment
    on performance yet. (We're holding off moving this into production as we
    are trying to provide high availability for our applications and services
    across the board, which in some cases means we will be using one VM for
    multiple services. Since the other services aren't ready to go yet, we
    haven't pushed this up.)

    - jrob
    -------------- next part --------------
    An HTML attachment was scrubbed...
    URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20140116/f6e5ef62/attachment.html>

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
grouprabbitmq-discuss @
categoriesrabbitmq
postedJan 12, '12 at 5:30p
activeJan 17, '14 at 5:04a
posts7
users5
websiterabbitmq.com
irc#rabbitmq

People

Translate

site design / logo © 2022 Grokbase