[rabbitmq-discuss] RabbitMQ HA

Mark

Does this help? http://www.rabbitmq.com/ha.html

alexis

In order to do this in our setup, we use HAProxy <http://haproxy.1wt.eu/> in
front of three virtual machines (our systems are all deployed on RackSpace
Cloud VMs). All of our software only knows about the Virtual IP that
points at our HAProxy installation; HAProxy knows about the three VMs, each
of which is running one node of a RabbitMQ cluster. All of our Queues are
created as HA queues so that messages are replicated across all nodes in
the event of a single node failure.

This method allows us to simply add another VM and bring it into the
cluster if we want to expand capacity, and no updates are required to our
code base as it continues to interact with HAProxy for connections. The
trick was to make sure we set everything up properly to automatically
reconnect in our code in the event of a disconnection, as HAProxy times out
connections at an interval determined by your settings.

Having said all that, we have yet to move this into production, so I can't
speak to whether or not it will behave as anticipated in the wild.
  However, initial testing is very promising, and when we force a node to
fail, HAProxy automatically detects that it has gone down, and just stops
handing out connections to that node. Just in case anyone complains about
the single point of failure that is HAProxy, that has also been setup to
failover on a Virtual IP; should our main HAProxy machine ever go down, the
other HAProxy instance will take over and continue handing out node
connections.

This has been an interesting experiment, feel free to bug me if you want
more details on our setup.

- jrob

Sorry for the multi-post, but I meant to give props to the book RabbitMQ in
Action which provides a very detailed example of the setup I described. I
really only had to plug in the Ruby equivalents in our environment, and
stomp a few bugs, and we were good to go.

John,

Does this include automatic recovery mode that amqp gem supports for publishers in 0.9.0? I am always curious how well it works in
for other people.

Thanks.

MK

http://github.com/michaelklishin
http://twitter.com/michaelklishin

Yup, that was it exactly. I just had the diff up, so here are the two
major changes I had to make:

- channel.queue( @queue, :durable => true ).subscribe(:ack=>true) do
+ channel.queue( @queue, :durable => true, :arguments => {'x-ha-policy'
=> 'all'} ).subscribe(:ack=>true) do |header,message|

    def self.get_channel(host)
      connection = AMQP.connect( :host=>host )
+ connection.on_connection_interruption do |conn, settings|
+ conn.reconnect(false,1)
+ end
+
      channel = AMQP::Channel.new( connection )
+ channel.auto_recovery = true
+
+ channel
    end

As I said, this hasn't gone into production yet, but it seems to be doing
well in our staging environment. I don't think our message rate is high
enough to really exercise this setup in staging, so I can't really comment
on performance yet. (We're holding off moving this into production as we
are trying to provide high availability for our applications and services
across the board, which in some cases means we will be using one VM for
multiple services. Since the other services aren't ready to go yet, we
haven't pushed this up.)

- jrob

Hi jrob,


Curious to know whether you pushed this setup in production and how has it
been? Any surprises? We are also testing this in perf lab and plan to push
to production in couple of weeks.


Anup