[rabbitmq-discuss] Keeping track of exchanges -- Any Advice

First of all, can you get the routing you need without every user
requiring their own exchange? What routing do you need to do?

Cheers, Simon

Hello Simon, thanks for the response.

I am not really sure to be honest. I am not very familiar with message
brokers and their performance characteristics. I suppose I could have
a topic exchange and then bind queues to various topics... Such as
have a queue for user1.queu1, user2.queue1, user3.queue1, etc

My question is how expensive is routing? Also, regarding the routing,
is it more performant to have one master topic exchange for all the
users or to distribute users across multiple topic exchanges or does
it not matter?

I assume rabbitmq first looks up the exchange and then goes through
each queue to see if there is a match for routing. If this is the
case, I imagine it would be more performant to create a thousand such
topic exchanges and divide all the users across them as opposed to one
topic exchange for all users.


In the end, I plan to run several rabbitmq nodes with a bucket of
users assigned to each rabbit node. Assigning each user to a node
based on a hash. Is this best practice for scaling rabbitmq for this
type of scenerio? Currently rabbitmq is the bottle neck in my
application benchmarks so this was my plan to scale it. Please correct
me if there is a better practice.

Unless you need wildcards you may not even need a topic exchange. *If*
routing turns out to be an expensive thing for you then a direct
exchange can give a notable performance boost.
Depends :) If you're sending lots of small non-persistent messages fast
then it can become a somewhat significant factor, just because RabbitMQ
isn't doing mush *else* per message.

Usually making routing simpler will make it faster.
No, it's not quite like that - topic routing is O(log(n)). But I think
the n there is the total number of topic bindings, not per exchange.
This sort of sharding is probably the most scalable solution if it's
practical. If it's not practical then clustering or federation would be
alternative routes...

Cheers, Simon

Direct routing would be great since all my code is already written for
direct routing. The problem with direct routing is rabbitmq (based on
the amqp spec) keeps throwing a channel exception if a message is sent
to an exchange that does not exist. Is there an extension within
rabbitmq that says "Pretend messages that get sent to a non-existant
exchange are sent and just discard the messages?"

If not, then I have to keep track of a LOT of exchanges that are
created and destroyed in an unpredictable matter. The only advantage
to the topic exchange routing is to work around the amqp throwing
channel exceptions as I can guarantee xxx topic exchanges can exist.

Okay, I feel like a real idiot. For some reason I was thinking that
direct exchanges required a 1:1 relationship to exchanges / queues.
Pretty silly of me considering I implemented the client/protocol for
our AMQP client.

That vastly simplifies the problem. Direct exchanges will work
marvellously. My question is, if there are going to be 50,000 queues,
is there any performance advantage to dividing these queues across
multiple exchanges on the same box? I do not know what kind of
algorithmns rabbit uses for exchange lookup and queue matching for the
direct exchange. So, is it better for 1 exchange to have 50,000 queues
on the same machine or 100 exchanges to have 500 queues on the same
machine?

Hi, Marcus:

The actual work of doing the routing through an exchange is actually
done by the Rabbit Erlang process associated with the *channel* you're
publishing on... in other words there's no "exchange process."

The work that your channel process does varies depending on the type of
exchange. In the case of topic exchanges there's a nice and well tuned
trie-based matching algorithm that Matthew has blogged about on the
Rabbit website in the past. Fan out and direct exchanges work about as
you'd expect.

Unlike exchanges, queues *do* have an Erlang process associated with them,
that receives messages that have been routed through the appropriate
exchange, by the channel process, which passes the AMQP message to the
queue process using Erlang messaging.

As always, it pays to perform an experiment that lines up with the scenario
you're expecting to face in your production system. But the above
observations should get you started on how to set your expectations...

Best regards,
Jerry

----- Original Message -----
From: "MarcusR" <marcus_rubeus at hotmail.com>
To: rabbitmq-discuss at lists.rabbitmq.com
Sent: Friday, February 3, 2012 4:49:34 PM
Subject: Re: [rabbitmq-discuss] Keeping track of exchanges -- Any Advice

Okay, I feel like a real idiot. For some reason I was thinking that
direct exchanges required a 1:1 relationship to exchanges / queues.
Pretty silly of me considering I implemented the client/protocol for
our AMQP client.

That vastly simplifies the problem. Direct exchanges will work
marvellously. My question is, if there are going to be 50,000 queues,
is there any performance advantage to dividing these queues across
multiple exchanges on the same box? I do not know what kind of
algorithmns rabbit uses for exchange lookup and queue matching for the
direct exchange. So, is it better for 1 exchange to have 50,000 queues
on the same machine or 100 exchanges to have 500 queues on the same
machine?
_______________________________________________
rabbitmq-discuss mailing list
rabbitmq-discuss at lists.rabbitmq.com
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss