[rabbitmq-discuss] SimpleRpcServer send generated queue name to client

The Erlang amqp client lib has an example RPC server.

In essence, when your client fires the "new subscription please" request it
also has to create a reply queue. The initial message from client to server
has to contain your "subscription request" plus your reply_to queue name
and, ideally, a correlation_id. The reply_to and correlation_id fields, I
think, are already part of the AMPQ message properties.

Your client can then receive the desired "new subscription" queue name as a
message from the server on the reply_to queue with the appropriate
correlation_id.

Hope this gets you into the right direction,
Robby

Hi,

glenner wrote:
It sounds reasonable; but it might be possible to leverage the features
of the RabbitMQ server even more than you have in mind.

For example, could it work to design a topic hierarchy for your system's
events, and then simply use an AMQP topic exchange to broadcast them to
the correct consumers?

The application-level server would then blindly send events, with an
appropriate routing key, to a named exchange it had set up, and the
clients would create private queues (using queueDeclare()) and bind them
(using queueBind()) to the exchange with appropriate binding key for
selecting the subset of events they are interested in. There need then
be no direct communication between the clients and the application-level
server.

You mentioned there were custom filter predicates. These are a potential
missing piece. If the event types don't fit nicely onto the routing keys
usable with topic exchanges, then you will be relying on client-side
filtering, which may or may not be suitable for your application. On the
other hand, if routing keys and a topic exchange get you 99% of the way
there, then the remaining small amount of filtering can easily be done
on the client side. A final option would be to implement a custom
exchange type which is capable of understanding your filter predicates
within the AMQP server itself.

Regards,
Tony

Hi,

I thought about that option, but I think it is not flexible enough for our application.
Like you said the custom filer predicates are very flexible. It could be possible to map them
to routing keys and use client side filtering as you said, but for security reasons I want control exactly which client receives which messages.
So this rules out client side filtering I guess?

I did not know about the option of implementing a custom exchange type.. Is it hard to do?
I don't know if this would be an ideal solution.. it feels strange pulling this application specific logic into the communication infrastructure (so tying our application to RabbitMQ) to me.. ?

What would be the disadvantages of doing it with the multiple queues as first suggested?
Is it performance related? Would it make a big difference using a topic exchange compared to for example 20 queues + subscriptions?
Are there other disadvantages?

Thanks,
Glenn

-----Oorspronkelijk bericht-----
Van: Tony Garnock-Jones [mailto:tonyg at lshift.net]
Verzonden: donderdag 4 maart 2010 0:20
Aan: Ergeerts Glenn
CC: rabbitmq-discuss at lists.rabbitmq.com
Onderwerp: Re: [rabbitmq-discuss] SimpleRpcServer send generated queue name to client

Hi,

glenner wrote:
It sounds reasonable; but it might be possible to leverage the features
of the RabbitMQ server even more than you have in mind.

For example, could it work to design a topic hierarchy for your system's
events, and then simply use an AMQP topic exchange to broadcast them to
the correct consumers?

The application-level server would then blindly send events, with an
appropriate routing key, to a named exchange it had set up, and the
clients would create private queues (using queueDeclare()) and bind them
(using queueBind()) to the exchange with appropriate binding key for
selecting the subset of events they are interested in. There need then
be no direct communication between the clients and the application-level
server.

You mentioned there were custom filter predicates. These are a potential
missing piece. If the event types don't fit nicely onto the routing keys
usable with topic exchanges, then you will be relying on client-side
filtering, which may or may not be suitable for your application. On the
other hand, if routing keys and a topic exchange get you 99% of the way
there, then the remaining small amount of filtering can easily be done
on the client side. A final option would be to implement a custom
exchange type which is capable of understanding your filter predicates
within the AMQP server itself.

Regards,
Tony

Glenn,

Ergeerts Glenn wrote:
Do these security considerations only apply to event types or the event
filtering predicates too?

If the former, you could go for a hybrid solution where the server
publishes events to a direct exchange, with the event type as the
routing key. Each client's queue would then be bound to that exchange
with binding keys for each of the event types the client is interested
in. And client-side filtering would take care of the more complex
filtering predicates.

The creation/deletion of the client queues could be left up to the
clients, but the access to the exchange could be restricted to the
server, with all bindings of client queues to the exchange being
established by that server.
It's fairly straightforward - if you know a bit about Erlang and the
rabbit internals ;) See
http://www.lshift.net/blog/2010/01/22/plugin-exchange-types-for-rabbitmq
for an example.
The standard, but less efficient, approach to implementing arbitrary
complex routing logic is to introduce a routing client. That is an
ordinary AMQP client which

a) consumes messages from, say, a single queue bound to a fanout
exchange to which producer send all publishes,

b) applies the necessary routing logic to compute a set of queues to
send the message to,

c) publishes copies of the message to the default exchange - one copy
per queue it needs routing too.

That's the basic setup, which can be suitably embellished by, say,
consuming from multiple exchanges/queues, and publishing to exchanges
other than the default exchange. Plus of course you need logic to deal
with subscription management.
You end up with one queue per client in all solutions proposed so far.
But there is still a performance advantage of getting rabbit to do as
much of the routing as possible. That's what it's designed to do, and it
(mostly) does it very efficiently. For example, if the same message
needs to go to 10 different queues, doing the routing in application
code results in ten separate copies of the message, all getting routed
separately, whereas if rabbit does the routing there will be just one
copy of the message and the distribution of that message to individual
queues is done very efficiently.


Regards,

Matthias.

Hi Glenn,

Ergeerts Glenn wrote:
No, it's now quite straightforward, though a bit bleeding-edge. The
required features have landed on default, meaning they're release-ready,
and will be in the next major release of rabbit.
I understand completely :-) It's a new kind of thing, I think, and
certainly a new way of thinking about networks---to have a programmable
network, how odd!---to me.

The question is *how much code* would you be embedding in the broker. If
it's not very much, then the amount you save by not having to
reimplement subscription management, expiry etc at the application level
perhaps outweighs the potential cost of migrating to another broker later.
It's not a performance issue: it's that you end up reimplementing
subscription and message distribution logic at the application level,
even though it is already available at the AMQP level. That's why it
makes sense to put the application logic *into* the AMQP level.

Tony