Hi folks,

I'm using rabbitmq as an event notification system. In my application
each client will subscribe to a number of exchanges. Exchanges may
come and go. The client cannot be expected to know in advance which
exchanges no longer exist, so they will use exchange.declare(...,
passive=True,...) to find out which ones exist. However,
exchange.declare in passive mode raises a channel exception and kills
the channel if the exchange doesn't exist. Assuming I want to use one
channel per client I'm stuck with ugly, racy code to subscribe to
multiple exchanges. (See the end of the post for the python code to
do it.) And if something goes wrong and kills the channel I have to
go through the whole dance again. This seems crazy.

My question is, what's the right way for one client to get messages
from multiple exchanges? Can one auto-delete queue be bound in
multiple channels, and if so is that the proper approach? Should I be
making N queues & channels to subscribe to N exchanges?

Also, exchange-exchange binding [1] would solve a lot of problems for
me -- does it work or is it just a proposal?

Thanks,
-n8

--
http://n8gray.org

[1]: https://dev.rabbitmq.com/wiki/ExchangeToExchangeBindings

while True:
chan = connection.channel()
found = []
for xc in exchanges:
try:
chan.exchange_declare(exchange=xc, passive=True, ...)
found.append(xc)
except amqp.exceptions.AMQPChannelException, e:
if e.amqp_reply_code == 404:
# Channel got closed. Open a new channel.
chan = connection.channel()
else:
raise e
# Sure hope nothing gets deleted at this point!
qname, _, _ = chan.queue_declare(exclusive=True)
try:
for xc in found:
chan.queue_bind(queue=qname, exchange=xc)
except amqp.exceptions.AMQPChannelException, e:
if e.amqp_reply_code == 404:
# Oops, an exchange got deleted. Start over.
continue
else:
raise e
# No exception raised, so we're done (until something
# else goes wrong & kills the channel...)
break

Search Discussions

  • Paul Jones at Aug 13, 2009 at 9:54 am
    Hi Nathan,

    Would it be possible to detail a bit more about your use case, and thus why
    you don't know when exchanges will come and go? In most scenarios, we'd
    recommend that you just declare the exchanges in a non-passive manner - but
    I take it that there is something special about your use case that prevents
    this.

    If your requirement definitely does stand, there isn't really a cleaner way
    to do this from the queue-binding end. Speculating a bit on your use case,
    if you're trying to bind every exchange in your system to a single queue,
    would it be possible to perform the binding at the point of exchange
    creation (ie, instead of a queue binding to a well-known exchange, bind the
    exchange to a well-known queue).

    Exchange-to-exchange bindings are still just a proposal at the moment I'm
    afraid.

    Thanks,
    Paul.
    On Tue, Aug 11, 2009 at 8:20 PM, Nathan Gray wrote:

    Hi folks,

    I'm using rabbitmq as an event notification system. In my application
    each client will subscribe to a number of exchanges. Exchanges may
    come and go. The client cannot be expected to know in advance which
    exchanges no longer exist, so they will use exchange.declare(...,
    passive=True,...) to find out which ones exist. However,
    exchange.declare in passive mode raises a channel exception and kills
    the channel if the exchange doesn't exist. Assuming I want to use one
    channel per client I'm stuck with ugly, racy code to subscribe to
    multiple exchanges. (See the end of the post for the python code to
    do it.) And if something goes wrong and kills the channel I have to
    go through the whole dance again. This seems crazy.

    My question is, what's the right way for one client to get messages
    from multiple exchanges? Can one auto-delete queue be bound in
    multiple channels, and if so is that the proper approach? Should I be
    making N queues & channels to subscribe to N exchanges?

    Also, exchange-exchange binding [1] would solve a lot of problems for
    me -- does it work or is it just a proposal?

    Thanks,
    -n8

    --
    http://n8gray.org

    [1]: https://dev.rabbitmq.com/wiki/ExchangeToExchangeBindings

    while True:
    chan = connection.channel()
    found = []
    for xc in exchanges:
    try:
    chan.exchange_declare(exchange=xc, passive=True, ...)
    found.append(xc)
    except amqp.exceptions.AMQPChannelException, e:
    if e.amqp_reply_code == 404:
    # Channel got closed. Open a new channel.
    chan = connection.channel()
    else:
    raise e
    # Sure hope nothing gets deleted at this point!
    qname, _, _ = chan.queue_declare(exclusive=True)
    try:
    for xc in found:
    chan.queue_bind(queue=qname, exchange=xc)
    except amqp.exceptions.AMQPChannelException, e:
    if e.amqp_reply_code == 404:
    # Oops, an exchange got deleted. Start over.
    continue
    else:
    raise e
    # No exception raised, so we're done (until something
    # else goes wrong & kills the channel...)
    break

    _______________________________________________
    rabbitmq-discuss mailing list
    rabbitmq-discuss at lists.rabbitmq.com
    http://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
    -------------- next part --------------
    An HTML attachment was scrubbed...
    URL: http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20090813/6e7d2c05/attachment.htm
  • Nathan Gray at Aug 14, 2009 at 5:29 pm
    Hi Paul,

    On Thu, Aug 13, 2009 at 2:54 AM, Paul Joneswrote:
    Hi Nathan,

    Would it be possible to detail a bit more about your use case, and thus why
    you don't know when exchanges will come and go? In most scenarios, we'd
    recommend that you just declare the exchanges in a non-passive manner - but
    I take it that there is something special about your use case that prevents
    this.
    Maybe, maybe not. I've written a multiplayer boardgame. I used
    Google App Engine for the server, which works fine but requires that
    the client poll the server to detect when someone has made a move,
    sent a chat, or otherwise generated an event. (GAE doesn't allow
    long-lived connections or raw socket access.) I'm hoping to use
    rabbitmq as an "event server" to get real-time updates to clients
    without pounding GAE with HTTP requests. When an event is generated
    for a game, GAE publishes to the event server, which tells any
    connected clients of the game to ask GAE for an update.

    There will be many games running, and each game has 2-4 players. The
    natural mapping (IMHO) to AMQP is to have one fanout exchange per
    game. The server on GAE creates this (durable) exchange when a new
    game starts. When the game ends there will be no more events from it,
    so the exchange is deleted. It's possible for a game to end before
    the client connects (the last player can play while the client is
    offline). Each player can play multiple games, thus the requirement
    to connect to multiple exchanges. Clients connect with auto-delete
    queues, since message content is uninteresting -- it's only the timing
    they care about.

    If there's a better way to do this I'm happy to hear it. I could use
    one big direct exchange with player names as routing keys, but that
    would require my publisher to send multiple messages for each game
    event, one for each player in the game. I could use one big topic
    exchange with routing keys like "joe.sally.bob" and subscribe with
    patterns like "#.bob.#", but putting all the messages through one
    topic exchange doesn't seem like it would scale well.

    Declaring exchanges non-passively could probably be made to work, but
    my intention was to do my best to disallow creation of exchanges by
    users in order to prevent abuse. (I'll save my rant about rabbitmq's
    permissions system for another time...)
    If your requirement definitely does stand, there isn't really a cleaner way
    to do this from the queue-binding end. Speculating a bit on your use case,
    if you're trying to bind every exchange in your system to a single queue,
    would it be possible to perform the binding at the point of exchange
    creation (ie, instead of a queue binding to a well-known exchange, bind the
    exchange to a well-known queue).

    Exchange-to-exchange bindings are still just a proposal at the moment I'm
    afraid.
    That's a shame. They're a perfect fit for many-to-many relationships
    like mine. I need one fan-out exchange per game and one fan-in
    exchange per user.

    Thanks,
    -n8

    --
    http://n8gray.org
    On Tue, Aug 11, 2009 at 8:20 PM, Nathan Gray wrote:

    Hi folks,

    I'm using rabbitmq as an event notification system. ?In my application
    each client will subscribe to a number of exchanges. ?Exchanges may
    come and go. ?The client cannot be expected to know in advance which
    exchanges no longer exist, so they will use exchange.declare(...,
    passive=True,...) to find out which ones exist. ?However,
    exchange.declare in passive mode raises a channel exception and kills
    the channel if the exchange doesn't exist. ?Assuming I want to use one
    channel per client I'm stuck with ugly, racy code to subscribe to
    multiple exchanges. ?(See the end of the post for the python code to
    do it.) ?And if something goes wrong and kills the channel I have to
    go through the whole dance again. ?This seems crazy.

    My question is, what's the right way for one client to get messages
    from multiple exchanges? ?Can one auto-delete queue be bound in
    multiple channels, and if so is that the proper approach? ?Should I be
    making N queues & channels to subscribe to N exchanges?

    Also, exchange-exchange binding [1] would solve a lot of problems for
    me -- does it work or is it just a proposal?

    Thanks,
    -n8

    --
    http://n8gray.org

    [1]: https://dev.rabbitmq.com/wiki/ExchangeToExchangeBindings

    while True:
    ? ?chan = connection.channel()
    ? ?found = []
    ? ?for xc in exchanges:
    ? ? ? ?try:
    ? ? ? ? ? ?chan.exchange_declare(exchange=xc, passive=True, ...)
    ? ? ? ? ? ?found.append(xc)
    ? ? ? ?except amqp.exceptions.AMQPChannelException, e:
    ? ? ? ? ? ?if e.amqp_reply_code == 404:
    ? ? ? ? ? ? ? ?# Channel got closed. ?Open a new channel.
    ? ? ? ? ? ? ? ?chan = connection.channel()
    ? ? ? ? ? ?else:
    ? ? ? ? ? ? ? ?raise e
    ? ?# Sure hope nothing gets deleted at this point!
    ? ?qname, _, _ = chan.queue_declare(exclusive=True)
    ? ?try:
    ? ? ? ?for xc in found:
    ? ? ? ? ? ?chan.queue_bind(queue=qname, exchange=xc)
    ? ?except amqp.exceptions.AMQPChannelException, e:
    ? ? ? ?if e.amqp_reply_code == 404:
    ? ? ? ? ? ?# Oops, an exchange got deleted. ?Start over.
    ? ? ? ? ? ?continue
    ? ? ? ?else:
    ? ? ? ? ? ?raise e
    ? ?# No exception raised, so we're done (until something
    ? ?# else goes wrong & kills the channel...)
    ? ?break

    _______________________________________________
    rabbitmq-discuss mailing list
    rabbitmq-discuss at lists.rabbitmq.com
    http://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
  • Paul Jones at Aug 14, 2009 at 5:53 pm
    Hi Nathan,

    On Fri, Aug 14, 2009 at 6:29 PM, Nathan Gray wrote:


    Maybe, maybe not. I've written a multiplayer boardgame. I used
    Google App Engine for the server, which works fine but requires that
    the client poll the server to detect when someone has made a move,
    sent a chat, or otherwise generated an event. (GAE doesn't allow
    long-lived connections or raw socket access.) I'm hoping to use
    rabbitmq as an "event server" to get real-time updates to clients
    without pounding GAE with HTTP requests. When an event is generated
    for a game, GAE publishes to the event server, which tells any
    connected clients of the game to ask GAE for an update.

    There will be many games running, and each game has 2-4 players. The
    natural mapping (IMHO) to AMQP is to have one fanout exchange per
    game. The server on GAE creates this (durable) exchange when a new
    game starts. When the game ends there will be no more events from it,
    so the exchange is deleted. It's possible for a game to end before
    the client connects (the last player can play while the client is
    offline). Each player can play multiple games, thus the requirement
    to connect to multiple exchanges. Clients connect with auto-delete
    queues, since message content is uninteresting -- it's only the timing
    they care about.

    If there's a better way to do this I'm happy to hear it. I could use
    one big direct exchange with player names as routing keys, but that
    would require my publisher to send multiple messages for each game
    event, one for each player in the game. I could use one big topic
    exchange with routing keys like "joe.sally.bob" and subscribe with
    patterns like "#.bob.#", but putting all the messages through one
    topic exchange doesn't seem like it would scale well.
    Assuming that there is a known identifier for each game (database id, guid,
    or equivalent), what about using a single direct exchange, with the routing
    key being set to the game id each time. When a player connects, they
    subscribe to the exchange with all of the routing keys for the games that
    they are interested in.

    As far as scalability is concerned, using a direct exchange with many
    routing keys should hopefully be no worse than many fanout exchanges with no
    routing keys. If you're considering the clustering scenario, exchange
    information is replicated to all nodes - so it isn't like there is a single
    point that all messages need to go through.

    Thanks,
    Paul.
    -------------- next part --------------
    An HTML attachment was scrubbed...
    URL: http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20090814/0d729d01/attachment.htm
  • Nathan Gray at Aug 14, 2009 at 6:17 pm

    On Fri, Aug 14, 2009 at 10:53 AM, Paul Joneswrote:
    Hi Nathan,

    On Fri, Aug 14, 2009 at 6:29 PM, Nathan Gray wrote:

    Maybe, maybe not. ?I've written a multiplayer boardgame. ?I used
    Google App Engine for the server, which works fine but requires that
    the client poll the server to detect when someone has made a move,
    sent a chat, or otherwise generated an event. ?(GAE doesn't allow
    long-lived connections or raw socket access.) ?I'm hoping to use
    rabbitmq as an "event server" to get real-time updates to clients
    without pounding GAE with HTTP requests. ?When an event is generated
    for a game, GAE publishes to the event server, which tells any
    connected clients of the game to ask GAE for an update.

    There will be many games running, and each game has 2-4 players. ?The
    natural mapping (IMHO) to AMQP is to have one fanout exchange per
    game. ?The server on GAE creates this (durable) exchange when a new
    game starts. ?When the game ends there will be no more events from it,
    so the exchange is deleted. ?It's possible for a game to end before
    the client connects (the last player can play while the client is
    offline). ?Each player can play multiple games, thus the requirement
    to connect to multiple exchanges. ?Clients connect with auto-delete
    queues, since message content is uninteresting -- it's only the timing
    they care about.

    If there's a better way to do this I'm happy to hear it. ?I could use
    one big direct exchange with player names as routing keys, but that
    would require my publisher to send multiple messages for each game
    event, one for each player in the game. ?I could use one big topic
    exchange with routing keys like "joe.sally.bob" and subscribe with
    patterns like "#.bob.#", but putting all the messages through one
    topic exchange doesn't seem like it would scale well.
    Assuming that there is a known identifier for each game (database id, guid,
    or equivalent), what about using a single direct exchange, with the routing
    key being set to the game id each time. When a player connects, they
    subscribe to the exchange with all of the routing keys for the games that
    they are interested in.
    Can you subscribe one queue to one exchange with multiple keys, or is
    the idea that each player would use one queue per game?
    As far as scalability is concerned, using a direct exchange with many
    routing keys should hopefully be no worse than many fanout exchanges with no
    routing keys. If you're considering the clustering scenario, exchange
    information is replicated to all nodes - so it isn't like there is a single
    point that all messages need to go through.
    Good to know. I wasn't too worried about using one direct exchange,
    but using one topic exchange seemed like a bad idea.

    Thanks,
    -n8
  • Paul Jones at Aug 14, 2009 at 6:52 pm

    Assuming that there is a known identifier for each game (database id, guid,
    or equivalent), what about using a single direct exchange, with the routing
    key being set to the game id each time. When a player connects, they
    subscribe to the exchange with all of the routing keys for the games that
    they are interested in.
    Can you subscribe one queue to one exchange with multiple keys, or is
    the idea that each player would use one queue per game?
    The one queue can be bound to a single exchange with multiple routing keys;
    so each player would only need to have a single queue. Depending on how you
    access Rabbit, it might also be worth putting the routing key into the
    message body, just to make it a bit more accessible (since some apis don't
    provide the easiest access to the routing key).

    As far as scalability is concerned, using a direct exchange with many
    routing keys should hopefully be no worse than many fanout exchanges with no
    routing keys. If you're considering the clustering scenario, exchange
    information is replicated to all nodes - so it isn't like there is a single
    point that all messages need to go through.
    Good to know. I wasn't too worried about using one direct exchange,
    but using one topic exchange seemed like a bad idea.
    Paul.
    -------------- next part --------------
    An HTML attachment was scrubbed...
    URL: http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20090814/f7819b25/attachment.htm
  • Nathan Gray at Aug 14, 2009 at 7:41 pm
    On Fri, Aug 14, 2009 at 11:52 AM, Paul Joneswrote:
    Assuming that there is a known identifier for each game (database id,
    guid,
    or equivalent), what about using a single direct exchange, with the
    routing
    key being set to the game id each time. When a player connects, they
    subscribe to the exchange with all of the routing keys for the games
    that
    they are interested in.
    Can you subscribe one queue to one exchange with multiple keys, or is
    the idea that each player would use one queue per game?
    The one queue can be bound to a single exchange with multiple routing keys;
    so each player would only need to have a single queue. Depending on how you
    access Rabbit, it might also be worth putting the routing key into the
    message body, just to make it a bit more accessible (since some apis don't
    provide the easiest access to the routing key).
    Cool! That should solve my problem. I think I'm ready to write a bit
    of code. BTW, I'd appreciate any pointers on using the C client code
    from anybody who's had experience with it.

    Thanks Paul, you've been very helpful. :)
    Cheers,
    -n8
  • Nathan Gray at Aug 15, 2009 at 5:33 am

    On Fri, Aug 14, 2009 at 12:41 PM, Nathan Graywrote:
    On Fri, Aug 14, 2009 at 11:52 AM, Paul Joneswrote:
    The one queue can be bound to a single exchange with multiple routing keys;
    so each player would only need to have a single queue. Depending on how you
    access Rabbit, it might also be worth putting the routing key into the
    message body, just to make it a bit more accessible (since some apis don't
    provide the easiest access to the routing key).
    Cool! ?That should solve my problem. ?I think I'm ready to write a bit
    of code. ?BTW, I'd appreciate any pointers on using the C client code
    from anybody who's had experience with it.
    Oops, more questions. Assume I've created a queue, bound it to an
    exchange with some keys, and started a consumer on it. If I want to
    bind on yet another key do I need to cancel the consumer first, or can
    I just keep issuing queue_bind at arbitrary times?

    Also, is there a nicely-formatted non-xml version of amqp0-8.xml somewhere?

    Thanks,
    -n8
  • Paul Jones at Aug 15, 2009 at 7:28 am

    On Sat, Aug 15, 2009 at 6:33 AM, Nathan Gray wrote:
    Oops, more questions. Assume I've created a queue, bound it to an
    exchange with some keys, and started a consumer on it. If I want to
    bind on yet another key do I need to cancel the consumer first, or can
    I just keep issuing queue_bind at arbitrary times?
    No, you can just bind whenever you like.

    Also, is there a nicely-formatted non-xml version of amqp0-8.xml somewhere?
    The AMQP website has a PDF'ed version at
    http://jira.amqp.org/confluence/download/attachments/720900/amqp-xml-doc0-8.pdf?version=1

    Paul.
    -------------- next part --------------
    An HTML attachment was scrubbed...
    URL: http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20090815/0d260422/attachment.htm
  • Nathan Gray at Aug 15, 2009 at 4:09 pm
    On Sat, Aug 15, 2009 at 12:28 AM, Paul Joneswrote:
    On Sat, Aug 15, 2009 at 6:33 AM, Nathan Gray wrote:

    Oops, more questions. ?Assume I've created a queue, bound it to an
    exchange with some keys, and started a consumer on it. ?If I want to
    bind on yet another key do I need to cancel the consumer first, or can
    I just keep issuing queue_bind at arbitrary times?
    No, you can just bind whenever you like.
    Ok, good.
    Also, is there a nicely-formatted non-xml version of amqp0-8.xml
    somewhere?
    The AMQP website has a PDF'ed version at
    http://jira.amqp.org/confluence/download/attachments/720900/amqp-xml-doc0-8.pdf?version=1
    That document seems to be missing most of the interesting information.
    For example here is the complete documentation for simple.get:

    ====
    This method provides a direct access to the messages in a queue using
    a synchronous dialogue that is designed for ? specific types of
    application where synchronous functionality is more important than
    performance.

    The?Get?method?has?the?following?specific?fields:

    This?is?the?Get?pseudo?structure:
    ====

    That's it. No description of the fields or pseudostructure.

    Thanks,
    -n8
  • Matthias Radestock at Aug 15, 2009 at 4:23 pm
    Nathan,

    Nathan Gray wrote:
    On Sat, Aug 15, 2009 at 12:28 AM, Paul Joneswrote:
    On Sat, Aug 15, 2009 at 6:33 AM, Nathan Gray wrote:
    Also, is there a nicely-formatted non-xml version of amqp0-8.xml
    somewhere?
    The AMQP website has a PDF'ed version at
    http://jira.amqp.org/confluence/download/attachments/720900/amqp-xml-doc0-8.pdf?version=1
    That document seems to be missing most of the interesting information.
    Try the 0-9-1 version instead:

    http://jira.amqp.org/confluence/download/attachments/720900/amqp-xml-doc0-9-1.pdf

    That, together with the 0-9-1 spec at

    http://jira.amqp.org/confluence/download/attachments/720900/amqp0-9-1.pdf

    is more readable than the 0-8 version. There are of course differences
    between 0-8 and 0-9-1, but the two version are very close indeed.


    Regards,

    Matthias.
  • Nathan Gray at Aug 19, 2009 at 4:32 pm

    On Sat, Aug 15, 2009 at 9:23 AM, Matthias Radestockwrote:
    Nathan,

    Nathan Gray wrote:
    On Sat, Aug 15, 2009 at 12:28 AM, Paul Joneswrote:
    On Sat, Aug 15, 2009 at 6:33 AM, Nathan Gray wrote:

    Also, is there a nicely-formatted non-xml version of amqp0-8.xml
    somewhere?
    The AMQP website has a PDF'ed version at

    http://jira.amqp.org/confluence/download/attachments/720900/amqp-xml-doc0-8.pdf?version=1
    That document seems to be missing most of the interesting information.
    Try the 0-9-1 version instead:

    http://jira.amqp.org/confluence/download/attachments/720900/amqp-xml-doc0-9-1.pdf

    That, together with the 0-9-1 spec at

    http://jira.amqp.org/confluence/download/attachments/720900/amqp0-9-1.pdf

    is more readable than the 0-8 version. There are of course differences
    between 0-8 and 0-9-1, but the two version are very close indeed.
    Ok, thanks for the tip!

    BTW, I'm happy to report that my prototype is working nicely. Thanks
    for the help everyone!

    Cheers,
    -n8

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
grouprabbitmq-discuss @
categoriesrabbitmq
postedAug 11, '09 at 7:20p
activeAug 19, '09 at 4:32p
posts12
users3
websiterabbitmq.com
irc#rabbitmq

People

Translate

site design / logo © 2022 Grokbase