We have a comet-like server that has four basic tiers of messages:
page-level, user-level, topic, and global. When a page is rendered
that hooks into the comet system, it initially subscribes to the comet
server using a POST with the tiers that it wants, and then it uses GET
to wait on new messages using long-polling. To support this, I've
made a somewhat overly-complicated server that does a lot of internal
queueing, but is ultimately backed by Rabbit.

I've been looking at simplifying this, but first I thought I'd run it
past you guys, since I think my idea may have some components that
rabbit would hate. Basically, what I want to do is make a short-lived
(and non-durable) queue per page (where a single page is dynamic, so
it tends to live for a while), and then have that queue be subscribed
to a "global" exchange, a "user" exchange, and a "topics" exchange. I
see that Rabbit now has a queue-expiry extension, so unused queues
could be automatically garbage collected (I was thinking of basically
having a 10s idle timeout on my temp queues). My concern is whether
rabbit likes having thousands or tens of thousands of queues created
and torn down every few minutes. I seem to recall that it didn't like
that at all in the past, but Rabbit's been progressing like mad
lately, so I figured I'd ask if that's a sane thing to do these days.

Search Discussions

  • Marek Majkowski at Feb 24, 2012 at 1:37 pm

    On Thu, Feb 23, 2012 at 16:54, tsuraan wrote:
    We have a comet-like server that has four basic tiers of messages:
    page-level, user-level, topic, and global. ?When a page is rendered
    that hooks into the comet system, it initially subscribes to the comet
    server using a POST with the tiers that it wants, and then it uses GET
    to wait on new messages using long-polling. ?To support this, I've
    made a somewhat overly-complicated server that does a lot of internal
    queueing, but is ultimately backed by Rabbit.

    I've been looking at simplifying this, but first I thought I'd run it
    past you guys, since I think my idea may have some components that
    rabbit would hate. ?Basically, what I want to do is make a short-lived
    (and non-durable) queue per page (where a single page is dynamic, so
    it tends to live for a while), and then have that queue be subscribed
    to a "global" exchange, a "user" exchange, and a "topics" exchange.
    Does it mean that every page is user specific? Or is "user" exchange
    not for per-user messages but rather global ones?
    I see that Rabbit now has a queue-expiry extension, so unused queues
    could be automatically garbage collected (I was thinking of basically
    having a 10s idle timeout on my temp queues).
    I'm not sure what your scale is, but this setup has a good property -
    it should be relatively easy to shard it. So it's generally reasonable.
    ?My concern is whether
    rabbit likes having thousands or tens of thousands of queues created
    and torn down every few minutes. ?I seem to recall that it didn't like
    that at all in the past, but Rabbit's been progressing like mad
    lately, so I figured I'd ask if that's a sane thing to do these days.
    Like always - you need to prototype and benchmark it :)

    But generally, I would probably go for a different setup.

    I think we can assume with reasonable confidence that number
    of concurrent users on your site is constant. (as opposed to
    dynamic pages AFAIU)

    So why not create queue-per-user? (for users actively browsing).

    When user enters a page - you can subscribe the queue to a per-page
    exchange. When user navigates away - you unsubscribe.

    Of course, this has some limitations - you may want to buffer
    data per-page, and this setup can't do that easily (you would
    need to use redis or something for that).

    Hope that helps,
    Marek
  • Tsuraan at Feb 24, 2012 at 4:13 pm

    Does it mean that every page is user specific? Or is "user" exchange
    not for per-user messages but rather global ones?
    The user exchange is for messages that are addressed to specific
    users, so every web page that a given user has open (on all their
    different browsers, etc) would get all messages addressed to that
    user. Global messages get sent to all pages that are opened in all
    browsers, topic messages get sent to all subscribers to a topic, and
    there are "page" messages that are targeted towards a specific
    instance of a page (i.e. a specific tab that's opened on one of our
    pages).
    Like always - you need to prototype and benchmark it :)
    Yeah, I'm really wondering about hidden things, like "that will work
    for a week, but then mnesia will start on fire" or something to that
    effect :)
    But generally, I would probably go for a different setup.

    I think we can assume with reasonable confidence that number
    of concurrent users on your site is constant. (as opposed to
    dynamic pages AFAIU)

    So why not create queue-per-user? (for users actively browsing).

    When user enters a page - you can subscribe the queue to a per-page
    exchange. When user navigates away - you unsubscribe.
    Ok, I think I'm missing something. Are you saying to create per-page
    exchanges rather than per-page queues, and then subscribe the
    exchanges to queues? Is it cheaper to create exchanges than to create
    queues? Also, binding a queue to an exchange isn't something I was
    aware could be done; does it just use the normal queue.bind command,
    but putting the exchange name where the queue name normally goes, and
    vice versa?
    Hope that helps,
    ?Marek
    It's an idea that I had never thought of, anyhow :) Thanks!
  • Marek Majkowski at Feb 24, 2012 at 4:57 pm

    On Fri, Feb 24, 2012 at 16:13, tsuraan wrote:
    Does it mean that every page is user specific? Or is "user" exchange
    not for per-user messages but rather global ones?
    The user exchange is for messages that are addressed to specific
    users, so every web page that a given user has open (on all their
    different browsers, etc) would get all messages addressed to that
    user. ?Global messages get sent to all pages that are opened in all
    browsers, topic messages get sent to all subscribers to a topic, and
    there are "page" messages that are targeted towards a specific
    instance of a page (i.e. a specific tab that's opened on one of our
    pages).
    Like always - you need to prototype and benchmark it :)
    Yeah, I'm really wondering about hidden things, like "that will work
    for a week, but then mnesia will start on fire" or something to that
    effect :)
    But generally, I would probably go for a different setup.

    I think we can assume with reasonable confidence that number
    of concurrent users on your site is constant. (as opposed to
    dynamic pages AFAIU)

    So why not create queue-per-user? (for users actively browsing).

    When user enters a page - you can subscribe the queue to a per-page
    exchange. When user navigates away - you unsubscribe.
    Ok, I think I'm missing something. ?Are you saying to create per-page
    exchanges rather than per-page queues, and then subscribe the
    exchanges to queues? ?Is it cheaper to create exchanges than to create
    queues?
    Yes, exchanges have no state, while queues actually store data.
    Exchanges - light, Queues - heavy.
    Also, binding a queue to an exchange isn't something I was
    aware could be done; does it just use the normal queue.bind command,
    but putting the exchange name where the queue name normally goes, and
    vice versa?
    I said:
    When user enters a page - you can subscribe the queue to a per-page
    exchange. When user navigates away - you unsubscribe.
    So, I suggest to bind the per-user-queue to an per-page-exchange.
    (or, bind per-user-queue to a all-pages-exchange but with a binding
    key that will catch only messages flying to the particular page
    on which the user is right now)

    Marek

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
grouprabbitmq-discuss @
categoriesrabbitmq
postedFeb 23, '12 at 4:54p
activeFeb 24, '12 at 4:57p
posts4
users2
websiterabbitmq.com
irc#rabbitmq

2 users in discussion

Tsuraan: 2 posts Marek Majkowski: 2 posts

People

Translate

site design / logo © 2022 Grokbase