I'm new to RabbitMQ (and AMQP), but I'm considering its use in an application.

I currently have a web application (Rails), and a mobile client (J2ME)
communicating with the web server via HTTP. The mobile client has to
operate in offline mode, so it keeps a persistent queue of messages to
be sent to the server. It periodically tries to send these messages to
the server, and keeps them if there is a network or server error.

I'd like to use RabbitMQ to improve the reliability, and to move to
socket connections instead of HTTP. I'd like to do the following:

1. Have the mobile client send messages to RabbitMQ using the STOMP plugin.
2. Have a Ruby script that consumes the messages on the server.
3. Keep a backup of all the messages sent to the server.

I have some questions about the above:

1 a. The client needs to be authenticated. I already have a simple
authentication system in the web application. Is it possible to use
this for authentication, or would I need to copy the authentication
data to RabbitMQ?

1 b. Is there a simple way to use SSL/TLS for the STOMP connection?

2 a. Are there any recommended practices for running a (Ruby) process
to consume messages in an infinite loop?

2 b. The Ruby script would occasionally be unable to process a
message. This could be because of a temporary lock on the database, an
invalid message from the mobile client, or a bug on the server. The
ideal would be for these messages to be retried a few times, then
moved to a different queue if it still fails. The messages in the
queue could then be inspected manually, and moved back to the original
queue when the bug was fixed.
How would this be accomplished? Does RabbitMQ have something like a
dead letter queue? Or would I have to move the message to a different
queue myself (in the Ruby script), in case of an error?
Note that I can't return any messages to the sender, I want to keep it
on the server in a different queue.

3. What is the best way to create a backup of all the message sent to
the server? Does RabbitMQ automatically do this, or should I write an
additional consumer to create a backup?


Regards,
Ralf Kistner

Search Discussions

  • Jon Brisbin at Sep 25, 2010 at 12:59 pm

    On Sep 25, 2010, at 6:07 AM, Ralf Kistner wrote:

    2 a. Are there any recommended practices for running a (Ruby) process
    to consume messages in an infinite loop?
    Check out the AMQP stuff in Ruby. It uses EventMachine. EM will run until you tell it to stop, which is what you're looking for.
    2 b. The Ruby script would occasionally be unable to process a
    message. This could be because of a temporary lock on the database, an
    invalid message from the mobile client, or a bug on the server. The
    ideal would be for these messages to be retried a few times, then
    moved to a different queue if it still fails. The messages in the
    queue could then be inspected manually, and moved back to the original
    queue when the bug was fixed.
    How would this be accomplished? Does RabbitMQ have something like a
    dead letter queue? Or would I have to move the message to a different
    queue myself (in the Ruby script), in case of an error?
    Note that I can't return any messages to the sender, I want to keep it
    on the server in a different queue.
    Redelivery is something I'm working on in my own application. I use it to update databases at remote locations that might or might not be online. If I get a timeout error, I write my message into a Riak bucket. There is a Quartz-based job that runs every five minutes that checks this retry bucket and simply deletes the messages and dumps them back into the broker. I haven't put it in yet, but I'm planning on implementing a "maxRetries" so that when a message exceeds the retry number, it will be either sent to a notification queue where it gets emailed, or moved to another Riak bucket where I can write an RSS feed over the top of it.
    3. What is the best way to create a backup of all the message sent to
    the server? Does RabbitMQ automatically do this, or should I write an
    additional consumer to create a backup?

    A NoSQL store where you keep the entire message in JSON format seems to me the safest and most performant way to persist messages with enough information to be able to resubmit the message at a later time. I have security tokens in the headers (I don't use logins but tokens) and of course I have to capture the exchange and routing key information. I use JSON data for my messages, so this fits nicely together but you could also base64 encode your data.

    You could use an additional consumer (e.g. on routing key "#") but I let the consumer decide whether to persist a message to Riak or not.

    Thanks!

    J. Brisbin
    http://jbrisbin.com/






    -------------- next part --------------
    An HTML attachment was scrubbed...
    URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20100925/315aeb74/attachment.htm>
  • Ralf Kistner at Oct 3, 2010 at 5:25 pm
    Thanks for the response.

    EventMachine looks perfect for my Ruby script.

    I'm already using MongoDB for other parts of my application, so I'll
    use it to implement a similar approach to what you suggested.
    My messages will mostly be binary data, but that shouldn't make a big
    difference.

    I'll use an additional consumer (on routing key "#") , to ensure that
    the messages are all persisted even when my consumer is broken and
    silently discards messages (this happened to me in the past).

    Ralf

    On Sat, Sep 25, 2010 at 2:59 PM, Jon Brisbin wrote:
    On Sep 25, 2010, at 6:07 AM, Ralf Kistner wrote:

    2 a. Are there any recommended practices for running a (Ruby) process
    to consume messages in an infinite loop?

    Check out the AMQP stuff in Ruby. It uses EventMachine. EM will run until
    you tell it to stop, which is what you're looking for.

    2 b. The Ruby script would occasionally be unable to process a
    message. This could be because of a temporary lock on the database, an
    invalid message from the mobile client, or a bug on the server. The
    ideal would be for these messages to be retried a few times, then
    moved to a different queue if it still fails. The messages in the
    queue could then be inspected manually, and moved back to the original
    queue when the bug was fixed.
    How would this be accomplished? Does RabbitMQ have something like a
    dead letter queue? Or would I have to move the message to a different
    queue myself (in the Ruby script), in case of an error?
    Note that I can't return any messages to the sender, I want to keep it
    on the server in a different queue.

    Redelivery is something I'm working on in my own application. I use it to
    update databases at remote locations that might or might not be online. If I
    get a timeout error, I write my message into a Riak bucket. There is a
    Quartz-based job that runs every five minutes that checks this retry bucket
    and simply deletes the messages and dumps them back into the broker. I
    haven't put it in yet, but I'm planning on implementing a "maxRetries" so
    that when a message exceeds the retry number, it will be either sent to a
    notification queue where it gets emailed, or moved to another Riak bucket
    where I can write an RSS feed over the top of it.

    3. What is the best way to create a backup of all the message sent to
    the server? Does RabbitMQ automatically do this, or should I write an
    additional consumer to create a backup?

    A NoSQL store where you keep the entire message in JSON format seems to me
    the safest and most performant way to persist messages with enough
    information to be able to resubmit the message at a later time. I have
    security tokens in the headers (I don't use logins but tokens) and of course
    I have to capture the exchange and routing key information. I use JSON data
    for my messages, so this fits nicely together but you could also base64
    encode your data.
    You could use an additional consumer (e.g. on routing key "#") but I let the
    consumer decide whether to persist a message to Riak or not.

    Thanks!
    J. Brisbin
    http://jbrisbin.com/




Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
grouprabbitmq-discuss @
categoriesrabbitmq
postedSep 25, '10 at 11:07a
activeOct 3, '10 at 5:25p
posts3
users2
websiterabbitmq.com
irc#rabbitmq

2 users in discussion

Ralf Kistner: 2 posts Jon Brisbin: 1 post

People

Translate

site design / logo © 2022 Grokbase