I am looking at using RabbitMQ for a prototype project I am doing. I
have a couple of questions:

1. Is there a way to have a queue that deletes messages if there are
no consumers attached? I don't want to use autoDelete, as I want
consumers to be able to attach and detach at any time and have the
queue exist.

2. Is there any way to attach a filter to the end of a queue, so that
when a client wants the next message, the message passes through the
filter and may be discarded, and the next message in the queue
considered for delivery to the client. Ideally, it would be possible
to hook into a Java program so that I can check that the message is
one the program should send. Obviously, this will introduce a
potentially large performance overhead, but that is unlikely to be a
problem in my case, as I am not dealing with a particularly high
throughput.

The use case is when I am receiving results from workers for a task,
where each task may produce many results that are put in an output
queue. In order to make the system fault tolerant to a worker becoming
unresponsive or failing, I would like to be able to filter messages
from the output queue that were sent before the worker failed so they
are not delivered twice.
Buffering the messages on the workers is not an option as the output
is a stream, so we need to be able to send results to the output queue
as soon as they are found, and avoid sending any duplicated results if
a worker is restarted. The order of the results is not important, just
that they all arrive eventually.

Cheers,
Andrew

Search Discussions

  • Benjamin Bennett at Nov 17, 2011 at 1:44 am
    For the first item in our deployed system we used the xmessagettl option
    when the queue is created. Messages expire after a certain amount of time.
    http://www.rabbitmq.com/extensions.html

    For the second one I would do it in the client and I don't know if rabbit
    supports it. Example in our system there are 10 consumer machines reading
    from the same queue, it goes quick when every node can filter out
    messages, if you have to parse through the contents of the message.
    On Nov 16, 2011 6:13 PM, "Andrew Esler" wrote:
    -------------- next part --------------
    An HTML attachment was scrubbed...
    URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20111116/a1efe561/attachment.htm>
  • Andrew Esler at Nov 17, 2011 at 2:49 am
    Thanks for the info about the ttl, it seems like a good solution.

    The reason I want to do filtering server side is then I have easy
    access to the info I need for filtering. If filtering is done client
    side, then all clients have to know about each other and tasks they
    are working on in order to be able to find the information they need
    to make filtering decisions.
    On Nov 17, 2:44?pm, Benjamin Bennett wrote:
    For the first item in our deployed system we used the xmessagettl option
    when the queue is created. ?Messages expire after a certain amount of time.http://www.rabbitmq.com/extensions.html

    For the second one I would do it in the client and I don't know if rabbit
    supports it. ?Example in our system there are 10 consumer machines reading
    from the same queue, ?it goes quick when every node can filter out
    messages, if you have to parse through the contents of the message.
    On Nov 16, 2011 6:13 PM, "Andrew Esler" wrote:

    _______________________________________________
    rabbitmq-discuss mailing list
    rabbitmq-disc... at lists.rabbitmq.comhttps://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
  • Benjamin Bennett at Nov 17, 2011 at 3:01 am
    The application we have is server side clustered application . The
    database server and message server are on the same cluster and we have
    10 application servers behind a h proxy.
    On Wed, Nov 16, 2011 at 8:49 PM, Andrew Esler wrote:
    Thanks for the info about the ttl, it seems like a good solution.

    The reason I want to do filtering server side is then I have easy
    access to the info I need for filtering. If filtering is done client
    side, then all clients have to know about each other and tasks they
    are working on in order to be able to find the information they need
    to make filtering decisions.
    On Nov 17, 2:44?pm, Benjamin Bennett wrote:
    For the first item in our deployed system we used the xmessagettl option
    when the queue is created. ?Messages expire after a certain amount of time.http://www.rabbitmq.com/extensions.html

    For the second one I would do it in the client and I don't know if rabbit
    supports it. ?Example in our system there are 10 consumer machines reading
    from the same queue, ?it goes quick when every node can filter out
    messages, if you have to pacrse through the contents of the message.
    On Nov 16, 2011 6:13 PM, "Andrew Esler" wrote:

    _______________________________________________
    rabbitmq-discuss mailing list
    rabbitmq-disc... at lists.rabbitmq.comhttps://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
    _______________________________________________
    rabbitmq-discuss mailing list
    rabbitmq-discuss at lists.rabbitmq.com
    https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
  • Matthias Radestock at Nov 17, 2011 at 8:39 am
    Andrew,
    On 17/11/11 02:49, Andrew Esler wrote:
    The reason I want to do filtering server side is then I have easy
    access to the info I need for filtering. If filtering is done client
    side, then all clients have to know about each other and tasks they
    are working on in order to be able to find the information they need
    to make filtering decisions.
    It is not obvious from your earlier problem statement why there would be
    a need for clients to know about each other.

    You said "I would like to be able to filter messages from the output
    queue that were sent before the worker failed so they are not delivered
    twice.". So there must be some mechanism in place that detects worker
    failure. Could that not trigger an event that is broadcast to all the
    consumers of the output queue, so that they can subsequently filter out
    the results of the failed worker?

    I wonder though whether this complexity is really necessary. The failure
    detection can never prevent duplicates completely - after all, some of
    the results of the dying worker may already have been processed. So
    given that the system needs to cope with that anyway, is really worth
    putting a secondary mechanism in place? That would only make sense if
    the worker failure frequency is high and the cost of ordinary duplicate
    handling is high.

    Regards,

    Matthias.
  • Andrew Esler at Nov 17, 2011 at 10:27 am
    Matthias,

    Ideally, I dont want my clients to know about each other.

    " So there must be some mechanism in place that detects worker
    failure. Could that not trigger an event that is broadcast to all the
    consumers of the output queue, so that they can subsequently filter
    out
    the results of the failed worker?"

    This is pretty much what I am planning to do now. Each client
    subscribe to a queue, and when a worker fails (expected to be very
    infrequently), then an update is broadcast to all clients saying
    ignore any messages with id of XXXXXX.
    The only problem I can see with this method is that we cant know when
    its safe to delete the notices saying 'ignore anything with id of
    XXXXX' as we cant search the messages in the system to say that there
    are no more messages with id XXXXX. However, this is unlikely to be a
    major problem in practice.

    As for the added complexity, I was only going to do it server side if
    it was possible to do simply. As such, its not a problem.

    Thanks for such a simple and easy to use yet powerful messaging
    system.

    Andrew
    On Nov 17, 9:39?pm, Matthias Radestock wrote:
    Andrew,
    On 17/11/11 02:49, Andrew Esler wrote:

    The reason I want to do filtering server side is then I have easy
    access to the info I need for filtering. If filtering is done client
    side, then all clients have to know about each other and tasks they
    are working on in order to be able to find the information they need
    to make filtering decisions.
    It is not obvious from your earlier problem statement why there would be
    a need for clients to know about each other.

    You said "I would like to be able to filter messages from the output
    queue that were sent before the worker failed so they are not delivered
    twice.". So there must be some mechanism in place that detects worker
    failure. Could that not trigger an event that is broadcast to all the
    consumers of the output queue, so that they can subsequently filter out
    the results of the failed worker?

    I wonder though whether this complexity is really necessary. The failure
    detection can never prevent duplicates completely - after all, some of
    the results of the dying worker may already have been processed. So
    given that the system needs to cope with that anyway, is really worth
    putting a secondary mechanism in place? That would only make sense if
    the worker failure frequency is high and the cost of ordinary duplicate
    handling is high.

    Regards,

    Matthias.
    _______________________________________________
    rabbitmq-discuss mailing list
    rabbitmq-disc... at lists.rabbitmq.comhttps://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
grouprabbitmq-discuss @
categoriesrabbitmq
postedNov 16, '11 at 11:42p
activeNov 17, '11 at 10:27a
posts6
users3
websiterabbitmq.com
irc#rabbitmq

People

Translate

site design / logo © 2022 Grokbase