Hi,

I am developing a python web crawler using RabbitMQ. The server sends
list of urls to be crawled to client and he will send the results after
crawling. My problem is that when client is processing the request and it
dies suddenly all the information abt the request is lost. i.e. rabbitmq
removes the message from queue when client takes a request. I want it to be
relaible. Even if the client dies message should be available in message
queue which can be allocated to some other client. Please reply asap.


Hoping for a positive response on this issue.

Thanks
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20120214/11eb28f9/attachment.htm>

Search Discussions

  • Emile Joubert at Feb 14, 2012 at 10:35 am
    Hi mohammed,
    On 14/02/12 09:22, mohammed zubeir wrote:
    I am developing a python web crawler using RabbitMQ. The server sends
    list of urls to be crawled to client and he will send the results after
    crawling. My problem is that when client is processing the request and
    it dies suddenly all the information abt the request is lost. i.e.
    rabbitmq removes the message from queue when client takes a request. I
    want it to be relaible. Even if the client dies message should be
    available in message queue which can be allocated to some other client.
    Please reply asap.
    The solution in this case is not to auto-ack messages when consuming.
    Your client should acknowledge the message only after all processing is
    complete. If the client dies then all unacknowledged messages will be
    returned to the queue, so that another client can receive them.

    If you also need to guard against broker failure then you should publish
    the messages as persistent, mark the queue as durable and make use of
    publisher confirms.



    -Emile

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
grouprabbitmq-discuss @
categoriesrabbitmq
postedFeb 14, '12 at 9:22a
activeFeb 14, '12 at 10:35a
posts2
users2
websiterabbitmq.com
irc#rabbitmq

2 users in discussion

Emile Joubert: 1 post Mohammed zubeir: 1 post

People

Translate

site design / logo © 2022 Grokbase