Hi,

I am developing a python web crawler using RabbitMQ. The server sends list
of urls to be crawled to client and he will send the results after
crawling. My problem is that when client is processing the request and it
dies suddenly all the information abt the request is lost. i.e. rabbitmq
removes the message from queue when client takes a request. I want it to be
relaible. Even if the client dies message should be available in message
queue which can be allocated to some other client. *


I am doing crawling using RPC. I am using pika.blockingConnection. I want
to do the communication asynchronously* instead of blocking connection. It
would be better if you provide an example of Asynchronous communication
using RPC on rabbitmq website.


Thanks for UR help.



Hoping for a positive response on this issue.


--
*
*
*
*
*
*
*---------------------------------------*
Mohammed Zubeir.A.
AkaTsuKi
*---------------------------------------*
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20120214/324529d3/attachment.htm>

Search Discussions

  • Ask Solem at Feb 14, 2012 at 11:12 am

    On 14 Feb 2012, at 11:03, mohammed zubeir wrote:

    Hi,

    I am developing a python web crawler using RabbitMQ. The server sends list of urls to be crawled to client and he will send the results after crawling. My problem is that when client is processing the request and it dies suddenly all the information abt the request is lost. i.e. rabbitmq removes the message from queue when client takes a request. I want it to be relaible. Even if the client dies message should be available in message queue which can be allocated to some other client.


    I am doing crawling using RPC. I am using pika.blockingConnection. I want to do the communication asynchronously instead of blocking connection. It would be better if you provide an example of Asynchronous communication using RPC on rabbitmq website.
    Writing an asynchronous application means that you should choose an async framework first.
    There are many of these to choose from: eventlet, gevent, twisted, tornado and others.
    We don't want the tutorials to bless any of them, and writing manual async code is very cumbersome.

    There are many examples in the Pika source code repository:
    https://github.com/pika/pika/tree/master/examples

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
grouprabbitmq-discuss @
categoriesrabbitmq
postedFeb 14, '12 at 11:03a
activeFeb 14, '12 at 11:12a
posts2
users2
websiterabbitmq.com
irc#rabbitmq

2 users in discussion

Ask Solem: 1 post Mohammed zubeir: 1 post

People

Translate

site design / logo © 2022 Grokbase