Adam

Would this solve your problem?

http://twitter.com/#!/hylomorphism/status/117667626880221184

Apologies if I misunderstood.

a

On Tue, Oct 11, 2011 at 2:48 AM, Adam Rabung wrote:
Hello,
I am trying to implement a work queue in RabbitMQ. ?I have several machines
that will act as worker pools for consuming my queues. ?I would like each
job to be processed only once. ?Some jobs have?environmental?requirements,
ie must be executed on a machine with an SSD. ?Not all worker pools will
meet these requirements. ?A first approach would be to have a queue for
every permutation of requirements: "No Requirements", "Requires SSD",
"Requires Certificate", etc and have each worker pool subscribe to all
queues which it can handle.
A majority of the jobs will have no requirements, so many worker pools will
be underutilized. ?To fight this, I am hoping to distribute jobs in a
round-robin fashion to all queues that can handle them. ?For example, a
no-requirements job can be handled by any worker pool, and therefore should
be distributed equally to all of them. ?That is, jobs w/ no requirements
will be evenly distributed to?"No Requirements", "Requires SSD", "Requires
Certificate", etc.
I tried this with topic exchanges, but of course they will distribute the
job to ALL matching queues, rather than a single arbitrary queue.
Some terrible alternative approaches I have so far:
1. Have producers submit to a single queue, and have a home-grown "load
balancer" consumer on the other end which distributes jobs to appropriate
queues.
2. When a worker pool spins up, rather than have each worker consume a
dedicate queue, have at least some of them hop between queues in an attempt
to keep them busy.
3. Hacks with timestamps/modulo arithmetic in the routing key to trick the
exchange into being round robin.
Being new to RabbitMQ, my intuition tells me #1 will be a big performance
problem, #2 will be a performance and maintenance problem, and #3 just won't
end up working. ?I suspect I'm missing something big here :)
Has anyone successfully load-balanced their workers in a situation like
this?
Any advice much appreciated.
Adam
_______________________________________________
rabbitmq-discuss mailing list
rabbitmq-discuss at lists.rabbitmq.com
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss

Search Discussions

Discussion Posts

Previous

Follow ups

Related Discussions

Discussion Navigation
viewthread | post
posts ‹ prev | 2 of 4 | next ›
Discussion Overview
grouprabbitmq-discuss @
categoriesrabbitmq
postedOct 11, '11 at 1:48a
activeOct 11, '11 at 2:52p
posts4
users3
websiterabbitmq.com
irc#rabbitmq

People

Translate

site design / logo © 2022 Grokbase