I am trying to implement a work queue in RabbitMQ. I have several machines
that will act as worker pools for consuming my queues. I would like each
job to be processed only once. Some jobs have environmental requirements,
ie must be executed on a machine with an SSD. Not all worker pools will
meet these requirements. A first approach would be to have a queue for
every permutation of requirements: "No Requirements", "Requires SSD",
"Requires Certificate", etc and have each worker pool subscribe to all
queues which it can handle.

A majority of the jobs will have no requirements, so many worker pools will
be underutilized. To fight this, I am hoping to distribute jobs in a
round-robin fashion to all queues that can handle them. For example, a
no-requirements job can be handled by any worker pool, and therefore should
be distributed equally to all of them. That is, jobs w/ no requirements
will be evenly distributed to "No Requirements", "Requires SSD", "Requires
Certificate", etc.

I tried this with topic exchanges, but of course they will distribute the
job to ALL matching queues, rather than a single arbitrary queue.

Some terrible alternative approaches I have so far:
1. Have producers submit to a single queue, and have a home-grown "load
balancer" consumer on the other end which distributes jobs to appropriate
2. When a worker pool spins up, rather than have each worker consume a
dedicate queue, have at least some of them hop between queues in an attempt
to keep them busy.
3. Hacks with timestamps/modulo arithmetic in the routing key to trick the
exchange into being round robin.

Being new to RabbitMQ, my intuition tells me #1 will be a big performance
problem, #2 will be a performance and maintenance problem, and #3 just won't
end up working. I suspect I'm missing something big here :)

Has anyone successfully load-balanced their workers in a situation like

Any advice much appreciated.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20111010/031ba60c/attachment.htm>

Search Discussions

Discussion Posts

Follow ups

Related Discussions

Discussion Navigation
viewthread | post
posts ‹ prev | 1 of 4 | next ›
Discussion Overview
grouprabbitmq-discuss @
postedOct 11, '11 at 1:48a
activeOct 11, '11 at 2:52p



site design / logo © 2023 Grokbase