Adam,
On 11/10/11 02:48, Adam Rabung wrote:
Hello,
I am trying to implement a work queue in RabbitMQ. I have several
machines that will act as worker pools for consuming my queues. I would
like each job to be processed only once. Some jobs
have environmental requirements, ie must be executed on a machine with
an SSD. Not all worker pools will meet these requirements. A first
approach would be to have a queue for every permutation of requirements:
"No Requirements", "Requires SSD", "Requires Certificate", etc and have
each worker pool subscribe to all queues which it can handle.
That is a sound approach.
A majority of the jobs will have no requirements, so many worker pools
will be underutilized.
As you say, the worker pools would subscribe to all the queues with
combinations of requirements they can handle. That includes the "No
Requirements" queue. The subscriptions can be active *simultaneously*,
so work in the "No Requirements" queue would be round-robin routed to
all workers, jobs in the "Require SSD" queue will be routed to all
workers that have SSDs, etc.

If each worker uses a single channel with a basic.qos prefetch setting
of 1 and subscribes to all the relevant queues on that channel, it will
be fed work items one at a time from the subset of all these queues that
have messages in them. There's some logic at the server that ensures
this is reasonably fair, though you will most likely find that queues
which can be handled by many workers drain faster than others.

Regards,

Matthias.

Search Discussions

Discussion Posts

Previous

Follow ups

Related Discussions

Discussion Navigation
viewthread | post
posts ‹ prev | 3 of 4 | next ›
Discussion Overview
grouprabbitmq-discuss @
categoriesrabbitmq
postedOct 11, '11 at 1:48a
activeOct 11, '11 at 2:52p
posts4
users3
websiterabbitmq.com
irc#rabbitmq

People

Translate

site design / logo © 2022 Grokbase