At 2:45 PM -0400 9/26/06, Barry Warsaw wrote:
Actually, Mailman does implement a hashed queue of sorts for its
queue runners. Every queue file is assigned a hash and a timestamp,
encoded in the file name. The timestamp is so that qrunners can
handle the files in FIFO order. The hash creates a "hash space" for
when multiple qrunners are used per queue. In that case, each
qrunner is responsible for a slice of the hash space, so that they
can run concurrently without having to deal with expensive and
But you're still using a single directory as an on-disk queue, and
that single directory has to be completely locked, operated on, and
then unlocked every single time you want to create a new file, delete
an old file, or rename a file.
These synchronous meta-data operations are what *kill* the
performance of programs like postfix and sendmail in large sites,
where the cost differential can be thousands, tens of thousands,
hundreds of thousands, or even millions of times when compared to a
true hashed directory scheme.
Do an "ls" on a directory with millions of files on an SGI box
running Irix and XFS (which incorporates it's own hashed directory
scheme internally). Then do the same command on virtually any other
box on virtually any other filesystem. Compare the difference in
There's a reason why postfix ships out-of-the-box with directory
hashing turned on.
Even just two levels of hashed directories with hexadecimal
subdirectory names will mean that you could have over a hundred queue
runners all going at once, with very little likelihood of them
stepping on each others toes. If you locked down each queue runner
to its own subdirectory, you could have 256 of them. Using two
characters of base-32 hashing at each level, you could get 1024 queue
runners with just one level of hashing.
I've done lots of MTA tuning in my time, and directory hashing has to
be the single biggest performance win that I have ever encountered.
Multiple qrunners are a supported configuration, although I
believe their use is rare. In fact, qrunners can contend
for list locks which can reduce their concurrency, but the
OutgoingRunner is one place where write access to the list
data isn't necessary and care was taken so that multiple
OutgoingRunners could maximize their concurrency.
It's good that we allow an application level of concurrency within
the Outgoing queue runners, and that we can avoid file locking. But
we're not getting any real concurrency within the filesystem until we
can physically break the queue down into multiple chunks and operate
on each chunk in a manner that is completely and totally independant
of all the other chunks.
Brad Knowles, <brad at stop.mail-abuse.org>
"Those who would give up essential Liberty, to purchase a little
temporary Safety, deserve neither Liberty nor Safety."
-- Benjamin Franklin (1706-1790), reply of the Pennsylvania
Assembly to the Governor, November 11, 1755
Founding Individual Sponsor of LOPSA. See <http://www.lopsa.org/