FAQ
Hello,

I have configured mailman and we're about to start populating a mailing list
with almost 20,000 members.

Any special considerations for lists this large?

Any advice appreciated.

Thanks,
Hunter

Search Discussions

  • Brad Knowles at Feb 5, 2004 at 8:39 pm

    At 12:24 PM -0800 2004/02/05, Hunter Hillegas wrote:

    I have configured mailman and we're about to start populating a mailing list
    with almost 20,000 members.

    Any special considerations for lists this large?
    See Rob Kolstad's paper "Tuning Sendmail for Large Mailing Lists" at
    <http://www.usenix.org/publications/library/proceedings/lisa97/21.kolstad.html>
    and Strata Chalup's paper "Drinking from the Fire(walls) Hose:
    Another Approach to Very Large Mailing Lists" at
    <http://www.usenix.org/publications/library/proceedings/lisa98/chalup.html>.

    See also the book _Sendmail Performance Tuning_ by Nick Christensen (see
    <http://www.jetcafe.org/~npc/book/sendmail/>). Also note the slides
    from related talks at
    <http://www.shub-internet.org/brad/papers/sendmail-tuning/> and
    <http://www.jetcafe.org/~npc/doc/performance_tuning.pdf>.


    A number of these issues have been addressed by more modern MTAs,
    Mailing List Managers, and configurations shipped with them, but
    there are still lots of good issues raised.

    --
    Brad Knowles, <brad.knowles at skynet.be>

    "They that can give up essential liberty to obtain a little temporary
    safety deserve neither liberty nor safety."
    -Benjamin Franklin, Historical Review of Pennsylvania.

    GCS/IT d+(-) s:+(++)>: a C++(+++)$ UMBSHI++++$ P+>++ L+ !E-(---) W+++(--) N+
    !w--- O- M++ V PS++(+++) PE- Y+(++) PGP>+++ t+(+++) 5++(+++) X++(+++) R+(+++)
    tv+(+++) b+(++++) DI+(++++) D+(++) G+(++++) e++>++++ h--- r---(+++)* z(+++)
  • Jon Carnes at Feb 5, 2004 at 9:28 pm

    On Thu, 2004-02-05 at 15:24, Hunter Hillegas wrote:
    Hello,

    I have configured mailman and we're about to start populating a mailing list
    with almost 20,000 members.

    Any special considerations for lists this large?

    Any advice appreciated.

    Thanks,
    Hunter
    Depending on your Disk sub-system, Processor, and amount of RAM, 20k
    members may be not be a lot for Mailman to handle.

    If however you find that messages spend a long time in the queue before
    being processed (or web-access to the list configuration is very slow to
    load) then you might want to consider putting aside some ram for use as
    a disk. You can copy your ~mailman/lists/... to this RAM drive and
    mount it over the ~mailman/lists (then startup Mailman). Access to the
    lists and configuration will then be much faster.

    Note: the RAM drive *must* be at least twice the size as the data stored
    in it, so that Mailman can make backup copies of the config files while
    doing changes.

    Good Luck!

    Jon Carnes
  • Brad Knowles at Feb 5, 2004 at 10:11 pm

    At 4:28 PM -0500 2004/02/05, Jon Carnes wrote:

    Depending on your Disk sub-system, Processor, and amount of RAM, 20k
    members may be not be a lot for Mailman to handle.
    Yes. Keep in mind that we've had people complain about
    performance with Mailman on some systems with 50k users, and yet
    there are other Mailman systems out there with > 200k users.

    It all depends on your hardware and software configuration.
    Anything can be configured to run dead-dog slow (not uncommon in
    out-of-the-box configurations), while most anything can be tweaked to
    be significantly faster.
    If however you find that messages spend a long time in the queue before
    being processed (or web-access to the list configuration is very slow to
    load) then you might want to consider putting aside some ram for use as
    a disk. You can copy your ~mailman/lists/... to this RAM drive and
    mount it over the ~mailman/lists (then startup Mailman). Access to the
    lists and configuration will then be much faster.
    Hmm. Is this really safe? What happens if there is a crash?
    Are those files safely written to physical media somewhere before the
    dangerous stuff is started? If you had a spontaneous crash, how much
    could you lose?
    Note: the RAM drive *must* be at least twice the size as the data stored
    in it, so that Mailman can make backup copies of the config files while
    doing changes.
    I would say that it should be considerably more than twice, to
    leave some slack for the filesystem code to do it's job properly.

    --
    Brad Knowles, <brad.knowles at skynet.be>

    "They that can give up essential liberty to obtain a little temporary
    safety deserve neither liberty nor safety."
    -Benjamin Franklin, Historical Review of Pennsylvania.

    GCS/IT d+(-) s:+(++)>: a C++(+++)$ UMBSHI++++$ P+>++ L+ !E-(---) W+++(--) N+
    !w--- O- M++ V PS++(+++) PE- Y+(++) PGP>+++ t+(+++) 5++(+++) X++(+++) R+(+++)
    tv+(+++) b+(++++) DI+(++++) D+(++) G+(++++) e++>++++ h--- r---(+++)* z(+++)
  • Bernd Petrovitsch at Feb 5, 2004 at 11:06 pm
    On Thu, 2004-02-05 at 22:28, Jon Carnes wrote:
    [...]
    If however you find that messages spend a long time in the queue before
    being processed (or web-access to the list configuration is very slow to
    load) then you might want to consider putting aside some ram for use as
    a disk. You can copy your ~mailman/lists/... to this RAM drive and
    mount it over the ~mailman/lists (then startup Mailman). Access to the
    lists and configuration will then be much faster.

    Note: the RAM drive *must* be at least twice the size as the data stored
    in it, so that Mailman can make backup copies of the config files while
    doing changes.
    Which raises the question if simply adding the RAM to the system thus
    increasing the disk cache (and not dedicated to just one part of the app
    and - given the above description - wasting 50% of it almost completely,
    not considering fault-tolerance and similar) won't be as good (if not
    batter)??

    Bernd
    --
    Firmix Software GmbH http://www.firmix.at/
    mobil: +43 664 4416156 fax: +43 1 7890849-55
    Embedded Linux Development and Services
  • Jon Carnes at Feb 5, 2004 at 11:18 pm

    On Thu, 2004-02-05 at 18:06, Bernd Petrovitsch wrote:
    On Thu, 2004-02-05 at 22:28, Jon Carnes wrote:
    [...]
    If however you find that messages spend a long time in the queue before
    being processed (or web-access to the list configuration is very slow to
    load) then you might want to consider putting aside some ram for use as
    a disk. You can copy your ~mailman/lists/... to this RAM drive and
    mount it over the ~mailman/lists (then startup Mailman). Access to the
    lists and configuration will then be much faster.

    Note: the RAM drive *must* be at least twice the size as the data stored
    in it, so that Mailman can make backup copies of the config files while
    doing changes.
    Which raises the question if simply adding the RAM to the system thus
    increasing the disk cache (and not dedicated to just one part of the app
    and - given the above description - wasting 50% of it almost completely,
    not considering fault-tolerance and similar) won't be as good (if not
    batter)??

    Bernd
    Alas, there is a whole lot of writing going on that just adding RAM does
    not address. If all Mailman did was read the list from the same process
    each time, then adding RAM would be the best bet.

    Jon Carnes
  • Brad Knowles at Feb 6, 2004 at 12:19 am

    At 12:06 AM +0100 2004/02/06, Bernd Petrovitsch wrote:

    Which raises the question if simply adding the RAM to the system thus
    increasing the disk cache (and not dedicated to just one part of the app
    and - given the above description - wasting 50% of it almost completely,
    not considering fault-tolerance and similar) won't be as good (if not
    batter)??
    Any cache is going to have to hit the disk at some point, and
    disks are many orders of magnitude slower than RAM. Moreover,
    coordinating that effort of periodically hitting the disk would
    really, really slow things down. Using pure RAM really will be more
    effective.

    --
    Brad Knowles, <brad.knowles at skynet.be>

    "They that can give up essential liberty to obtain a little temporary
    safety deserve neither liberty nor safety."
    -Benjamin Franklin, Historical Review of Pennsylvania.

    GCS/IT d+(-) s:+(++)>: a C++(+++)$ UMBSHI++++$ P+>++ L+ !E-(---) W+++(--) N+
    !w--- O- M++ V PS++(+++) PE- Y+(++) PGP>+++ t+(+++) 5++(+++) X++(+++) R+(+++)
    tv+(+++) b+(++++) DI+(++++) D+(++) G+(++++) e++>++++ h--- r---(+++)* z(+++)

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupmailman-users @
categoriespython
postedFeb 5, '04 at 8:24p
activeFeb 6, '04 at 12:19a
posts7
users4
websitelist.org

People

Translate

site design / logo © 2022 Grokbase