FAQ
The organization I work for has a membership (list) base of in excess of
90,000 users. In the past, we've outsourced our mailings (not spam!
but legitimate newsletter mailings that our users sign up for) to
external companies that have sophisticated bounce handling, etc. etc.

I have been asked if we could bring this task in-house.

I wonder what resources (and perhaps fine-tuning) would be required to
get Mailman to accomodate these sorts of large tasks.

Earlier, I posted a question about High Availability and received only
one response (thank you, though). There has to be a way to scale
Mailman into a large infrastructure (?). Given that Mailman is, in of
itself, an API, there must be some way to hook into the MTA (Postfix, in
our case).

I'd appreciate information from someone who has implemented something
this significant.

The outsourced company we used in the past had sophisticated queue
monitoring tools (php-based) as well as queue management (Postfix used
as the MTA).

Thanks in advance....

Search Discussions

  • Forrest Aldrich at Mar 14, 2005 at 8:36 pm
    It seems that many of the scalability issues could be addressed by
    changing the backend database that Mailman uses... to SQL.

    If we had the ability to store content in a backend SQL database - we
    could further scale this using standard mechanisms.

    Thoughts?
  • John Dennis at Mar 14, 2005 at 8:42 pm

    On Mon, 2005-03-14 at 15:36 -0500, Forrest Aldrich wrote:
    It seems that many of the scalability issues could be addressed by
    changing the backend database that Mailman uses... to SQL.

    If we had the ability to store content in a backend SQL database - we
    could further scale this using standard mechanisms.

    Thoughts?
    I believe this is on the to-do list for Mailman Version 3.0 (a.k.a MM3)
    --
    John Dennis <jdennis at redhat.com>
  • Mark Sapiro at Mar 14, 2005 at 8:57 pm

    John Dennis wrote:
    On Mon, 2005-03-14 at 15:36 -0500, Forrest Aldrich wrote:
    It seems that many of the scalability issues could be addressed by
    changing the backend database that Mailman uses... to SQL.

    If we had the ability to store content in a backend SQL database - we
    could further scale this using standard mechanisms.

    Thoughts?
    I believe this is on the to-do list for Mailman Version 3.0 (a.k.a MM3)
    It is. See http://www.list.org/todo.html

    In the mean time there is a MySQL MemberAdaptor for Mailman 2.1 at
    http://sourceforge.net/tracker/index.php?funcÞtail&aidƒ9386&group_id3&atid00103

    --
    Mark Sapiro <msapiro at value.net> The highway is for gamblers,
    San Francisco Bay Area, California better use your sense - B. Dylan
  • Brad Knowles at Mar 14, 2005 at 10:22 pm

    At 1:09 PM -0500 2005-03-14, Forrest Aldrich wrote:

    I wonder what resources (and perhaps fine-tuning) would be required to
    get Mailman to accomodate these sorts of large tasks.
    See <http://www.python.org/cgi-bin/faqw-mm.py?req=show&file=faq01.015.htp>.

    Also search for "performance" within the FAQ Wizard.
    Earlier, I posted a question about High Availability and received only
    one response (thank you, though). There has to be a way to scale Mailman
    into a large infrastructure (?). Given that Mailman is, in of itself,
    an API, there must be some way to hook into the MTA (Postfix, in our case).
    In terms of doing high-availability, I think most people have
    been splitting the front-end web service from the back-end list
    processing, as well as splitting the MTA services onto separate
    machines.

    If you need the back-end Mailman-only stuff to also be highly
    available, you should be able to mostly do that with NFS, but this
    may run into some problems, and will certainly cost you in terms of
    performance.
    I'd appreciate information from someone who has implemented something
    this significant.
    Most of the information about large-scale mailing lists is
    already found in the FAQ entry mentioned above, or in the archives.
    There's a lot less information in the FAQ Wizard or in the archives
    with regards to high-availability configurations.

    I don't think anyone anywhere has publicly talked about doing
    both high-volume and high-availability, at least not with Mailman.
    The outsourced company we used in the past had sophisticated queue
    monitoring tools (php-based) as well as queue management (Postfix
    used as the MTA).
    Postfix can be a really good MTA for mailing lists, at least for
    handling outbound e-mail.

    If you get into lots of message scanning and having to pass
    through multiple scanning systems (e.g., multiple anti-spam and
    anti-virus scanning systems), then I think sendmail would scale
    better (due to the milter interface), but sendmail would also take
    more work to configure, and more care and feeding to keep going once
    it's configured.

    --
    Brad Knowles, <brad at stop.mail-abuse.org>

    "Those who would give up essential Liberty, to purchase a little
    temporary Safety, deserve neither Liberty nor Safety."

    -- Benjamin Franklin (1706-1790), reply of the Pennsylvania
    Assembly to the Governor, November 11, 1755

    SAGE member since 1995. See <http://www.sage.org/> for more info.
  • Forrest Aldrich at Mar 15, 2005 at 5:05 am

    Brad Knowles wrote:

    [ ... ]
    In terms of doing high-availability, I think most people have been
    splitting the front-end web service from the back-end list processing,
    as well as splitting the MTA services onto separate machines.

    If you need the back-end Mailman-only stuff to also be highly
    available, you should be able to mostly do that with NFS, but this may
    run into some problems, and will certainly cost you in terms of
    performance.
    [ ... ]
    I don't think anyone anywhere has publicly talked about doing both
    high-volume and high-availability, at least not with Mailman.

    I gather this, from all the searching I've done. I've not really
    found any useful information about scaling and
    high-availabilty/redundancy in the FAQ or anywhere else.

    The HA solution will very likely be solved with a stable
    database/backend hook where we can store/retrieve Mailman's information
    -- standard scalablity on that side will be relatively straightforward.

    I've not looked into the MyslqMembership hack yet. Has anyone used this?


    Thanks,
    Forrest
  • Philippe Landau at Mar 15, 2005 at 5:51 am

    Forrest Aldrich wrote:
    I don't think anyone anywhere has publicly talked about doing both
    high-volume and high-availability, at least not with Mailman.
    pair.com has investigated this and put considerable
    resources into these questions.
    their newsgroups are semipublic,
    but i am sure if you ask Kevin Martin
    he can point you in the right direction.
    excellent people there.

    kind regards philippe

    --
    I gather this, from all the searching I've done. I've not really
    found any useful information about scaling and
    high-availabilty/redundancy in the FAQ or anywhere else.

    The HA solution will very likely be solved with a stable
    database/backend hook where we can store/retrieve Mailman's information
    -- standard scalablity on that side will be relatively straightforward.

    I've not looked into the MyslqMembership hack yet. Has anyone used this?


    Thanks,
    Forrest
  • Brad Knowles at Mar 15, 2005 at 9:11 am

    At 12:05 AM -0500 2005-03-15, Forrest Aldrich wrote:

    I gather this, from all the searching I've done. I've not really found
    any useful information about scaling and high-availabilty/redundancy in
    the FAQ or anywhere else.
    There is some scalability information in the FAQ. Go to the FAQ
    Wizard and search for "performance". If you want more details,
    you'll need to go to the archives of the mailing list where people
    talked about things that have been summarized in the FAQ.

    There's also been some information that's been shared privately
    -- if you find the people that have talked publicly about how they
    have gone about handling scalability or HA issues, they may be
    willing/able to share additional details with you that they have not
    made public.
    The HA solution will very likely be solved with a stable database/backend
    hook where we can store/retrieve Mailman's information -- standard
    scalablity on that side will be relatively straightforward.
    There is a MySQL MemberAdapter.
    I've not looked into the MyslqMembership hack yet. Has anyone used this?
    I know they have, because it wouldn't exist otherwise. However,
    I'll let the appropriate speak up for themselves.

    --
    Brad Knowles, <brad at stop.mail-abuse.org>

    "Those who would give up essential Liberty, to purchase a little
    temporary Safety, deserve neither Liberty nor Safety."

    -- Benjamin Franklin (1706-1790), reply of the Pennsylvania
    Assembly to the Governor, November 11, 1755

    SAGE member since 1995. See <http://www.sage.org/> for more info.

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupmailman-users @
categoriespython
postedMar 14, '05 at 6:09p
activeMar 15, '05 at 9:11a
posts8
users5
websitelist.org

People

Translate

site design / logo © 2022 Grokbase