FAQ
Hello world,

some of my list members have a mail address like recipient at example.com
- nothing special about that. When mail is delivered to their address,
their ISPs mail server rewrite the address internally to
recipient at intern.example.com and try to deliver them to the users
mailbox. If for any reason, e.g. quota, this fails, a NDR/DSN is sent
with the new name:

Received: from mout3.example.com (mout3.example.com [195.4.92.93])
(using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits))
(No client certificate requested)
by mail.incertum.net (Postfix) with ESMTPS
for <invalidlist-bounces at lists.incertum.net>; Tue, 7 Jul 2009 23:09:54 +0200 (CEST)
Received: from [195.4.92.10] (helo=0.mx.example.com)
by mout3.example.com with esmtpa (ID exim) (port 25) (Exim 4.69 #92)
id 1MOHvF-000306-UO
for invalidlist-bounces at lists.incertum.net; Tue, 07 Jul 2009 23:09:53 +0200
Received: from mbox134.example.com ([195.4.93.134]:43037)
by 0.mx.example.com with esmtpa (ID exim) (port 25) (Exim 4.69 #79)
id 1MOHvF-0001Qz-H2
for invalidlist-bounces at lists.incertum.net; Tue, 07 Jul 2009 23:09:53 +0200
Received: from exim by mbox134.example.com with local (Exim 4.69 #91)
id 1MOHvF-0007qp-Dc
for invalidlist-bounces at lists.incertum.net; Tue, 07 Jul 2009 23:09:53 +0200
X-Failed-Recipients: recipient at intern.example.com
Auto-Submitted: auto-replied
From: Mail Delivery System <Mailer-Daemon at example.com>
To: invalidlist-bounces at lists.incertum.net
Subject: Mail delivery failed: returning message to sender
Message-Id: <E1MOHvF-0007qp-Dc at mbox134.example.com>
Date: Tue, 07 Jul 2009 23:09:53 +0200
X-purgate-ID: 149285::1247000993-000022A6-9D72E1E7/0-0/0-18

This message was created automatically by mail delivery software.

A message that you sent could not be delivered to one or more of its
recipients. This is a permanent error. The following address(es) failed:

save to inbox
generated by recipient at intern.example.com
mailbox is full: retry timeout exceeded

Is there any way to rewrite those addresses or to help Mailman
identify those bounces correctly?

And yes, this a pretty stupid thing to do on their ISPs part...


Ciao
Stefan
--
Stefan F?rster http://www.incertum.net/ Public Key: 0xBBE2A9E9
You're here because you know something...you can't explain but you feel it...
that there's something wrong with the world.
--Morpheus, from The Matrix

Search Discussions

  • Mark Sapiro at Jul 7, 2009 at 11:20 pm

    Stefan F?rster wrote:
    some of my list members have a mail address like recipient at example.com
    - nothing special about that. When mail is delivered to their address,
    their ISPs mail server rewrite the address internally to
    recipient at intern.example.com and try to deliver them to the users
    mailbox. If for any reason, e.g. quota, this fails, a NDR/DSN is sent
    with the new name: [...]
    X-Failed-Recipients: recipient at intern.example.com
    Auto-Submitted: auto-replied
    From: Mail Delivery System <Mailer-Daemon at example.com>
    To: invalidlist-bounces at lists.incertum.net
    Subject: Mail delivery failed: returning message to sender
    Message-Id: <E1MOHvF-0007qp-Dc at mbox134.example.com>
    Date: Tue, 07 Jul 2009 23:09:53 +0200
    X-purgate-ID: 149285::1247000993-000022A6-9D72E1E7/0-0/0-18

    This message was created automatically by mail delivery software.

    A message that you sent could not be delivered to one or more of its
    recipients. This is a permanent error. The following address(es) failed:

    save to inbox
    generated by recipient at intern.example.com
    mailbox is full: retry timeout exceeded

    Is there any way to rewrite those addresses or to help Mailman
    identify those bounces correctly?

    And yes, this a pretty stupid thing to do on their ISPs part...

    Mailman has a whole host of heuristic recognizers for non-compliant DSNs,
    but it doesn't try to guess if a non-member bouncing address might be a
    munged version of a member address.

    The 'best' solution for recognizing original recipients of bounced
    messages is Mailman's VERP like capability. This puts the original
    recipient's address in the envelope sender address so the envelope in this
    case is from invalidlist-bounces+recipient=example.com at lists.incertum.net.
    When the bounce is returned to that address, the MTA delivers it to the
    list-bounces address and Mailman recognizes the original recipient as
    recipient at example.com from the envelope recipient of the DSN.

    The cost of this approach is that Mailman has to send an individual
    message to the outgoing MTA for each recipient.

    --
    Mark Sapiro <mark at msapiro.net> The highway is for gamblers,
    San Francisco Bay Area, California better use your sense - B. Dylan
  • Stefan Förster at Jul 8, 2009 at 4:56 pm

    * Mark Sapiro wrote:
    Stefan F?rster wrote:
    generated by recipient at intern.example.com
    mailbox is full: retry timeout exceeded

    Is there any way to rewrite those addresses or to help Mailman
    identify those bounces correctly?

    And yes, this a pretty stupid thing to do on their ISPs part...

    Mailman has a whole host of heuristic recognizers for non-compliant DSNs,
    but it doesn't try to guess if a non-member bouncing address might be a
    munged version of a member address.
    Because it's a pretty stupid thing to do - at least the fact that
    addresses are not rewritten twice.
    The 'best' solution for recognizing original recipients of bounced
    messages is Mailman's VERP like capability. This puts the original
    recipient's address in the envelope sender address so the envelope in this
    case is from invalidlist-bounces+recipient=example.com at lists.incertum.net.
    When the bounce is returned to that address, the MTA delivers it to the
    list-bounces address and Mailman recognizes the original recipient as
    recipient at example.com from the envelope recipient of the DSN.

    The cost of this approach is that Mailman has to send an individual
    message to the outgoing MTA for each recipient.
    Bummer. There is no way to implement that, our mailing list server is
    already suffering from too much traffic, it's quite old hardware,
    after all.


    Thanks anyways
    Stefan
    --
    Stefan F?rster http://www.incertum.net/ Public Key: 0xBBE2A9E9
    FdI #283: whoami - Whoami ist nur was f?r Leute mit Alzheimer! (Begr?ndung von
    Microsoft Deutschland f?r das Entfernen des Befehls aus Windows NT)
  • Brad Knowles at Jul 8, 2009 at 10:30 pm

    on 7/8/09 11:56 AM, Stefan F?rster said:

    Bummer. There is no way to implement that, our mailing list server is
    already suffering from too much traffic, it's quite old hardware,
    after all.
    Take a look at the stuff on "performance tuning" in the FAQ. Even
    really old hardware can perform amazingly well, if it's tuned correctly.
    I guarantee that you would be astonished at what I can achieve with an
    ancient Compaq Pentium-133 laptop with 64MB of RAM and a 10GB hard drive
    (upgraded from 1GB).

    If you've already done as much performance tuning as you can do, then
    obviously you're not going to squeeze any more blood out of that turnip.
    But I would suggest that you familiarize yourself with these sections
    to make sure that you're actually doing as much as you can.

    --
    Brad Knowles <brad at shub-internet.org>
    LinkedIn Profile: <http://tinyurl.com/y8kpxu>
  • Stefan Förster at Jul 8, 2009 at 11:12 pm

    * Brad Knowles wrote:
    on 7/8/09 11:56 AM, Stefan F?rster said:
    Bummer. There is no way to implement that, our mailing list server is
    already suffering from too much traffic, it's quite old hardware,
    after all.
    Take a look at the stuff on "performance tuning" in the FAQ. Even
    really old hardware can perform amazingly well, if it's tuned correctly.
    I guarantee that you would be astonished at what I can achieve with an
    ancient Compaq Pentium-133 laptop with 64MB of RAM and a 10GB hard drive
    (upgraded from 1GB).

    If you've already done as much performance tuning as you can do, then
    obviously you're not going to squeeze any more blood out of that turnip.
    But I would suggest that you familiarize yourself with these sections
    to make sure that you're actually doing as much as you can.
    Thanks for your advice, Brad. The problem is that, due to policy
    reasons, outgoing mail has to pass a content filter, running locally
    on the Mailman box. With VERP...

    I guess I will simply move the list server to another computer (and a
    different network).


    Cheers
    Stefan
  • Brad Knowles at Jul 9, 2009 at 5:08 am

    on 7/8/09 6:12 PM, Stefan F?rster said:

    Thanks for your advice, Brad. The problem is that, due to policy
    reasons, outgoing mail has to pass a content filter, running locally
    on the Mailman box. With VERP...
    Chuq von Rospach wrote some stuff in the FAQ detailing his experience
    with how VERP impacted performance on the systems he was managing. Of
    course, this doesn't necessarily apply directly to your case, but it is
    instructive to read.


    My recollection is that, in his case, he found that without VERP he got
    about two recipients per copy of each message transmitted -- due to the
    fact that some recipients are all on the same system and only one copy
    is sent to that system for multiple recipients, while others get unique
    copies because no one else is subscribed from that system.

    That meant that enabling VERP roughly doubled the number of copies of
    messages that had to be sent (so that each person is guaranteed to get
    their own personal unique copy), but that this didn't actually affect
    the overall performance very much (since so much of e-mail is I/O bound
    and waiting for the system at the other end to respond).

    However, enabling VERP also meant that it was now much, much easier for
    the system to automatically manage bounces (a.k.a., Non-Delivery
    Notices, or NDNs), delivery status notices (DSNs), etc.... This made
    overall management of the system much easier, and greatly reduced the
    amount of work that the system had to try to do to parse the bounces to
    try to figure out which recipient(s) it was in relation to.


    If you throw a content scanning system into that mix, most of the
    content of each of those mailing list messages will be the same, so
    depending on how that content scanning system is configured, it
    shouldn't be that much more expensive to process 100 virtually identical
    messages as it is to process the first message in that group.
    I guess I will simply move the list server to another computer (and a
    different network).
    OTOH, moving the mailing list function to a different server and
    separating that from the content scanning system is also a good idea,
    including lots of other reasons.


    Good luck, and I hope that this works out for you.

    --
    Brad Knowles <brad at shub-internet.org>
    LinkedIn Profile: <http://tinyurl.com/y8kpxu>
  • Stefan Förster at Jul 9, 2009 at 11:00 am

    * Brad Knowles wrote:
    on 7/8/09 6:12 PM, Stefan F?rster said:
    Thanks for your advice, Brad. The problem is that, due to policy
    reasons, outgoing mail has to pass a content filter, running locally
    on the Mailman box. With VERP...
    Chuq von Rospach wrote some stuff in the FAQ detailing his experience
    with how VERP impacted performance on the systems he was managing. Of
    course, this doesn't necessarily apply directly to your case, but it is
    instructive to read.
    To be honest - I never understood where he got his numbers from. Or to
    be more precisely, how he got that lucky (note that I obfuscated the listname):

    # list_members mylist | cut -d@ -f2 | sort | uniq -c | sort -rn | head -3
    1414 gmx.de
    1009 web.de
    404 freenet.de
    # list_members mylist | wc -l
    2966

    For that list, there would be 5 connections from Mailman to the local
    MTA for 2827 recipients, followed by <= 139 connections for the
    remainder of the whole list, i.e. no more than 144 delivery attempts.

    With VERP, there would be 2966 connections from Mailman to the local
    MTA.

    With the content filter set up the way it is, without VERP, the MTA
    would create no more than 144*2 queue files and the content filter
    would in turn create 144 subdirectories with about 3 files each, for a
    total of 720 files and 144 directories created/unlink'd.

    VERP'd, I'd have to create/unlink 14830 files and 2966 directories.

    And yes, I know that the MTA might create more than one file per
    message delivery - I was assuming the "best case". Perhaps I just got
    "unlucky" with the recipient distribution for that particular list -
    if it is of any importance to the lsit archives, I'd gladly
    investigate other lists I'm running. Or perhaps you could provide some
    live data from python.org lists?

    BTW - will MM3 be able to utilize the VERP features offered by many
    modern MTAs?
    I guess I will simply move the list server to another computer (and a
    different network).
    OTOH, moving the mailing list function to a different server and
    separating that from the content scanning system is also a good idea,
    including lots of other reasons.
    My main reason to move the whole installation is mainly that in
    another network, I don't have to scan every outgoing message. After
    all, it's not Mailman's fault that the box is heavily loaded.
    Good luck, and I hope that this works out for you.
    Thank you, Brad.


    Cheers
    Stefan
  • Mark Sapiro at Jul 9, 2009 at 2:22 pm

    Stefan F?rster wrote:
    BTW - will MM3 be able to utilize the VERP features offered by many
    modern MTAs?

    There are patches for Postfix which can probably be adapted for other
    MTAs. See the thread "Patch for use of Postfix VERP support" starting at
    <http://mail.python.org/pipermail/mailman-users/2009-January/064588.html>.

    This seems to be a good idea for future Mailman. I don't know if Barry has
    thought about it for MM3, but I will look at it for 2.2.

    --
    Mark Sapiro <mark at msapiro.net> The highway is for gamblers,
    San Francisco Bay Area, California better use your sense - B. Dylan
  • Barry Warsaw at Jul 9, 2009 at 3:26 pm
    [Adding mailman-developers]
    On Jul 9, 2009, at 10:22 AM, Mark Sapiro wrote:

    Stefan F?rster wrote:
    BTW - will MM3 be able to utilize the VERP features offered by many
    modern MTAs?

    There are patches for Postfix which can probably be adapted for other
    MTAs. See the thread "Patch for use of Postfix VERP support"
    starting at
    <http://mail.python.org/pipermail/mailman-users/2009-January/064588.html
    .
    This seems to be a good idea for future Mailman. I don't know if
    Barry has
    thought about it for MM3, but I will look at it for 2.2.
    I have, and it doesn't look difficult to do, although I really want to
    rework the current delivery architecture so that it's more modular.
    This would also allow different lists to have different delivery
    mechanisms (e.g. bulk vs. individual) and should be more efficient and
    more extensible.

    VERPing is really only one half of the "personalization" or individual
    delivery story. Ideally it's the half that would by default always be
    enabled since it improves the reliability of your memberships. It's
    this part that would benefit greatly from adding MTA-specific
    support. I think Exim supports VERPing without an SMTP extension, and
    I get the impression it can be made to work in Sendmail too.

    The other half is really a "mail merge" operation, where the headers
    and footers (currently, but maybe someday the email body) contains
    placeholders that get filled in with personal information for the
    specific user. I'm not aware of a standard way to do this in any MTA,
    so I think if you want to do this kind of message body
    personalization, you still have to do that in Mailman. I'd like to
    support this better and would consider MTA-specific approaches.

    Mailman also supports something called "full personalization" which
    hacks other headers too. I put this in the category of mail merging
    since I'm not aware of any MTAs providing support for that either.

    -Barry

    -------------- next part --------------
    A non-text attachment was scrubbed...
    Name: PGP.sig
    Type: application/pgp-signature
    Size: 832 bytes
    Desc: This is a digitally signed message part
    URL: <http://mail.python.org/pipermail/mailman-users/attachments/20090709/f6337dd4/attachment.pgp>

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupmailman-users @
categoriespython
postedJul 7, '09 at 9:22p
activeJul 9, '09 at 3:26p
posts9
users4
websitelist.org

People

Translate

site design / logo © 2022 Grokbase