Grokbase Groups Perl qpsmtpd May 2005
FAQ
I like the fact that qmail puts everything into the queue as
separate transactions, even for a single message with multiple recipients
(one sender, one recipient, one message, one connection). However, it is
not an SMTP requirement, and many MTAs will send the same message to
multiple recipients in a single connection, if the MXes are the same for
all.

If I'm going to reject an email, I generally do it during the
connection, so that I don't have to accept responsibility for it. It also
means I don't have to process it further internally, when only some of the
recipients want to accept it.

However, this leaves me with a quandry: If I reject the
connection, I reject it for everyone, including a recipient who may have
wanted it; if I accept the connection (full transaction), I accept it for
everyone, including a recipient who may not have wanted it - the internal
processing mentioned earlier.

If I only allowed a single sender/recipient/message per
connection, then it would be easy to apply the rules and either accept or
reject the connections individually.

<QUESTION> what would be the implication for multiple recipient
connections if each additional recipient generated a 4xx error?</QUESTION>

Would the sending server DATA for the first recipient and
reconnect for the next ones?

Would it depend on the actual error code?

Would it retry, later, and do the same thing again? (I'd hate to
see it keep doing this until a timeout, only to fail completely, when only
some of the recipients wanted to reject the connection.)

Enquiring minds (mine!) want to know. Thanks for your opinions,
theories, and most of all, facts.

Search Discussions

  • David Nicol at May 11, 2005 at 10:27 pm

    On 5/11/05, roger-fake-address@rope.net wrote:
    <QUESTION> what would be the implication for multiple recipient
    connections if each additional recipient generated a 4xx error?</QUESTION>

    Would the sending server DATA for the first recipient and
    reconnect for the next ones?
    That what they're supposed to do. There could be delays, or failure, depending
    on the sender's software.
    Would it depend on the actual error code?

    Would it retry, later, and do the same thing again? (I'd hate to
    see it keep doing this until a timeout, only to fail completely, when only
    some of the recipients wanted to reject the connection.)
    the one that succeeded should not get tried again.


    --
    David L Nicol
    Like a bird on a wire,
    Like a drunk in a midnight choir
    I have tried in my way to be free.
  • Roger-Fake-Address at May 11, 2005 at 10:43 pm

    On Wed, 11 May 2005, David Nicol wrote:
    On 5/11/05, roger-fake-address@rope.net wrote:
    <QUESTION> what would be the implication for multiple recipient
    connections if each additional recipient generated a 4xx error?</QUESTION>

    Would the sending server DATA for the first recipient and
    reconnect for the next ones?
    That what they're supposed to do. There could be delays, or failure, depending
    on the sender's software.
    I presume you are saying that the SMTP standard says that's what's
    supposed to happen.

    My worry is that there might be some major MTA (sendmail?) that
    drops the connection on the temp error, only to keep trying the same thing
    until it times out and bounces. Delays I can live with, but if lots of
    legit email is going to fail, then it's not worth it.

    So, if anyone can add to this (MTAs not acting this way), I'd
    appreciate it.

    Thanks.
  • John Peacock at May 12, 2005 at 2:17 am

    roger-fake-address@rope.net wrote:
    Would the sending server DATA for the first recipient and
    reconnect for the next ones?
    If memory serves, there are MTA's in the wild (old Lotus Notes I think it was)
    which would freak out something fierce in response to 4xx in the RCPT TO: phase.
    On the other hand, some spammers refuse to take the hint, so I've been
    thinking of altering dnsbl to return DENY_DISCONNECT instead of just DENY (or
    rather to add a parameter to switch that behavior on at will).

    The RFC's definitely require that a 4xx in the RCPT TO: phase be treated as a
    temporary error for that recipient only (think disk quotas), so any MTA that
    doesn't requeue is broken. But, accepting for all is more conservative. You
    could add a custom plugin which notes which recipients would have denied the
    message and then remove that RCPT in data_post (or is it post_data?).

    HTH

    John
  • Bryan Scott at May 12, 2005 at 6:28 pm

    Would the sending server DATA for the first recipient and
    reconnect for the next ones?
    The RFC's definitely require that a 4xx in the RCPT TO: phase be treated
    as a temporary error for that recipient only (think disk quotas), so any
    MTA that doesn't requeue is broken.
    Consider also that some recipients may not exist (account deleted, typo
    in original sender's message).

    Why not just send a 5xx for the specific recipient? I don't know why
    you would want a 4xx if you're going force a reconnect just to reject it
    anyway.

    I eventually plan to get to a per-user config (like Gavin built) that
    would allow some users to get everything (spam trap) and others to block
    all of it, and this would certainly help that.

    -- Bryan
  • Roger-fake-address at May 12, 2005 at 6:50 pm
    Thanks to all for all the feedback so far.
    On Thu, 12 May 2005, Bryan Scott wrote:

    Would the sending server DATA for the first recipient and
    reconnect for the next ones?
    The RFC's definitely require that a 4xx in the RCPT TO: phase be treated
    as a temporary error for that recipient only (think disk quotas), so any
    MTA that doesn't requeue is broken.
    Consider also that some recipients may not exist (account deleted, typo
    in original sender's message).

    Why not just send a 5xx for the specific recipient? I don't know why
    you would want a 4xx if you're going force a reconnect just to reject it
    anyway.
    I'm trying to test things, coming from as many different MTAs as
    possible. So far, it looks fairly good. Several split things up into
    separate transactions/connections like qmail. One will RSET and (if I
    reset the recipient counter for each "mail from") end up delivering them
    all in one connection, anyway - "data" for each sender/receiver pair.

    As to 5xx errors, sometimes I need to wait for the "data" before
    issuing a permanent error. Most 5xx errors occur well before that, though,
    and they can certainly be used. I think it's just a matter of what order
    the "rcpt" plugins are found in the config file, as to whether a
    connection will get a temporary or permanent error.

    Anyway, still trying to round up more MTAs to send a test message
    from...
  • John Peacock at May 12, 2005 at 7:03 pm

    Bryan Scott wrote:
    Why not just send a 5xx for the specific recipient? I don't know why
    you would want a 4xx if you're going force a reconnect just to reject it
    anyway.
    Please read the original message again. Roger was asking if most MTA's
    would requeue messages for multiple recipients, so that he could
    serialize incoming mail (i.e. only the first recipient would be accepted
    in any given transaction). And at least my impression was that they
    _should_ by the RFC's but that isn't necessarily a good way to write
    your code.

    One thing I forgot to point out was that if a MTA does requeue 4xx
    recipients, it may use its internal retry intervals as if the *entire*
    message had failed. Consequently, recipient #10 might not get delivered
    for some hours or even days later...

    John
  • Bryan Scott at May 12, 2005 at 7:14 pm
    I read it about 4 times, and ended up addressing this part:

    "If I'm going to reject an email, I generally do it during the
    connection, so that I don't have to accept responsibility for it. It
    also means I don't have to process it further internally, when only some
    of the recipients want to accept it.

    "However, this leaves me with a quandry: If I reject the
    connection, I reject it for everyone, including a recipient who may have
    wanted it; if I accept the connection (full transaction), I accept it
    for everyone, including a recipient who may not have wanted it - the
    internal processing mentioned earlier.

    "If I only allowed a single sender/recipient/message per connection,
    then it would be easy to apply the rules and either accept or reject the
    connections individually."

    Guess I should have stuck to the markup and answered that. ;)

    But his later response explains in more detail his desire to deal with
    as few owners of broken MTA's as possible when splitting up individual
    responses. So question answered, either way.

    -- Bryan

    John Peacock wrote:
    Bryan Scott wrote:
    Why not just send a 5xx for the specific recipient? I don't know why
    you would want a 4xx if you're going force a reconnect just to reject
    it anyway.

    Please read the original message again. Roger was asking if most MTA's
    would requeue messages for multiple recipients, so that he could
    serialize incoming mail (i.e. only the first recipient would be accepted
    in any given transaction). And at least my impression was that they
    _should_ by the RFC's but that isn't necessarily a good way to write
    your code.

    One thing I forgot to point out was that if a MTA does requeue 4xx
    recipients, it may use its internal retry intervals as if the *entire*
    message had failed. Consequently, recipient #10 might not get delivered
    for some hours or even days later...

    John
  • Peter J. Holzer at May 12, 2005 at 7:31 pm

    On 2005-05-12 12:28:09 -0600, Bryan Scott wrote:
    Would the sending server DATA for the first recipient and
    reconnect for the next ones?
    The RFC's definitely require that a 4xx in the RCPT TO: phase be treated
    as a temporary error for that recipient only (think disk quotas), so any
    MTA that doesn't requeue is broken.
    Consider also that some recipients may not exist (account deleted, typo
    in original sender's message).

    Why not just send a 5xx for the specific recipient? I don't know why
    you would want a 4xx if you're going force a reconnect just to reject it
    anyway.
    Because you don't know yet if you are going to reject it. Consider the
    use of a content-filter which is configurable per user.

    You are receiving an incoming SMTP connection:

    EHLO foo.sender.invalid
    MAIL FROM:<someone@sender.invalid>
    RCPT TO:<alice@recipient.invalid>
    RCPT TO:<bob@recipient.invalid>
    RCPT TO:<charlie@recipient.invalid>
    RCPT TO:<debbie@recipient.invalid>
    DATA
    Subject: Enlarge your INBOX!

    We will send you 200MB of spam per month, guaranteed!
    .

    According to their different filter settings, alice and charlie want the
    mail rejected, bob wants it to be marked up as possible spam, and debbie
    wants to receive it unaltered.

    How do you handle that? You don't know that the message has to be
    rejected for some recipients until after you have received the message
    and run the content filter - but then you can only accept or reject it
    for all recipients.

    The easy solution is to accept it for only one recipient at a time:

    EHLO foo.sender.invalid
    250
    MAIL FROM:<someone@sender.invalid>
    250 sender ok
    RCPT TO:<alice@recipient.invalid>
    250 recipient ok
    RCPT TO:<bob@recipient.invalid>
    450 only one at a time, please!
    RCPT TO:<charlie@recipient.invalid>
    450 only one at a time, please!
    RCPT TO:<debbie@recipient.invalid>
    450 only one at a time, please!
    DATA
    354
    Subject: Enlarge your INBOX!

    We will send you 200MB of spam per month, guaranteed!
    .
    550 Spam detected.

    If the client is a legitimate MTA, it will generate a DSN, stating that
    the message could not be delivered to <alice@recipient.invalid> and will
    retry for the other three recipients:

    EHLO foo.sender.invalid
    250
    MAIL FROM:<someone@sender.invalid>
    250 sender ok
    RCPT TO:<bob@recipient.invalid>
    250 sender ok
    RCPT TO:<charlie@recipient.invalid>
    450 only one at a time, please!
    RCPT TO:<debbie@recipient.invalid>
    450 only one at a time, please!
    DATA
    354
    Subject: Enlarge your INBOX!

    We will send you 200MB of spam per month, guaranteed!
    .
    250 Queued!

    Now it has successfully delivered the mail for bob and will retry for
    the other two.

    The main problem with this solution is that many MTAs will retry only after
    the queue interval (typically several minutes to a few hours, but
    sometimes with an exponential backoff). If the list of recipients is
    large, it will take a long time to deliver the mail to all recipients (I
    have seen legitimate mails to 100+ recipients in our logs - assuming a
    queue interval of 30 minutes, that means more than two days until it has
    been delivered to the last recipient).

    A smarter approach (which I think I proposed on this list before) is to
    assume that for most messages with multiple recipients either all
    recipients want it or none. And for those messages which only some want,
    the decision will be the same for each recipient as for other messages
    with the same sender. So we do the following:

    For the first message from this recipient, accept all RCPT commands.
    Run the filter on the message. If it returns OK for all recipients,
    accept the message. If it returns DENY for all recipients, reject the
    message permanently. Otherwise, reject the message temporarily, and
    store the tupels (sender, recipient, set) in a database, where set has
    one value for all recipients which would have accepted the message and
    another value for all those who would have rejected it.

    The next time the the message is received, we can look up the set values
    from the database: We accept all RCPT commands for those recipients
    with the same set value as the first recipient, and reject them
    temporarily for all others. So the chance that content filters will now
    "vote" the same for all recipients is very high.

    This should reduce the number of retries from n-1 to 2 in the typical
    case.
    I eventually plan to get to a per-user config (like Gavin built) that
    would allow some users to get everything (spam trap) and others to block
    all of it, and this would certainly help that.
    At the time of the RCPT command you generally don't have enough
    information to decide whether you want to accept or reject.

    hp

    --
    _ | Peter J. Holzer | Weil wir die materielle Welt nicht so
    _|_) | Sysadmin WSR | wahrnehmen, wie sie ist, sind Sachen wie
    hjp@hjp.at | JPEG, MP3 usw. möglich.
    __/ | http://www.hjp.at/ | -- Heiko Schlenker in drsm.
  • Roger-fake-address at May 12, 2005 at 7:50 pm

    On Thu, 12 May 2005, Peter J. Holzer wrote:

    The main problem with this solution is that many MTAs will retry only after
    the queue interval (typically several minutes to a few hours, but
    sometimes with an exponential backoff). If the list of recipients is
    large, it will take a long time to deliver the mail to all recipients (I
    have seen legitimate mails to 100+ recipients in our logs - assuming a
    queue interval of 30 minutes, that means more than two days until it has
    been delivered to the last recipient).
    So far in my testing, this is only the case with postfix. qmail,
    Intermail, and sendmail split everything into separate queue entries, so
    they are all separate connections in roughly the same time span. With
    iPlanet, the MTA does a RSET between each retry, so it can still all be
    dealt with by a single connection.

    I'm still waiting for a couple of EXCHANGE users for test
    messages, and my Exim user may have a configuration problem - it's being
    routed through the ISP's servers.

    Still missing: Courier. Others?
    A smarter approach (which I think I proposed on this list before) is to
    assume that for most messages with multiple recipients either all
    recipients want it or none. And for those messages which only some want,
    the decision will be the same for each recipient as for other messages
    with the same sender.
    My philosphy, at this point, is to err on the conservative side,
    and allow everything through. This does mean some cleanup after-the-fact,
    though.

    Though I like your suggested implementation, it wouldn't work in
    my case, since legit mail has to get through, period. We can't have the
    lone recipient who wants delivery to be overruled by the majority.
    At the time of the RCPT command you generally don't have enough
    information to decide whether you want to accept or reject.
    It depends on what you do, actually, but generally speaking, you
    are correct. I don't do any DNSBL or other DNS or public DB stuff, and I
    can typically reject 98% of spam connections without doing DATA (over 90%
    of all connections here are spam), so I have very little to deal with
    internally. Still, for my setup, it would be best to be able to deal with
    one sender/receiver/message at a time.
  • Peter J. Holzer at May 12, 2005 at 8:37 pm

    On 2005-05-12 19:50:32 +0000, roger-fake-address@rope.net wrote:
    On Thu, 12 May 2005, Peter J. Holzer wrote:
    The main problem with this solution is that many MTAs will retry only after
    the queue interval (typically several minutes to a few hours, but
    sometimes with an exponential backoff). If the list of recipients is
    large, it will take a long time to deliver the mail to all recipients (I
    have seen legitimate mails to 100+ recipients in our logs - assuming a
    queue interval of 30 minutes, that means more than two days until it has
    been delivered to the last recipient).
    So far in my testing, this is only the case with postfix. qmail,
    Intermail, and sendmail split everything into separate queue entries, so
    they are all separate connections in roughly the same time span.
    Which version of sendmail? I specifically had sendmail in mind when I
    wrote that.

    hp

    --
    _ | Peter J. Holzer | Weil wir die materielle Welt nicht so
    _|_) | Sysadmin WSR | wahrnehmen, wie sie ist, sind Sachen wie
    hjp@hjp.at | JPEG, MP3 usw. möglich.
    __/ | http://www.hjp.at/ | -- Heiko Schlenker in drsm.
  • Roger-fake-address at May 12, 2005 at 9:45 pm

    On Thu, 12 May 2005, Peter J. Holzer wrote:
    On 2005-05-12 19:50:32 +0000, roger-fake-address@rope.net wrote:
    On Thu, 12 May 2005, Peter J. Holzer wrote:
    The main problem with this solution is that many MTAs will retry only after
    the queue interval (typically several minutes to a few hours, but
    sometimes with an exponential backoff). If the list of recipients is
    large, it will take a long time to deliver the mail to all recipients (I
    have seen legitimate mails to 100+ recipients in our logs - assuming a
    queue interval of 30 minutes, that means more than two days until it has
    been delivered to the last recipient).
    So far in my testing, this is only the case with postfix. qmail,
    Intermail, and sendmail split everything into separate queue entries, so
    they are all separate connections in roughly the same time span.
    Which version of sendmail? I specifically had sendmail in mind when I
    wrote that.
    Sendmail (8.12.11)

    Also got back mixed results with Exchange - one of them, directly
    connected to the net, sent them as qmail does, all separately. The other
    sent them all as a single package, via a smarthost (InterMail, which
    previously split them apart when originated on that MTA), which forwarded
    them on as a complete package. So far, I have only gotten the 1st of 3
    from that smarthosted Exchange server.
  • David Sparks at May 12, 2005 at 10:00 pm
    550 Spam detected.
    This almost never does what you think it does or want it to do.

    On a bad day I personally get 100+ bounces caused by people who though
    rejecting spams (and virii) was a good idea.

    Accept and discard.

    ds
  • Gavin Carr at May 12, 2005 at 10:21 pm

    On Thu, May 12, 2005 at 09:31:44PM +0200, Peter J. Holzer wrote:
    On 2005-05-12 12:28:09 -0600, Bryan Scott wrote:
    According to their different filter settings, alice and charlie want the
    mail rejected, bob wants it to be marked up as possible spam, and debbie
    wants to receive it unaltered.

    How do you handle that? You don't know that the message has to be
    rejected for some recipients until after you have received the message
    and run the content filter - but then you can only accept or reject it
    for all recipients.

    The easy solution is to accept it for only one recipient at a time:
    Another variant is to accept it for all recipients who share the same
    config, so that in this case you could accept it for both alice and
    charlie, and denysoft the others.

    This works particularly well if you only allow per-domain configurations,
    rather than per-user ones, since in practice a large proportion of
    multi-rcpt emails going to the same mailserver are going to be all for
    the same domain.

    My denysoft_multi_rcpt plugin supports both individual and shared config
    denials.

    Cheers,
    Gavin

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupqpsmtpd @
categoriesperl
postedMay 11, '05 at 9:30p
activeMay 12, '05 at 10:21p
posts14
users7
websitesmtpd.develooper.com

People

Translate

site design / logo © 2021 Grokbase