[qpsmtpd] Opinion regarding multiple recipients per connection

That what they're supposed to do. There could be delays, or failure, depending
on the sender's software.
the one that succeeded should not get tried again.

I presume you are saying that the SMTP standard says that's what's
supposed to happen.

My worry is that there might be some major MTA (sendmail?) that
drops the connection on the temp error, only to keep trying the same thing
until it times out and bounces. Delays I can live with, but if lots of
legit email is going to fail, then it's not worth it.

So, if anyone can add to this (MTAs not acting this way), I'd
appreciate it.

Thanks.

If memory serves, there are MTA's in the wild (old Lotus Notes I think it was)
which would freak out something fierce in response to 4xx in the RCPT TO: phase.
On the other hand, some spammers refuse to take the hint, so I've been
thinking of altering dnsbl to return DENY_DISCONNECT instead of just DENY (or
rather to add a parameter to switch that behavior on at will).

The RFC's definitely require that a 4xx in the RCPT TO: phase be treated as a
temporary error for that recipient only (think disk quotas), so any MTA that
doesn't requeue is broken. But, accepting for all is more conservative. You
could add a custom plugin which notes which recipients would have denied the
message and then remove that RCPT in data_post (or is it post_data?).

HTH

John

Consider also that some recipients may not exist (account deleted, typo
in original sender's message).

Why not just send a 5xx for the specific recipient? I don't know why
you would want a 4xx if you're going force a reconnect just to reject it
anyway.

I eventually plan to get to a per-user config (like Gavin built) that
would allow some users to get everything (spam trap) and others to block
all of it, and this would certainly help that.

-- Bryan

Thanks to all for all the feedback so far.
I'm trying to test things, coming from as many different MTAs as
possible. So far, it looks fairly good. Several split things up into
separate transactions/connections like qmail. One will RSET and (if I
reset the recipient counter for each "mail from") end up delivering them
all in one connection, anyway - "data" for each sender/receiver pair.

As to 5xx errors, sometimes I need to wait for the "data" before
issuing a permanent error. Most 5xx errors occur well before that, though,
and they can certainly be used. I think it's just a matter of what order
the "rcpt" plugins are found in the config file, as to whether a
connection will get a temporary or permanent error.

Anyway, still trying to round up more MTAs to send a test message
from...

Please read the original message again. Roger was asking if most MTA's
would requeue messages for multiple recipients, so that he could
serialize incoming mail (i.e. only the first recipient would be accepted
in any given transaction). And at least my impression was that they
_should_ by the RFC's but that isn't necessarily a good way to write
your code.

One thing I forgot to point out was that if a MTA does requeue 4xx
recipients, it may use its internal retry intervals as if the *entire*
message had failed. Consequently, recipient #10 might not get delivered
for some hours or even days later...

John

I read it about 4 times, and ended up addressing this part:

"If I'm going to reject an email, I generally do it during the
connection, so that I don't have to accept responsibility for it. It
also means I don't have to process it further internally, when only some
of the recipients want to accept it.

"However, this leaves me with a quandry: If I reject the
connection, I reject it for everyone, including a recipient who may have
wanted it; if I accept the connection (full transaction), I accept it
for everyone, including a recipient who may not have wanted it - the
internal processing mentioned earlier.

"If I only allowed a single sender/recipient/message per connection,
then it would be easy to apply the rules and either accept or reject the
connections individually."

Guess I should have stuck to the markup and answered that. ;)

But his later response explains in more detail his desire to deal with
as few owners of broken MTA's as possible when splitting up individual
responses. So question answered, either way.

-- Bryan

John Peacock wrote:

Because you don't know yet if you are going to reject it. Consider the
use of a content-filter which is configurable per user.

You are receiving an incoming SMTP connection:

EHLO foo.sender.invalid
MAIL FROM:<someone@sender.invalid>
RCPT TO:<alice@recipient.invalid>
RCPT TO:<bob@recipient.invalid>
RCPT TO:<charlie@recipient.invalid>
RCPT TO:<debbie@recipient.invalid>
DATA
Subject: Enlarge your INBOX!

We will send you 200MB of spam per month, guaranteed!
.

According to their different filter settings, alice and charlie want the
mail rejected, bob wants it to be marked up as possible spam, and debbie
wants to receive it unaltered.

How do you handle that? You don't know that the message has to be
rejected for some recipients until after you have received the message
and run the content filter - but then you can only accept or reject it
for all recipients.

The easy solution is to accept it for only one recipient at a time:

EHLO foo.sender.invalid
250
MAIL FROM:<someone@sender.invalid>
250 sender ok
RCPT TO:<alice@recipient.invalid>
250 recipient ok
RCPT TO:<bob@recipient.invalid>
450 only one at a time, please!
RCPT TO:<charlie@recipient.invalid>
450 only one at a time, please!
RCPT TO:<debbie@recipient.invalid>
450 only one at a time, please!
DATA
354
Subject: Enlarge your INBOX!

We will send you 200MB of spam per month, guaranteed!
.
550 Spam detected.

If the client is a legitimate MTA, it will generate a DSN, stating that
the message could not be delivered to <alice@recipient.invalid> and will
retry for the other three recipients:

EHLO foo.sender.invalid
250
MAIL FROM:<someone@sender.invalid>
250 sender ok
RCPT TO:<bob@recipient.invalid>
250 sender ok
RCPT TO:<charlie@recipient.invalid>
450 only one at a time, please!
RCPT TO:<debbie@recipient.invalid>
450 only one at a time, please!
DATA
354
Subject: Enlarge your INBOX!

We will send you 200MB of spam per month, guaranteed!
.
250 Queued!

Now it has successfully delivered the mail for bob and will retry for
the other two.

The main problem with this solution is that many MTAs will retry only after
the queue interval (typically several minutes to a few hours, but
sometimes with an exponential backoff). If the list of recipients is
large, it will take a long time to deliver the mail to all recipients (I
have seen legitimate mails to 100+ recipients in our logs - assuming a
queue interval of 30 minutes, that means more than two days until it has
been delivered to the last recipient).

A smarter approach (which I think I proposed on this list before) is to
assume that for most messages with multiple recipients either all
recipients want it or none. And for those messages which only some want,
the decision will be the same for each recipient as for other messages
with the same sender. So we do the following:

For the first message from this recipient, accept all RCPT commands.
Run the filter on the message. If it returns OK for all recipients,
accept the message. If it returns DENY for all recipients, reject the
message permanently. Otherwise, reject the message temporarily, and
store the tupels (sender, recipient, set) in a database, where set has
one value for all recipients which would have accepted the message and
another value for all those who would have rejected it.

The next time the the message is received, we can look up the set values
from the database: We accept all RCPT commands for those recipients
with the same set value as the first recipient, and reject them
temporarily for all others. So the chance that content filters will now
"vote" the same for all recipients is very high.

This should reduce the number of retries from n-1 to 2 in the typical
case.
At the time of the RCPT command you generally don't have enough
information to decide whether you want to accept or reject.

hp

--
_ | Peter J. Holzer | Weil wir die materielle Welt nicht so
__/ | http://www.hjp.at/ | -- Heiko Schlenker in drsm.

So far in my testing, this is only the case with postfix. qmail,
Intermail, and sendmail split everything into separate queue entries, so
they are all separate connections in roughly the same time span. With
iPlanet, the MTA does a RSET between each retry, so it can still all be
dealt with by a single connection.

I'm still waiting for a couple of EXCHANGE users for test
messages, and my Exim user may have a configuration problem - it's being
routed through the ISP's servers.

Still missing: Courier. Others?
My philosphy, at this point, is to err on the conservative side,
and allow everything through. This does mean some cleanup after-the-fact,
though.

Though I like your suggested implementation, it wouldn't work in
my case, since legit mail has to get through, period. We can't have the
lone recipient who wants delivery to be overruled by the majority.
It depends on what you do, actually, but generally speaking, you
are correct. I don't do any DNSBL or other DNS or public DB stuff, and I
can typically reject 98% of spam connections without doing DATA (over 90%
of all connections here are spam), so I have very little to deal with
internally. Still, for my setup, it would be best to be able to deal with
one sender/receiver/message at a time.

Which version of sendmail? I specifically had sendmail in mind when I
wrote that.

hp

--
_ | Peter J. Holzer | Weil wir die materielle Welt nicht so
__/ | http://www.hjp.at/ | -- Heiko Schlenker in drsm.

Sendmail (8.12.11)

Also got back mixed results with Exchange - one of them, directly
connected to the net, sent them as qmail does, all separately. The other
sent them all as a single package, via a smarthost (InterMail, which
previously split them apart when originated on that MTA), which forwarded
them on as a complete package. So far, I have only gotten the 1st of 3
from that smarthosted Exchange server.

This almost never does what you think it does or want it to do.

On a bad day I personally get 100+ bounces caused by people who though
rejecting spams (and virii) was a good idea.

Accept and discard.

ds

Another variant is to accept it for all recipients who share the same
config, so that in this case you could accept it for both alice and
charlie, and denysoft the others.

This works particularly well if you only allow per-domain configurations,
rather than per-user ones, since in practice a large proportion of
multi-rcpt emails going to the same mailserver are going to be all for
the same domain.

My denysoft_multi_rcpt plugin supports both individual and shared config
denials.

Cheers,
Gavin