[Mailman-Users] Attachment URL in the archive is mangled.

That's URL-encoded line folding that is not being properly interpreted
by the recipient.
Good question. I'm not sure that this problem can be fixed, short of
fixing the clients to properly understand URL-encoded line folding.

But I do hold out hope that Mark Sapiro or one of the other core
developers can prove me wrong.

This appears to be a problem with encoding/decoding of quoted-printable.

What Mailman version is this?

If Mailman is recent enough to have it, is Non-digest options ->
scrub-nondigest Yes or No? (your "The attachments are travelling okay
with the messages" implies no, but just checking.)

Is your list digestable and if so, are the links in the "plain" digest
OK or are they like the archive?

Does this happen with every attachment, or only in some messages?

Hi Mark

In answer to your questions ...

I am using Mailman version: 2.1.5

Subscribers can choose digest if they want.

The link is broken in the digest messages, just like in the archive.

The broken link happens for every PDF file, even though the PDF arrives
with the email. (Doesn't arrive in the digest, just a broken link)

The link is almost okay, it's just the "=" sign that gets inserted that
messes things up.

If it is a Word.doc that gets attached, no URL appears at all in the
archive, not is there even a "scrubbed" message. Also, Word.doc files
don't even arrive with the email.

cheers

MArk Dale


Mark Sapiro wrote:

There have been many changes in Scrubber.py (the module which does
this) since 2.1.5. I will look into it, but I don't know if I will be
able to duplicate the problem.


Because these attachments are removed by content filtering before the
message is archived and delivered.

For what it's worth Mark, I copied a Scrubber.py from a Version 2.19 on
the wild chance that it might do some good - but as was to be expected,
no magic occurred.

Here's hoping you do find a way to duplicate the problem. Upgrading this
installation would be like changing a flat tyre on a moving car.

Understood. Thanks.



cheers
Mark Dale
--

Did you remember to restart Mailman after replacing Scrubber.py?

Hi Mark
(Apologies for late reply, have been away)

Yes, I sure did. And also rebooted the server (and checked that the
qrunner was starting up on boot)
I also noticed that with the new Scrubber.py, emails with attachments
were neither delivered nor archived. Emails without attachments were okay.

Once I returned the original Scrubber.py and restarted, emails with
attachments got delivered and also sent to archive. (Broken link to
attachment still in archived message though).

Mark Dale



Mark Sapiro wrote:

There is probably some incompatibility between the 2.1.9 Scrubber.py
and the rest of your installation that threw an exception on mail with
attachments. Check Mailman's error log and shunt queue.


I have been able to duplicate the problem with the 2.1.5 version of
Scrubber.py. I'll try to come up with a simple patch that you can
apply to fix it. It is fixed in recent Scrubber.py versions.

The issue is as I thought. The scrubbed message is quoted-printable
encoded, but the Content-Transfer-Encoding: header says 8bit and not
quoted-printable so the message is not properly decoded for the
archive.

In my case, I could see other symptoms of this besides just the URL of
the scrubbed attachment. For example, the "-- " separator before my
signature was rendered as "-- =" followed by a blank line.

I think you'd be better off upgrading your Mailman, but if you want to
try a patch to the 2.1.5 Scrubber.py, I think this should do it. It is
only lightly tested, but I think it's OK.

--- Scrubber.py 2008-11-22 20:21:38.375000000 -0800
+++ Scrubberx.py 2008-11-22 20:54:47.250000000 -0800
@@ -326,9 +326,8 @@
# Now join the text and set the payload
sep = _('-------------- next part --------------\n')
del msg['content-type']
- msg.set_payload(sep.join(text), charset)
del msg['content-transfer-encoding']
- msg.add_header('Content-Transfer-Encoding', '8bit')
+ msg.set_payload(sep.join(text), charset)
return msg

Hi Mark

Your patch has worked a treat. Thank you very much for your patience and
generosity. Many of us would be a long way up the proverbial creek if it
weren't for your support.

In summary, for those interested:

Using Mailman 2.1.5 - in the Archives, the URL to attachments was being
malformed by it getting wrapped and an "=" sign inserted at the line break.

This was fixed (albeit lightly tested so far) by replacing, in the
Scrubber.py file, as follows:

--------
OLD CODE
--------
# Now join the text and set the payload
sep = _('-------------- next part --------------\n')
del msg['content-type']
msg.set_payload(sep.join(text), charset)
del msg['content-transfer-encoding']
msg.add_header('Content-Transfer-Encoding', '8bit')
return msg

--------
NEW CODE
--------
# Now join the text and set the payload
sep = _('-------------- next part --------------\n')
del msg['content-type']
del msg['content-transfer-encoding']
msg.set_payload(sep.join(text), charset)
return msg


cheers
Mark



Mark Sapiro wrote:

Yes, it was a shot in the dark replacing 2.1.5 Srubber with a 2.1.9
There are a number of errors in the logfile e.g.

Nov 23 03:05:27 2008 (1800) SHUNTING:
1227409525.3357961+31e6f6226a3b5cbc4a8dadaf3f4dde064947822e

Nov 23 03:07:07 2008 (1800) uncaught archiver exception at filepos: 0

Nov 23 03:07:07 2008 (1800) Uncaught runner exception: 'module' object
has no attribute 'SCRUBBER_USE_ATTACHMENT_FILENAME_EXTENSION'
Yes. Grant Taylor mentioned this. He made reference to format=flow being
the go, rather that quoted=printable. I confess it's something I don't
understand. I had thought f=f was something the email client decided.
Yes. I saw that also, I was pretending not to. ;-)

cheers
Mark Dale

It is not clear if you saw my followup post with the patch. It's at
<http://mail.python.org/pipermail/mailman-users/2008-November/064117.html>
in case you missed it.


Format=flowed and/or quoted-printable encoding are all things decided
by a mail client or in this case by the Python email library.
Scrubber.py is building a new, text/plain message containing the
text/plain parts of the original messages and the notes and URLs for
scrubbed parts. It creates the body by concatenating these parts
separated by the '-------------- next part --------------\n' separator.

It then calls an email message method to set this as the message body,
and the email library decides on the appropriate
Content-Transfer-Encoding and sets the appropriate header. The bug in
Scrubber is that it removed that header and replaced it with
"Content-Transfer-Encoding: 8bit".

The current email library will create quoted-printable encoded parts
where appropriate, but won't create format=flowed which is a good
thing because the pipermail archiver doesn't understand it.

*nod*

If I recall correctly when "quoted=printable" encoding is used, the "="
at the end of the line by its self is an indication to the receiving MUA
that the line is suppose to be unwrapped. I.e. remove the "=" from the
end of the "...attachments/200" line and add the "81119..." line to the end.

However if the quoted=printable MIME encoding is broken, things like
this will happen.

Will you please forward me one of the problem messages (as an attachment
because I need to see the message source) and I will verify this.
*nod* This is as I would expect.

Actually I bet that you are removing the "=" and adding the subsequent
line to the end of the URL, which is what quoted=printable is suppose to
do for you.
I think this is probably another small bug somewhere in Mailman in how
it handles / folds lines of text. Format=flowed (my preference) suffers
equally in older versions of Mailman.
*nod*



Grant. . . .