FAQ
Hi,

I just noticed a message in the "bad" directory. I looked at it with
show_qfiles and it looked harmless. Probably it was broken WRT its MIME
encoding, but it wasn't a virus or anything. On a whim I ran unshunt on the
"bad" directory and the message was then delivered to the list.

Basically I'm curious why Mailman didn't consider the message "bad" that
time around. Is unshunt-processing somehow different from normal processing?

Thank, Sebastian
--
.:.Sebastian Hagedorn - RZKR-R1 (Geb?ude 52), Zimmer 18.:.
Zentrum f?r angewandte Informatik - Universit?tsweiter Service RRZK
.:.Universit?t zu K?ln / Cologne University - ? +49-221-478-5587.:.
.:.:.:.Skype: shagedorn.:.:.:.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 186 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/mailman-users/attachments/20080425/974ddd48/attachment.pgp>

Search Discussions

  • Mark Sapiro at Apr 25, 2008 at 4:21 pm

    Sebastian Hagedorn wrote:
    I just noticed a message in the "bad" directory. I looked at it with
    show_qfiles and it looked harmless. Probably it was broken WRT its MIME
    encoding, but it wasn't a virus or anything. On a whim I ran unshunt on
    the "bad" directory and the message was then delivered to the list.

    Basically I'm curious why Mailman didn't consider the message "bad" that
    time around. Is unshunt-processing somehow different from normal
    processing?

    I'm guessing that you are running Mailman 2.1.10 with the patch from
    <http://mail.python.org/pipermail/mailman-users/2008-April/061360.html>,
    since there's little else these days that would put anything in the
    'bad' queue. Did the file have a .psv extension?

    You should really look in the 'error' log to see the message associated
    with preserving the entry. It may have been an actual unparseable
    message due to defective MIME structure, or some possibly transient
    issue. In the first case, I'd expect the same error to occur upon
    unshunting, but not necessarily in the second case. There is nothing
    really in bin/unshunt that would make it succeed without correcting the
    underlying problem.

    The main problem with unshunting a message from the 'bad' queue (other
    than having to change the extension from .psv to .pck or .bak) is that
    true shunted messages have an item in the metadata indicating the
    original queue. Preserved messages in the 'bad' queue don't have this,
    so they will be unshunted to the 'in' queue by default, which may not be
    correct - e.g. they may have come from the bounce queue.

    - --
    Mark Sapiro <mark at msapiro.net> The highway is for gamblers,
    San Francisco Bay Area, California better use your sense - B. Dylan
  • Sebastian Hagedorn at Apr 25, 2008 at 4:34 pm
    Hi Mark,

    thanks for your reply.

    --On 25. April 2008 09:21:12 -0700 Mark Sapiro wrote:
    -----BEGIN PGP SIGNED MESSAGE-----
    Hash: SHA1

    Sebastian Hagedorn wrote:
    I just noticed a message in the "bad" directory. I looked at it with
    show_qfiles and it looked harmless. Probably it was broken WRT its MIME
    encoding, but it wasn't a virus or anything. On a whim I ran unshunt on
    the "bad" directory and the message was then delivered to the list.

    Basically I'm curious why Mailman didn't consider the message "bad" that
    time around. Is unshunt-processing somehow different from normal
    processing?

    I'm guessing that you are running Mailman 2.1.10 with the patch from
    <http://mail.python.org/pipermail/mailman-users/2008-April/061360.html>,
    since there's little else these days that would put anything in the
    'bad' queue.
    No, it's 2.1.9 with two patches you sent me a few weeks ago.
    Did the file have a .psv extension?
    No, it was a .pck file. I found this entry in the vette log:

    (3438) Message discarded, msgid: <c39b437d2808.47ee2e26 at wmich.edu>

    Then I read this comment in Defaults.py:

    # When a message that is unparsable (by the email package) is received, what
    # should we do with it? The most common cause of unparsable messages is
    # broken MIME encapsulation, and the most common cause of that is viruses
    like
    # Nimda. Set this variable to No to discard such messages, or to Yes to
    store
    # them in qfiles/bad subdirectory.
    QRUNNER_SAVE_BAD_MESSAGES = Yes

    So I guess that's what happened, although the log line "message discarded"
    is a bit misleading. BTW, is there a reason why the list a message was
    addressed to isn't logged when messages are discarded?
    You should really look in the 'error' log to see the message associated
    with preserving the entry.
    There was nothing in "error", only that line in "vette".
    It may have been an actual unparseable
    message due to defective MIME structure, or some possibly transient
    issue. In the first case, I'd expect the same error to occur upon
    unshunting, but not necessarily in the second case. There is nothing
    really in bin/unshunt that would make it succeed without correcting the
    underlying problem. OK.
    The main problem with unshunting a message from the 'bad' queue (other
    than having to change the extension from .psv to .pck or .bak) is that
    true shunted messages have an item in the metadata indicating the
    original queue. Preserved messages in the 'bad' queue don't have this,
    so they will be unshunted to the 'in' queue by default, which may not be
    correct - e.g. they may have come from the bounce queue.
    But that should be obvious after looking at it with show_qfiles, right?

    Cheers, Sebastian
    --
    .:.Sebastian Hagedorn - RZKR-R1 (Geb?ude 52), Zimmer 18.:.
    Zentrum f?r angewandte Informatik - Universit?tsweiter Service RRZK
    .:.Universit?t zu K?ln / Cologne University - ? +49-221-478-5587.:.
    .:.:.:.Skype: shagedorn.:.:.:.
    -------------- next part --------------
    A non-text attachment was scrubbed...
    Name: not available
    Type: application/pgp-signature
    Size: 186 bytes
    Desc: not available
    URL: <http://mail.python.org/pipermail/mailman-users/attachments/20080425/5e4feb2b/attachment.pgp>
  • Mark Sapiro at Apr 25, 2008 at 5:16 pm

    Sebastian Hagedorn wrote:
    --On 25. April 2008 09:21:12 -0700 Mark Sapiro wrote:
    I'm guessing that you are running Mailman 2.1.10 with the patch from
    <http://mail.python.org/pipermail/mailman-users/2008-April/061360.html>,
    since there's little else these days that would put anything in the
    'bad' queue.
    No, it's 2.1.9 with two patches you sent me a few weeks ago.
    Did the file have a .psv extension?
    No, it was a .pck file. I found this entry in the vette log:

    (3438) Message discarded, msgid: <c39b437d2808.47ee2e26 at wmich.edu>

    This should have nothing to do with anything in the 'bad' queue. This
    message is logged when a handler decides to discard an incoming post,
    e.g. a post from a non-member in discard_these_nonmembers or any other
    condition with a discard action.

    Then I read this comment in Defaults.py:

    # When a message that is unparsable (by the email package) is received,
    what
    # should we do with it? The most common cause of unparsable messages is
    # broken MIME encapsulation, and the most common cause of that is
    viruses like
    # Nimda. Set this variable to No to discard such messages, or to Yes to
    store
    # them in qfiles/bad subdirectory.
    QRUNNER_SAVE_BAD_MESSAGES = Yes

    So I guess that's what happened, although the log line "message
    discarded" is a bit misleading. BTW, is there a reason why the list a
    message was addressed to isn't logged when messages are discarded?

    That setting in Defaults.py and the qfiles/bad directory have not been
    used by anything other than the bin/update script for a long time (since
    2.0.x I think). I just resurrected them in the 2.1.10 patch I posted
    this week.

    How old was the file you unshunted or how old was the message in it?

    The list isn't logged in the case of a 'qfiles/bad' entry because some
    error prevents the original entry from being properly read so we can't
    get all the nice information like what list it was for.

    The list name isn't logged in the case of the vette log entry, because
    whoever created that code didn't think it would be useful or just didn't
    think to do it (You could find the message-id in the MTA log if you
    really wanted to know the list). Changing long standing log messages is
    not something to be done lightly as it potentially breaks peoples log
    analysis tools.

    You should really look in the 'error' log to see the message associated
    with preserving the entry.
    There was nothing in "error", only that line in "vette".

    This is not surprising given that this is Mailman 2.1.9, as I think that
    means the message must have been quite old.

    The main problem with unshunting a message from the 'bad' queue (other
    than having to change the extension from .psv to .pck or .bak) is that
    true shunted messages have an item in the metadata indicating the
    original queue. Preserved messages in the 'bad' queue don't have this,
    so they will be unshunted to the 'in' queue by default, which may not be
    correct - e.g. they may have come from the bounce queue.
    But that should be obvious after looking at it with show_qfiles, right?

    In that example, yes, but not necessarily if it came from the 'out' or
    'archive' queues. At least in these cases it would be more subtle, and
    you might need to see the metadata which show_qfiles doesn't show you
    (but dumpdb does).

    - --
    Mark Sapiro <mark at msapiro.net> The highway is for gamblers,
    San Francisco Bay Area, California better use your sense - B. Dylan
  • Sebastian Hagedorn at Apr 25, 2008 at 5:36 pm

    --On 25. April 2008 10:16:27 -0700 Mark Sapiro wrote:

    Sebastian Hagedorn wrote:
    No, it was a .pck file. I found this entry in the vette log:

    (3438) Message discarded, msgid: <c39b437d2808.47ee2e26 at wmich.edu>

    This should have nothing to do with anything in the 'bad' queue.
    But it was definitely that message!
    This
    message is logged when a handler decides to discard an incoming post,
    e.g. a post from a non-member in discard_these_nonmembers or any other
    condition with a discard action.
    That's what I thought initially, but how did it end up in "bad" then? Could
    it be caused by this patch to 2.1.9 that I applied on March 8:

    syslog('error', 'Ignoring unparseable message: %s',filebase)
    + self._switchboard.finish(filebase)

    I see this line in "error" shortly after the message that I found in "bad"
    arrived:

    Mar 29 17:01:05 2008 (3438) Ignoring unparseable message:
    1206806465.417665+d798425535212058c480aebf7900add235ac4cb9
    Mar 29 17:01:05 2008 (3438) Ignoring unparseable message:
    1206806465.4255731+c874adc58f17010096988e4e388e31260a56677a

    But how can I tell what messages those really are?
    How old was the file you unshunted or how old was the message in it?
    The message was from March 29. I wouldn't have bothered unshunting it, but
    it seemed as though it might still be relevant. Unfortunately I don't
    remember the timestamp of the file and now it's gone, of course.
    The list name isn't logged in the case of the vette log entry, because
    whoever created that code didn't think it would be useful or just didn't
    think to do it (You could find the message-id in the MTA log if you
    really wanted to know the list).
    That's true.
    Changing long standing log messages is
    not something to be done lightly as it potentially breaks peoples log
    analysis tools.
    I can see that, but still I'd put it on my wishlist for 2.2 or 3.0, because
    it's happened a few times that list admins complained about "lost
    messages". It would've been much easier to see that mail to those lists had
    been discarded if the list name had been logged. The way it is now we first
    had to look at the MTA's log and then search the error log for the
    message-id. And when a single message is sent to more than one list, that
    still doesn't necessarily tell you the whole story ...
    You should really look in the 'error' log to see the message associated
    with preserving the entry.
    There was nothing in "error", only that line in "vette".
    This is not surprising given that this is Mailman 2.1.9, as I think that
    means the message must have been quite old.
    Not that old ...
    --
    .:.Sebastian Hagedorn - RZKR-R1 (Geb?ude 52), Zimmer 18.:.
    Zentrum f?r angewandte Informatik - Universit?tsweiter Service RRZK
    .:.Universit?t zu K?ln / Cologne University - ? +49-221-478-5587.:.
    .:.:.:.Skype: shagedorn.:.:.:.
    -------------- next part --------------
    A non-text attachment was scrubbed...
    Name: not available
    Type: application/pgp-signature
    Size: 186 bytes
    Desc: not available
    URL: <http://mail.python.org/pipermail/mailman-users/attachments/20080425/404ddcb9/attachment.pgp>
  • Mark Sapiro at Apr 25, 2008 at 6:19 pm

    Sebastian Hagedorn wrote:
    --On 25. April 2008 10:16:27 -0700 Mark Sapiro wrote:
    Sebastian Hagedorn wrote:
    No, it was a .pck file. I found this entry in the vette log:

    (3438) Message discarded, msgid: <c39b437d2808.47ee2e26 at wmich.edu>

    This should have nothing to do with anything in the 'bad' queue.
    But it was definitely that message!

    I have been overlooking something. I think it was the same message in
    the 'bad' queue and in the vette log entry. This happens when the list's
    Content filtering -> filter_action is Preserve and the message is
    discarded because content filtering removed all its content.

    If it was accepted when unshunted, the list's content filtering settings
    must have been changed in the interim.

    This
    message is logged when a handler decides to discard an incoming post,
    e.g. a post from a non-member in discard_these_nonmembers or any other
    condition with a discard action.
    That's what I thought initially, but how did it end up in "bad" then?
    Could it be caused by this patch to 2.1.9 that I applied on March 8:

    syslog('error', 'Ignoring unparseable message:
    %s',filebase)
    + self._switchboard.finish(filebase)

    This patch doesn't put anything in 'bad'. In fact, without this patch,
    the original unparseable queue entry is left in the original queue with
    a .bak extension. The patch just removes the .bak file unless you also
    have patches to the finish() method in Mailman/Queue/Switchboard.py.

    I see this line in "error" shortly after the message that I found in
    "bad" arrived:

    Mar 29 17:01:05 2008 (3438) Ignoring unparseable message:
    1206806465.417665+d798425535212058c480aebf7900add235ac4cb9
    Mar 29 17:01:05 2008 (3438) Ignoring unparseable message:
    1206806465.4255731+c874adc58f17010096988e4e388e31260a56677a

    But how can I tell what messages those really are?

    Those are two separate messages/queue entries. In 2.1.9 with the patch
    above, you can't tell anything, because the queue entries are gone. In
    2.1.10 as released last Monday, the error log messages would have been
    for example,

    'Skipping and preserving unparseable message:
    1206806465.4255731+c874adc58f17010096988e4e388e31260a56677a'

    and the queue entry would have been saved as
    1206806465.4255731+c874adc58f17010096988e4e388e31260a56677a.psv in
    qfiles/shunt.

    The patch I posted at
    <http://mail.python.org/pipermail/mailman-users/2008-April/061360.html>
    makes this conditional on QRUNNER_SAVE_BAD_MESSAGES and saves the entry
    in qfiles/bad instead.


    Changing long standing log messages is
    not something to be done lightly as it potentially breaks peoples log
    analysis tools.
    I can see that, but still I'd put it on my wishlist for 2.2 or 3.0,
    because it's happened a few times that list admins complained about
    "lost messages". It would've been much easier to see that mail to those
    lists had been discarded if the list name had been logged. The way it is
    now we first had to look at the MTA's log and then search the error log
    for the message-id. And when a single message is sent to more than one
    list, that still doesn't necessarily tell you the whole story ...

    I understand. There are a number of log messages that could benefit from
    additional information. I've put it on my 2.2 list to review these.

    - --
    Mark Sapiro <mark at msapiro.net> The highway is for gamblers,
    San Francisco Bay Area, California better use your sense - B. Dylan
  • Sebastian Hagedorn at Apr 25, 2008 at 7:43 pm
    --On 25. April 2008 11:19:08 -0700 Mark Sapiro wrote:
    This should have nothing to do with anything in the 'bad' queue.
    But it was definitely that message!

    I have been overlooking something. I think it was the same message in
    the 'bad' queue and in the vette log entry. This happens when the list's
    Content filtering -> filter_action is Preserve and the message is
    discarded because content filtering removed all its content.

    If it was accepted when unshunted, the list's content filtering settings
    must have been changed in the interim.
    Thanks, that makes sense! The list does have content filtering and
    "preserve" as its filter_action.
    --
    .:.Sebastian Hagedorn - RZKR-R1 (Geb?ude 52), Zimmer 18.:.
    Zentrum f?r angewandte Informatik - Universit?tsweiter Service RRZK
    .:.Universit?t zu K?ln / Cologne University - ? +49-221-478-5587.:.
    .:.:.:.Skype: shagedorn.:.:.:.
    -------------- next part --------------
    A non-text attachment was scrubbed...
    Name: not available
    Type: application/pgp-signature
    Size: 186 bytes
    Desc: not available
    URL: <http://mail.python.org/pipermail/mailman-users/attachments/20080425/45334b5e/attachment.pgp>

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupmailman-users @
categoriespython
postedApr 25, '08 at 1:39p
activeApr 25, '08 at 7:43p
posts7
users2
websitelist.org

2 users in discussion

Sebastian Hagedorn: 4 posts Mark Sapiro: 3 posts

People

Translate

site design / logo © 2022 Grokbase