FAQ
Hi all,

Recently I found a fairly huge amount of files in the shunt queue (around
1500). All I could gather from the error log is that most of them were from an
old bug in parsing weird charsets and the later ones were due to the post and
smtp-failur logfiles being owned by root. So I fixed the perms (had already
upgraded to 2.1.8 a month or so ago) and all was well.

Then I ran bin/unshunt.

Then people started complaining that they were getting a few hundred old
messages. :)

What am I missing here? From what I've been able to dig up on the list
archives the qfiles/shunt directory should only contain messages that never
made it out to any lists due to some error in config, perms or some bug in
mailman itself. But it's looking like these all did somehow make it through
mailman previously and were sent a second time. A few other tidbits:

-looking at some samples, msgids are the same, so it wasn't a user getting a
rejection and then posting again

-the dupes DO NOT show up in the web archive

Any other info I can supply here? Mailman 2.1.8, ruby 1.8.2, FreeBSD 4.7.

Thanks,

Charles

___
Charles Sprickman
NetEng/SysAdmin
Bway.net - New York's Best Internet - www.bway.net
spork at bway.net - 212.655.9344

Search Discussions

  • Mark Sapiro at Jul 31, 2006 at 11:48 pm

    Charles Sprickman wrote:
    Then I ran bin/unshunt.

    Then people started complaining that they were getting a few hundred old
    messages. :)

    There are several possibilities here.

    1) users are complaining that they received messages posted long ago in
    threads long since dead, not necessarily that they previously received
    the same message (message-id).

    2) when the original didn't reach the list, the OP 'fixed' the problem
    and reposted, thus the current message only appears to be a dupe.

    3) if a message is shunted because SMTPDirect.py can't write the 'post'
    log, it has already been sent, but OutgoingRunner.py doesn't know this
    when it catches the exception and shunts the message. So those
    messages which were shunted because of ownership/permission issues on
    the 'post' log probably were duplicated when you ran unshunt


    What am I missing here? From what I've been able to dig up on the list
    archives the qfiles/shunt directory should only contain messages that never
    made it out to any lists due to some error in config, perms or some bug in
    mailman itself.

    This is not always true, as in case 3) above. Thus, you have to use
    judgement in unshunting.

    But it's looking like these all did somehow make it through
    mailman previously and were sent a second time. A few other tidbits:

    -looking at some samples, msgids are the same, so it wasn't a user getting a
    rejection and then posting again

    OK, so that rules out 2).

    -the dupes DO NOT show up in the web archive

    Archiving is a separate process. Depending on where in processing the
    error occurred, lots of possibilities ensue.

    If the error occurs early in IncomingRunner, the post will be neither
    archived nor delivered until the problem is fixed and the message is
    unshunted.

    In the case where the error occurs in delivery under OutgoingRunner,
    the message will have been archived, and when unshunted will be
    processed only by OutgoingRunner and not by ArchRunner, so no
    duplicate archive entry results. But, in this case, it is possible
    that some or all receipients received the original delivery and will
    receive duplicates when the message is unshunted.

    --
    Mark Sapiro <msapiro at value.net> The highway is for gamblers,
    San Francisco Bay Area, California better use your sense - B. Dylan
  • Charles Sprickman at Aug 1, 2006 at 8:27 pm

    On Mon, 31 Jul 2006, Mark Sapiro wrote:

    There are several possibilities here.

    3) if a message is shunted because SMTPDirect.py can't write the 'post'
    log, it has already been sent, but OutgoingRunner.py doesn't know this
    when it catches the exception and shunts the message. So those
    messages which were shunted because of ownership/permission issues on
    the 'post' log probably were duplicated when you ran unshunt
    I think that solves the mystery then.
    What am I missing here? From what I've been able to dig up on the list
    archives the qfiles/shunt directory should only contain messages that never
    made it out to any lists due to some error in config, perms or some bug in
    mailman itself.
    This is not always true, as in case 3) above. Thus, you have to use
    judgement in unshunting.
    Will do. Also set up an snmp monitoring process to watch the shunt queue
    and let me know when things are piling up so I can investigate.
    In the case where the error occurs in delivery under OutgoingRunner,
    the message will have been archived, and when unshunted will be
    processed only by OutgoingRunner and not by ArchRunner, so no
    duplicate archive entry results. But, in this case, it is possible
    that some or all receipients received the original delivery and will
    receive duplicates when the message is unshunted.
    Excellent info. I put a question regarding this in the FAQ, so now I'll
    answer it.

    Thanks for the insight Mark, very much appreciated.

    Charles

    --
    Mark Sapiro <msapiro at value.net> The highway is for gamblers,
    San Francisco Bay Area, California better use your sense - B. Dylan

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupmailman-users @
categoriespython
postedJul 31, '06 at 11:08p
activeAug 1, '06 at 8:27p
posts3
users2
websitelist.org

2 users in discussion

Charles Sprickman: 2 posts Mark Sapiro: 1 post

People

Translate

site design / logo © 2022 Grokbase