FAQ
It seems I am getting duplicate messages from mailman v2.1.5 when senders send both TO and CC to the list. For some reason no one else in the list is acknowledging this issue so I look like an idiot complaining about it, but I'm seeing several instances of it per day.

I do not get dupes of all mails otherwise I'd suspect double subscription. The Avoid Duplicates flag is set. This only happens for mails that have the list in the CC - I understand the dupe flag is for when people use ReplyAll and the recipient is in both the list of subscribers and the CC for the mail. I believe the code is supposed to filter duplicate Message-IDs to a given recipient no matter how those dupes are generated, but I'm getting them. Any ideas?

Thank you kindly.

Search Discussions

  • Brad Knowles at Apr 30, 2006 at 12:45 pm

    At 8:28 AM +0000 2006-04-30, Tony G wrote:

    I do not get dupes of all mails otherwise I'd suspect double subscription.
    The Avoid Duplicates flag is set. This only happens for mails that have
    the list in the CC - I understand the dupe flag is for when people use
    ReplyAll and the recipient is in both the list of subscribers and the CC
    for the mail.
    It's not just the recipient, but the exact same recipient
    address. So, if you're subscribed to the list with one address, and
    they cc: a different address of yours, then the system has no way to
    tell that these are both the same person.
    I believe the code is supposed to filter duplicate
    Message-IDs to a given recipient no matter how those dupes are generated,
    but I'm getting them. Any ideas?
    No, the code doesn't work at the Message-ID level. It works at
    the level of the listed recipients in the To: and CC: headers.

    If you want filtering done at the Message-ID level, then you need
    to apply that on your end -- either use a program like procmail to
    filter all your incoming mail, or set up your MUA to discard
    duplicates, etc....

    There's nothing more that Mailman can do to help you.

    --
    Brad Knowles, <brad at stop.mail-abuse.org>

    "Those who would give up essential Liberty, to purchase a little
    temporary Safety, deserve neither Liberty nor Safety."

    -- Benjamin Franklin (1706-1790), reply of the Pennsylvania
    Assembly to the Governor, November 11, 1755

    LOPSA member since December 2005. See <http://www.lopsa.org/>.
  • Tony G at Apr 30, 2006 at 11:03 pm

    Brad Knowles wrote:
    Tony G wrote:
    I do not get dupes of all mails otherwise I'd suspect double
    subscription. The Avoid Duplicates flag is set. This only happens
    for mails that have the list in the CC - I understand the dupe
    flag is for when people use ReplyAll and the recipient is in both
    the list of subscribers and the CC for the mail.
    It's not just the recipient, but the exact same recipient
    address. So, if you're subscribed to the list with one address, and
    they cc: a different address of yours, then the system has no way to
    tell that these are both the same person.
    No one on this list has any other address of mine and the dupes aren't to
    threads where I have posted. Checking with individuals and the headers I
    am not being sent mail from anywhere except mailman.

    I believe the code is supposed to filter duplicate
    Message-IDs to a given recipient no matter how those dupes are
    generated, but I'm getting them. Any ideas?
    No, the code doesn't work at the Message-ID level. It works at
    the level of the listed recipients in the To: and CC: headers.
    I was mislead by discussions like these:
    <http://mail.python.org/pipermail/mailman-developers/2001-August/009467.htm
    l>
    <http://mail.python.org/pipermail/mailman-developers/2002-March/010911.html
    >
    And this code referred to in one of those threads:
    <http://www.opensource.apple.com/darwinsource/Current/mailman-117/mailman/M
    ailman/Handlers/AvoidDuplicates.py>

    If you want filtering done at the Message-ID level, then you need
    to apply that on your end -- either use a program like procmail to
    filter all your incoming mail, or set up your MUA to discard
    duplicates, etc....

    There's nothing more that Mailman can do to help you.
    I am filtering the dupes. What confuses me is that they are only coming
    from one list that uses mailman. You sent TO me and CC'd the list but I
    only got one copy as one would hope. It's for this reason that I suspect
    that this other list (mono-list at ximian) have some broken version of
    Mailman. Of course they put the blame back on my server or MUA. All I'm
    trying to do here is diagnose and help people to fix open source software.
    Isn't that why we're all here?

    Thanks.
  • Mark Sapiro at Apr 30, 2006 at 11:22 pm

    Tony G wrote:
    I was mislead by discussions like these:
    <http://mail.python.org/pipermail/mailman-developers/2001-August/009467.html>
    <http://mail.python.org/pipermail/mailman-developers/2002-March/010911.html

    Yes, but these are discussions of a suggested patch that isn't in the
    current distribution.


    This is presumably Apple's code, not ours, but it does essentially what
    our AvoidDuplicates handler does which is not send to any list member
    who has requested to avoid duplicates and who's address is in the To:
    or Cc: of the message. This doesn't avoid sending two messages with
    the same message id to the same person. It only avoids sending a post
    from the list to the user if the user might have received it directly
    and we can determine that.


    More in my next reply.

    --
    Mark Sapiro <msapiro at value.net> The highway is for gamblers,
    San Francisco Bay Area, California better use your sense - B. Dylan
  • Mark Sapiro at Apr 30, 2006 at 2:13 pm

    Tony G wrote:
    It seems I am getting duplicate messages from mailman v2.1.5 when senders send both TO and CC to the list. For some reason no one else in the list is acknowledging this issue so I look like an idiot complaining about it, but I'm seeing several instances of it per day.
    Brad has already provided good information in his reply. I just want to
    add that it looks to me like in this case, the sender's MUA is
    actually giving the list address twice (once for To: and once for Cc:)
    to the MTA, and the MTA is not dropping the duplicate recipient. Thus,
    two copies of the post are actually being sent to the list. Do the
    duplicates appear twice in archives and/or digests? (I think the
    duplicate may be dropped from the archives if the message-id is the
    same, but I'm not sure).

    Other subscribers may not see this because their incoming MTA/MUA may
    drop duplicate message-ids.

    --
    Mark Sapiro <msapiro at value.net> The highway is for gamblers,
    San Francisco Bay Area, California better use your sense - B. Dylan
  • Tony G at Apr 30, 2006 at 11:15 pm

    Mark Sapiro wrote:
    Brad has already provided good information in his reply. I just want
    to add that it looks to me like in this case, the sender's MUA is
    actually giving the list address twice (once for To: and once for Cc:)
    to the MTA, and the MTA is not dropping the duplicate recipient. Thus,
    two copies of the post are actually being sent to the list.
    As I mentioned in my response to Brad, I'm not even involved in the
    discussions where I'm getting the dupes. This would be an obvious source
    of duplication. I'm just another quiet recipient to the list (hehe, until
    now anyway) and there's no reason for me to be explicitly identified in the
    TO or CC.
    Do the
    duplicates appear twice in archives and/or digests? (I think the
    duplicate may be dropped from the archives if the message-id is the
    same, but I'm not sure).
    No, only one copy of every post is in the archives and no one else is
    reporting dupes.
    Other subscribers may not see this because their incoming MTA/MUA may
    drop duplicate message-ids.
    As with most OSS projects there is a heavy bias away from Microsoft
    software, including Outlook which I use. Chances are slim that others in
    this OSS forum are using Outlook and thus their MUAs might be filtering
    away the problem. So I was suspecting my MUA. However, I do not get dupes
    when people in this mailman-users list CC the list but I do get some dupes
    from that group - obviously the same MTA/MUA tiers are involved. This is
    why I have been suspecting something in the list software that is
    different. Maybe there's a patch or addin that they've installed that
    corrupted this dupe-filtering code? That's what brought me here - Until
    now I've had no idea if this was an issue specific to that group or
    something that was introduced in the v2.1.5 code that you guys would know
    or want to know about.

    Thanks again.
  • Mark Sapiro at Apr 30, 2006 at 11:42 pm

    Tony G wrote:
    Mark Sapiro wrote:
    Brad has already provided good information in his reply. I just want
    to add that it looks to me like in this case, the sender's MUA is
    actually giving the list address twice (once for To: and once for Cc:)
    to the MTA, and the MTA is not dropping the duplicate recipient. Thus,
    two copies of the post are actually being sent to the list.
    As I mentioned in my response to Brad, I'm not even involved in the
    discussions where I'm getting the dupes. This would be an obvious source
    of duplication. I'm just another quiet recipient to the list (hehe, until
    now anyway) and there's no reason for me to be explicitly identified in the
    TO or CC.

    Right, but you said when you receive duplicates, the list was addressed
    twice in the post, once in To: and once in Cc:. Thus, I am suggesting
    that the list actually receives the post twice and sends it twice and
    that's why you get two copies.

    Do the
    duplicates appear twice in archives and/or digests? (I think the
    duplicate may be dropped from the archives if the message-id is the
    same, but I'm not sure).
    No, only one copy of every post is in the archives and no one else is
    reporting dupes.

    Does anyone receive digests to confirm or deny the presense of dups in
    a digest. I haven't looked at the code for this right now, but as I
    said, I think pipermail may drop dups from the archive, so the absence
    of dups in the archive is not definitive.

    There may be something here with digests too, but I don't think so.

    Other subscribers may not see this because their incoming MTA/MUA may
    drop duplicate message-ids.
    As with most OSS projects there is a heavy bias away from Microsoft
    software, including Outlook which I use. Chances are slim that others in
    this OSS forum are using Outlook and thus their MUAs might be filtering
    away the problem. So I was suspecting my MUA. However, I do not get dupes
    when people in this mailman-users list CC the list but I do get some dupes
    from that group - obviously the same MTA/MUA tiers are involved.

    The process of your not getting a dup when I address my reply to both
    you and the list IS a Mailman process. You only get the direct message
    from me because you have requested no dups, and Mailman sees your
    address in the To: of my post so it doesn't send to you.

    This is not the case on the other list where maybe the list receives
    the post twice.

    This is
    why I have been suspecting something in the list software that is
    different. Maybe there's a patch or addin that they've installed that
    corrupted this dupe-filtering code? That's what brought me here - Until
    now I've had no idea if this was an issue specific to that group or
    something that was introduced in the v2.1.5 code that you guys would know
    or want to know about.

    I don't think it's Mailman per se, but I can't be sure without knowing
    what is happening. I still think it could be actually two copies of
    the post reaching the list, but the fact that no one else on the list
    admits to receiving dups makes this seem unlikely. I would like to
    help you get to the bottom of this if I can.

    I think the next thing is to collect both copies of some duplicated
    posts with full headers and examine the Received: chains in each to
    try to determine where the duplication is occurring. If you need help
    with this, you can post headers from duplicated posts to this list or
    send them to me off list, and I will try to help.

    --
    Mark Sapiro <msapiro at value.net> The highway is for gamblers,
    San Francisco Bay Area, California better use your sense - B. Dylan
  • Mark Sapiro at May 1, 2006 at 12:33 am

    Mark Sapiro wrote:
    I think the next thing is to collect both copies of some duplicated
    posts with full headers and examine the Received: chains in each to
    try to determine where the duplication is occurring.

    Another thing you can do if you have access to Mailman's logs is to
    check the post and smtp logs to see if the entire post is being sent
    twice.

    --
    Mark Sapiro <msapiro at value.net> The highway is for gamblers,
    San Francisco Bay Area, California better use your sense - B. Dylan
  • Tony G at May 1, 2006 at 12:49 am

    Mark Sapiro wrote:
    .. you said when you receive duplicates, the list was
    addressed twice in the post, once in To: and once in Cc:. Thus, I am
    suggesting that the list actually receives the post twice and sends
    it twice and that's why you get two copies.
    I mis-spoke. It only occurs in that one list when someone CC's the list. I
    would understand if it were in both TO _and_ CC. Since all lists and MUA's
    are different, people don't know whether to Reply or Reply-All, so it's
    50/50 whether the list address will be in the TO _or_ CC, but most people
    know to not double up.

    There may be something here with digests too, but I don't think so.
    I have not asked about digests but I don't think I'll find anything there
    either.

    The process of your not getting a dup when I address my reply to both
    you and the list IS a Mailman process. You only get the direct message
    from me because you have requested no dups, and Mailman sees your
    address in the To: of my post so it doesn't send to you.

    This is not the case on the other list where maybe the list receives
    the post twice.
    Remember that I am not involved in the discussions where I get the dupes.
    I understand the mailman code filters possible dupes when TO = a
    subscription address.

    I have confirmation from a single Outlook user that I get dupes when they
    do not. (I wish I had a larger sampling of responses over a longer period
    of time but I'll take what I can get.) This points to an MTA issue and not
    list or MUA. I am now checking in that area. Again the problem is that I
    only get dupes to this one address when the list is in the CC. I can't
    believe an MTA bug would be that specific so it might be some timing issue,
    a hiccup during greylisting processing, etc.


    I don't think it's Mailman per se, but I can't be sure without knowing
    what is happening. I still think it could be actually two copies of
    the post reaching the list, but the fact that no one else on the list
    admits to receiving dups makes this seem unlikely. I would like to
    help you get to the bottom of this if I can.

    I think the next thing is to collect both copies of some duplicated
    posts with full headers and examine the Received: chains in each to
    try to determine where the duplication is occurring. If you need help
    with this, you can post headers from duplicated posts to this list or
    send them to me off list, and I will try to help.
    Over the last month or two I have been going over the headers myself and
    the headers on each copy of the mails are exactly the same: same Received
    path and time stamps, and same Message-ID (!!). The only pattern between
    what generates dupes and what does not is that the list address is in the
    CC of duped mails - but only on that list, not in this mailman-users list.
    :(

    Please allow me a few days to pursue the MTA route and I'll post a note
    here to let you know if I find anything. Based on your comments and info
    from the other list I'm fairly convinced that I shouldn't be wasting any
    more of your collective time. But I will let you know what I find out.

    Thank you kindly for your dedicated interest.
  • Bryan Carbonnell at May 1, 2006 at 1:45 am

    On 4/30/06, Tony G wrote:

    Over the last month or two I have been going over the headers myself and
    the headers on each copy of the mails are exactly the same: same Received
    path and time stamps, and same Message-ID (!!). The only pattern between
    what generates dupes and what does not is that the list address is in the
    CC of duped mails - but only on that list, not in this mailman-users list.
    :(

    Please allow me a few days to pursue the MTA route and I'll post a note
    here to let you know if I find anything. Based on your comments and info
    from the other list I'm fairly convinced that I shouldn't be wasting any
    more of your collective time. But I will let you know what I find out.

    Thank you kindly for your dedicated interest.
    I haven't been following this thread too carefully, but a couple of
    things clicked together based on some duplicate issues I've seen with
    my users.

    Just to recap the details as I recall them

    1) You are the only one seeing the duplicate e-mails
    2) You are using Outlook.
    3) Duplicate e-mails are idendical down to the header level.
    4) Only happens on 1 list.
    5) Only happens when the list is in the CC header.

    If all the above are correct, check your Outlook rules. Outlook may be
    the cause of the duplicates.

    I've spent a TON of time trying to diagnose this for one user and it
    turned out it was her Outlook rules that caused the duplicate.

    I don't use Outlook myself, but try disabling the rule for a while and
    see if the duplicates still happen. If not, then recreate the rule.

    I know this isn't Mailman related, but I think that this may be a good
    cause for the archive.

    --
    Bryan Carbonnell - carbonnb at gmail.com
    Life's journey is not to arrive at the grave safely in a well
    preserved body, but rather to skid in sideways, totally worn out,
    shouting "What a great ride!"
  • Tony G at May 17, 2006 at 10:49 pm
    I just wanted to close this old topic with a thanks to Bryan Carbonnell
    (and anyone else) who successfully diagnosed the issue of mail duplication
    as mis-configured Outlook rules rather than a problem with mailman
    processing CC, reply-to, etc..

    Thanks to anyone who responded. I'll go flog myself once for every person
    who's time has been wasted here. :)
    Tony


    Bryan Carbonnell wrote:
    Just to recap the details as I recall them

    1) You are the only one seeing the duplicate e-mails
    2) You are using Outlook.
    3) Duplicate e-mails are idendical down to the header level.
    4) Only happens on 1 list.
    5) Only happens when the list is in the CC header.

    If all the above are correct, check your Outlook rules. Outlook may be
    the cause of the duplicates.

    I've spent a TON of time trying to diagnose this for one user and it
    turned out it was her Outlook rules that caused the duplicate.

    I don't use Outlook myself, but try disabling the rule for a while and
    see if the duplicates still happen. If not, then recreate the rule.

    I know this isn't Mailman related, but I think that this may be a good
    cause for the archive.
  • Brad Knowles at May 1, 2006 at 2:48 am

    At 5:49 PM -0700 2006-04-30, Tony G wrote:

    Over the last month or two I have been going over the headers myself and
    the headers on each copy of the mails are exactly the same: same Received
    path and time stamps, and same Message-ID (!!). The only pattern between
    what generates dupes and what does not is that the list address is in the
    CC of duped mails - but only on that list, not in this mailman-users list.
    :(
    If the headers really are completely identical, including all
    queue ids and time stamps in the "Received:" headers, then the only
    machine that could be causing the duplication is the MTA of your mail
    services provider.

    With a few header samples of various messages, we should be able
    to figure out what's going on. Or, at least get a better idea of
    what additional information we may need.

    --
    Brad Knowles, <brad at stop.mail-abuse.org>

    "Those who would give up essential Liberty, to purchase a little
    temporary Safety, deserve neither Liberty nor Safety."

    -- Benjamin Franklin (1706-1790), reply of the Pennsylvania
    Assembly to the Governor, November 11, 1755

    LOPSA member since December 2005. See <http://www.lopsa.org/>.
  • John W. Baxter at May 1, 2006 at 2:47 pm

    On 4/30/06 1:28 AM, "Tony G" wrote:

    It seems I am getting duplicate messages from mailman v2.1.5 when senders send
    both TO and CC to the list. For some reason no one else in the list is
    acknowledging this issue so I look like an idiot complaining about it, but I'm
    seeing several instances of it per day.
    I'm seeing duplicates too, reading in Microsoft Entourage (which differs
    greatly from Outlook). I haven't taken the time to analyze them (delete
    works fine). Since some of them are in threads crossposted to users and
    developers lists, I see four copies of those.

    Two copies of a message with this Message-Id:
    Message-Id: <1146489146.11009.13.camel at geddy.wooz.org>
    Message-Id: <1146489146.11009.13.camel at geddy.wooz.org>
    reached me through the developers list (based on the List-xx headers)

    This is the one in which Barry started out
    "I agree that we need a lot more data"

    And two more copies came via the users list.

    --John
  • John W. Baxter at May 1, 2006 at 3:14 pm

    On 5/1/06 7:47 AM, "John W. Baxter" wrote:
    On 4/30/06 1:28 AM, "Tony G" wrote:

    It seems I am getting duplicate messages from mailman v2.1.5 when senders
    send
    both TO and CC to the list. For some reason no one else in the list is
    acknowledging this issue so I look like an idiot complaining about it, but
    I'm
    seeing several instances of it per day.
    I'm seeing duplicates too, reading in Microsoft Entourage (which differs
    greatly from Outlook). I haven't taken the time to analyze them (delete
    works fine). Since some of them are in threads crossposted to users and
    developers lists, I see four copies of those.

    Two copies of a message with this Message-Id:
    Message-Id: <1146489146.11009.13.camel at geddy.wooz.org>
    Message-Id: <1146489146.11009.13.camel at geddy.wooz.org>
    reached me through the developers list (based on the List-xx headers)

    This is the one in which Barry started out
    "I agree that we need a lot more data"

    And two more copies came via the users list.
    Tony, I think you need to review your Outlook rules.

    My duplications are produced in our mail server's mail sorting process, and
    the key is the presence of prefixes for both -users and -developers in the
    Subject: lines of the problem messages.

    Considering only the two copies from the Developers list, the divergence
    appears to be in our mail processing. The messages arrived our our
    world-facing machine with the same Exim Id:

    Received: from smtp-vbr8.xs4all.nl ([194.109.24.28]:3979)
    by entiat.olympus.net with esmtp (Exim 4.42)
    id 1FaYCW-0006pK-O1
    for jwblist at loricamail.com; Mon, 01 May 2006 06:12:33 -0700

    and

    Received: from smtp-vbr8.xs4all.nl ([194.109.24.28]:3979)
    by entiat.olympus.net with esmtp (Exim 4.42)
    id 1FaYCW-0006pK-O1
    for jwblist at loricamail.com; Mon, 01 May 2006 06:12:33 -0700

    (ie, the same).

    And having looked at the server logs, I now know what is happening in my
    case. I have our mail processing sort list mail into IMAP folders based
    on--in the case of these lists--the presence of the subject prefix, and
    these messages have prefixes for both lists in the Subject: header.

    So my copy for -users gets sorted into my folders for -users and for
    -developers, and my copy for -developers gets sorted into -users and into
    -developers.

    Note: the next revision of our mail processing will allow sorting based on
    arbitrary headers--the List- headers for example.

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupmailman-users @
categoriespython
postedApr 30, '06 at 8:28a
activeMay 17, '06 at 10:49p
posts14
users5
websitelist.org

People

Translate

site design / logo © 2022 Grokbase