FAQ
Dear mailman guys..

is there way to tell mailman to send certain number of messages per hour? I have 57 lists on 18 servers for my customers and I?m having trouble with hotmail and yahoo rejecting mail from my servers and causing a lot of subscribed addressed to be removed from the lists because of automatic bounce processing done by mailman.. when contacting my server provider they advised to control the amount of messages sent per hour or something like that ..

is that possible and how?

Thanks a million :)

Search Discussions

  • Mark Sapiro at Jan 18, 2011 at 2:18 am

    On 1/17/2011 3:00 PM, Khalil Abbas wrote:
    is there way to tell mailman to send certain number of messages per
    hour?

    The short answer is No, but see the FAQ at <http://wiki.list.org/x/j4A9>
    for other possible solutions.

    --
    Mark Sapiro <mark at msapiro.net> The highway is for gamblers,
    San Francisco Bay Area, California better use your sense - B. Dylan
  • Khalil Abbas at Jan 20, 2011 at 11:30 am
    by the way, I keep deleting the following:

    /var/spool/mailman/bounces/*.pck
    /var/lib/mailman/data/*.pck

    because they fill up all the space on my small hard drives on my VPS
    servers.. is that why mailman's bounce processor removes the subscribers or
    because they does actually bounce until they reach the points limit?

    Thanks..

    -----Original Message-----
    From: Mark Sapiro
    Sent: Tuesday, January 18, 2011 4:18 AM
    To: Khalil Abbas
    Cc: mailman-users at python.org
    Subject: Re: [Mailman-Users] control mail distribution ..
    On 1/17/2011 3:00 PM, Khalil Abbas wrote:

    is there way to tell mailman to send certain number of messages per
    hour?

    The short answer is No, but see the FAQ at <http://wiki.list.org/x/j4A9>
    for other possible solutions.

    --
    Mark Sapiro <mark at msapiro.net> The highway is for gamblers,
    San Francisco Bay Area, California better use your sense - B. Dylan
  • Mark Sapiro at Jan 20, 2011 at 8:53 pm

    Khalil Abbas wrote:
    by the way, I keep deleting the following:

    /var/spool/mailman/bounces/*.pck

    It looks like BounceRunner has died and is not processing the bounce
    queue. Chack Mailman's qrunner and error logs for more information.

    /var/lib/mailman/data/*.pck

    Are these heldmsg-listname-*.pck files or bounce-events-*.pck files. If
    they are held messages, they should be removed with bin/discard or
    through the moderator interface, and setting the list's
    max_days_to_hold may help.

    If they are bounce-events-*.pck files, most of them are probably stale
    because of dying BounceRunners. Assuming you are not slicing queues,
    i.e., you have only one qrunner per queue, there should never be more
    than two of these, one with the PID of the current OutgoingRunner and
    one with the PID of the current BounceRunner. All the others are dead
    files left behind when a runner died or was killed.

    --
    Mark Sapiro <mark at msapiro.net> The highway is for gamblers,
    San Francisco Bay Area, California better use your sense - B. Dylan
  • Khalil Abbas at Jan 21, 2011 at 9:15 am
    well, attached are parts of the error and qrunner logs.. I also forgot to mention that I also delete the following files:

    rm /var/lib/mailman/data/*.pck -f
    rm /var/lock/mailman/*.* -f

    the first one also eats up my little space on the hard drive, and the other locks the lists n they can?t send..

    what I need to know, why the subscribers are being removed? is it because of misconfiguration of mailman? or because of outside MTA?s like yahoo and hotmail rejecting mail from my servers because I need to control the flow of messages going out from my servers?

    another thing worth to mention, the servers are not blocked by hotmail because I sent single test messages from all my servers to some of my subscribers that were removed from my lists and they received them with no problems..

    Thanks..

    -----Original Message-----
    From: Mark Sapiro
    Sent: Friday, January 21, 2011 12:21 AM
    To: Khalil Abbas
    Subject: Re: [Mailman-Users] control mail distribution ..

    Khalil Abbas wrote:
    if the bounce runner keeps dying, then why the subscribers keep being
    removed??

    Because it doesn't die always? You're the one with the logs and other
    evidence that might enable someone to answer this.

    and I understand the conclusion to my previously asked question,
    deleting these files does not get the subscribers to be removed right?

    Right. Deleting *.pck files from /var/spool/mailman/bounce will result
    in those specific bounce messages (each file is one queued bounce
    message) being lost and therefore not processed.

    Likewise, deleting bounce-events-*.pck files from
    /var/lib/mailman/data/ will in the case of active files result in the
    bounce information in the file being unprocessed, and in the case of
    "dead" files have no effect because whatever residual bounce info is
    in those files is lost.

    In any event, removing these files will result only in bounce scores
    not being incremented and subscribers not being removed, not the
    reverse.

    --
    Mark Sapiro <mark at msapiro.net> The highway is for gamblers,
    San Francisco Bay Area, California better use your sense - B. Dylan
    -------------- next part --------------
    An embedded and charset-unspecified text was scrubbed...
    Name: error.txt
    URL: <http://mail.python.org/pipermail/mailman-users/attachments/20110121/427967ed/attachment-0002.txt>
    -------------- next part --------------
    An embedded and charset-unspecified text was scrubbed...
    Name: qrunnerx.txt
    URL: <http://mail.python.org/pipermail/mailman-users/attachments/20110121/427967ed/attachment-0003.txt>
  • Mark Sapiro at Jan 23, 2011 at 2:02 am

    On 1/21/2011 1:15 AM, Khalil Abbas wrote:
    well, attached are parts of the error and qrunner logs.. I also forgot
    to mention that I also delete the following files:

    rm /var/lib/mailman/data/*.pck -f
    This was covered in my reply at
    <http://mail.python.org/pipermail/mailman-users/2011-January/071032.html>
    rm /var/lock/mailman/*.* -f
    Doing this also removes the master lock whose purpose is to prevent you
    from starting multiple sets of qrunners. This is bad.

    the first one also eats up my little space on the hard drive, and the
    other locks the lists n they can?t send..

    Also, if lists are being left locked by dead processes, this indicates
    there are other problems. Do you suffer frequent power outages?

    what I need to know, why the subscribers are being removed? is it
    because of misconfiguration of mailman? or because of outside MTA?s like
    yahoo and hotmail rejecting mail from my servers because I need to
    control the flow of messages going out from my servers?

    You also need to know the underlying causes of all the anomalous things
    that are occurring in your installation. There are serious problems.
    Your logs may help. Also see FAQs <http://wiki.list.org/x/_4A9> for info
    on completely stopping Mailman and starting only after completely
    stopped and <http://wiki.list.org/x/noA9> for lock information.

    If you set bounce_notify_owner_on_disable to Yes on your lists, the list
    owner will be sent a notice whenever a member's delivery is disabled by
    bounce. This notice will contain the triggering bounce message. That
    should help you diagnose the cause.

    another thing worth to mention, the servers are not blocked by hotmail
    because I sent single test messages from all my servers to some of my
    subscribers that were removed from my lists and they received them with
    no problems..

    Possibly your MTA is treating the "temporary failure" throttling status
    used by some large ISPs to rate limit your mail as a hard failure and
    returning a failure DSN to Mailman.

    The only thing in your error log was one unparseable message which has
    nothing to do with any of this.
    From your qrunner log, I see a SIGHUP from log rotation at Jan 16
    04:02:54 2011. Then Mailman is stopped at Jan 16 18:17:53 2011, and
    started at Jan 16 18:19:41 2011.

    Then I see

    Jan 17 00:08:48 2011 (2666) Master qrunner detected subprocess exit
    (pid: 2712, sig: 6, sts: None, class: BounceRunner, slice: 1/1) [restarting]

    which says BounceRunner died with Signal 6 (SIGABRT). If it wasn't
    manually killed, there should be a corresponding error log entry, but
    the only entry in the error log you provided is from Dec 26 which
    indicates you weren't looking at the current error log.

    Then we see a sequence

    Jan 17 17:11:34 2011 Mailman stopped

    Jan 17 17:13:29 2011 Mailman started

    Jan 17 20:29:32 2011 Mailman stopped

    Jan 17 20:31:34 2011 Mailman started

    Why all the stopping and starting?

    --
    Mark Sapiro <mark at msapiro.net> The highway is for gamblers,
    San Francisco Bay Area, California better use your sense - B. Dylan
  • Mark Sapiro at Jan 23, 2011 at 2:12 am

    On 1/22/2011 6:02 PM, Mark Sapiro wrote:
    Jan 17 00:08:48 2011 (2666) Master qrunner detected subprocess exit
    (pid: 2712, sig: 6, sts: None, class: BounceRunner, slice: 1/1) [restarting]

    which says BounceRunner died with Signal 6 (SIGABRT).

    Actually, I think this is wrong. I think it died with Error 6

    006 ENXIO No such device or address

    and not SIGABRT.

    --
    Mark Sapiro <mark at msapiro.net> The highway is for gamblers,
    San Francisco Bay Area, California better use your sense - B. Dylan
  • Khalil Abbas at Jan 23, 2011 at 2:24 am
    well I did send an old error file cause the current one is huge..

    I think mailman keeps stopping because I reboot the servers everyday to free up the memory.. I have a TV station as a customer having 6 lists with a total of 300,000 subscribers, 50,000 each on the following server:

    Centos
    CPU Intel P4 3000
    2 GBs RAM
    120 Gbs hard drive

    and they post daily to their lists..

    I also have another local Newspaper as a customer having about 100,000 subscribers on 3 lists with a virtual private server with the following specs:
    Centos
    256 MBs Ram
    10 GBs Hard drive

    which I have a hard time cleaning up because of the little space that keeps filling up..

    is that the problem? do I need to use bigger hardware ?

    another thing, the servers doesn?t send directly, the messages are being sent to my hosting servers provider's relay server..

    Thanks for everything..



    -----Original Message-----
    From: Mark Sapiro
    Sent: Sunday, January 23, 2011 4:02 AM
    To: Khalil Abbas
    Cc: mailman-users at python.org
    Subject: Re: [Mailman-Users] control mail distribution ..
    On 1/21/2011 1:15 AM, Khalil Abbas wrote:
    well, attached are parts of the error and qrunner logs.. I also forgot
    to mention that I also delete the following files:

    rm /var/lib/mailman/data/*.pck -f
    This was covered in my reply at
    <http://mail.python.org/pipermail/mailman-users/2011-January/071032.html>
    rm /var/lock/mailman/*.* -f
    Doing this also removes the master lock whose purpose is to prevent you
    from starting multiple sets of qrunners. This is bad.

    the first one also eats up my little space on the hard drive, and the
    other locks the lists n they can?t send..

    Also, if lists are being left locked by dead processes, this indicates
    there are other problems. Do you suffer frequent power outages?

    what I need to know, why the subscribers are being removed? is it
    because of misconfiguration of mailman? or because of outside MTA?s like
    yahoo and hotmail rejecting mail from my servers because I need to
    control the flow of messages going out from my servers?

    You also need to know the underlying causes of all the anomalous things
    that are occurring in your installation. There are serious problems.
    Your logs may help. Also see FAQs <http://wiki.list.org/x/_4A9> for info
    on completely stopping Mailman and starting only after completely
    stopped and <http://wiki.list.org/x/noA9> for lock information.

    If you set bounce_notify_owner_on_disable to Yes on your lists, the list
    owner will be sent a notice whenever a member's delivery is disabled by
    bounce. This notice will contain the triggering bounce message. That
    should help you diagnose the cause.

    another thing worth to mention, the servers are not blocked by hotmail
    because I sent single test messages from all my servers to some of my
    subscribers that were removed from my lists and they received them with
    no problems..

    Possibly your MTA is treating the "temporary failure" throttling status
    used by some large ISPs to rate limit your mail as a hard failure and
    returning a failure DSN to Mailman.

    The only thing in your error log was one unparseable message which has
    nothing to do with any of this.
    From your qrunner log, I see a SIGHUP from log rotation at Jan 16
    04:02:54 2011. Then Mailman is stopped at Jan 16 18:17:53 2011, and
    started at Jan 16 18:19:41 2011.

    Then I see

    Jan 17 00:08:48 2011 (2666) Master qrunner detected subprocess exit
    (pid: 2712, sig: 6, sts: None, class: BounceRunner, slice: 1/1) [restarting]

    which says BounceRunner died with Signal 6 (SIGABRT). If it wasn't
    manually killed, there should be a corresponding error log entry, but
    the only entry in the error log you provided is from Dec 26 which
    indicates you weren't looking at the current error log.

    Then we see a sequence

    Jan 17 17:11:34 2011 Mailman stopped

    Jan 17 17:13:29 2011 Mailman started

    Jan 17 20:29:32 2011 Mailman stopped

    Jan 17 20:31:34 2011 Mailman started

    Why all the stopping and starting?

    --
    Mark Sapiro <mark at msapiro.net> The highway is for gamblers,
    San Francisco Bay Area, California better use your sense - B. Dylan
  • Mark Sapiro at Jan 23, 2011 at 3:45 am

    Khalil Abbas wrote:
    well I did send an old error file cause the current one is huge..

    Because of some underlying issue that can't be diagnosed without seeing
    what's there.

    For example, what are the error log entries with time stamps around the

    Jan 17 00:08:48 2011 (2666) Master qrunner detected subprocess exit
    (pid: 2712, sig: 6, sts: None, class: BounceRunner, slice: 1/1) [restarting]

    (i.e. between 00:08:45 and 00:08:48) that might tell what the issue
    with BounceRunner was.

    I think mailman keeps stopping because I reboot the servers everyday to
    free up the memory..

    That's probably at least part of why you wind up with orphaned
    /var/lib/mailman/data/bounce-events-*.pck and stale locks.

    I have a TV station as a customer having 6 lists
    with a total of 300,000 subscribers, 50,000 each on the following
    server:

    Centos
    CPU Intel P4 3000
    2 GBs RAM
    120 Gbs hard drive

    and they post daily to their lists..

    I also have another local Newspaper as a customer having about 100,000 >subscribers on 3 lists with a virtual private server with the following >specs:
    Centos
    256 MBs Ram
    10 GBs Hard drive

    which I have a hard time cleaning up because of the little space that >keeps filling up..

    is that the problem? do I need to use bigger hardware ?

    Maybe. I can't begin to tell you what the underlying issues are
    without seeing the logs.

    another thing, the servers doesn't send directly, the messages
    are being sent to my hosting servers provider's relay server..

    So there may be issues with this. There might be indications in
    Mailman's smtp-failure log, and the disable notices to the owner from
    Mailman should help.


    --
    Mark Sapiro <mark at msapiro.net> The highway is for gamblers,
    San Francisco Bay Area, California better use your sense - B. Dylan
  • Mark Sapiro at Jan 23, 2011 at 6:21 pm

    On 1/23/2011 7:46 AM, Khalil Abbas wrote:
    attached are the Jan 16, and Jan 17th logs: error, smtp, qrunner,
    locks.. I replaced my domain name with the word (DOMAIN) cause I believe
    this is shown to public ..

    there are several smtp-failure files (failure.1, .2, .3...) but they are
    all empty (0 bytes size)..
    The locks log shows one broken expired lock, possibly due to your
    frequent reboots.

    The error log shows only unparseable messages but there are over 3000 of
    them in about 26 hours. Correlation with the qrunner log shows these are
    all processed by BounceRunner. Thus they are unparseable messages sent
    to the LIST-bounces addresses. Normally, unparseable messages are spam,
    but possible they are real bounces from some broken MTA.

    The next time you have an accumulation of
    /var/spool/mailman/bounce/*.pck files, examine some of them with
    Mailman's bin/dumpdb or bin/show_qfiles tool. If they look like
    legitimate bounce messages, they may give you some information about
    your bounces, but note that the particular messages that are unparseable
    do not get scored for any user.

    If they look like spam, then your problem may just be massive amounts of
    spam sent to your LIST-bounces addresses.

    --
    Mark Sapiro <mark at msapiro.net> The highway is for gamblers,
    San Francisco Bay Area, California better use your sense - B. Dylan
  • Khalil Abbas at Jan 23, 2011 at 6:36 pm
    well this is much comforting, meaning that there is nothing to worry about
    the configuration.. I turned on the "Should Mailman notify you, the list
    owner, when bounces cause a member's subscription to be disabled?" to yes to
    see what happens..

    but still this does not answer the main question, what causes the existing
    real members to be unsubscribed? is it because hotmail and yahoo are
    rejecting the mail flow that needs to be controlled?

    this leads to another question, should this (control mail flow) be
    considered in the next version of mailman? this will also answer the million
    dollars question: CAN mailman handle huge lists like those of TV stations
    like mine (300,000 subscribers)?

    final question: is there any advice about my current situation (hardware,
    configuration...etc.)?

    Thanks mate, you're the best :)




    -----Original Message-----
    From: Mark Sapiro
    Sent: Sunday, January 23, 2011 8:21 PM
    To: Khalil Abbas
    Cc: mailman-users at python.org
    Subject: Re: [Mailman-Users] control mail distribution ..
    On 1/23/2011 7:46 AM, Khalil Abbas wrote:
    attached are the Jan 16, and Jan 17th logs: error, smtp, qrunner,
    locks.. I replaced my domain name with the word (DOMAIN) cause I believe
    this is shown to public ..

    there are several smtp-failure files (failure.1, .2, .3...) but they are
    all empty (0 bytes size)..
    The locks log shows one broken expired lock, possibly due to your
    frequent reboots.

    The error log shows only unparseable messages but there are over 3000 of
    them in about 26 hours. Correlation with the qrunner log shows these are
    all processed by BounceRunner. Thus they are unparseable messages sent
    to the LIST-bounces addresses. Normally, unparseable messages are spam,
    but possible they are real bounces from some broken MTA.

    The next time you have an accumulation of
    /var/spool/mailman/bounce/*.pck files, examine some of them with
    Mailman's bin/dumpdb or bin/show_qfiles tool. If they look like
    legitimate bounce messages, they may give you some information about
    your bounces, but note that the particular messages that are unparseable
    do not get scored for any user.

    If they look like spam, then your problem may just be massive amounts of
    spam sent to your LIST-bounces addresses.

    --
    Mark Sapiro <mark at msapiro.net> The highway is for gamblers,
    San Francisco Bay Area, California better use your sense - B. Dylan
  • Mark Sapiro at Jan 23, 2011 at 8:33 pm

    Khalil Abbas wrote:
    well this is much comforting, meaning that there is nothing to worry about
    the configuration.. I turned on the "Should Mailman notify you, the list
    owner, when bounces cause a member's subscription to be disabled?" to yes to
    see what happens..

    but still this does not answer the main question, what causes the existing
    real members to be unsubscribed? is it because hotmail and yahoo are
    rejecting the mail flow that needs to be controlled?

    When you see some actual disabling bounce notifications, you will
    hopefully be able to answer that for yourself. Until then, I don't
    know what causes it.

    this leads to another question, should this (control mail flow) be
    considered in the next version of mailman? this will also answer the million
    dollars question: CAN mailman handle huge lists like those of TV stations
    like mine (300,000 subscribers)?

    Mailman handles large lists and large volumes of mail just fine with
    appropriately sized hardware and properly configured MTAs. See the FAQ
    at <http://wiki.list.org/x/NoA9> and note the following quoted from
    the end of that article.

    If you're going to be running lists with more than a few thousand
    members, then you need to have a thorough understanding of how your
    MTA and Mailman work, and a deep and intimate familiarity with
    Internet e-mail in general. And of course, you will have to
    carefully consider how best to tune your MTA and Mailman to work
    best together. If you're missing any of these prerequisites, you're
    in for a difficult time.


    final question: is there any advice about my current situation (hardware,
    configuration...etc.)?

    Increase your disk storage. These days, when a 500GB drive costs well
    under $100 US, a production server with 10GB of storage is way too
    cramped, and your lack of disk space is causing you do do all sorts of
    questionable things that shouldn't be necessary.

    --
    Mark Sapiro <mark at msapiro.net> The highway is for gamblers,
    San Francisco Bay Area, California better use your sense - B. Dylan
  • Mark Sapiro at Jan 23, 2011 at 9:20 pm

    On 1/23/2011 12:33 PM, Mark Sapiro wrote:
    Khalil Abbas wrote:
    well this is much comforting, meaning that there is nothing to worry about
    the configuration.. I turned on the "Should Mailman notify you, the list
    owner, when bounces cause a member's subscription to be disabled?" to yes to
    see what happens..

    but still this does not answer the main question, what causes the existing
    real members to be unsubscribed? is it because hotmail and yahoo are
    rejecting the mail flow that needs to be controlled?

    When you see some actual disabling bounce notifications, you will
    hopefully be able to answer that for yourself. Until then, I don't
    know what causes it.

    I also looked more closely at the smtp log you sent earlier. There are a
    huge number of mailman generated notices that originate from
    BounceRunner. All lists appear affected. I think these are unrecognized
    bounce notifications. I have suggested before that there may be a large
    amount of spam being sent to the LIST-bounces addresses, but there may
    also be a mail loop of some kind. I suggest that you make sure that each
    list's (including the 'mailman' list) owner attribute contains only
    deliverable addresses and that none of the LIST-* addresses are among
    them. Similarly, if there are any moderator addresses, they must be
    deliverable and must not include any LIST-* addresses and shouldn't
    duplicate any owner addresses for the same list.

    Also try sending an email to the LIST-owner addresses to verify such
    mail is received.

    --
    Mark Sapiro <mark at msapiro.net> The highway is for gamblers,
    San Francisco Bay Area, California better use your sense - B. Dylan
  • Khalil Abbas at Jan 23, 2011 at 10:11 pm
    I also forgot to mention something.. I'm managing 57 lists on 18 servers..
    some are small lists and some are huge but the thing in common is that they
    all send a single message to their lists per day.. but I have put my own
    email address (khillo100 at hotmail dot com) in all of the 57 lists.. and
    everyday I receive between 30 and 40 emails from the lists but never 57!

    another thing in common is that all 18 servers never send directly, they go
    thru the same relay server ..

    I tried to something, I removed my address from all lists and re-subscribed
    and I received 57 welcome messages.. but when sending to the lists, only 30
    to 40 are received..

    any thoughts?





    -----Original Message-----
    From: Mark Sapiro
    Sent: Sunday, January 23, 2011 11:20 PM
    To: Khalil Abbas
    Cc: mailman-users at python.org
    Subject: Re: [Mailman-Users] control mail distribution ..
    On 1/23/2011 12:33 PM, Mark Sapiro wrote:
    Khalil Abbas wrote:
    well this is much comforting, meaning that there is nothing to worry
    about
    the configuration.. I turned on the "Should Mailman notify you, the list
    owner, when bounces cause a member's subscription to be disabled?" to yes
    to
    see what happens..

    but still this does not answer the main question, what causes the
    existing
    real members to be unsubscribed? is it because hotmail and yahoo are
    rejecting the mail flow that needs to be controlled?

    When you see some actual disabling bounce notifications, you will
    hopefully be able to answer that for yourself. Until then, I don't
    know what causes it.

    I also looked more closely at the smtp log you sent earlier. There are a
    huge number of mailman generated notices that originate from
    BounceRunner. All lists appear affected. I think these are unrecognized
    bounce notifications. I have suggested before that there may be a large
    amount of spam being sent to the LIST-bounces addresses, but there may
    also be a mail loop of some kind. I suggest that you make sure that each
    list's (including the 'mailman' list) owner attribute contains only
    deliverable addresses and that none of the LIST-* addresses are among
    them. Similarly, if there are any moderator addresses, they must be
    deliverable and must not include any LIST-* addresses and shouldn't
    duplicate any owner addresses for the same list.

    Also try sending an email to the LIST-owner addresses to verify such
    mail is received.

    --
    Mark Sapiro <mark at msapiro.net> The highway is for gamblers,
    San Francisco Bay Area, California better use your sense - B. Dylan
  • Mark Sapiro at Jan 24, 2011 at 1:14 am

    halil Abbas wrote:
    I also forgot to mention something.. I'm managing 57 lists on 18 servers..
    some are small lists and some are huge but the thing in common is that they
    all send a single message to their lists per day.. but I have put my own
    email address (khillo100 at hotmail dot com) in all of the 57 lists.. and
    everyday I receive between 30 and 40 emails from the lists but never 57!

    another thing in common is that all 18 servers never send directly, they go
    thru the same relay server ..

    I agree that mails are getting bounced/dropped, but I can't begin to
    guess why without seeing actual bounce messages and/or logs from the
    relay server and/or logs from your local MTA if you use a local MTA
    between Mailman and the relay server.

    You may need to talk to the operators of the relay. I know you did and
    they told you to rate limit your sending, but do they know that is the
    problem, or is that a non-response?

    If in fact you do have a local MTA which Mailman delivers to and which
    then relays to the relay server, you can investigate rate limiting
    there as discussed in the FAQ at <http://wiki.list.org/x/j4A9> to
    which you have been previously referred.


    Let us see some actual bounce messages. We know there are some because
    bounce processing is removing subscribers, so what do they look like?


    Also, have you followed up on my following suggestion to verify that
    all the owner addresses are deliverable. I.e., what happened to the
    nearly 3000 notices from BounceRunner that were indicated in the smtp
    log you sent me.

    I also looked more closely at the smtp log you sent earlier. There are a
    huge number of mailman generated notices that originate from
    BounceRunner. All lists appear affected. I think these are unrecognized
    bounce notifications. I have suggested before that there may be a large
    amount of spam being sent to the LIST-bounces addresses, but there may
    also be a mail loop of some kind. I suggest that you make sure that each
    list's (including the 'mailman' list) owner attribute contains only
    deliverable addresses and that none of the LIST-* addresses are among
    them. Similarly, if there are any moderator addresses, they must be
    deliverable and must not include any LIST-* addresses and shouldn't
    duplicate any owner addresses for the same list.

    Also try sending an email to the LIST-owner addresses to verify such
    mail is received.
    --
    Mark Sapiro <mark at msapiro.net> The highway is for gamblers,
    San Francisco Bay Area, California better use your sense - B. Dylan
  • Khalil Abbas at Jan 24, 2011 at 7:35 am
    ah yes, these to list owner notices I guess are unsubscribe notices.. for I
    turned on the "Should Mailman notify you, the list owner, when bounces cause
    a member to be unsubscribed?" to yes.. I tried to send test messages to
    these addresses, most of them are real subscribers that got kicked off and
    received my test messages, and some returned back for different reasons like
    mailbox full and mailbox has been closed and stuff like that ..

    checking some random files in the /var/spool/mailman/bounces/*.pck I found :

    ******************
    This is the qmail-send program at ______________.\nI\'m afraid I wasn\'t
    able to deliver your message to the following addresses.\nThis is a
    permanent error; I\'ve given up. Sorry it didn\'t work out.\n\n<___________
    (hotmail.fr address) >:\nConnected to 65.54.188.110 but sender was
    rejected.\nRemote host said: 421 RP-001 Unfortunately, some messages from
    ____________ weren\'t sent. Please try again. We have limits for how many
    messages can be sent per hour and per day. You can also refer to
    http://mail.live.com/mail/troubleshooting.aspx#errors.\nI\'m not going to
    try again; this message has been in the queue too long.
    ******************

    checking the error code 421 RP-001 on hotmail I found:

    ***************
    The mail server IP connecting to Windows Live Hotmail server has exceeded
    the rate limit allowed. Reason for rate limitation is related to IP/domain
    reputation. If you are not an email/network admin please contact your
    Email/Internet Service Provider for help.
    ***************

    so it is a rate limit no?

    what can be done about this? it says something about ip/domain reputation,
    is it my domain's reputation or my service provider's relay server's
    reputation?

    Thanks..

    -----Original Message-----
    From: Mark Sapiro
    Sent: Monday, January 24, 2011 3:14 AM
    To: Khalil Abbas ; mailman-users at python.org
    Subject: Re: [Mailman-Users] control mail distribution ..

    halil Abbas wrote:
    I also forgot to mention something.. I'm managing 57 lists on 18 servers..
    some are small lists and some are huge but the thing in common is that they
    all send a single message to their lists per day.. but I have put my own
    email address (khillo100 at hotmail dot com) in all of the 57 lists.. and
    everyday I receive between 30 and 40 emails from the lists but never 57!

    another thing in common is that all 18 servers never send directly, they go
    thru the same relay server ..

    I agree that mails are getting bounced/dropped, but I can't begin to
    guess why without seeing actual bounce messages and/or logs from the
    relay server and/or logs from your local MTA if you use a local MTA
    between Mailman and the relay server.

    You may need to talk to the operators of the relay. I know you did and
    they told you to rate limit your sending, but do they know that is the
    problem, or is that a non-response?

    If in fact you do have a local MTA which Mailman delivers to and which
    then relays to the relay server, you can investigate rate limiting
    there as discussed in the FAQ at <http://wiki.list.org/x/j4A9> to
    which you have been previously referred.


    Let us see some actual bounce messages. We know there are some because
    bounce processing is removing subscribers, so what do they look like?


    Also, have you followed up on my following suggestion to verify that
    all the owner addresses are deliverable. I.e., what happened to the
    nearly 3000 notices from BounceRunner that were indicated in the smtp
    log you sent me.

    I also looked more closely at the smtp log you sent earlier. There are a
    huge number of mailman generated notices that originate from
    BounceRunner. All lists appear affected. I think these are unrecognized
    bounce notifications. I have suggested before that there may be a large
    amount of spam being sent to the LIST-bounces addresses, but there may
    also be a mail loop of some kind. I suggest that you make sure that each
    list's (including the 'mailman' list) owner attribute contains only
    deliverable addresses and that none of the LIST-* addresses are among
    them. Similarly, if there are any moderator addresses, they must be
    deliverable and must not include any LIST-* addresses and shouldn't
    duplicate any owner addresses for the same list.

    Also try sending an email to the LIST-owner addresses to verify such
    mail is received.
    --
    Mark Sapiro <mark at msapiro.net> The highway is for gamblers,
    San Francisco Bay Area, California better use your sense - B. Dylan
  • Mark Sapiro at Jan 24, 2011 at 7:00 pm

    Khalil Abbas wrote:
    checking some random files in the /var/spool/mailman/bounces/*.pck I found :

    ******************
    This is the qmail-send program at ______________.\nI\'m afraid I wasn\'t
    able to deliver your message to the following addresses.\nThis is a
    permanent error; I\'ve given up. Sorry it didn\'t work out.\n\n<___________
    (hotmail.fr address) >:\nConnected to 65.54.188.110 but sender was
    rejected.\nRemote host said: 421 RP-001 Unfortunately, some messages from
    ____________ weren\'t sent. Please try again. We have limits for how many
    messages can be sent per hour and per day. You can also refer to
    http://mail.live.com/mail/troubleshooting.aspx#errors.\nI\'m not going to
    try again; this message has been in the queue too long.
    ******************

    checking the error code 421 RP-001 on hotmail I found:

    ***************
    The mail server IP connecting to Windows Live Hotmail server has exceeded
    the rate limit allowed. Reason for rate limitation is related to IP/domain
    reputation. If you are not an email/network admin please contact your
    Email/Internet Service Provider for help.
    ***************

    so it is a rate limit no?

    Yes it is. But, hotmail is limiting the rate by returning a 421 status
    which is retryable, but qmail on the relay host is either giving up
    too soon, or you will never be able to deliver all your mail no matter
    what you do.

    what can be done about this? it says something about ip/domain reputation,
    is it my domain's reputation or my service provider's relay server's
    reputation?

    It's your service provider's relay server. I'm surprised they haven't
    cut you off. Consider that hotmail is rate limiting that server and
    rejecting (with 421) mail over the limit. This would affect not only
    your list mail, but mail to hotmail from any other customer of the
    service provider being sent at the same time.

    Now, consider what is happening. The relay server gives up retrying the
    421 because the "message has been in the queue too long". Either it's
    "give up" time is way to short, something only the service provider
    can adjust, but maybe you can tell from when the rejected message was
    sent, or it keeps retrying for more than a day. If the latter, the
    implication would seem that hotmail's rate limit does not allow for
    all the days messages to be delivered in a day. The only solution to
    that is to get hotmail to raise the rate limit, and only the service
    provider can do that.

    Now, there is one other point. When you resubscribed, you got all your
    notices, so mail sent to one recipient at a time seems to get through.
    You can cause Mailman to send more messages with fewer recipients per
    message by putting say

    SMTP_MAX_RCPTS = 10

    in mm_cfg.py. That may help. Otherwise, I don't think there's much you
    can do except maybe convince your customers to reduce the frequency of
    their announcements.

    --
    Mark Sapiro <mark at msapiro.net> The highway is for gamblers,
    San Francisco Bay Area, California better use your sense - B. Dylan
  • Khalil Abbas at Jan 24, 2011 at 7:46 pm
    I'm discussing this with my service provider right now.. I'll tell u what
    comes up..

    by the way, what's the default value for SMTP_MAX_RCPTS ??

    Thanks..

    -----Original Message-----
    From: Mark Sapiro
    Sent: Monday, January 24, 2011 9:00 PM
    To: Khalil Abbas ; mailman-users at python.org
    Subject: Re: [Mailman-Users] control mail distribution ..

    Khalil Abbas wrote:
    checking some random files in the /var/spool/mailman/bounces/*.pck I found
    :

    ******************
    This is the qmail-send program at ______________.\nI\'m afraid I wasn\'t
    able to deliver your message to the following addresses.\nThis is a
    permanent error; I\'ve given up. Sorry it didn\'t work out.\n\n<___________
    (hotmail.fr address) >:\nConnected to 65.54.188.110 but sender was
    rejected.\nRemote host said: 421 RP-001 Unfortunately, some messages from
    ____________ weren\'t sent. Please try again. We have limits for how many
    messages can be sent per hour and per day. You can also refer to
    http://mail.live.com/mail/troubleshooting.aspx#errors.\nI\'m not going to
    try again; this message has been in the queue too long.
    ******************

    checking the error code 421 RP-001 on hotmail I found:

    ***************
    The mail server IP connecting to Windows Live Hotmail server has exceeded
    the rate limit allowed. Reason for rate limitation is related to IP/domain
    reputation. If you are not an email/network admin please contact your
    Email/Internet Service Provider for help.
    ***************

    so it is a rate limit no?

    Yes it is. But, hotmail is limiting the rate by returning a 421 status
    which is retryable, but qmail on the relay host is either giving up
    too soon, or you will never be able to deliver all your mail no matter
    what you do.

    what can be done about this? it says something about ip/domain reputation,
    is it my domain's reputation or my service provider's relay server's
    reputation?

    It's your service provider's relay server. I'm surprised they haven't
    cut you off. Consider that hotmail is rate limiting that server and
    rejecting (with 421) mail over the limit. This would affect not only
    your list mail, but mail to hotmail from any other customer of the
    service provider being sent at the same time.

    Now, consider what is happening. The relay server gives up retrying the
    421 because the "message has been in the queue too long". Either it's
    "give up" time is way to short, something only the service provider
    can adjust, but maybe you can tell from when the rejected message was
    sent, or it keeps retrying for more than a day. If the latter, the
    implication would seem that hotmail's rate limit does not allow for
    all the days messages to be delivered in a day. The only solution to
    that is to get hotmail to raise the rate limit, and only the service
    provider can do that.

    Now, there is one other point. When you resubscribed, you got all your
    notices, so mail sent to one recipient at a time seems to get through.
    You can cause Mailman to send more messages with fewer recipients per
    message by putting say

    SMTP_MAX_RCPTS = 10

    in mm_cfg.py. That may help. Otherwise, I don't think there's much you
    can do except maybe convince your customers to reduce the frequency of
    their announcements.

    --
    Mark Sapiro <mark at msapiro.net> The highway is for gamblers,
    San Francisco Bay Area, California better use your sense - B. Dylan
  • Mark Sapiro at Jan 24, 2011 at 11:18 pm

    Khalil Abbas wrote:
    by the way, what's the default value for SMTP_MAX_RCPTS ??

    All the settings default values and their documentation is in
    Defaults.py.

    # Ceiling on the number of recipients that can be specified in a single
    SMTP
    # transaction. Set to 0 to submit the entire recipient list in one
    # transaction. Only used with the SMTPDirect DELIVERY_MODULE.
    SMTP_MAX_RCPTS = 500

    --
    Mark Sapiro <mark at msapiro.net> The highway is for gamblers,
    San Francisco Bay Area, California better use your sense - B. Dylan
  • Khalil Abbas at Jan 26, 2011 at 4:15 pm
    well I decreased the SMTP_MAX_RCPTS to only 5 (five) and when checking the
    mailq every now and then .. some of the 18 servers gave the following:

    -Queue ID- --Size-- ----Arrival Time---- -Sender/Recipient-------
    335BB14BB8CA 111316 Wed Jan 26 09:04:29 MAILER-DAEMON
    (unknown mail transport
    error)

    and some of them:

    -Queue ID- --Size-- ----Arrival Time---- -Sender/Recipient-------
    7B477146C78 108725 Tue Jan 25 21:14:27
    balligho-bounces at ballighoseven.info
    (system resource
    problem)

    I kept rebooting the servers every while to get the mail moving .. and so
    far I received 27 messages out of 57 lists ..

    I'll raise it to 10 and try again..

    any thoughts ?


    -----Original Message-----
    From: Mark Sapiro
    Sent: Tuesday, January 25, 2011 1:18 AM
    To: Khalil Abbas ; mailman-users at python.org
    Subject: Re: [Mailman-Users] control mail distribution ..

    Khalil Abbas wrote:
    by the way, what's the default value for SMTP_MAX_RCPTS ??

    All the settings default values and their documentation is in
    Defaults.py.

    # Ceiling on the number of recipients that can be specified in a single
    SMTP
    # transaction. Set to 0 to submit the entire recipient list in one
    # transaction. Only used with the SMTPDirect DELIVERY_MODULE.
    SMTP_MAX_RCPTS = 500

    --
    Mark Sapiro <mark at msapiro.net> The highway is for gamblers,
    San Francisco Bay Area, California better use your sense - B. Dylan
  • Mark Sapiro at Jan 26, 2011 at 4:29 pm

    Khalil Abbas wrote:
    well I decreased the SMTP_MAX_RCPTS to only 5 (five) and when checking the
    mailq every now and then .. some of the 18 servers gave the following:

    -Queue ID- --Size-- ----Arrival Time---- -Sender/Recipient-------
    335BB14BB8CA 111316 Wed Jan 26 09:04:29 MAILER-DAEMON
    (unknown mail transport
    error)

    This looks like the MTA has some problem delivering a bounce DSN.

    and some of them:

    -Queue ID- --Size-- ----Arrival Time---- -Sender/Recipient-------
    7B477146C78 108725 Tue Jan 25 21:14:27
    balligho-bounces at ballighoseven.info
    (system resource
    problem)

    The MTA has been unable to deliver this message for the indicated
    reason. It has been successfully delivered from Mailman to the MTA.
    SMTP_MAX_RCPTS has nothing to do with this.

    I kept rebooting the servers every while to get the mail moving

    The need to do this indicates serious problems with your mail setup in
    general. I can't help you with this. I don't think I can help you
    further at all.


    .. and so
    far I received 27 messages out of 57 lists ..

    I'll raise it to 10 and try again..

    any thoughts ?

    Just the above.

    --
    Mark Sapiro <mark at msapiro.net> The highway is for gamblers,
    San Francisco Bay Area, California better use your sense - B. Dylan

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupmailman-users @
categoriespython
postedJan 17, '11 at 11:00p
activeJan 26, '11 at 4:29p
posts21
users2
websitelist.org

2 users in discussion

Mark Sapiro: 12 posts Khalil Abbas: 9 posts

People

Translate

site design / logo © 2022 Grokbase