FAQ
How is some processing working?

Someone send a mail to the list.
Mailserver received it.
Wrapper handled it and put the message to the qfiles dir.
So far so good here...

qrunner is running..
What does this qrunner script exacly doing with the qfiles dir?
I need to understand some things.

See my problem i described before on the list. It is now proccessing qfiles files but it is not yet good and fast enough.

I did change the default values to :

SMTP_MAX_RCPTS = 1
LIST_LOCK_LIFETIME = minutes(5)
LIST_LOCK_TIMEOUT = seconds(10)
QRUNNER_LOCK_LIFETIME = minutes(10)
QRUNNER_PROCESS_LIFETIME = minutes(5)
QRUNNER_MAX_MESSAGES = 5

With the default values from mailman it processes 1 message from the qfiles dir between 20 and 45 minutes.
Now it is processing 1 message from the qfiles dir between 4 and 10 minutes.

What i want is process all the files at the qfiles dir at once till its empty. So lets say, 50 messages at the qfiles dir is processed in a few minutes, and not 1 message in 10 minutes..
There are more incomming messages then qrunner processing for out. The qfiles dir is growing and growing and growing...

I am not talking about sending messages though the MTA. It is just qrunner that us doing nothing while there are so many *.msg/*.db files are to be processed.

I hope i am now clear enough :-)


btw, there are still al lot of *.msg/*.db files more than 4 ours old.. when are these processed?





Danny Terweij


-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.python.org/pipermail/mailman-users/attachments/20020426/886d3e34/attachment.htm

Search Discussions

  • J C Lawrence at Apr 26, 2002 at 10:34 pm

    On Fri, 26 Apr 2002 23:17:42 +0200 Danny Terweij wrote:

    qrunner is running.. What does this qrunner script exacly doing with
    the qfiles dir? I need to understand some things.
    See my problem i described before on the list. It is now proccessing
    qfiles files but it is not yet good and fast enough.
    Please see the FAQ:

    http://www.python.org/cgi-bin/faqw-mm.py

    --
    J C Lawrence
    ---------(*) Satan, oscillate my metallic sonatas.
    claw at kanga.nu He lived as a devil, eh?
    http://www.kanga.nu/~claw/ Evil is a name of a foeman, as I live.
  • Danny Terweij at Apr 26, 2002 at 10:55 pm
    Please explain where to look? (section, paragraph and so on..)

    Every one can say "see/read the FAQ". But i did not see any relevanting on
    my questions in the FAQ.

    Searching on : qrunner is slow, technical details qrunner, qfiles
    And more search strings.

    I got returned:

    Mailman FAQ Search Results
    No hits.

    Danny.

    ----- Original Message -----
    From: "J C Lawrence" <claw at kanga.nu>
    To: "Danny Terweij" <danny at terweij.nl>
    Cc: <mailman-users at python.org>
    Sent: Saturday, April 27, 2002 12:34 AM
    Subject: Re: [Mailman-Users] Still want it faster :-)

    On Fri, 26 Apr 2002 23:17:42 +0200
    Danny Terweij wrote:
    qrunner is running.. What does this qrunner script exacly doing with
    the qfiles dir? I need to understand some things.
    See my problem i described before on the list. It is now proccessing
    qfiles files but it is not yet good and fast enough.
    Please see the FAQ:

    http://www.python.org/cgi-bin/faqw-mm.py

    --
    J C Lawrence
    ---------(*) Satan, oscillate my metallic sonatas.
    claw at kanga.nu He lived as a devil, eh?
    http://www.kanga.nu/~claw/ Evil is a name of a foeman, as I live.
  • J C Lawrence at Apr 27, 2002 at 12:32 am

    On Sat, 27 Apr 2002 00:55:08 +0200 Danny Terweij wrote:

    Please explain where to look? (section, paragraph and so on..)
    The FAQ is not that long...
    Every one can say "see/read the FAQ". But i did not see any
    relevanting on my questions in the FAQ.
    You're not having a qrunner problem, but an MTA and DNS lookup problem.

    --
    J C Lawrence
    ---------(*) Satan, oscillate my metallic sonatas.
    claw at kanga.nu He lived as a devil, eh?
    http://www.kanga.nu/~claw/ Evil is a name of a foeman, as I live.
  • Danny Terweij at Apr 27, 2002 at 1:07 am
    From: "J C Lawrence" <claw at kanga.nu>
    Please explain where to look? (section, paragraph and so on..)
    The FAQ is not that long...
    Nope, just read almost all but no answers :-(
    Every one can say "see/read the FAQ". But i did not see any
    relevanting on my questions in the FAQ.
    You're not having a qrunner problem, but an MTA and DNS lookup problem.
    My question is , what are the technical details of qrunner?
    From starting qrunner till sending out the message.
    Then i maybe got the working of qrunner. Now i do not see any clue why
    MTA/DNS is related to picking up a message from the qfiles dir.

    My vision on qrunner :
    Heyyy i have got a new file in the qfile dir. lets doing something with it.
    When finished that mail.. lets check if there is another.. yeah i found
    another one.. lets do something with it... lets see. hmmzz. nothing
    left..EXIT. In meanwhile when nothing is there and qrunner did exit its
    proc, crontab starts every minute qrunner. When nothing found.. exit. When
    found files.. start the whole process again.
    (So i see it)

    So if anyone can me explain how it is working then i am happy.

    For now i do not think it is an MTA/DNS problem, but qrunner is the problem.
    It sits there in memory and is doing nothing at all. (besides checking
    between 4 and 10 minutes the qfiles dir).

    Why is every minute the qrunner logfile files with :
    Apr 25 02:00:15 2002 (10140) Could not acquire qrunner lock
    Apr 25 02:01:09 2002 (10203) Could not acquire qrunner lock
    Apr 25 02:02:11 2002 (10239) Could not acquire qrunner lock
    Apr 25 02:03:09 2002 (10251) Could not acquire qrunner lock
    Apr 25 02:04:09 2002 (10260) Could not acquire qrunner lock
    Apr 25 02:05:20 2002 (10270) Could not acquire qrunner lock
    Apr 25 02:06:09 2002 (10348) Could not acquire qrunner lock
    Apr 25 02:07:10 2002 (10384) Could not acquire qrunner lock
    Apr 25 02:08:09 2002 (10396) Could not acquire qrunner lock
    Apr 25 02:09:09 2002 (10404) Could not acquire qrunner lock
    Apr 25 02:10:23 2002 (10415) Could not acquire qrunner lock
    Apr 25 02:12:11 2002 (10520) Could not acquire qrunner lock
    Apr 25 02:13:09 2002 (10530) Could not acquire qrunner lock
    Apr 25 02:14:07 2002 (10534) Could not acquire qrunner lock
    Apr 25 02:15:17 2002 (10543) Could not acquire qrunner lock
    Apr 25 02:16:11 2002 (10605) Could not acquire qrunner lock
    Apr 25 02:17:09 2002 (10642) Could not acquire qrunner lock
    Apr 25 02:18:09 2002 (10646) Could not acquire qrunner lock
    Apr 25 02:19:08 2002 (10650) Could not acquire qrunner lock
    Apr 25 02:20:14 2002 (10664) Could not acquire qrunner lock
    Apr 25 02:21:10 2002 (10727) Could not acquire qrunner lock
    Apr 25 02:22:08 2002 (10770) Could not acquire qrunner lock
    Apr 25 02:23:08 2002 (10774) Could not acquire qrunner lock
    Apr 25 02:24:08 2002 (10778) Could not acquire qrunner lock

    This costs me lots of hard drive space for useless information.
    Is it a faillure?

    Danny.
  • J C Lawrence at Apr 27, 2002 at 1:24 am

    On Sat, 27 Apr 2002 03:07:42 +0200 Danny Terweij wrote:
    From: "J C Lawrence" <claw at kanga.nu>
    You're not having a qrunner problem, but an MTA and DNS lookup
    problem.
    My question is, what are the technical details of qrunner?
    Not much really, for the bit you're interested in it iterates across the
    qfiles directory taking each message in turn and processing/delivering
    it appropriately. There's little magic or mystery there -- look at the
    source -- its quite trivial.
    Then i maybe got the working of qrunner. Now i do not see any clue why
    MTA/DNS is related to picking up a message from the qfiles dir.
    Sure -- again, read the FAQ, and read the archives of this list.

    The basic story is that sendmail (which I recommend you move away from)
    is not accepting each address handed to it on a RCPT TO until it has
    done a DNS verify against the domain. This is slow. It is especially
    slow if you don't have a local cacheing name server. Even if you do it
    is slow. If name resolution is broken or poorly set up on your machine
    it can take up to 2 minutes per address just to attempt the name
    resolution.

    Summary: Install a cacheing name server and configure your MTA to not do
    DNS verifies on RCPT TOs. This is covered clearly and in great detail
    in the FAQ.
    My vision on qrunner : Heyyy i have got a new file in the qfile
    dir. lets doing something with it. When finished that mail.. lets
    check if there is another.. yeah i found another one.. lets do
    something with it... lets see. hmmzz. nothing left..EXIT. In meanwhile
    when nothing is there and qrunner did exit its proc, crontab starts
    every minute qrunner. When nothing found.. exit. When found
    files.. start the whole process again. (So i see it) Right.
    For now i do not think it is an MTA/DNS problem, but qrunner is the
    problem. It sits there in memory and is doing nothing at
    all. (besides checking between 4 and 10 minutes the qfiles dir). <sigh>
    This costs me lots of hard drive space for useless information. Is it
    a faillure?
    No, a warning.

    --
    J C Lawrence
    ---------(*) Satan, oscillate my metallic sonatas.
    claw at kanga.nu He lived as a devil, eh?
    http://www.kanga.nu/~claw/ Evil is a name of a foeman, as I live.
  • Danny Terweij at Apr 27, 2002 at 1:37 am
    ----- Original Message -----
    From: "J C Lawrence" <claw at kanga.nu>
    Then i maybe got the working of qrunner. Now i do not see any clue why
    MTA/DNS is related to picking up a message from the qfiles dir.
    Sure -- again, read the FAQ, and read the archives of this list.
    The basic story is that sendmail (which I recommend you move away from)
    is not accepting each address handed to it on a RCPT TO until it has
    done a DNS verify against the domain. This is slow. It is especially
    slow if you don't have a local cacheing name server. Even if you do it
    is slow. If name resolution is broken or poorly set up on your machine
    it can take up to 2 minutes per address just to attempt the name
    resolution.

    Summary: Install a cacheing name server and configure your MTA to not do
    DNS verifies on RCPT TOs. This is covered clearly and in great detail
    in the FAQ.
    You are reffering to :

    /usr/sbin/sendmail -bd -ODeliveryModeÞfer \
    -ODaemonPortOptions=Name=MSA,Port=NNNN,M=E,Addr7.0.0.1
    ?

    And i use :

    DELIVERY_MODULE = 'SMTPDirect'
    # SMTP host and port, when DELIVERY_MODULE is 'SMTPDirect'
    SMTPHOST = '127.0.0.1'
    SMTPPORT = 25

    127.0.0.1 is sendmail.
    This is doing the DNS lookups?
    Can i just do SMTPHOST = '192.168.0.1' witch is another SMTP server at my
    LAN (windows 2000 server with Mailtraq )?
    Is it then not using sendmail and DNS troubles at all?
    My vision on qrunner : Heyyy i have got a new file in the qfile
    [cut]
    Right.
    Thank you.
    This costs me lots of hard drive space for useless information. Is it
    a faillure?
    No, a warning.
    What are the actions to prevent it?
    So this file must be in normal operation empty?


    Danny.
  • J C Lawrence at Apr 27, 2002 at 2:31 am

    On Sat, 27 Apr 2002 03:37:58 +0200 Danny Terweij wrote:

    You are reffering to :
    /usr/sbin/sendmail -bd -ODeliveryModeÞfer \
    -ODaemonPortOptions=Name=MSA,Port=NNNN,M=E,Addr7.0.0.1 ?
    That's one possible address, tho not one I'm fond of.
    And i use :
    DELIVERY_MODULE = 'SMTPDirect' # SMTP host and port, when
    DELIVERY_MODULE is 'SMTPDirect' SMTPHOST = '127.0.0.1' SMTPPORT = 25
    127.0.0.1 is sendmail. This is doing the DNS lookups?
    Note that NNN above is not 25.

    In the case that you are running against a reasonably default Sendmail
    setup, yes, sendmail will be calling gethostbyname() and other
    sillinesses during the SMTP transaction. Do an strace to see.
    Can i just do SMTPHOST = '192.168.0.1' witch is another SMTP server at
    my LAN (windows 2000 server with Mailtraq )? Is it then not using
    sendmail and DNS troubles at all?
    1) Yes.

    2) No, it won't use your local MTA at all.

    3) That depends on the configuration of your other MTA (I don't know or
    use Microsoft tools and so can't answer there).
    What are the actions to prevent it? So this file must be in normal
    operation empty?
    Get your mail system fast enough that deliveries are able to happen
    before new instances of qrunner are spawned.

    Note: I run several multi-thousand subscriber lists and I have no lock
    contention.

    --
    J C Lawrence
    ---------(*) Satan, oscillate my metallic sonatas.
    claw at kanga.nu He lived as a devil, eh?
    http://www.kanga.nu/~claw/ Evil is a name of a foeman, as I live.
  • J C Lawrence at Apr 27, 2002 at 2:50 am

    On Fri, 26 Apr 2002 19:31:57 -0700 J C Lawrence <J> wrote:

    Note: I run several multi-thousand subscriber lists and I have no lock
    contention.
    Minor stats:

    On a not very chunky system (PII-333) running a fairly untuned postfix
    can reliably sustain 1400 remote deliveries per minute (its actually a
    pretty stock Debian/testing install). The same system, which has a
    reasonable but not great IO chain (F/W SCSI III, single partition for
    /var which also holds /var/logs), can reliably spool (for remote
    delivery) in excess of 8,000 messages per minute (ie discrete SMTP
    transactions, not just RCPT TO's). I don't know if it can do more or
    what the actual top value is as I've not tried to find it and I've not
    had enough traffic to sustain loads at that level long enough.

    Note: Qrunner in 2.0 won't sustain that rate of spool entries as it is
    single threaded. I've created and sustained that level in previous
    tests running local custom qrunner variants against a hashed/split
    qfiles directory and thus plexing Postfix that way.

    You've talked previously of lists with low double digit numbers of
    subscribers, and of having what you insist are qrunner performance
    problems at that level.

    Again, qrunner is not your problem. An i486-33 with a bad IDE IO chain
    can easily, trivially, reliably sustain 20+ SMTP transactions (spool or
    remote delivery) per minute (I've done this).

    --
    J C Lawrence
    ---------(*) Satan, oscillate my metallic sonatas.
    claw at kanga.nu He lived as a devil, eh?
    http://www.kanga.nu/~claw/ Evil is a name of a foeman, as I live.
  • Danny Terweij at Apr 27, 2002 at 8:39 am
    From: "J C Lawrence" <claw at kanga.nu>
    On Sat, 27 Apr 2002 03:37:58 +0200
    Danny Terweij wrote:
    You are reffering to :
    /usr/sbin/sendmail -bd -ODeliveryModeÞfer \
    -ODaemonPortOptions=Name=MSA,Port=NNNN,M=E,Addr7.0.0.1 ?
    That's one possible address, tho not one I'm fond of.
    And i use :
    DELIVERY_MODULE = 'SMTPDirect' # SMTP host and port, when
    DELIVERY_MODULE is 'SMTPDirect' SMTPHOST = '127.0.0.1' SMTPPORT = 25
    127.0.0.1 is sendmail. This is doing the DNS lookups?
    Note that NNN above is not 25.

    I did implement above.(started sendmail with port 250 and SMTPPORT = 250)
    Still same result. processing 1 message between 4-10 minutes.
    Then i did SMTPHOST = '192.162.0.1' SMTPPORT = 25 .... No results. also
    slow.

    ** Now i am stuck with it, i am sure when i install exim or postfix it will
    be slow also. **

    My System :
    [root at lnx root]# free
    total used free shared buffers cached
    Mem: 94120 91832 2288 4 2292 12052
    -/+ buffers/cache: 77488 16632
    Swap: 192772 54628 138144
    [root at lnx root]#

    Pentium 120
    RH 7.2 (No GUI).
    2x 40Gb HD , 7200RPM IDE

    Stats :
    10:42am up 21 min, 2 users, load average: 0.64, 1.17, 0.87
    70 processes: 68 sleeping, 2 running, 0 zombie, 0 stopped
    CPU states: 42.4% user, 4.8% system, 0.0% nice, 52.7% idle
    Mem: 94120K av, 56748K used, 37372K free, 4K shrd, 2684K
    buff
    Swap: 192772K av, 29868K used, 162904K free 37316K
    cached



    Danny.
  • Harold Paulson at Apr 27, 2002 at 12:17 am
    Danny,

    Here are some ways I have discovered of making Mailman really painfully slow:

    1) Use Sendmail for your outgoing MTA and have it do DNS checks on
    relayed mail. Pick an email address out on the net and test it from
    your Mailman machine with something like 'mail -v
    youraccount at hotmail.com'. By the way, turning off DNS checks in
    sendmail usually makes it an open relay. Yeah!

    2) Have slow or broken DNS. What happens when you do something like,
    'nslookup hotmail.com' from your Mailman server?

    3) Leave old lock files lying around in ~mailman/locks/. This works
    especially well on really old versions of mailman.

    4) Populate your mailing lists with email addresses from the corners of
    the earth with the worst connections and shakiest DNS.

    Good luck!

    - H
    How is some processing working?

    Someone send a mail to the list.
    Mailserver received it.
    Wrapper handled it and put the message to the qfiles dir.
    So far so good here...

    qrunner is running..
    What does this qrunner script exacly doing with the qfiles dir?
    I need to understand some things.

    See my problem i described before on the list. It is now proccessing
    qfiles files but it is not yet good and fast enough.

    I did change the default values to :

    SMTP_MAX_RCPTS = 1
    LIST_LOCK_LIFETIME = minutes(5)
    LIST_LOCK_TIMEOUT = seconds(10)
    QRUNNER_LOCK_LIFETIME = minutes(10)
    QRUNNER_PROCESS_LIFETIME = minutes(5)
    QRUNNER_MAX_MESSAGES = 5

    With the default values from mailman it processes 1 message from the
    qfiles dir between 20 and 45 minutes.
    Now it is processing 1 message from the qfiles dir between 4 and 10 minutes.

    What i want is process all the files at the qfiles dir at once till
    its empty. So lets say, 50 messages at the qfiles dir is processed
    in a few minutes, and not 1 message in 10 minutes..
    There are more incomming messages then qrunner processing for out.
    The qfiles dir is growing and growing and growing...

    I am not talking about sending messages though the MTA. It is just
    qrunner that us doing nothing while there are so many *.msg/*.db
    files are to be processed.

    I hope i am now clear enough :-)


    btw, there are still al lot of *.msg/*.db files more than 4 ours
    old.. when are these processed?





    Danny Terweij

    --

    Harold Paulson Sierra Web Design
    haroldp at sierraweb.com http://www.sierraweb.com
    VOICE: 775.833.9500 FAX: 775-201-4458
  • Danny Terweij at Apr 27, 2002 at 12:50 am
    From: "Harold Paulson" <haroldp at sierraweb.com>
    Danny, Hi Paul.
    Here are some ways I have discovered of making Mailman really painfully slow:
    1) Use Sendmail for your outgoing MTA and have it do DNS checks on
    relayed mail. Pick an email address out on the net and test it from
    your Mailman machine with something like 'mail -v
    youraccount at hotmail.com'. By the way, turning off DNS checks in
    sendmail usually makes it an open relay. Yeah!
    [root at lnx root]# mail -v xxx at hotmail.com
    Subject: Test
    Cc:
    xxx at hotmail.com... Connecting to xx.xx.xx. via relay...
    220 xx.xx.xx Ready for action (Mailtraq 1.1.6.1177/SMTP)
    EHLO lnx.xx.xx.xx
    250 xx.xx.xx
    MAIL From:<root at lnx.xx.xx.xx>
    250 receiving from root at lnx.xx.xx.xx
    RCPT To:<xxx at hotmail.com>
    250 will send to xxx at hotmail.com
    DATA
    354 send the message, terminate with "."
    .
    250 received the message, thanks
    xx at hotmail.com... Sent (received the message, thanks)
    Closing connection to xx.xx.xx.
    QUIT
    221 have a nice day (SMTP Closing)
    [root at lnx root]#

    Very fast. No problem here.
    2) Have slow or broken DNS. What happens when you do something like,
    'nslookup hotmail.com' from your Mailman server?
    [root at lnx root]# nslookup hotmail.com
    Note: nslookup is deprecated and may be removed from future releases.
    Consider using the `dig' or `host' programs instead. Run nslookup with
    the `-sil[ent]' option to prevent this message from appearing.
    Server: 192.168.0.1
    Address: 192.168.0.1#53

    Non-authoritative answer:
    Name: hotmail.com
    Address: 64.4.54.7
    Name: hotmail.com
    Address: 64.4.43.7
    Name: hotmail.com
    Address: 64.4.44.7
    Name: hotmail.com
    Address: 64.4.45.7
    Name: hotmail.com
    Address: 64.4.52.7
    Name: hotmail.com
    Address: 64.4.53.7

    [root at lnx root]#

    And with dig :

    [root at lnx root]# dig hotmail.com

    ; <<>> DiG 9.1.3 <<>> hotmail.com
    ;; global options: printcmd
    ;; Got answer:
    ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 24642
    ;; flags: qr aa rd ra; QUERY: 1, ANSWER: 6, AUTHORITY: 4, ADDITIONAL: 4

    ;; QUESTION SECTION:
    ;hotmail.com. IN A

    ;; ANSWER SECTION:
    hotmail.com. 3600 IN A 64.4.54.7
    hotmail.com. 3600 IN A 64.4.43.7
    hotmail.com. 3600 IN A 64.4.44.7
    hotmail.com. 3600 IN A 64.4.45.7
    hotmail.com. 3600 IN A 64.4.52.7
    hotmail.com. 3600 IN A 64.4.53.7

    ;; AUTHORITY SECTION:
    hotmail.com. 3600 IN NS ns1.hotmail.com.
    hotmail.com. 3600 IN NS ns2.hotmail.com.
    hotmail.com. 3600 IN NS ns3.hotmail.com.
    hotmail.com. 3600 IN NS ns4.hotmail.com.

    ;; ADDITIONAL SECTION:
    ns1.hotmail.com. 3600 IN A 216.200.206.140
    ns2.hotmail.com. 3600 IN A 216.200.206.139
    ns3.hotmail.com. 3600 IN A 209.185.130.68
    ns4.hotmail.com. 3600 IN A 64.4.29.24

    ;; Query time: 189 msec
    ;; SERVER: 192.168.0.1#53(192.168.0.1)
    ;; WHEN: Sat Apr 27 02:48:49 2002
    ;; MSG SIZE rcvd: 261

    [root at lnx root]#

    Is this good or bad?

    3) Leave old lock files lying around in ~mailman/locks/. This works
    especially well on really old versions of mailman.
    No old lock files there.
    4) Populate your mailing lists with email addresses from the corners of
    the earth with the worst connections and shakiest DNS.
    :-) I am in holland and only dutch and some belgium people subscribed.
    So no strange corners here also :-)

    Good luck!
    I did change the default values to :

    SMTP_MAX_RCPTS = 1
    LIST_LOCK_LIFETIME = minutes(5)
    LIST_LOCK_TIMEOUT = seconds(10)
    QRUNNER_LOCK_LIFETIME = minutes(10)
    QRUNNER_PROCESS_LIFETIME = minutes(5)
    QRUNNER_MAX_MESSAGES = 5
    But are these fine or not? What are good value's?


    Danny.

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupmailman-users @
categoriespython
postedApr 26, '02 at 9:17p
activeApr 27, '02 at 8:39a
posts12
users3
websitelist.org

People

Translate

site design / logo © 2022 Grokbase