FAQ

[Tomcat-users] Error 503 ocurring when server under load

Rob G
Oct 7, 2010 at 4:59 pm
Hey all,

Recently migrated a production site (mixture of Servlets and JSPs)
from Oracle Application Server to Apache/Tomcat. Since then we have
seen numerous HTTP Error 503 - Service unavailable errors at peak
times when site is under load. mod_jk.log has the following error
message(s):

[2184:1952] [error] jk_lb_worker.c (1473): All tomcat instances
failed, no more workers left

I'm looking for help in trying to tweak settings to prevent this, or
confirmation that I've configured the setup correctly.

Many thanks
Rob

Platform:
Windows Server 2003 SP2

Setup:
Two tomcat instances with a single Apache front end, all on the same server

Versions
Tomcat: 6.0.24
Apache: 2.2.16
mod SSL: 2.2.16
Open SSL: 0.9.8
mod_JK:1.2.30

Conf:
httpd.conf:

<IfModule mod_jk.c>

JkWorkersFile conf/workers.properties
JkLogFile logs/mod_jk.log
JkLogLevel error
JkLogStampFormat "[%a %b %d %H:%M:%S %Y] "
JkOptions +ForwardKeySize +ForwardURICompat -ForwardDirectories
JkRequestLogFormat "%w %V %T"

<Location /*/WEB-INF/*>
AllowOverride None
deny from all
</Location>

JkMount /examples balancer
JkMount /examples/* balancer
JkMount /App1/* balancer
JkMount /App2/* balancer
</IfModule>

Workers.properties:
workers.tomcat_home="C:\tomcat1"
workers.java_home="C:\Program Files\Java\jdk1.6.0_18"
ps=\

# Define workers
#worker.list=worker1,worker2
worker.list=balancer

# Set properties for worker worker1 (ajp13)
worker.worker1.type=ajp13
worker.worker1.host=localhost
worker.worker1.port=8009
worker.worker1.lbfactor=1

worker.worker1.cachesize=10
worker.worker1.connection_pool_timeout=300
worker.worker1.socket_keepalive=FALSE
#worker.worker1.recycle_timeout=300

# Set properties for worker worker2 (ajp13)
worker.worker2.type=ajp13
worker.worker2.host=localhost
worker.worker2.port=8019
worker.worker2.lbfactor=1

worker.worker2.cachesize=10
worker.worker2.connection_pool_timeout=300
worker.worker2.socket_keepalive=FALSE
#worker.worker2.recycle_timeout=300

worker.balancer.type=lb
worker.balancer.balance_workers=worker1,worker2
worker.balancer.method=R
worker.balancer.sticky_session=True

Server.xml (same for both tomcat instances apart from jvmRoute):
<?xml version='1.0' encoding='utf-8'?>
<Server port="8005" shutdown="SHUTDOWN">

<Listener className="org.apache.catalina.core.AprLifecycleListener"
SSLEngine="on" />
<Listener className="org.apache.catalina.core.JasperListener" />
<Listener className="org.apache.catalina.core.JreMemoryLeakPreventionListener"
/>
<Listener className="org.apache.catalina.mbeans.ServerLifecycleListener" />
<Listener className="org.apache.catalina.mbeans.GlobalResourcesLifecycleListener"
/>


<GlobalNamingResources>

<Resource name="UserDatabase" auth="Container"
type="org.apache.catalina.UserDatabase"
description="User database that can be updated and saved"
factory="org.apache.catalina.users.MemoryUserDatabaseFactory"
pathname="conf/tomcat-users.xml" />
</GlobalNamingResources>

<Service name="Catalina">

<Connector port="8080" protocol="HTTP/1.1"
connectionTimeout="40000"
redirectPort="8443" />

<Connector port="8009" protocol="AJP/1.3" redirectPort="8443" />

<Engine name="Catalina" defaultHost="localhost" jvmRoute="worker1">

<Realm className="org.apache.catalina.realm.UserDatabaseRealm"
resourceName="UserDatabase"/>

<Host name="localhost" appBase="webapps"
unpackWARs="true" autoDeploy="true"
xmlValidation="false" xmlNamespaceAware="false">

</Host>
</Engine>
</Service>
</Server>

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
reply

Search Discussions

16 responses

  • André Warnier at Oct 7, 2010 at 5:32 pm

    Rob G wrote:
    Hey all,

    Recently migrated a production site (mixture of Servlets and JSPs)
    from Oracle Application Server to Apache/Tomcat. Since then we have
    seen numerous HTTP Error 503 - Service unavailable errors at peak
    times when site is under load. mod_jk.log has the following error
    message(s):

    [2184:1952] [error] jk_lb_worker.c (1473): All tomcat instances
    failed, no more workers left

    I'm looking for help in trying to tweak settings to prevent this, or
    confirmation that I've configured the setup correctly.

    Many thanks
    Rob

    Platform:
    Windows Server 2003 SP2

    Setup:
    Two tomcat instances with a single Apache front end, all on the same server

    Versions
    Tomcat: 6.0.24
    Apache: 2.2.16
    mod SSL: 2.2.16
    Open SSL: 0.9.8
    mod_JK:1.2.30
    Your configuration looks very clean to me (no unnecessary settings etc.), which in this
    case is a plus (a good base to start tuning).
    You may want to upgrade Tomcat to the latest version (6.0.29).

    But before you start tuning, you should get some idea of what is actually going on.

    For example, at the moment these errors happen, what are these Tomcats really doing ?
    Are they really busy each processing 200 requests, with 200 threads running and actually
    doing something ? (200 is the default for the "maxThreads" attribute of the AJP Connector).

    If yes, then you may just need a leaner application, or a bigger system (more RAM, faster
    CPU), or more systems. What does the Task Manager tell you about the total system load ?

    If not, and many of these threads are waiting, then you may have an issue with a keepAlive
    that is too long.
    See http://tomcat.apache.org/tomcat-6.0-doc/config/ajp.html, for the connectionTimeout and
    keepAliveTimeout attributes.

    Whatever you do, first get an idea of the starting situation. Then modify one setting at
    a time, and observe (and note) the effects.


    ---------------------------------------------------------------------
    To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
    For additional commands, e-mail: users-help@tomcat.apache.org
  • Pid at Oct 7, 2010 at 9:41 pm

    On 07/10/2010 18:31, André Warnier wrote:
    Rob G wrote:
    Hey all,

    Recently migrated a production site (mixture of Servlets and JSPs)
    from Oracle Application Server to Apache/Tomcat. Since then we have
    seen numerous HTTP Error 503 - Service unavailable errors at peak
    times when site is under load. mod_jk.log has the following error
    message(s):
    OK. Is there anything else different apart from the Servlet container?
    [2184:1952] [error] jk_lb_worker.c (1473): All tomcat instances
    failed, no more workers left
    Seems like your Tomcats are maxed out.
    I'm looking for help in trying to tweak settings to prevent this, or
    confirmation that I've configured the setup correctly.
    Platform:
    Windows Server 2003 SP2

    Setup:
    Two tomcat instances with a single Apache front end, all on the same
    server
    Why do you have two Tomcat instances? (It's not a trick question, I'm
    interested in your reasoning.)
    Versions
    Tomcat: 6.0.24
    Apache: 2.2.16
    mod SSL: 2.2.16
    Open SSL: 0.9.8
    There's newer OpenSSL available, with important security fixes, if I'm
    not mistaken.
    mod_JK:1.2.30
    Your configuration looks very clean to me (no unnecessary settings
    etc.), which in this case is a plus (a good base to start tuning).
    Was there an attachment I didn't see?
    You may want to upgrade Tomcat to the latest version (6.0.29). +1
    But before you start tuning, you should get some idea of what is
    actually going on. +1
    For example, at the moment these errors happen, what are these Tomcats
    really doing ?
    Are they really busy each processing 200 requests, with 200 threads
    running and actually doing something ? (200 is the default for the
    "maxThreads" attribute of the AJP Connector).
    Q: How many threads & server instances did you have before?
    If yes, then you may just need a leaner application, or a bigger system
    (more RAM, faster CPU), or more systems. What does the Task Manager
    tell you about the total system load ?
    What were your Java -Xmx etc settings before, and now?
    If not, and many of these threads are waiting, then you may have an
    issue with a keepAlive that is too long.
    Or several other things.

    What is your Connector config?

    Are you using an Executor?

    Is there a database behind this, if so, what are the DataSource pool
    size settings?
    See http://tomcat.apache.org/tomcat-6.0-doc/config/ajp.html, for the
    connectionTimeout and keepAliveTimeout attributes.

    Whatever you do, first get an idea of the starting situation. Then
    modify one setting at a time, and observe (and note) the effects.
    Thread dumps from a maxed out Tomcat will tell you what each Thread is
    waiting for. Collect a series of these during high load periods to find
    out what's happening.

    Enable JMX and check the Connector, (Executor if enabled), and
    DataSources; I'd be looking at backlog of requests, active + idle pool
    members, total pool size.


    p
    ---------------------------------------------------------------------
    To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
    For additional commands, e-mail: users-help@tomcat.apache.org
  • Rob G at Oct 7, 2010 at 11:07 pm
    Thanks to all for their replies to date. Much appreciated, I'll
    respond to some of the points raised. Apologies if I'm asking basic
    stuff but still getting my ahead around Tomcat and Apache integration.
    On 7 October 2010 18:31, André Warnier wrote:

    Your configuration looks very clean to me (no unnecessary settings etc.),
    which in this case is a plus (a good base to start tuning). Thank you
    You may want to upgrade Tomcat to the latest version (6.0.29). Noted
    For example, at the moment these errors happen, what are these Tomcats
    really doing ?
    Are they really busy each processing 200 requests, with 200 threads running
    and actually doing something ? (200 is the default for the "maxThreads"
    attribute of the AJP Connector).
    Unfortunately I'm not in a position to conduct realtime monitoring of
    the server and Tomcat. Is there a way to log this thread data?
    If yes, then you may just need a leaner application, or a bigger system
    (more RAM, faster CPU), or more systems.  What does the Task Manager tell
    you about the total system load ?
    Again, unfortunately not in a position to monitor in real time.
    If not, and many of these threads are waiting, then you may have an issue
    with a keepAlive that is too long.
    See http://tomcat.apache.org/tomcat-6.0-doc/config/ajp.html, for the
    connectionTimeout and keepAliveTimeout attributes. Will do
    Whatever you do, first get an idea of the starting situation.  Then modify
    one setting at a time, and observe (and note) the effects.
    Sensible advice

    Pid wrote:
    OK. Is there anything else different apart from the Servlet container?
    I'm not sure what you mean?
    Why do you have two Tomcat instances? (It's not a trick question, I'm interested in your reasoning.)
    Load balancing. I thought/guessed that two instances would be
    sufficent. Perhaps this was incorrect and I need more?
    Was there an attachment I didn't see?
    No, all details for .conf files and server.xml was posted inline.
    How many threads & server instances did you have before?
    What were your Java -Xmx etc settings before, and now?
    Are you using an Executor?
    I'll have to check, but I think it's safe to say that they are set to
    the default values as I don't recall changing anything relating to
    them.
    Thread dumps from a maxed out Tomcat will tell you what each Thread is waiting for. Collect a series of these during high load periods to find
    out what's happening.
    I'll have to figure out how to do this, but sounds a good idea.

    Christopher Schultz wrote
    I think you might want to compare the request processor counts you have for Tomcat versus mod_jk: the default maxThreads for a Tomcat <Connector> is 200, while your MaxClients settings might be different on your Apache httpd configuration.
    Excuse my ignorance, but can you clarify where I check these settings?
    I believe that only the Worker MPM is available on Microsoft Windows. Have you explicitly configured an MPM?
    Since I don't know what an MPM is I'm going to say no.
    Since you're using Apache 2.x, mod_jk should be able to auto-configure this value for you.
    Do I need to set anything to allow this auto config to happen?
    From my reading, you may be limiting yourself to 10 simultaneous connections from Apache httpd to Tomcat, and others will get a 503.
    What lead you to that conclusion?

    ---------------------------------------------------------------------
    To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
    For additional commands, e-mail: users-help@tomcat.apache.org
  • Pid at Oct 8, 2010 at 8:32 am

    On 08/10/2010 00:07, Rob G wrote:
    Thanks to all for their replies to date. Much appreciated, I'll
    respond to some of the points raised. Apologies if I'm asking basic
    stuff but still getting my ahead around Tomcat and Apache integration.
    On 7 October 2010 18:31, André Warnier wrote:

    Your configuration looks very clean to me (no unnecessary settings etc.),
    which in this case is a plus (a good base to start tuning). Thank you
    You may want to upgrade Tomcat to the latest version (6.0.29). Noted
    For example, at the moment these errors happen, what are these Tomcats
    really doing ?
    Are they really busy each processing 200 requests, with 200 threads running
    and actually doing something ? (200 is the default for the "maxThreads"
    attribute of the AJP Connector).
    Unfortunately I'm not in a position to conduct realtime monitoring of
    the server and Tomcat. Is there a way to log this thread data?
    How do you get log data?
    If yes, then you may just need a leaner application, or a bigger system
    (more RAM, faster CPU), or more systems. What does the Task Manager tell
    you about the total system load ?
    Again, unfortunately not in a position to monitor in real time.
    You can enable JMX and connect remotely on a specified port to get a
    look inside the JVM>
    If not, and many of these threads are waiting, then you may have an issue
    with a keepAlive that is too long.
    See http://tomcat.apache.org/tomcat-6.0-doc/config/ajp.html, for the
    connectionTimeout and keepAliveTimeout attributes. Will do
    Whatever you do, first get an idea of the starting situation. Then modify
    one setting at a time, and observe (and note) the effects.
    Sensible advice

    Pid wrote:
    OK. Is there anything else different apart from the Servlet container?
    I'm not sure what you mean?
    Did you change DB too, or is was it a simple migration?
    Why do you have two Tomcat instances? (It's not a trick question, I'm interested in your reasoning.)
    Load balancing. I thought/guessed that two instances would be
    sufficent. Perhaps this was incorrect and I need more?
    Maybe, maybe not.

    I was trying to understand why you had two instances, rather than one
    with, say, more threads configured.

    Load balancing two Tomcat instances which are both using the same
    underlying computing resource will not offer a tangible benefit in terms
    of throughput / response time compared to a suitably configured single
    instance.

    If you're on a 32bit system and are hitting process memory limits, you
    might need two instances to cope with the load.

    How many threads & server instances did you have before?
    What were your Java -Xmx etc settings before, and now?
    Are you using an Executor?
    I'll have to check, but I think it's safe to say that they are set to
    the default values as I don't recall changing anything relating to
    them.
    Thread dumps from a maxed out Tomcat will tell you what each Thread is waiting for. Collect a series of these during high load periods to find
    out what's happening.
    I'll have to figure out how to do this, but sounds a good idea.
    Do you have remote access to the machine?

    There are command line tools with JDK 6 (6.0.21 is the latest) which are
    useful. E.g. jstack, jmap.

    If you have remote access to the GUI, JDK 6 also has JConsole and
    VisualVM which provide metric tonnes of useful information.

    Otherwise you can connect those tools, for slightly less output,
    remotely via JMX. Be sure to use :

    org.apache.catalina.mbeans.JmxRemoteLifecycleListener
    http://tomcat.apache.org/tomcat-6.0-doc/config/listeners.html


    Please also check the port numbers you have in use for each Tomcat
    instance. Are the Tomcats installed and run as a service?


    p
    Christopher Schultz wrote
    I think you might want to compare the request processor counts you have for Tomcat versus mod_jk: the default maxThreads for a Tomcat <Connector> is 200, while your MaxClients settings might be different on your Apache httpd configuration.
    Excuse my ignorance, but can you clarify where I check these settings?
    I believe that only the Worker MPM is available on Microsoft Windows. Have you explicitly configured an MPM?
    Since I don't know what an MPM is I'm going to say no.
    Since you're using Apache 2.x, mod_jk should be able to auto-configure this value for you.
    Do I need to set anything to allow this auto config to happen?
    From my reading, you may be limiting yourself to 10 simultaneous connections from Apache httpd to Tomcat, and others will get a 503.
    What lead you to that conclusion?

    ---------------------------------------------------------------------
    To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
    For additional commands, e-mail: users-help@tomcat.apache.org
  • Christopher Schultz at Oct 8, 2010 at 2:21 pm
    Rob,
    On 10/7/2010 7:07 PM, Rob G wrote:
    Christopher Schultz wrote
    I think you might want to compare the request processor counts you
    have for Tomcat versus mod_jk: the default maxThreads for a Tomcat
    <Connector> is 200, while your MaxClients settings might be
    different on your Apache httpd configuration.
    Excuse my ignorance, but can you clarify where I check these settings?
    Obviously, the worker settings are in your Workers.properties file that
    you've already posted. The settings for Apache httpd are usually in a
    file called conf/httpd.conf in the Apache httpd installation directory.
    On win32, that might be something like C:\Program Files\Apache
    Foundation\Apache Web Server\conf\httpd.conf

    Tomcat's server settings are in conf/server.xml in the Apache Tomcat
    installation directory. On win32, that might be something like
    C:\Program Files\Apache Foundation\Apache Tomcat\conf\server.xml

    Since you have two copies of Tomcat working, you should check both of
    them. In fact, if you could post the server.xml files from both
    instances and tell us how you have them configured to run side-by-side,
    that would be helpful.

    Pid pointed out that you might not have both instances running due to a
    shutdownPort conflict. That might also be the case if you have other
    ports that conflict.
    I believe that only the Worker MPM is available on Microsoft Windows. Have you explicitly configured an MPM?
    Since I don't know what an MPM is I'm going to say no.
    :) That stands for "Multi-Processing Module" and it's how the Apache Web
    Server serves more than one request at a time. On *NIX, you can choose
    the "prefork" module which gives you one child process per simultaneous
    request, or the "worker" module which gets you a single (child?) process
    with many threads running within it. I just checked the documentation
    and it looks like on windows, you get mpm_winnt, which is different from
    even those, though it quite resembles the mpm_worker.
    Since you're using Apache 2.x, mod_jk should be able to
    auto-configure this value for you.
    Do I need to set anything to allow this auto config to happen?
    No. If you don't set any value, mod_jk will compute a reasonable (and
    usually correct) default. For a prefork MPM, having more than a single
    connection in the connection pool is a huge waste, since a single httpd
    request processor can't make more than one connection to Tomcat on the
    backend. With the worker MPM, the connection pool should probably be the
    same size as the number of worker threads you have configured.
    Otherwise, you'll have a deficit that may cause the errors you are
    encountering.
    From my reading, you may be limiting yourself to 10 simultaneous
    connections from Apache httpd to Tomcat, and others will get a
    503.
    What lead you to that conclusion?
    Here's the text from the docs:

    "
    This directive has been deprecated since 1.2.16.
    Cachesize defines the number of connections made to the AJP backend that
    are maintained as a connection pool. It will limit the number of those
    connection that each web server child process can make.

    Cachesize property is used only for multi threaded web servers such as
    Apache 2.0 (worker), IIS and Netscape. The cachesize property should
    reflect the number of threads per child process. JK will discover the
    number of threads per child process on Apache 2 web server with
    worker-mpm and set its default value to match the ThreadsPerChild
    Apache directive. [...]
    "

    Since mpm_winnt (like mpm_worker) uses ThreadsPerChild to set the number
    of worker threads (and therefore simultaneous requests that can be
    accepted), you'll have to know what ThreadsPerChild is set to. That's
    easy: just search for that directive in httpd.conf. If it's not there,
    or commented-out, the default is 64 on mpm_winnt
    (http://httpd.apache.org/docs/2.0/mod/mpm_common.html#threadsperchild).

    If you've left it as the default (64) and you've set worker.cachesize
    (really worker.connection_pool_size) to 10, then you can accept 64
    connections from clients but only call-out to Tomcat over 10 of those
    connections: the others will have to wait. I'm not sure what the
    scheduling algorithm is for choosing which thread gets the next
    available connection from the pool (FIFO?), but it's very possible to
    starve your clients even after 10 simultaneous connections.

    - -chris
    ---------------------------------------------------------------------
    To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
    For additional commands, e-mail: users-help@tomcat.apache.org
  • Rob G at Oct 11, 2010 at 8:41 pm
    Christopher Schultz wrote> This directive has been deprecated since 1.2.16.
    Cachesize defines the number of connections made to the AJP backend that
    are maintained as a connection pool. It will limit the number of those
    connection that each web server child process can make.

    Cachesize property is used only for multi threaded web servers such as
    Apache 2.0 (worker), IIS and Netscape. The cachesize property should
    reflect the number of threads per child process. JK will discover the
    number of threads per child process on Apache 2 web server with
    worker-mpm and set its default value to match the ThreadsPerChild
    Apache directive. [...]
    "

    Since mpm_winnt (like mpm_worker) uses ThreadsPerChild to set the number
    of worker threads (and therefore simultaneous requests that can be
    accepted), you'll have to know what ThreadsPerChild is set to. That's
    easy: just search for that directive in httpd.conf. If it's not there,
    or commented-out, the default is 64 on mpm_winnt
    (http://httpd.apache.org/docs/2.0/mod/mpm_common.html#threadsperchild).

    If you've left it as the default (64) and you've set worker.cachesize
    (really worker.connection_pool_size) to 10, then you can accept 64
    connections from clients but only call-out to Tomcat over 10 of those
    connections: the others will have to wait. I'm not sure what the
    scheduling algorithm is for choosing which thread gets the next
    available connection from the pool (FIFO?), but it's very possible to
    starve your clients even after 10 simultaneous connections.

    - -chris
    So if I'm reading your email and the docs correctly. I should just
    comment out the cachesize=10 from the workers.properties. And since
    for connection_pool_size (that replaced it) JK will discover this
    number for the Apache web server automatically and set the pool size
    to this value, I don't need to add anything to the workers.properties
    file?

    ---------------------------------------------------------------------
    To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
    For additional commands, e-mail: users-help@tomcat.apache.org
  • Christopher Schultz at Oct 12, 2010 at 6:47 pm
    Rob,
    On 10/11/2010 4:40 PM, Rob G wrote:
    So if I'm reading your email and the docs correctly. I should just
    comment out the cachesize=10 from the workers.properties. I would.
    And since for connection_pool_size (that replaced it) JK will
    discover this number for the Apache web server automatically and set
    the pool size to this value, I don't need to add anything to the
    workers.properties file?
    I believe that is true.

    On the other hand, there is another case where you might have problems.
    If you have, say, 512 worker threads in Apache httpd but you only have,
    say, 200 request processor threads configured in Tomcat, then you will
    get mod_jk connection failures on the httpd side.

    I would recommend that you have enough request processors configured in
    Tomcat to handle the expected load.

    - -chris
    ---------------------------------------------------------------------
    To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
    For additional commands, e-mail: users-help@tomcat.apache.org
  • Jeffrey Janner at Oct 12, 2010 at 7:23 pm
    I just occurred to me that I don't think anyone's asked if these are net-mounted file systems. I've seen this timestamp-shifting before, but only on net-mounted filesystems. Usually the source and local systems are set to different timezones (or DST settings).
    -----Original Message-----
    From: Christopher Schultz
    Sent: Tuesday, October 12, 2010 1:47 PM
    To: Tomcat Users List
    Subject: Re: Error 503 ocurring when server under load

    -----BEGIN PGP SIGNED MESSAGE-----
    Hash: SHA1

    Rob,
    On 10/11/2010 4:40 PM, Rob G wrote:
    So if I'm reading your email and the docs correctly. I should just
    comment out the cachesize=10 from the workers.properties. I would.
    And since for connection_pool_size (that replaced it) JK will
    discover this number for the Apache web server automatically and set
    the pool size to this value, I don't need to add anything to the
    workers.properties file?
    I believe that is true.

    On the other hand, there is another case where you might have problems.
    If you have, say, 512 worker threads in Apache httpd but you only have,
    say, 200 request processor threads configured in Tomcat, then you will
    get mod_jk connection failures on the httpd side.

    I would recommend that you have enough request processors configured in
    Tomcat to handle the expected load.

    - -chris
    -----BEGIN PGP SIGNATURE-----
    Version: GnuPG v1.4.10 (MingW32)
    Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

    iEYEARECAAYFAky0rToACgkQ9CaO5/Lv0PCvjACgwttZ9YfINxpWP+DI1+VlKfvI
    OTAAoK+2RQRibL56GdYWlaWxx6obZVln
    =hY3s
    -----END PGP SIGNATURE-----

    ---------------------------------------------------------------------
    To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
    For additional commands, e-mail: users-help@tomcat.apache.org
    __________________________________________________________________________

    Confidentiality Notice: This Transmission (including any attachments) may contain information that is privileged, confidential, and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient you are hereby notified that any dissemination, distribution, or copying of this communication is strictly prohibited.

    If you have received this transmission in error, please immediately reply to the sender or telephone (512) 343-9100 and delete this transmission from your system.
  • Christopher Schultz at Oct 12, 2010 at 8:27 pm
    Jeffrey,
    On 10/12/2010 3:22 PM, Jeffrey Janner wrote:
    I just occurred to me that I don't think anyone's asked if these are
    net-mounted file systems. I've seen this timestamp-shifting before,
    but only on net-mounted filesystems. Usually the source and local
    systems are set to different timezones (or DST settings).
    That's a very good thought.

    I've never seen good things happen when using a network filesystem for
    webapp deployments. Feel free to store your data on NFS, just don't use
    NFS to store your WAR files ;)

    - -chris
    ---------------------------------------------------------------------
    To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
    For additional commands, e-mail: users-help@tomcat.apache.org
  • Jeffrey Janner at Oct 13, 2010 at 10:18 pm
    I just realized that I somehow replied to the wrong thread on this one.
    It was meant for Jane Muse's thread.
    I'll repost there in case someone missed it.
    Sorry to interrupt your thread with irrelevant information/spleculation.
    Jeff
    -----Original Message-----
    From: Jeffrey Janner
    Sent: Tuesday, October 12, 2010 2:23 PM
    To: Tomcat Users List
    Subject: RE: Error 503 ocurring when server under load

    I just occurred to me that I don't think anyone's asked if these are
    net-mounted file systems. I've seen this timestamp-shifting before,
    but only on net-mounted filesystems. Usually the source and local
    systems are set to different timezones (or DST settings).
    -----Original Message-----
    From: Christopher Schultz
    Sent: Tuesday, October 12, 2010 1:47 PM
    To: Tomcat Users List
    Subject: Re: Error 503 ocurring when server under load

    -----BEGIN PGP SIGNED MESSAGE-----
    Hash: SHA1

    Rob,
    On 10/11/2010 4:40 PM, Rob G wrote:
    So if I'm reading your email and the docs correctly. I should just
    comment out the cachesize=10 from the workers.properties. I would.
    And since for connection_pool_size (that replaced it) JK will
    discover this number for the Apache web server automatically and
    set
    the pool size to this value, I don't need to add anything to the
    workers.properties file?
    I believe that is true.

    On the other hand, there is another case where you might have problems.
    If you have, say, 512 worker threads in Apache httpd but you only have,
    say, 200 request processor threads configured in Tomcat, then you will
    get mod_jk connection failures on the httpd side.

    I would recommend that you have enough request processors configured in
    Tomcat to handle the expected load.

    - -chris
    -----BEGIN PGP SIGNATURE-----
    Version: GnuPG v1.4.10 (MingW32)
    Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

    iEYEARECAAYFAky0rToACgkQ9CaO5/Lv0PCvjACgwttZ9YfINxpWP+DI1+VlKfvI
    OTAAoK+2RQRibL56GdYWlaWxx6obZVln
    =hY3s
    -----END PGP SIGNATURE-----

    ---------------------------------------------------------------------
    To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
    For additional commands, e-mail: users-help@tomcat.apache.org
    _______________________________________________________________________
    ___

    Confidentiality Notice: This Transmission (including any attachments)
    may contain information that is privileged, confidential, and exempt
    from disclosure under applicable law. If the reader of this message is
    not the intended recipient you are hereby notified that any
    dissemination, distribution, or copying of this communication is
    strictly prohibited.

    If you have received this transmission in error, please immediately
    reply to the sender or telephone (512) 343-9100 and delete this
    transmission from your system.
    __________________________________________________________________________

    Confidentiality Notice: This Transmission (including any attachments) may contain information that is privileged, confidential, and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient you are hereby notified that any dissemination, distribution, or copying of this communication is strictly prohibited.

    If you have received this transmission in error, please immediately reply to the sender or telephone (512) 343-9100 and delete this transmission from your system.
  • Rob G at Oct 13, 2010 at 7:01 am

    On 12 October 2010 19:47, Christopher Schultz wrote:
    I would.

    I believe that is true.

    On the other hand, there is another case where you might have problems.
    If you have, say, 512 worker threads in Apache httpd but you only have,
    say, 200 request processor threads configured in Tomcat, then you will
    get mod_jk connection failures on the httpd side.

    I would recommend that you have enough request processors configured in
    Tomcat to handle the expected load.

    - -chris
    Thanks Chris (and everyone else for their comments).

    As an update: I updated Tomcat from 6.0.24 to 6.0.29 and commented out
    the cachesize directive from workers.properties. The server ran for a
    full day yesterday and not a single 503 error. :) So with a bit of
    luck it will continue to work as expected. Of course I'll monitor for
    any other issues for the next few days.

    Rob

    ---------------------------------------------------------------------
    To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
    For additional commands, e-mail: users-help@tomcat.apache.org
  • Christopher Schultz at Oct 7, 2010 at 10:18 pm
    Rob,
    On 10/7/2010 12:59 PM, Rob G wrote:
    Recently migrated a production site (mixture of Servlets and JSPs)
    from Oracle Application Server to Apache/Tomcat. Since then we have
    seen numerous HTTP Error 503 - Service unavailable errors at peak
    times when site is under load. mod_jk.log has the following error
    message(s):

    [2184:1952] [error] jk_lb_worker.c (1473): All tomcat instances
    failed, no more workers left
    Just to reinforce a comment by Andre, I think you might want to compare
    the request processor counts you have for Tomcat versus mod_jk: the
    default maxThreads for a Tomcat <Connector> is 200, while your
    MaxClients settings might be different on your Apache httpd configuration.

    If you have httpd.MaxClients=250 and connector.maxThreads=200 then you
    have a deficit of 50 who will get 503 errors if they try to access your
    webapp during peak load.
    Apache: 2.2.16
    I believe that only the Worker MPM is available on Microsoft Windows.
    Have you explicitly configured an MPM?
    worker.worker1.cachesize=10
    You might want to read the documentation for worker.cachesize in
    http://tomcat.apache.org/connectors-doc/reference/workers.html

    Note that it's deprecated and you should use connection_pool_size instead.

    Since you're using Apache 2.x, mod_jk should be able to auto-configure
    this value for you.

    - From my reading, you may be limiting yourself to 10 simultaneous
    connections from Apache httpd to Tomcat, and others will get a 503.
    Rainer or Mladen will have to comment to clarify: I am certainly no
    expert in mod_jk configurations.

    Hope that helps,
    - -chris
    ---------------------------------------------------------------------
    To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
    For additional commands, e-mail: users-help@tomcat.apache.org
  • Pid at Oct 7, 2010 at 10:29 pm

    On 07/10/2010 23:17, Christopher Schultz wrote:
    Rob,
    On 10/7/2010 12:59 PM, Rob G wrote:
    Recently migrated a production site (mixture of Servlets and JSPs)
    from Oracle Application Server to Apache/Tomcat. Since then we have
    seen numerous HTTP Error 503 - Service unavailable errors at peak
    times when site is under load. mod_jk.log has the following error
    message(s):
    [2184:1952] [error] jk_lb_worker.c (1473): All tomcat instances
    failed, no more workers left
    Just to reinforce a comment by Andre, I think you might want to compare
    the request processor counts you have for Tomcat versus mod_jk: the
    default maxThreads for a Tomcat <Connector> is 200, while your
    MaxClients settings might be different on your Apache httpd configuration.

    If you have httpd.MaxClients=250 and connector.maxThreads=200 then you
    have a deficit of 50 who will get 503 errors if they try to access your
    webapp during peak load.
    Apache: 2.2.16
    I believe that only the Worker MPM is available on Microsoft Windows.
    Have you explicitly configured an MPM?
    worker.worker1.cachesize=10
    Is this info in the original email? (If so, I think my mailer isn't
    behaving well).


    p


    ---------------------------------------------------------------------
    To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
    For additional commands, e-mail: users-help@tomcat.apache.org
  • André Warnier at Oct 8, 2010 at 9:09 am
    Pid wrote:
    ...
    Is this info in the original email?
    Yes.

    (If so, I think my mailer isn't
    behaving well).
    It seems also somewhat spastic. We've been receiving multiple copies of what you send.
    Or did you just have too much coffee lately ?


    ---------------------------------------------------------------------
    To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
    For additional commands, e-mail: users-help@tomcat.apache.org
  • Pid at Oct 9, 2010 at 8:43 am

    On 08/10/2010 10:08, André Warnier wrote:
    Pid wrote:
    ...
    Is this info in the original email?
    Yes.

    (If so, I think my mailer isn't
    behaving well).
    It seems also somewhat spastic. We've been receiving multiple copies of
    what you send.
    Or did you just have too much coffee lately ?
    Hmm. My apologies.

    I was on a tiny & unreliable network connection; I think client & server
    dropped & restarted transactions.


    p
  • Pid at Oct 7, 2010 at 10:30 pm

    On 07/10/2010 17:59, Rob G wrote:
    Hey all,

    Recently migrated a production site (mixture of Servlets and JSPs)
    from Oracle Application Server to Apache/Tomcat. Since then we have
    seen numerous HTTP Error 503 - Service unavailable errors at peak
    times when site is under load. mod_jk.log has the following error
    message(s):

    [2184:1952] [error] jk_lb_worker.c (1473): All tomcat instances
    failed, no more workers left

    I'm looking for help in trying to tweak settings to prevent this, or
    confirmation that I've configured the setup correctly.

    Many thanks
    Rob

    Platform:
    Windows Server 2003 SP2

    Setup:
    Two tomcat instances with a single Apache front end, all on the same server

    Versions
    Tomcat: 6.0.24
    Apache: 2.2.16
    mod SSL: 2.2.16
    Open SSL: 0.9.8
    mod_JK:1.2.30

    Conf:
    httpd.conf:

    <IfModule mod_jk.c>

    JkWorkersFile conf/workers.properties
    JkLogFile logs/mod_jk.log
    JkLogLevel error
    JkLogStampFormat "[%a %b %d %H:%M:%S %Y] "
    JkOptions +ForwardKeySize +ForwardURICompat -ForwardDirectories
    JkRequestLogFormat "%w %V %T"

    <Location /*/WEB-INF/*>
    AllowOverride None
    deny from all
    </Location>

    JkMount /examples balancer
    JkMount /examples/* balancer
    JkMount /App1/* balancer
    JkMount /App2/* balancer
    </IfModule>

    Workers.properties:
    workers.tomcat_home="C:\tomcat1"
    workers.java_home="C:\Program Files\Java\jdk1.6.0_18"
    ps=\

    # Define workers
    #worker.list=worker1,worker2
    worker.list=balancer

    # Set properties for worker worker1 (ajp13)
    worker.worker1.type=ajp13
    worker.worker1.host=localhost
    worker.worker1.port=8009
    worker.worker1.lbfactor=1

    worker.worker1.cachesize=10
    worker.worker1.connection_pool_timeout=300
    worker.worker1.socket_keepalive=FALSE
    #worker.worker1.recycle_timeout=300

    # Set properties for worker worker2 (ajp13)
    worker.worker2.type=ajp13
    worker.worker2.host=localhost
    worker.worker2.port=8019
    worker.worker2.lbfactor=1

    worker.worker2.cachesize=10
    worker.worker2.connection_pool_timeout=300
    worker.worker2.socket_keepalive=FALSE
    #worker.worker2.recycle_timeout=300

    worker.balancer.type=lb
    worker.balancer.balance_workers=worker1,worker2
    worker.balancer.method=R
    worker.balancer.sticky_session=True

    Server.xml (same for both tomcat instances apart from jvmRoute):
    <?xml version='1.0' encoding='utf-8'?>
    <Server port="8005" shutdown="SHUTDOWN">

    <Listener className="org.apache.catalina.core.AprLifecycleListener"
    SSLEngine="on" />
    <Listener className="org.apache.catalina.core.JasperListener" />
    <Listener className="org.apache.catalina.core.JreMemoryLeakPreventionListener"
    />
    <Listener className="org.apache.catalina.mbeans.ServerLifecycleListener" />
    <Listener className="org.apache.catalina.mbeans.GlobalResourcesLifecycleListener"
    />


    <GlobalNamingResources>

    <Resource name="UserDatabase" auth="Container"
    type="org.apache.catalina.UserDatabase"
    description="User database that can be updated and saved"
    factory="org.apache.catalina.users.MemoryUserDatabaseFactory"
    pathname="conf/tomcat-users.xml" />
    </GlobalNamingResources>

    <Service name="Catalina">

    <Connector port="8080" protocol="HTTP/1.1"
    connectionTimeout="40000"
    redirectPort="8443" />

    <Connector port="8009" protocol="AJP/1.3" redirectPort="8443" />

    <Engine name="Catalina" defaultHost="localhost" jvmRoute="worker1">

    <Realm className="org.apache.catalina.realm.UserDatabaseRealm"
    resourceName="UserDatabase"/>

    <Host name="localhost" appBase="webapps"
    unpackWARs="true" autoDeploy="true"
    xmlValidation="false" xmlNamespaceAware="false">

    </Host>
    </Engine>
    </Service>
    </Server>
    [Answers own previous question] Weirdness.

    In addition to the previous relevant questions:

    Are both of the two Tomcat instances actually working? If the shutdown
    port is identical on both (8005), then I think the second one won't
    start up. The connectors certainly won't.

    Also, if you're using mod_jk 1.2.30 there's an example config in the
    install dir which uses the template config method and has good default
    settings.


    p

Related Discussions

Discussion Navigation
viewthread | post