Grokbase Groups Apache users May 2012
FAQ
Madriva 2010.2 running httpd apache 2.2.22

I am having trouble with httpd requests staying active and multiplying.
I just came off having 160 versions of httpd running and completely slowing
down the system. I upgraded to 2.2.22 and it still happens (it went from the
normal 10 servers running to 15 in about a 1/2 hour.) According to the start
times, these seem to be associated with totally bizarre requests from google
(forged addresses?)

Eg, here is one entry from the ps auxww list

apache 18137 0.0 0.5 26844 5744 ? S 09:34 0:00 /usr/sbin/httpd -f /etc/httpd/conf/httpd.conf -DAPACHE2 -DHAVE_PERL -DHAVE_PHP5 -DHAVE_ACTIONS -DHAVE_ALIAS -DHAVE_ASIS -DHAVE_AUTH_BASIC -DHAVE_AUTH_DIGEST -DHAVE_AUTHN_ALIAS -DHAVE_AUTHN_ANON -DHAVE_AUTHN_DBM -DHAVE_AUTHN_DEFAULT -DHAVE_AUTHN_FILE -DHAVE_AUTHZ_DBM -DHAVE_AUTHZ_DEFAULT -DHAVE_AUTHZ_GROUPFILE -DHAVE_AUTHZ_HOST -DHAVE_AUTHZ_OWNER -DHAVE_AUTHZ_USER -DHAVE_AUTOINDEX -DHAVE_BUCKETEER -DHAVE_CASE_FILTER -DHAVE_CASE_FILTER_IN -DHAVE_CERN_META -DHAVE_CGI -DHAVE_CGID -DHAVE_CHARSET_LITE -DHAVE_DIR -DHAVE_DUMPIO -DHAVE_ECHO -DHAVE_ENV -DHAVE_EXAMPLE -DHAVE_EXPIRES -DHAVE_EXT_FILTER -DHAVE_FILTER -DHAVE_HEADERS -DHAVE_IDENT -DHAVE_IMAGEMAP -DHAVE_INCLUDE -DHAVE_INFO -DHAVE_LOG_CONFIG -DHAVE_LOG_FORENSIC -DHAVE_LOGIO -DHAVE_MIME -DHAVE_MIME_MAGIC -DHAVE_NEGOTIATION -DHAVE_OPTIONAL_FN_EXPORT -DHAVE_OPTIONAL_FN_IMPORT -DHAVE_OPTIONAL_HOOK_EXPORT -DHAVE_OPTIONAL_HOOK_IMPORT -DHAVE_REWRITE -DHAVE_SETENVIF -DHAVE_SPELING -DHAVE_SSL -DHAVE_STATUS -DHAVE_SUBSTITUTE -DHAVE_SUEXEC -DHAVE_UNIQUE_ID -DHAVE_USERTRACK -DHAVE_VERSION -DHAVE_VHOST_ALIAS

At that time in the
access_log I have a whole bunch of entries like
::1 - - [22/May/2012:09:34:22 -0700] "OPTIONS * HTTP/1.0" 200 - "-" "Apache/2.2.22 (Mandriva Linux/PREFORK-0.1mdv2010.2) (internal dummy connection)"


In the past I have also had connections like
66.249.68.198 - - [22/May/2012:09:35:25 -0700] "GET /aggregator/www.umsl.edu/~keelr/010/www.twitter.com/www.iaea.org/Publications/Documents/Board/2008/www.environment-agency.gov.uk/homeandleisure/floods/node/www.guardian.co.uk/business/2012/feb/21/node/node/22?page=11 HTTP/1.1" 200 58609 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"

associated with the times of the startup of those persistant connections. This
looks to be a totally bizzare GET. since that address certainly has nothing to
do with my site.

In the error log around that time I get nothing that looks suspicious

[Tue May 22 09:31:54 2012] [error] [client 119.63.196.27] File does not exist: /usr/local/http/htdocs/robots.txt
[Tue May 22 09:32:25 2012] [error] [client 86.68.18.171] File does not exist: /usr/local/http/htdocs/favicon.ico
[Tue May 22 09:36:47 2012] [error] [client 89.144.206.157] File does not exist: /usr/local/http/htdocs/thirdman/reichs/blank.gif, referer: http://axion.physics.ubc.ca/thirdman/reichs/reichsbruecke.htm

--
William G. Unruh | Canadian Institute for| Tel: +1(604)822-3273
Physics&Astronomy | Advanced Research | Fax: +1(604)822-5324
UBC, Vancouver,BC | Program in Cosmology | unruh@physics.ubc.ca
Canada V6T 1Z1 | and Gravity | www.theory.physics.ubc.ca/

Search Discussions

  • William A. Rowe Jr. at May 22, 2012 at 10:05 pm

    On 5/22/2012 12:02 PM, Bill Unruh wrote:

    Eg, here is one entry from the ps auxww list

    apache 18137 0.0 0.5 26844 5744 ? S 09:34 0:00 /usr/sbin/httpd -f
    /etc/httpd/conf/httpd.conf -DAPACHE2 -DHAVE_PERL -DHAVE_PHP5 -DHAVE_ACTIONS -DHAVE_ALIAS
    -DHAVE_ASIS -DHAVE_AUTH_BASIC -DHAVE_AUTH_DIGEST -DHAVE_AUTHN_ALIAS -DHAVE_AUTHN_ANON
    -DHAVE_AUTHN_DBM -DHAVE_AUTHN_DEFAULT -DHAVE_AUTHN_FILE -DHAVE_AUTHZ_DBM
    -DHAVE_AUTHZ_DEFAULT -DHAVE_AUTHZ_GROUPFILE -DHAVE_AUTHZ_HOST -DHAVE_AUTHZ_OWNER
    -DHAVE_AUTHZ_USER -DHAVE_AUTOINDEX -DHAVE_BUCKETEER -DHAVE_CASE_FILTER
    -DHAVE_CASE_FILTER_IN -DHAVE_CERN_META -DHAVE_CGI -DHAVE_CGID -DHAVE_CHARSET_LITE
    -DHAVE_DIR -DHAVE_DUMPIO -DHAVE_ECHO -DHAVE_ENV -DHAVE_EXAMPLE -DHAVE_EXPIRES
    -DHAVE_EXT_FILTER -DHAVE_FILTER -DHAVE_HEADERS -DHAVE_IDENT -DHAVE_IMAGEMAP -DHAVE_INCLUDE
    -DHAVE_INFO -DHAVE_LOG_CONFIG -DHAVE_LOG_FORENSIC -DHAVE_LOGIO -DHAVE_MIME
    -DHAVE_MIME_MAGIC -DHAVE_NEGOTIATION -DHAVE_OPTIONAL_FN_EXPORT -DHAVE_OPTIONAL_FN_IMPORT
    -DHAVE_OPTIONAL_HOOK_EXPORT -DHAVE_OPTIONAL_HOOK_IMPORT -DHAVE_REWRITE -DHAVE_SETENVIF
    -DHAVE_SPELING -DHAVE_SSL -DHAVE_STATUS -DHAVE_SUBSTITUTE -DHAVE_SUEXEC -DHAVE_UNIQUE_ID
    -DHAVE_USERTRACK -DHAVE_VERSION -DHAVE_VHOST_ALIAS
    Never seen such a crazy startup line, I presume all of your modules have individual
    <IfDefine > blocks around each?
    At that time in the access_log I have a whole bunch of entries like
    ::1 - - [22/May/2012:09:34:22 -0700] "OPTIONS * HTTP/1.0" 200 - "-" "Apache/2.2.22
    (Mandriva Linux/PREFORK-0.1mdv2010.2) (internal dummy connection)"
    That's your own local loopback from a process running on this same box.
    In the past I have also had connections like 66.249.68.198 - - [22/May/2012:09:35:25
    -0700] "GET
    /aggregator/www.umsl.edu/~keelr/010/www.twitter.com/www.iaea.org/Publications/Documents/Board/2008/www.environment-agency.gov.uk/homeandleisure/floods/node/www.guardian.co.uk/business/2012/feb/21/node/node/22?page=11
    HTTP/1.1" 200 58609 "-" "Mozilla/5.0 (compatible; Googlebot/2.1;
    +http://www.google.com/bot.html)"
    No clue. Maybe playing with open proxies? The document seems to be 58k if that helps you
    at all (maybe a local index page?)
    In the error log around that time I get nothing that looks suspicious

    [Tue May 22 09:31:54 2012] [error] [client 119.63.196.27] File does not exist:
    /usr/local/http/htdocs/robots.txt
    [Tue May 22 09:32:25 2012] [error] [client 86.68.18.171] File does not exist:
    /usr/local/http/htdocs/favicon.ico
    Certainly harmless and most common from crawlers and web browsers respectively. You may
    want to add a simple one line robots.txt file, and a simple default icon.

    [Tue May 22 09:36:47 2012] [error] [client 89.144.206.157] File does not exist:
    /usr/local/http/htdocs/thirdman/reichs/blank.gif, referer:
    http://axion.physics.ubc.ca/thirdman/reichs/reichsbruecke.htm
    Your own mistake in the html, it seems.

    Can you interrupt one of the truly hosed processes using gdb and try 'thread apply all bt'

    ---------------------------------------------------------------------
    To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
    For additional commands, e-mail: users-help@httpd.apache.org
  • Bill Unruh at May 22, 2012 at 11:00 pm

    On Tue, 22 May 2012, William A. Rowe Jr. wrote:
    On 5/22/2012 12:02 PM, Bill Unruh wrote:

    Eg, here is one entry from the ps auxww list

    apache 18137 0.0 0.5 26844 5744 ? S 09:34 0:00 /usr/sbin/httpd -f
    /etc/httpd/conf/httpd.conf -DAPACHE2 -DHAVE_PERL -DHAVE_PHP5 -DHAVE_ACTIONS -DHAVE_ALIAS
    -DHAVE_ASIS -DHAVE_AUTH_BASIC -DHAVE_AUTH_DIGEST -DHAVE_AUTHN_ALIAS -DHAVE_AUTHN_ANON
    -DHAVE_AUTHN_DBM -DHAVE_AUTHN_DEFAULT -DHAVE_AUTHN_FILE -DHAVE_AUTHZ_DBM
    -DHAVE_AUTHZ_DEFAULT -DHAVE_AUTHZ_GROUPFILE -DHAVE_AUTHZ_HOST -DHAVE_AUTHZ_OWNER
    -DHAVE_AUTHZ_USER -DHAVE_AUTOINDEX -DHAVE_BUCKETEER -DHAVE_CASE_FILTER
    -DHAVE_CASE_FILTER_IN -DHAVE_CERN_META -DHAVE_CGI -DHAVE_CGID -DHAVE_CHARSET_LITE
    -DHAVE_DIR -DHAVE_DUMPIO -DHAVE_ECHO -DHAVE_ENV -DHAVE_EXAMPLE -DHAVE_EXPIRES
    -DHAVE_EXT_FILTER -DHAVE_FILTER -DHAVE_HEADERS -DHAVE_IDENT -DHAVE_IMAGEMAP -DHAVE_INCLUDE
    -DHAVE_INFO -DHAVE_LOG_CONFIG -DHAVE_LOG_FORENSIC -DHAVE_LOGIO -DHAVE_MIME
    -DHAVE_MIME_MAGIC -DHAVE_NEGOTIATION -DHAVE_OPTIONAL_FN_EXPORT -DHAVE_OPTIONAL_FN_IMPORT
    -DHAVE_OPTIONAL_HOOK_EXPORT -DHAVE_OPTIONAL_HOOK_IMPORT -DHAVE_REWRITE -DHAVE_SETENVIF
    -DHAVE_SPELING -DHAVE_SSL -DHAVE_STATUS -DHAVE_SUBSTITUTE -DHAVE_SUEXEC -DHAVE_UNIQUE_ID
    -DHAVE_USERTRACK -DHAVE_VERSION -DHAVE_VHOST_ALIAS
    Never seen such a crazy startup line, I presume all of your modules have individual
    <IfDefine > blocks around each?
    It is basically what Mandriva has as its default.

    At that time in the access_log I have a whole bunch of entries like
    ::1 - - [22/May/2012:09:34:22 -0700] "OPTIONS * HTTP/1.0" 200 - "-" "Apache/2.2.22
    (Mandriva Linux/PREFORK-0.1mdv2010.2) (internal dummy connection)"
    That's your own local loopback from a process running on this same box.
    There are no processes running on this same box. It is rarely used. and
    certainly did not have a browser running at that time.

    \
    In the past I have also had connections like 66.249.68.198 - - [22/May/2012:09:35:25
    -0700] "GET
    /aggregator/www.umsl.edu/~keelr/010/www.twitter.com/www.iaea.org/Publications/Documents/Board/2008/www.environment-agency.gov.uk/homeandleisure/floods/node/www.guardian.co.uk/business/2012/feb/21/node/node/22?page=11
    HTTP/1.1" 200 58609 "-" "Mozilla/5.0 (compatible; Googlebot/2.1;
    +http://www.google.com/bot.html)"
    No clue. Maybe playing with open proxies? The document seems to be 58k if that helps you
    at all (maybe a local index page?)
    There is no such file or path on my system. If I try to use it, I get file not
    found. I have nothing called /aggregator/

    In the error log around that time I get nothing that looks suspicious

    [Tue May 22 09:31:54 2012] [error] [client 119.63.196.27] File does not exist:
    /usr/local/http/htdocs/robots.txt
    [Tue May 22 09:32:25 2012] [error] [client 86.68.18.171] File does not exist:
    /usr/local/http/htdocs/favicon.ico
    Certainly harmless and most common from crawlers and web browsers respectively. You may
    want to add a simple one line robots.txt file, and a simple default icon.
    That was what I thought.

    [Tue May 22 09:36:47 2012] [error] [client 89.144.206.157] File does not exist:
    /usr/local/http/htdocs/thirdman/reichs/blank.gif, referer:
    http://axion.physics.ubc.ca/thirdman/reichs/reichsbruecke.htm
    Your own mistake in the html, it seems.
    Yup. Just displying it on the off chance it could be problematic. Since I have
    not the ghost of an idea what could be wrong, I also have not the ghost of an
    idea what could be a symptom either.

    Can you interrupt one of the truly hosed processes using gdb and try 'thread apply all bt'
    What does that do?

    Thread 1 (Thread 0xb760f700 (LWP 20861)):
    #0 0xffffe424 in __kernel_vsyscall ()
    #1 0xb77ece6b in fcntl () from /lib/i686/libpthread.so.0
    #2 0xb780f832 in ?? () from /usr/lib/libapr-1.so.0
    #3 0xb780f1ad in apr_proc_mutex_lock () from /usr/lib/libapr-1.so.0
    #4 0x0809294c in ?? ()
    #5 0x08092e0b in ?? ()
    #6 0x08093be4 in ap_mpm_run ()
    #7 0x08064cd1 in main ()


    ---------------------------------------------------------------------
    To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
    For additional commands, e-mail: users-help@httpd.apache.org
  • William A. Rowe Jr. at May 23, 2012 at 12:10 am

    On 5/22/2012 6:00 PM, Bill Unruh wrote:
    On Tue, 22 May 2012, William A. Rowe Jr. wrote:
    On 5/22/2012 12:02 PM, Bill Unruh wrote:

    At that time in the access_log I have a whole bunch of entries like
    ::1 - - [22/May/2012:09:34:22 -0700] "OPTIONS * HTTP/1.0" 200 - "-" "Apache/2.2.22
    (Mandriva Linux/PREFORK-0.1mdv2010.2) (internal dummy connection)"
    That's your own local loopback from a process running on this same box.
    There are no processes running on this same box. It is rarely used. and
    certainly did not have a browser running at that time.
    Then a server module is likely pinging itself. Any chance you set up an infinite proxy
    recursion here?
    In the past I have also had connections like 66.249.68.198 - - [22/May/2012:09:35:25
    -0700] "GET
    /aggregator/www.umsl.edu/~keelr/010/www.twitter.com/www.iaea.org/Publications/Documents/Board/2008/www.environment-agency.gov.uk/homeandleisure/floods/node/www.guardian.co.uk/business/2012/feb/21/node/node/22?page=11

    HTTP/1.1" 200 58609 "-" "Mozilla/5.0 (compatible; Googlebot/2.1;
    +http://www.google.com/bot.html)"
    No clue. Maybe playing with open proxies? The document seems to be 58k if that helps you
    at all (maybe a local index page?)
    There is no such file or path on my system. If I try to use it, I get file not
    found. I have nothing called /aggregator/
    Looking more and more like a proxy recursion/infinite looping sort of config error.
    Can you interrupt one of the truly hosed processes using gdb and try 'thread apply all bt'
    What does that do?
    Dumps all threads instead of just one of them.
    Thread 1 (Thread 0xb760f700 (LWP 20861)):
    #0 0xffffe424 in __kernel_vsyscall ()
    #1 0xb77ece6b in fcntl () from /lib/i686/libpthread.so.0
    #2 0xb780f832 in ?? () from /usr/lib/libapr-1.so.0
    #3 0xb780f1ad in apr_proc_mutex_lock () from /usr/lib/libapr-1.so.0
    #4 0x0809294c in ?? ()
    #5 0x08092e0b in ?? ()
    #6 0x08093be4 in ap_mpm_run ()
    #7 0x08064cd1 in main ()
    It might be helpful to first install the debuginfo for the apr/httpd packages, but this
    looks like it might be in the accept queue waiting on the MutexFile to unblock this one
    (and is probably a healthy process).

    If you are running prefork we would encourage you to try the worker mpm.

    ---------------------------------------------------------------------
    To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
    For additional commands, e-mail: users-help@httpd.apache.org
  • Bill Unruh at May 23, 2012 at 1:35 am

    On Tue, 22 May 2012, William A. Rowe Jr. wrote:
    On 5/22/2012 6:00 PM, Bill Unruh wrote:
    On Tue, 22 May 2012, William A. Rowe Jr. wrote:
    On 5/22/2012 12:02 PM, Bill Unruh wrote:

    At that time in the access_log I have a whole bunch of entries like
    ::1 - - [22/May/2012:09:34:22 -0700] "OPTIONS * HTTP/1.0" 200 - "-" "Apache/2.2.22
    (Mandriva Linux/PREFORK-0.1mdv2010.2) (internal dummy connection)"
    That's your own local loopback from a process running on this same box.
    There are no processes running on this same box. It is rarely used. and
    certainly did not have a browser running at that time.
    Then a server module is likely pinging itself. Any chance you set up an infinite proxy
    recursion here?
    In the past I have also had connections like 66.249.68.198 - - [22/May/2012:09:35:25
    -0700] "GET
    /aggregator/www.umsl.edu/~keelr/010/www.twitter.com/www.iaea.org/Publications/Documents/Board/2008/www.environment-agency.gov.uk/homeandleisure/floods/node/www.guardian.co.uk/business/2012/feb/21/node/node/22?page=11

    HTTP/1.1" 200 58609 "-" "Mozilla/5.0 (compatible; Googlebot/2.1;
    +http://www.google.com/bot.html)"
    No clue. Maybe playing with open proxies? The document seems to be 58k if that helps you
    at all (maybe a local index page?)
    There is no such file or path on my system. If I try to use it, I get file not
    found. I have nothing called /aggregator/
    Looking more and more like a proxy recursion/infinite looping sort of config error.
    OK, I do have a "virtual host", www.emergentgravity.org defined on this
    system.

    In /etc/httpd/conf/vhosts.d/00_default_vhosts.conf I have
    ----------------------------------------
    NameVirtualHost *:80
    <VirtualHost *:80>
    ServerName www.theory.physics.ubc.ca
    ServerAlias theory.physics.ubc.ca
    DocumentRoot /usr/local/http/htdocs
    </VirtualHost>

    <VirtualHost *:80>
    ServerName www.emergentgravity.org
    ServerAlias emergentgravity.org
    DocumentRoot /usr/local/http/htdocs/emergent/drupal.new
    </VirtualHost>
    -------------------------------------------

    This is the whole of that file.
    This is supposed to be so that a www.emergentgravity.org address goes to its
    drupal document root, and such that anything coming in to
    www.theory.physics.ubc.ca comes in to /usr/local/http/htdocs.

    If you suspect I have no idea what I am doing in setting this up, you are
    correct. Have I managed to stick in a feedback loop in here?

    Can you interrupt one of the truly hosed processes using gdb and try 'thread apply all bt'
    What does that do?
    Dumps all threads instead of just one of them.
    Thread 1 (Thread 0xb760f700 (LWP 20861)):
    #0 0xffffe424 in __kernel_vsyscall ()
    #1 0xb77ece6b in fcntl () from /lib/i686/libpthread.so.0
    #2 0xb780f832 in ?? () from /usr/lib/libapr-1.so.0
    #3 0xb780f1ad in apr_proc_mutex_lock () from /usr/lib/libapr-1.so.0
    #4 0x0809294c in ?? ()
    #5 0x08092e0b in ?? ()
    #6 0x08093be4 in ap_mpm_run ()
    #7 0x08064cd1 in main ()
    It might be helpful to first install the debuginfo for the apr/httpd packages, but this
    Unfortunately Mandriva never bothered to supply the apache-debug package for
    this version of apache.

    looks like it might be in the accept queue waiting on the MutexFile to unblock this one
    (and is probably a healthy process).

    If you are running prefork we would encourage you to try the worker mpm.
    Sorry, I have no idea what this sentence means. (prefork, worker mpm)


    ---------------------------------------------------------------------
    To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
    For additional commands, e-mail: users-help@httpd.apache.org
    --
    William G. Unruh | Canadian Institute for| Tel: +1(604)822-3273
    Physics&Astronomy | Advanced Research | Fax: +1(604)822-5324
    UBC, Vancouver,BC | Program in Cosmology | unruh@physics.ubc.ca
    Canada V6T 1Z1 | and Gravity | www.theory.physics.ubc.ca/

    ---------------------------------------------------------------------
    To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
    For additional commands, e-mail: users-help@httpd.apache.org
  • Bill Unruh at May 23, 2012 at 1:48 am
    Aha-- under "www.emergentgravity.org" I do have an aggregator apparently.
    (Just tried it and I do get a whole bunch of news items.) so this must be what
    is giving those strange "GET" entries.
    (This site was set up by someone else and thus my knowledge of what it does is
    pretty sketchy.)


    On Tue, 22 May 2012, Bill Unruh wrote:
    On Tue, 22 May 2012, William A. Rowe Jr. wrote:
    On 5/22/2012 6:00 PM, Bill Unruh wrote:
    On Tue, 22 May 2012, William A. Rowe Jr. wrote:
    On 5/22/2012 12:02 PM, Bill Unruh wrote:

    At that time in the access_log I have a whole bunch of entries like
    : : 1 - - [22/May/2012:09:34:22 -0700] "OPTIONS * HTTP/1.0" 200 - "-"
    : : "Apache/2.2.22
    (Mandriva Linux/PREFORK-0.1mdv2010.2) (internal dummy connection)"
    That's your own local loopback from a process running on this same
    box.
    There are no processes running on this same box. It is rarely used. and
    certainly did not have a browser running at that time.
    Then a server module is likely pinging itself. Any chance you set up an
    infinite proxy
    recursion here?
    In the past I have also had connections like 66.249.68.198 - -
    [22/May/2012:09:35:25
    -0700] "GET
    /aggregator/www.umsl.edu/~keelr/010/www.twitter.com/www.iaea.org/Publications/Documents/Board/2008/www.environment-agency.gov.uk/homeandleisure/floods/node/www.guardian.co.uk/business/2012/feb/21/node/node/22?page=11

    HTTP/1.1" 200 58609 "-" "Mozilla/5.0 (compatible; Googlebot/2.1;
    +http://www.google.com/bot.html)"
    No clue. Maybe playing with open proxies? The document seems to be
    58k if that helps you
    at all (maybe a local index page?)
    There is no such file or path on my system. If I try to use it, I get
    file not
    found. I have nothing called /aggregator/
    Looking more and more like a proxy recursion/infinite looping sort of
    config error.
    OK, I do have a "virtual host", www.emergentgravity.org defined on this
    system.

    In /etc/httpd/conf/vhosts.d/00_default_vhosts.conf I have
    ---------------------------------------- NameVirtualHost *:80
    <VirtualHost *:80>
    ServerName www.theory.physics.ubc.ca
    ServerAlias theory.physics.ubc.ca
    DocumentRoot /usr/local/http/htdocs
    </VirtualHost>

    <VirtualHost *:80>
    ServerName www.emergentgravity.org
    ServerAlias emergentgravity.org
    DocumentRoot /usr/local/http/htdocs/emergent/drupal.new
    </VirtualHost>
    -------------------------------------------

    This is the whole of that file.
    This is supposed to be so that a www.emergentgravity.org address goes to its
    drupal document root, and such that anything coming in to
    www.theory.physics.ubc.ca comes in to /usr/local/http/htdocs.

    If you suspect I have no idea what I am doing in setting this up, you are
    correct. Have I managed to stick in a feedback loop in here?

    Can you interrupt one of the truly hosed processes using gdb and try
    'thread apply all bt'
    What does that do?
    Dumps all threads instead of just one of them.
    Thread 1 (Thread 0xb760f700 (LWP 20861)):
    #0 0xffffe424 in __kernel_vsyscall ()
    #1 0xb77ece6b in fcntl () from /lib/i686/libpthread.so.0
    #2 0xb780f832 in ?? () from /usr/lib/libapr-1.so.0
    #3 0xb780f1ad in apr_proc_mutex_lock () from /usr/lib/libapr-1.so.0
    #4 0x0809294c in ?? ()
    #5 0x08092e0b in ?? ()
    #6 0x08093be4 in ap_mpm_run ()
    #7 0x08064cd1 in main ()
    It might be helpful to first install the debuginfo for the apr/httpd
    packages, but this
    Unfortunately Mandriva never bothered to supply the apache-debug package for
    this version of apache.

    looks like it might be in the accept queue waiting on the MutexFile to
    unblock this one
    (and is probably a healthy process).

    If you are running prefork we would encourage you to try the worker mpm.
    Sorry, I have no idea what this sentence means. (prefork, worker mpm)


    ---------------------------------------------------------------------
    To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
    For additional commands, e-mail: users-help@httpd.apache.org
    --
    William G. Unruh | Canadian Institute for| Tel: +1(604)822-3273
    Physics&Astronomy | Advanced Research | Fax: +1(604)822-5324
    UBC, Vancouver,BC | Program in Cosmology | unruh@physics.ubc.ca
    Canada V6T 1Z1 | and Gravity | www.theory.physics.ubc.ca/

    ---------------------------------------------------------------------
    To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
    For additional commands, e-mail: users-help@httpd.apache.org
  • Bill Unruh at May 24, 2012 at 2:48 am

    On Tue, 22 May 2012, Bill Unruh wrote:
    Madriva 2010.2 running httpd apache 2.2.22

    I am having trouble with httpd requests staying active and multiplying. I
    just came off having 160 versions of httpd running and completely slowing
    down the system. I upgraded to 2.2.22 and it still happens (it went from the
    normal 10 servers running to 15 in about a 1/2 hour.) According to the start
    times, these seem to be associated with totally bizarre requests from google
    (forged addresses?)

    Eg, here is one entry from the ps auxww list

    apache 18137 0.0 0.5 26844 5744 ? S 09:34 0:00
    /usr/sbin/httpd -f /etc/httpd/conf/httpd.conf -DAPACHE2 -DHAVE_PERL
    -DHAVE_PHP5 -DHAVE_ACTIONS -DHAVE_ALIAS -DHAVE_ASIS -DHAVE_AUTH_BASIC
    -DHAVE_AUTH_DIGEST -DHAVE_AUTHN_ALIAS -DHAVE_AUTHN_ANON -DHAVE_AUTHN_DBM
    -DHAVE_AUTHN_DEFAULT -DHAVE_AUTHN_FILE -DHAVE_AUTHZ_DBM -DHAVE_AUTHZ_DEFAULT
    -DHAVE_AUTHZ_GROUPFILE -DHAVE_AUTHZ_HOST -DHAVE_AUTHZ_OWNER -DHAVE_AUTHZ_USER
    -DHAVE_AUTOINDEX -DHAVE_BUCKETEER -DHAVE_CASE_FILTER -DHAVE_CASE_FILTER_IN
    -DHAVE_CERN_META -DHAVE_CGI -DHAVE_CGID -DHAVE_CHARSET_LITE -DHAVE_DIR
    -DHAVE_DUMPIO -DHAVE_ECHO -DHAVE_ENV -DHAVE_EXAMPLE -DHAVE_EXPIRES
    -DHAVE_EXT_FILTER -DHAVE_FILTER -DHAVE_HEADERS -DHAVE_IDENT -DHAVE_IMAGEMAP
    -DHAVE_INCLUDE -DHAVE_INFO -DHAVE_LOG_CONFIG -DHAVE_LOG_FORENSIC -DHAVE_LOGIO
    -DHAVE_MIME -DHAVE_MIME_MAGIC -DHAVE_NEGOTIATION -DHAVE_OPTIONAL_FN_EXPORT
    -DHAVE_OPTIONAL_FN_IMPORT -DHAVE_OPTIONAL_HOOK_EXPORT
    -DHAVE_OPTIONAL_HOOK_IMPORT -DHAVE_REWRITE -DHAVE_SETENVIF -DHAVE_SPELING
    -DHAVE_SSL -DHAVE_STATUS -DHAVE_SUBSTITUTE -DHAVE_SUEXEC -DHAVE_UNIQUE_ID
    -DHAVE_USERTRACK -DHAVE_VERSION -DHAVE_VHOST_ALIAS

    At that time in the access_log I have a whole bunch of entries like
    : : 1 - - [22/May/2012:09:34:22 -0700] "OPTIONS * HTTP/1.0" 200 - "-"
    : : "Apache/2.2.22 (Mandriva Linux/PREFORK-0.1mdv2010.2) (internal dummy
    : : connection)"


    In the past I have also had connections like 66.249.68.198 - -
    [22/May/2012:09:35:25 -0700] "GET
    /aggregator/www.umsl.edu/~keelr/010/www.twitter.com/www.iaea.org/Publications/Documents/Board/2008/www.environment-agency.gov.uk/homeandleisure/floods/node/www.guardian.co.uk/business/2012/feb/21/node/node/22?page=11
    HTTP/1.1" 200 58609 "-" "Mozilla/5.0 (compatible; Googlebot/2.1;
    +http://www.google.com/bot.html)"

    associated with the times of the startup of those persistant connections.
    This
    looks to be a totally bizzare GET. since that address certainly has nothing
    to
    do with my site.

    In the error log around that time I get nothing that looks suspicious

    [Tue May 22 09:31:54 2012] [error] [client 119.63.196.27] File does not
    exist: /usr/local/http/htdocs/robots.txt
    [Tue May 22 09:32:25 2012] [error] [client 86.68.18.171] File does not exist:
    /usr/local/http/htdocs/favicon.ico
    [Tue May 22 09:36:47 2012] [error] [client 89.144.206.157] File does not
    exist: /usr/local/http/htdocs/thirdman/reichs/blank.gif, referer:
    http://axion.physics.ubc.ca/thirdman/reichs/reichsbruecke.htm

    OK, I have closed down that virtual host I had set up ( which was the source
    of those aggregator web page requests), but I am still getting the same
    problems. It starts out with 9 copies of httpd daemon running. After a few
    hours it is up to 15 or 20. I have no idea what is causing this.
    I have now put in a cron job which checks every 10 min and if it finds more
    than 24 instances of httpd running, it restarts httpd (service httpd restart)
    But this is clearly a horrible kludge.
    Is there any way I can figure out what is triggering these versions of httpd
    to be piling up?




    ---------------------------------------------------------------------
    To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
    For additional commands, e-mail: users-help@httpd.apache.org
  • Pete Houston at May 24, 2012 at 8:45 am

    On Wed, May 23, 2012 at 07:47:37PM -0700, Bill Unruh wrote:
    But this is clearly a horrible kludge. Is there any way I can figure
    out what is triggering these versions of httpd
    to be piling up?
    Since you've looked at the logs and not found anything there (or at
    least eliminated what problems you have found), mod_status would be
    a good next step.

    If there is some reason why you cannot enable mod_status, then check who
    is holding the connections open with netstat.

    Pete
    --
    Openstrike - improving business through open source
    http://www.openstrike.co.uk/ or call 01722 770036 / 07092 020107
  • Bill Unruh at May 24, 2012 at 3:42 pm

    On Thu, 24 May 2012, Pete Houston wrote:
    On Wed, May 23, 2012 at 07:47:37PM -0700, Bill Unruh wrote:
    But this is clearly a horrible kludge. Is there any way I can figure
    out what is triggering these versions of httpd
    to be piling up?
    Since you've looked at the logs and not found anything there (or at
    least eliminated what problems you have found), mod_status would be
    a good next step.

    If there is some reason why you cannot enable mod_status, then check who
    is holding the connections open with netstat.
    Unfortunately I am a bit out of my depth here. How do I enable mod_status and
    then use it? And what exactly do I look for in netstat. I see nothing in
    netstat -pa just shows one httpd listaning on port www and one on port https

    which I presume is the master daemon listening for connections. (what do the
    other 8 daemons which are started up do?)

    tcp 0 0 *:www *:* LISTEN 6105/httpd
    tcp 0 0 *:https *:* LISTEN 6105/httpd

    It also shows
    tcp 0 0 boson:www mail5.alphameric.com:31040 FIN_WAIT2 -

    Why would it be waiting for a shutdown from the remote end?

    It does seem to be stuck. (ie, I do netstat -pa ten minutes apart and that
    line is still there)

    This address occurs in the logs as downloading my "thirdman" page
    (www.theory.physics.ubc.ca/thirdman/thirdman.html) which has a huge load of
    pictures on it.



    Pete
    --
    William G. Unruh | Canadian Institute for| Tel: +1(604)822-3273
    Physics&Astronomy | Advanced Research | Fax: +1(604)822-5324
    UBC, Vancouver,BC | Program in Cosmology | unruh@physics.ubc.ca
    Canada V6T 1Z1 | and Gravity | www.theory.physics.ubc.ca/

    ---------------------------------------------------------------------
    To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
    For additional commands, e-mail: users-help@httpd.apache.org
  • Bill Unruh at May 24, 2012 at 4:29 pm
    I was too quick to cry ignorance. I have now gotten mod_status working and
    here is the output. For some bizarre reason the number of servers is
    multiplying

    68.58.140.236 was downloading another page with lots of pictures on it.
    (http://axion.physics.ubc.ca/titanic/-- axion is an ancient pseudonym for
    www.theory.physics.ubc.ca)

    Why are these idle servers multiplying? As I said this all started out by my
    having 160 of them which was causing swapping death for the system.

    How do I tell apache not to do this multiplication of servers?

    Note that the aggregator requests keep coming even though the virtual host
    which hosted them is disconnected (commented out in the httpd configuratio
    nfiles)



    -------------------------------------------------------------

    Apache Server Status for www.theory.physics.ubc.ca

    Server Version: Apache/2.2.22 (Mandriva Linux/PREFORK-0.1mdv2010.2)
    mod_ssl/2.2.22 OpenSSL/1.0.0a PHP/5.3.13 with Suhosin-Patch mod_perl/2.0.4
    Perl/v5.10.1
    Server Built: Feb 1 2012 12:26:04

    Current Time: Thursday, 24-May-2012 09:18:07 PDT
    Restart Time: Thursday, 24-May-2012 08:55:18 PDT
    Parent Server Generation: 0
    Server uptime: 22 minutes 48 seconds
    Total accesses: 172 - Total Traffic: 3.9 MB
    CPU Usage: u.01 s.02 cu0 cs0 - .00219% CPU load
    .126 requests/sec - 2976 B/second - 23.1 kB/request
    1 requests currently being processed, 14 idle workers

    ______________W.................................................
    ................................................................
    ................................................................
    ................................................................

    Scoreboard Key:
    "_" Waiting for Connection, "S" Starting up, "R" Reading Request,
    "W" Sending Reply, "K" Keepalive (read), "D" DNS Lookup,
    "C" Closing connection, "L" Logging, "G" Gracefully finishing,
    "I" Idle cleanup of worker, "." Open slot with no current process

    Srv PID Acc M CPU SS Req Conn Child Slot Client VHost Request
    0-0 11536 0/38/38 _ 0.01 117 1 0.0 0.55 0.55 68.58.140.236 www.theory.physics.ubc.ca NULL
    1-0 11537 0/13/13 _ 0.00 117 332 0.0 0.51 0.51 68.58.140.236 www.theory.physics.ubc.ca NULL
    2-0 11538 0/11/11 _ 0.00 117 5 0.0 0.47 0.47 68.58.140.236 www.theory.physics.ubc.ca NULL
    3-0 11539 0/14/14 _ 0.00 117 1 0.0 0.61 0.61 68.58.140.236 www.theory.physics.ubc.ca NULL
    4-0 11540 0/19/19 _ 0.00 117 1 0.0 0.67 0.67 68.58.140.236 www.theory.physics.ubc.ca NULL
    5-0 11541 0/17/17 _ 0.00 117 0 0.0 0.30 0.30 66.249.68.41 www.theory.physics.ubc.ca GET /aggregator/www.guardian.co.uk/world/2012/jan/17/node/node/
    6-0 11542 0/6/6 _ 0.00 1159 1 0.0 0.01 0.01 24.82.146.42 www.theory.physics.ubc.ca NULL
    7-0 11544 0/49/49 _ 0.02 117 1 0.0 0.77 0.77 68.58.140.236 www.theory.physics.ubc.ca NULL
    8-0 11737 0/4/4 _ 0.00 3 0 0.0 0.00 0.00 66.249.68.193 www.theory.physics.ubc.ca GET /aggregator/www.youtube.com/www.iaea.org/Publications/Docum
    14-0 11743 0/1/1 W 0.00 0 0 0.0 0.00 0.00 24.82.146.42 www.theory.physics.ubc.ca GET /server-status HTTP/1.1
    Srv Child Server number - generation
    PID OS process ID
    Acc Number of accesses this connection / this child / this slot
    M Mode of operation
    CPU CPU usage, number of seconds
    SS Seconds since beginning of most recent request
    Req Milliseconds required to process most recent request
    Conn Kilobytes transferred this connection
    Child Megabytes transferred this child
    Slot Total megabytes transferred this slot




    On Thu, 24 May 2012, Pete Houston wrote:
    On Wed, May 23, 2012 at 07:47:37PM -0700, Bill Unruh wrote:
    But this is clearly a horrible kludge. Is there any way I can figure
    out what is triggering these versions of httpd
    to be piling up?
    Since you've looked at the logs and not found anything there (or at
    least eliminated what problems you have found), mod_status would be
    a good next step.

    If there is some reason why you cannot enable mod_status, then check who
    is holding the connections open with netstat.

    Pete
    --
    William G. Unruh | Canadian Institute for| Tel: +1(604)822-3273
    Physics&Astronomy | Advanced Research | Fax: +1(604)822-5324
    UBC, Vancouver,BC | Program in Cosmology | unruh@physics.ubc.ca
    Canada V6T 1Z1 | and Gravity | www.theory.physics.ubc.ca/

    ---------------------------------------------------------------------
    To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
    For additional commands, e-mail: users-help@httpd.apache.org
  • Pete Houston at May 24, 2012 at 5:03 pm
    A dozen or so idle processes is perfectly normal for prefork (which
    you are clearly running, BTW). Only worry about this if there are a
    consistently high number of idle processes (say 30 or more for a lightly
    loaded server) in which case you can tune the value of MaxSpareServers
    to suit.

    Have a read about the prefork MPM in the documentation:
    http://httpd.apache.org/docs/2.2/mod/prefork.html

    If the number of idle processes is consistently higher than
    MaxSpareServers you have a bug.

    HTH,

    Pete
    --
    Openstrike - improving business through open source
    http://www.openstrike.co.uk/ or call 01722 770036 / 07092 020107
  • Bill Unruh at May 24, 2012 at 5:58 pm

    On Thu, 24 May 2012, Pete Houston wrote:

    A dozen or so idle processes is perfectly normal for prefork (which
    you are clearly running, BTW). Only worry about this if there are a
    consistently high number of idle processes (say 30 or more for a lightly
    loaded server) in which case you can tune the value of MaxSpareServers
    to suit.
    Thanks. As I said that problem raised its head because there were 160 of them
    and the system was swapping itself to death. But that was on an earlier
    version (2.2.0 I think it was). I will look at MaxSpareServers.
    Have a read about the prefork MPM in the documentation:
    http://httpd.apache.org/docs/2.2/mod/prefork.html

    If the number of idle processes is consistently higher than
    MaxSpareServers you have a bug.
    OK, my MaxSpareServers is 20 and I had 21 (mind you that is counting the
    master, so maybe it was actually 20) yesterday all day.

    HTH,

    Pete
    --
    William G. Unruh | Canadian Institute for| Tel: +1(604)822-3273
    Physics&Astronomy | Advanced Research | Fax: +1(604)822-5324
    UBC, Vancouver,BC | Program in Cosmology | unruh@physics.ubc.ca
    Canada V6T 1Z1 | and Gravity | www.theory.physics.ubc.ca/

    ---------------------------------------------------------------------
    To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
    For additional commands, e-mail: users-help@httpd.apache.org

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupusers @
categoriesapache
postedMay 22, '12 at 5:02p
activeMay 24, '12 at 5:58p
posts12
users3
websitehttpd.apache.org
irc#httpd

People

Translate

site design / logo © 2022 Grokbase