FAQ
I just deployed our quite large Catalyst app and I'm seeing a constant increase
in memory usage. Here's a current snapshop from top:

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
29342 www 16 0 67156 60m 3180 S 1 3.0 0:13.87 perl
29332 www 25 0 41052 34m 1204 S 0 1.7 0:00.01 perl
29333 www 25 0 68424 60m 3172 S 0 3.0 0:13.42 perl
29334 www 22 0 68944 60m 3308 S 0 3.0 0:13.58 perl
29335 www 16 0 67428 60m 3168 S 0 3.0 0:13.86 perl
29336 www 16 0 68736 60m 3192 S 0 3.0 0:13.88 perl
29337 www 16 0 67392 59m 3172 S 0 2.9 0:14.02 perl
29338 www 18 0 66404 59m 3180 S 0 2.9 0:13.49 perl
29339 www 17 0 69216 62m 3172 S 0 3.1 0:14.12 perl
29340 www 16 0 67992 60m 3188 S 0 3.0 0:13.89 perl
29341 www 16 0 68812 60m 3180 S 0 3.0 0:14.88 perl

We're using lighttpd and handle the starting of the FCGI server manually. Is
there a way to have the FCGI children restart once in a while without having to
restart the whole FCGI process manager (and bringing down the application for a
few seconds)? Something like Apache's MaxRequestsPerChild directive would be
cool.

Then it's time to go memory leak hunting :(

Thanks a lot!

--Tobias

Search Discussions

  • Bernhard Graf at Aug 16, 2007 at 10:18 am

    On Thursday 16 August 2007 10:40, Tobias Kremer wrote:
    I just deployed our quite large Catalyst app and I'm seeing a
    constant increase in memory usage. Here's a current snapshop from
    top:
    We're using lighttpd and handle the starting of the FCGI server
    manually. Is there a way to have the FCGI children restart once in a
    while without having to restart the whole FCGI process manager (and
    bringing down the application for a few seconds)? Something like
    Apache's MaxRequestsPerChild directive would be cool.
    I use daemontools similar to
    http://www.catalystframework.org/calendar/2006/4 .

    My run file looks like this:

    ========================================================================
    #!/bin/sh

    cd /srv/www/vhosts/myapp

    exec 2>&1 \
    setuidgid myappuser \
    softlimit -m 150000000 \
    script/myapp_fastcgi.pl -e -l /var/tmp/myapp/fastcgi.socket -n 10
    ========================================================================

    Sets uid and gid to "myappuser ", Spawns 10 processes and listens on the
    specified UNIX socket. If a process gets too big, it gets terminated and
    replaced by a fresh one (softlimit is part of daemontools, but you could
    use ulimit instead).
    Then it's time to go memory leak hunting :(
    I don't care much since softlimit restarts the FCGI processes (under
    normal conditions approx. every 12 hours).

    BTW - here are the manpages for daemontools
    http://smarden.sunsite.dk/pape/djb/manpages/daemontools-0.76-man.tar.gz


    HTH
    --
    Bernhard Graf
  • Tobias Kremer at Aug 16, 2007 at 12:37 pm

    Quoting Bernhard Graf <catalyst2@augensalat.de>:
    On Thursday 16 August 2007 10:40, Tobias Kremer wrote:
    We're using lighttpd and handle the starting of the FCGI server
    manually. Is there a way to have the FCGI children restart once in a
    while without having to restart the whole FCGI process manager (and
    bringing down the application for a few seconds)? Something like
    Apache's MaxRequestsPerChild directive would be cool.
    I use daemontools similar to
    http://www.catalystframework.org/calendar/2006/4 .
    Thanks! I'll take a look at it.

    For now I'm gonna try running two independent FastCGI process managers and
    restart them on different hours via a cronjob so that one of them will always
    be available to lighttpd. Other solutions are still welcome! :)

    --Tobias
  • Bernhard Graf at Aug 16, 2007 at 1:06 pm

    On Thursday 16 August 2007 13:42, Tobias Kremer wrote:
    Quoting Bernhard Graf <catalyst2@augensalat.de>:
    Thanks! I'll take a look at it.
    You should definitely do that. Not only for this case - daemontools (or
    similar like runit) are superior for nearly every server service on *ix
    OSes. Check out http://smarden.org/runit/runscripts.html and compare
    the scripts with typical System-V run scripts (not mentioning the
    supervision concept in general).
    For now I'm gonna try running two independent FastCGI process
    managers and restart them on different hours via a cronjob so that
    one of them will always be available to lighttpd. Other solutions are
    still welcome! :)
    Why two FCGI process managers? To compensate FCGI downtimes? Did you
    find a way to tell lighttpd not to talk to an FCGI process that is
    down? I only get a 500 error in that case.

    --
    Bernhard Graf
  • Tobias Kremer at Aug 16, 2007 at 1:27 pm

    Quoting Bernhard Graf <catalyst2@augensalat.de>:
    You should definitely do that. Not only for this case - daemontools (or
    similar like runit) are superior for nearly every server service on *ix
    OSes. Check out http://smarden.org/runit/runscripts.html and compare
    the scripts with typical System-V run scripts (not mentioning the
    supervision concept in general).
    I suppose that the whole fcgi-pm process will be restarted instead of individual
    fcgi processes, correct? That'd mean a downtime of about 5 seconds which would
    render it useless to me :(
    Why two FCGI process managers? To compensate FCGI downtimes? Did you
    find a way to tell lighttpd not to talk to an FCGI process that is
    down? I only get a 500 error in that case.
    I just _hope_ that lighttpd will do the right thing as the error log tells me
    this when a backend server goes down:

    connect failed: Connection refused on unix:/srv/webapp.socket
    backend died; we'll disable it for 5 seconds and send the request to another
    backend instead: reconnects: 0 load: 1

    --Tobias
  • Daniel McBrearty at Aug 16, 2007 at 1:38 pm
    daemontools is brilliant. I use it for several services now. it is so
    easy to set up and use.

    hot tip : don't forget the -e to myapp_fastcgi.pl option to get
    multilog writing to your log files.



    On 8/16/07, Tobias Kremer wrote:
    Quoting Bernhard Graf <catalyst2@augensalat.de>:
    You should definitely do that. Not only for this case - daemontools (or
    similar like runit) are superior for nearly every server service on *ix
    OSes. Check out http://smarden.org/runit/runscripts.html and compare
    the scripts with typical System-V run scripts (not mentioning the
    supervision concept in general).
    I suppose that the whole fcgi-pm process will be restarted instead of individual
    fcgi processes, correct? That'd mean a downtime of about 5 seconds which would
    render it useless to me :(
    Why two FCGI process managers? To compensate FCGI downtimes? Did you
    find a way to tell lighttpd not to talk to an FCGI process that is
    down? I only get a 500 error in that case.
    I just _hope_ that lighttpd will do the right thing as the error log tells me
    this when a backend server goes down:

    connect failed: Connection refused on unix:/srv/webapp.socket
    backend died; we'll disable it for 5 seconds and send the request to another
    backend instead: reconnects: 0 load: 1

    --Tobias

    _______________________________________________
    List: Catalyst@lists.rawmode.org
    Listinfo: http://lists.rawmode.org/mailman/listinfo/catalyst
    Searchable archive: http://www.mail-archive.com/catalyst@lists.rawmode.org/
    Dev site: http://dev.catalyst.perl.org/

    --
    Daniel McBrearty
    email : danielmcbrearty at gmail.com
    http://www.engoi.com
    http://danmcb.vox.com
    http://danmcb.blogger.com
    find me on linkedin and facebook
    BTW : 0873928131
  • Bernhard Graf at Aug 16, 2007 at 2:10 pm

    On Thursday 16 August 2007 14:32, Tobias Kremer wrote:
    Quoting Bernhard Graf <catalyst2@augensalat.de>:
    You should definitely do that. Not only for this case - daemontools
    (or similar like runit) are superior for nearly every server
    service on *ix OSes. Check out
    http://smarden.org/runit/runscripts.html and compare the scripts
    with typical System-V run scripts (not mentioning the supervision
    concept in general).
    I suppose that the whole fcgi-pm process will be restarted instead of
    individual fcgi processes, correct? That'd mean a downtime of about 5
    seconds which would render it useless to me :(
    No. The softlimit applies to each of the processes forked by the master,
    so each process is terminated if it gets too fat.
    Why two FCGI process managers? To compensate FCGI downtimes? Did
    you find a way to tell lighttpd not to talk to an FCGI process that
    is down? I only get a 500 error in that case.
    I just _hope_ that lighttpd will do the right thing as the error log
    tells me this when a backend server goes down:

    connect failed: Connection refused on unix:/srv/webapp.socket
    backend died; we'll disable it for 5 seconds and send the request to
    another backend instead: reconnects: 0 load: 1
    Just by setting multiple "socket" entries in the lighttpd-conf?

    --
    Bernhard Graf
  • Tobias Kremer at Aug 16, 2007 at 2:25 pm

    Quoting Bernhard Graf <catalyst2@augensalat.de>:
    On Thursday 16 August 2007 14:32, Tobias Kremer wrote:
    Quoting Bernhard Graf <catalyst2@augensalat.de>:
    You should definitely do that. Not only for this case - daemontools
    (or similar like runit) are superior for nearly every server
    service on *ix OSes. Check out
    http://smarden.org/runit/runscripts.html and compare the scripts
    with typical System-V run scripts (not mentioning the supervision
    concept in general).
    I suppose that the whole fcgi-pm process will be restarted instead of
    individual fcgi processes, correct? That'd mean a downtime of about 5
    seconds which would render it useless to me :(
    No. The softlimit applies to each of the processes forked by the master,
    so each process is terminated if it gets too fat.
    Cool! :)
    I just _hope_ that lighttpd will do the right thing as the error log
    tells me this when a backend server goes down:
    connect failed: Connection refused on unix:/srv/webapp.socket
    backend died; we'll disable it for 5 seconds and send the request to
    another backend instead: reconnects: 0 load: 1
    Just by setting multiple "socket" entries in the lighttpd-conf?
    Yes, like so:

    $HTTP["host"] =~ "bla" {
    fastcgi.server = (
    "" => (
    "MyApp" => (
    "socket" => "/srv/socket_1",
    "check-local" => "disable"
    ),
    (
    "socket" => "/srv/socket_2",
    "check-local" => "disable"
    )
    )
    )
    }

    --Tobias
  • Daniel McBrearty at Aug 16, 2007 at 10:23 am
    IIRC you see this under mod_perl because of the way that memory is
    shared between processes until it is written to, when a seperate copy
    is created per process. It happens even with no leaks.

    I have no idea whether FCGI has a similar mechanism, but just
    something to maybe look at.

    On 8/16/07, Tobias Kremer wrote:
    I just deployed our quite large Catalyst app and I'm seeing a constant increase
    in memory usage. Here's a current snapshop from top:

    PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
    29342 www 16 0 67156 60m 3180 S 1 3.0 0:13.87 perl
    29332 www 25 0 41052 34m 1204 S 0 1.7 0:00.01 perl
    29333 www 25 0 68424 60m 3172 S 0 3.0 0:13.42 perl
    29334 www 22 0 68944 60m 3308 S 0 3.0 0:13.58 perl
    29335 www 16 0 67428 60m 3168 S 0 3.0 0:13.86 perl
    29336 www 16 0 68736 60m 3192 S 0 3.0 0:13.88 perl
    29337 www 16 0 67392 59m 3172 S 0 2.9 0:14.02 perl
    29338 www 18 0 66404 59m 3180 S 0 2.9 0:13.49 perl
    29339 www 17 0 69216 62m 3172 S 0 3.1 0:14.12 perl
    29340 www 16 0 67992 60m 3188 S 0 3.0 0:13.89 perl
    29341 www 16 0 68812 60m 3180 S 0 3.0 0:14.88 perl

    We're using lighttpd and handle the starting of the FCGI server manually. Is
    there a way to have the FCGI children restart once in a while without having to
    restart the whole FCGI process manager (and bringing down the application for a
    few seconds)? Something like Apache's MaxRequestsPerChild directive would be
    cool.

    Then it's time to go memory leak hunting :(

    Thanks a lot!

    --Tobias

    _______________________________________________
    List: Catalyst@lists.rawmode.org
    Listinfo: http://lists.rawmode.org/mailman/listinfo/catalyst
    Searchable archive: http://www.mail-archive.com/catalyst@lists.rawmode.org/
    Dev site: http://dev.catalyst.perl.org/

    --
    Daniel McBrearty
    email : danielmcbrearty at gmail.com
    http://www.engoi.com
    http://danmcb.vox.com
    http://danmcb.blogger.com
    find me on linkedin and facebook
    BTW : 0873928131
  • Antano Solar at Aug 19, 2007 at 6:27 am

    On 8/16/07, Tobias Kremer wrote:
    I just deployed our quite large Catalyst app and I'm seeing a constant
    increase
    in memory usage. Here's a current snapshop from top:

    PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
    29342 www 16 0 67156 60m 3180 S 1 3.0 0:13.87 perl
    29332 www 25 0 41052 34m 1204 S 0 1.7 0:00.01 perl
    29333 www 25 0 68424 60m 3172 S 0 3.0 0:13.42 perl
    29334 www 22 0 68944 60m 3308 S 0 3.0 0:13.58 perl
    29335 www 16 0 67428 60m 3168 S 0 3.0 0:13.86 perl
    29336 www 16 0 68736 60m 3192 S 0 3.0 0:13.88 perl
    29337 www 16 0 67392 59m 3172 S 0 2.9 0:14.02 perl
    29338 www 18 0 66404 59m 3180 S 0 2.9 0:13.49 perl
    29339 www 17 0 69216 62m 3172 S 0 3.1 0:14.12 perl
    29340 www 16 0 67992 60m 3188 S 0 3.0 0:13.89 perl
    29341 www 16 0 68812 60m 3180 S 0 3.0 0:14.88 perl

    We're using lighttpd and handle the starting of the FCGI server manually.
    Is
    there a way to have the FCGI children restart once in a while without
    having to
    restart the whole FCGI process manager (and bringing down the application
    for a
    few seconds)? Something like Apache's MaxRequestsPerChild directive would
    be
    cool.

    Then it's time to go memory leak hunting :(

    Thanks a lot!

    --Tobias

    Top adds memory usage for libraries for every instance even when they are
    shared libraries

    Antano Solar John
    -------------- next part --------------
    An HTML attachment was scrubbed...
    URL: http://lists.scsys.co.uk/pipermail/catalyst/attachments/20070819/e3d97963/attachment-0001.htm

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupcatalyst @
categoriescatalyst, perl
postedAug 16, '07 at 9:35a
activeAug 19, '07 at 6:27a
posts10
users4
websitecatalystframework.org
irc#catalyst

People

Translate

site design / logo © 2021 Grokbase