[Catalyst] memory usage of mod_perl process

It's Perl itself taking the memory, so I doubt you'll see much
difference, but go ahead and try it. I'd be curious.

There is plenty of documentation on how to reduce the memory size of
applications running on mod_perl. Take a look at this to get you
started:
http://modperlbook.org/html/ch10_01.html
http://modperlbook.org/html/ch14_01.html

- Perrin

We've got around 100+ pms and myapp_server.pl uses around 87mb of RAM.
Don't really know about usage under mod_perl, I'd need to check it out.

-Nilson Santos F. Jr.

+----[ Nilson Santos Figueiredo Junior (08.Feb.2007 11:37):
+----]

You might want to tune your apache/modperl (google 'apache
modperl tuning').
http://perl.apache.org/docs/1.0/guide/performance.html

But also, compare the memory usage of your modperl httpd
processes, and the one of the catalyst webapp_server.pl.
The comparison should give you an idea of how much overhead
is coming from apache. You should evaluate if the effort of
trimming apache functionality (for example by removing
dynamically loaded modules) is worth it.

Just as a reference, I've got ~13 Mb usage from a barebones
catalyst app (create a new app and run its standalone
server). While the app I'm working on (91 pms, of which ~80
are coming from the model) is running at ~150 Mb in
apache/modperl (but this is not fair to catalyst because
this is a heavy apache that is also running mod_python
(trac/svn)) and at ~86 Mb using the standalone web server.

Are you using DBIx::Class?

Our DBIx::Class model is ~80 tables. Because during
development we were also changing the underlying database,
at one point we were running our test server using three
different instances of these models, each against a
different db instance. In this case, the average httpd
process was huge. The mem size was reduced by ~70Mb for each
model instance that we eliminated.

So yes, you can gain a significant amount of memory if you trim
down your db tables/schema and your DBIx::Class model. In
our case those ~80 tables are there because the database is
also used outside of the catalyst webapp. When development
of the catalyst app stabilizes, we will go over the model
and remove all .pm files for tables that the catalyst app
doesn't use.

Hope this helps,

Fernan

Are pms number of entries in %INC?

I have 464 entries in %INC, but that's hardly a good measurement
considering how small some of the plugins are. So maybe pms means
something else? Sure does to my wife.

Top shows:

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
26074 www-data 15 0 72188 51m 6184 S 3.3 5.1 0:07.60 apache2

A lot of people seem to like FastCGI. I started out using a FastCGI
setup but restarts were taking a long time (seemed every FastCGI
process had to compile the application). I moved to mod_perl and
much happier now.

+----[ Bill Moseley (08.Feb.2007 12:16):
I think this is the number of .pm modules under your application's
own lib/. Or at least this is how I interpreted it.
yes, not the best way to measure it, agreed.

Fernan
+----]

I am the coworker of Fayland who posted the topic "memory usage of mod_perl
process".

Our company has invested quite some time on migrating our perl code to
catalyst (more than half year of time by 7 programmers). However, I am
starting to worry that moving to catalyst is causing too much overhead for
our processes and thus severely slowing down our site.

I remember the memory usage of each process before catalyst was around 25Mb.
And now the new code is using around 90MB as fayland said. During peak hour,
our servers were able to handle 30+ requests per sec before catalyst
upgrade. Now we are only able to handle around 10 requests per sec.

As a result, the response time of the site is much slower after the upgrade.
it takes at least 2 sec for the browser to start responding during non-peak
hours.

Here's the memory stats for a typical process:

Memory Usage (in bytes):

Size : 97107968 (92.6M)
Share : 6012928 ( 5.7M)
VSize : 97107968 (92.6M)
RSS : 85131264 (81.2M)

Memory Segments Usage (in bytes):

Text : 253952 ( 248K)
Shlib : 0 ( 0K)
Data : 78696448 (75.1M)
Stack : 0 ( 0K)



I have read practical mod_perl. I tried to preload many of our modules in
startup.pl. But the shared memory value doesnt change at all. Also, doesnt
5.7M shared memory usage sound too small comparing to the 92.6M total size?

I wonder how much is your shared memory? Is the memory overhead of catalyst
huge? Is catalyst inherently slow?

You're not looking at the right thing there. This is not actual
shared memory, but rather copy-on-write sharing. It will never show
in your SHARE section. On versions of Linux with a 2.6 kernel, you
can use the techniques in Apache::SizeLimit to get a sesne of how much
sharing is going. This doesn't work with older kernels.

A reliable way to tell if your sharing is improved is to start up
apache without preloading and look at the available memory from
"free", and then put in the preloading and restart apache and check it
again. You should have more free memory after preloading, allowing
you to run more processes.

- Perrin

when i run this "free" test, should i run it on the live server, or on a
test server with single process apache mode?

here's the result on a live server:

total used free shared buffers cached
Mem: 4150972 2654016 1496956 0 27548 1313344
-/+ buffers/cache: 1313124 2837848
Swap: 8385912 6920 8378992


and here's the top results on one of the server:

top - 14:21:55 up 144 days, 9:30, 0 users, load average: 19.30, 18.65,
12.27
Tasks: 87 total, 1 running, 86 sleeping, 0 stopped, 0 zombie
Cpu(s): 69.6% us, 1.3% sy, 0.0% ni, 27.7% id, 0.0% wa, 0.2% hi, 1.2% si
Mem: 4150972k total, 2623968k used, 1527004k free, 27380k buffers
Swap: 8385912k total, 6920k used, 8378992k free, 1312832k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
26343 apache 15 0 97196 83m 6236 S 10.0 2.1 0:08.69 httpd
26340 apache 15 0 94740 81m 5896 S 7.8 2.0 0:07.62 httpd
26315 apache 15 0 95552 81m 5896 S 7.4 2.0 0:08.31 httpd
26326 apache 15 0 94836 81m 5868 S 7.4 2.0 0:08.17 httpd
26371 apache 15 0 95496 81m 5872 S 7.4 2.0 0:08.21 httpd


I set the maxserver as 60.
Our CPU load is extremely high. But the actual RAM usage is not high at all.
we are using only 260 gig out of 400 gig. From our top results, does it look
like the memory usage of our processes is the bottleneck?

- Perrin

I wouldn't mess with things on your live server, but you want to run
in normal mode, not single-process.
Not even that much. You're using about 1.3 GB and the rest of that
2.6 is just being used to cache files. You should be able to run a
lot more apache processes on this machine, or maybe give some RAM to
whatever else needs it on there, like a database.
No, you have tons of free RAM. It sounds like something else is
slowing you down. You have a lot of CPU being used on there (about
60%) which might mean you need to profile your code for CPU
bottlenecks. You may also be waiting on database queries, which you
can find out with Devel::DProf or the DBI profiler.

- Perrin

hi how can you tell our apache is just using 1.3 GB? what king of files does
the rest of it cache?...

can we run devel::DProf with mod_perl catatlyst? or do we have to use
apache::Dprof?

- Perrin

[...]
Look at the second line of output from free (or free -m, for
megabytes). It's telling you that you're only using 1313124 bytes for
applications, and the rest is buffers and cache.

For more info on buffers and cache, try googling it:
http://www.google.com/search?hl=en&safe=off&q=linux+free+buffers+cache&btnG=Search

The first hit looks pretty good:
http://gentoo-wiki.com/FAQ_Linux_Memory_Management
Apache::DProf should work, but I think people often profile catalyst
apps on the test server using Devel::DProf. There's probably
something on the catalyst site about this if you search for it.

- Perrin

I too have found that my Catalyst apps are "huge" compared to previous mod_perl
or FastCGI apps. I had been using Class::DBI and a hand-rolled framework, but
now use Catalyst and DBIx::Class. The difference is at least a factor of 2.

It's not been a problem for me at my current traffic levels, and I'm more than
happy to trade some RAM for the joy the design and coding has become compared to
my previous situation.

But I do wonder where all the RAM is going. I especially concur that not much
is being shared which is curious given that the modules are preloaded by
mod_perl. I had done some profiling to measure how much RAM a dummy Cat app
took vs a dummy DBIC schema, maybe I'll dig that up and send it to the list.

Best,
Brian

Take a good, hard look at what's coming out of your web servers. Install
a Squid farm in front of the servers and let them cache anything suitable...

letting Apache fork just to serve an image can be really damaging to
performance. Double check the HTTP headers on all outgoing pages include
proper caching instructions for Squid so it'll work properly.

Even whole store-front pages from Catalyst sites can usually be cached
well, because users don't have state at that point, and perhaps most
hits are to your homepage (so remember to let Squid know your homepage
is cacheable, in the headers).

Remember mod_perl can be quite leaky as well, so kill off the child
processes after they've served a few requests. Oh, and switch off
hyperthreading if you have dual P4/Xeon boxes, and use PreFork MPM.

There's more info in the slides from this LPW talk:

http://london.pm.org/lpw/talks/2006/dave_hodgkinson-mod_perl_server_farm_architecture_done_right.pdf

HTH,

oliver.

yes we are thinking about that now. however, we are not sure how to update
the cache we we want to refresh the page. i guess we will adjust the http
header when we want to flush it? how does squid know if our page is
refreshed when it has already cached our page though?...

letting Apache fork just to serve an image can be really damaging to

yah, i know that. our catalyst apache servers are only for serving dynamic
pages. however, there are some ajax javascript files we have to host on the
exact same subdomain to work. i still feel that those javascript files are
wasting our apache resources. for ajax javascript like that, how should i
redirect the load to other servers while maintaining the same subdomain in
the location bar? i tried to use mod_rewrite, and seems like its even slower
than serving the file itself!

Even whole store-front pages from Catalyst sites can usually be cached

You can call the PURGE command on squid. You must allow this in the conf and
put in place the proper acl to allow it. I believe you can call squid with
the following options to force a purge of the cache of a given page.

-m PURGE http://mywebsite.com/index.html

We are not using catalyst in production yet so we don't have squid running in
front of any catalyst apps. We do however have it running in front of some
python/zope/plone sites. There are plugins that handle the purging directly
for us. We just provide the url to purge to a method that handles the purge
when needed.

to
am
overhead
Install
update
http
Or set the Time to live on the page headers to a reasonable amount of time
-- lets say 1 minute. This basically serves cache for 99.9% of the hits
and only goes to the app server once per minute for a refresh. No perge
needed, just tune the cache time to be the maximum you feel comfortable
with. Hitting the app server from your frontend every once in a while for
static or cacheable pages is no huge deal -- don't sacrifice administrative
costs for one or two hits a minute.

right! i just googled and found
http://search.cpan.org/~sock/Net-Squid-Purge-0.1/
that cpan module should be able to purge the cached.

There are a lot of good tuning tips in the Apache docs, as Perrin says.

I presume you've tried things like using a lightweight front end proxy (to
serve images/static files) and building a custom backend mod_perl Apache
that includes *only* the modules you need. If you use the stock
Fedora/Redhat Apache that's not optimal.
In your Apache config preload everything pre-fork (if you use that model)
including setting up the DB connection. Write code that caches expensive
calculated values/lookup files/SQL values in memory with a cheap
last-modified check before re-doing work.
Don't set MaxRequestsPerChild too high (I use 100) as the mem usage will
climb as Perl switches some shared memory variables to new memory, as the
Apache mod_perl docs describe.

For comparison, I'm using DBIx::Class across 15 tables and a really simple
hand-rolled MVC (not Cat) for a medium volume site. It gives 'top' lines
like:
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
21495 apache 15 0 203m 43m 6188 S 0.0 4.5 0:01.46 httpd
This can handle over 300 requests/sec with 1GB memory on an x86_64 single
processor.

Regards, Peter
www.dragonstaff.com

300/sec from the application server? That's impressive. I'm happy
when I see a see a tenth of that. Are you hitting the database very
hard? Complex joins? Are your pages very complex?

very complex?
No, it's a fairly simple backend processor that mostly handles XML
transactions for a Flash frontend. There are some XHTML admin pages
generated with Template::Toolkit (and so are slower) but they are rarely
hit.

For the database a typical run hits a session table, user table, writes a
log table and reads data from a couple of other tables. No complex joins.
The app handles user registration, some league tables, polls, competition
entries and users submitting images that are approved and shown on the site.

In this case I knew there was the potential for a huge number of users so
went for a lightweight approach that could easily be tuned further. For a
more complex app with fewer peak users I'd have used Catalyst.

Regards, Peter
www.dragonstaff.com

what other light weight approach would you go for?
so are you saying in general catalyst is pretty heavy, and its not for site
with many concurrent users?...

Regards, Peter

with many concurrent users?...

No, not at all. It all depends on what your application does, how many web
screens, how many peak concurrent users and equivalent requests/second, how
many discrete users, how many data feeds in/out, data volumes, network
transfer volumes.

Your limiting factor might be CPU, database I/O or network bandwidth.

Different architectural models are best for different cases.



There is a cost equation between using a powerful framework that increases
robustness, maintainability and speed of development versus how many servers
you need.

Say a server costs you $100 per month to rent and each Catalyst server maxes
at 30 requests per second but you get a peak load of 90 reqs/sec. You would
need to load balance your app across 3 servers. There would be an initial
configuration cost and then you'd be paying $200 p.m. = $2400 p.a. more for
the two extra servers.

Let's say a developer, including overhead, costs you $1000 per day. If you
move away from Catalyst to a lighter framework and it costs you more than
2.5 developer days extra then you're losing money. Catalyst has a lot of
tried and tested pluggable modules and if you end up re-writing any of those
you're almost certainly spending money.


First thing, set up proper caching (either Squid or a lightweight Apache)
and use a tool like Fiddler to examine the HTTP request Cache headers to
make sure it's working properly. That's a huge saving.

Then do profiling and benchmarking to determine exactly what the bottlenecks
are before making any changes.

If you're doing a lot of dynamic Ajax stuff with complex SQL, there might be
a case for hand-tuning that part of the app. Look at whether you can
pre-calculate and cache expensive queries or change the app design to allow
this. If necessary, you could write a separate lightweight app to serve
those requests.



I didn't find a suitable lightweight mod_perl framework so wrote one using
CGI::Session, DBIx::Class, Template toolkit. It's not that difficult.

If you would like some details please feel free to email off-list or call
me. However, that was an atypical case and as soon as you start needing
internationalization, RSS feeds etc. I think you're better off with
Catalyst.



You said you need to serve JS from the same server as the heavyweight app.
Assuming the JS pages are static or can be precalculated and stored in a
static dir, and you have the JS below /js and your app below /app in the URI
namespace:

On the same server run two Apaches (or one Apache + one Fastcgi)

1) Lightweight proxy. Build it yourself from source, turn off all
unnecessary options and shared libraries except mod_proxy/mod_rewrite.

Use rewrite rules to proxy /app/. on to heavy server, e.g. based on an
Apache 1 setup I have that's live:

RewriteEngine On

RewriteRule ^/app/(.*$) http://localhost:2000/app/$1 [P]

ProxyPassReverse / http://localhost:2000/

RewriteRule ^proxy:.* - [F]

The /js and /images get served by the proxy server (which has a <1MB memory
footprint per child)

2) Heavy mod_perl server listening at localhost:2000. Maps /app on to your
handler (40-100MB memory per child).



Say you had to handle the Ajax part separately. With this model you can add
another proxy entry that routes, say, /ajax/ on to another Apache server on
the same host (e.g. localhost:2001) or on another host altogether (e.g.
http://ajax1.zorpia.com <http://ajax1.zorpia.com/> ). If you need to scale
more, use DNS round robin to a set of front end proxies like this and route
requests to a farm of backend servers using a shared database server.



I'd be surprised if you couldn't get decent performance out of a
well-designed Catalyst app. There are businesses listed on the Catalyst
pages that offer consultancy help if you need further pointers. As a
business I used Shadowcat Systems last year and received good technical
design advice.



Regards,

Peter Edwards

www.dragonstaff.com <http://www.dragonstaff.com/> ~ Business IT Consultancy

Just a couple of comments on this topic.

If you're using apache2.2 then mod_cache is available. This can be used to
cache the result of a request either in memory or disk. Ideal for the
situations where you want to cache the front page of your site every minute.

http://httpd.apache.org/docs/2.2/mod/mod_cache.html

Also if you want to offload all your static resources to a non-Cat server
then the simplest way of doing that is to simply stick a full url in the
HTML.

<img src="http://static.example.com/images/logo.png" />

Use a config option so you can vary it:

<img src="[% c.config.static_path %]/images/logo.png" />

Using ConfigLoader you can have different devel/live values for static_path
too.

Carl