[nodejs] Memory leak, growing RSS and debugging

If you did read the list, you would come across the same issue being
discussed in this post:
https://groups.google.com/d/msg/nodejs/Aq9BId5Tff8/7G3JxaIVd2YJ

https://github.com/Jimbly/node-mtrace

Thanks Ilya, I'll have a second look at node-mtrace.

I tried this (as I did read the list ;-)), but it did not help much, and I
hoped someone could identify a specific class of problems with my
description (RSS grows, heap stays ok)...

Nico



Am Montag, 19. März 2012 16:06:38 UTC+1 schrieb Ilya Dmitrichenko:

Rising RSS and no JS heap growth is exactly the symptoms I was seeing
that caused me to expose mtrace in a node module, but my conclusion is
that it's a hard thing to track down, especially in C++ app. Sadly it
seems Buffers show up as just "operator new()" in an mtrace, so if
those are leaking, you won't know much for sure, as it'll just be
reported in the one big "new" bucket, though looking at the sizes of
outstanding allocations (using the GCC mtrace command line tool) you
might be able to deduce something useful.

I also had some luck just taking a core dump with gcore and looking at
the dump in a hex editor (in my case, the data was clearly big blocks
of triplets of 32 bit floats and then big blocks of integers, so I
could conclude it was vertex and index buffers, and combined with the
mtrace results I knew to look for places that were calling new on
these kinds of objects and might not be deleting them - in your case
perhaps you'll get lucky and see some repeating data that might
indicate what kind of data is leaking).

It's useful to note that any Buffers under 8kb in size end up coming
out of pooled 8kb buffers, and as long as any JS object is referencing
any slice of that buffer, the whole 8kb is still going to be in
memory, which can be misleading (or the cause of "leaks" if you're,
say, slicing out a few bytes of the buffer and hanging on to them for
some reason).

- Jimb

Nico did you have any luck in your debugging adventures? My symptoms seems
to very closely match yours. Was going to try mtrace.

Same problem here. Our app slowly increases the rss. and i could not
find anything with node-webkit-agent :(

Am 12.09.2012 02:58, schrieb wavded:

Mit freundlichen Grüßen

i solved my problems with an update to 0.8.9.

in 0.8.3 there was a bugfix
which might caused a steady growing _events-array.

Am 12.09.2012 07:47, schrieb Stefan Zehe:

I'll check if I can get information about the _events arrays, but as I wrote before, 0.8.8 did not change much:

http://cl.ly/image/2Y3N3w0t1I34


Nico

Having similar issue to what Nico is seeing...

(a not very helpful picture follows):
the blue is RSS growing to fill up available memory, while the tiny pink
and red lines are the heapTotal and heapUsed, which stays bounded (around
20-25 MB heap total in my case), data from 0.8.8 (haven't tried 0.8.9 yet)

https://plus.google.com/u/0/photos/106160293361349494652/albums/5787976828393231153/5787976827897869394

the common thread for me with Nico is that connections are constantly
created and destroyed. I've seen this behavior both using https connections
and then using only tls connections.

I get a similar issue using socket.io with WEBSOCKET transport enabled. It
also appear that RSS grows to fill up available memory but doesn't seem to
hit an OOM. It just stays at a very high memory usage. On a 2gig machine it
climbs to like 1960mb and is quite scary.

I've tried node up to 0.8.9 and still happens.

That's intentional. The V8 garbage collector tries very hard to do as
few sweeps as possible and it does that by growing the heap where
possible. The larger the heap is, the less V8 has to scavenge for
memory when new objects are created.

So this is what I was hoping. I agree with Ben, when a machine has enough memory, it should be used.
Lots of free memory are useless ...

So we can "ignore" this "problem", as it is no leak (and thus not causing the process to crash with OOM)...

Nico

So do I understand correctly: a node program could use up all the computer's memory?

If I then need to launch additional programs, will node notice this and reduce its memory usage so that the new program doesn't incur virtual memory penalties?

Is there a memory size at which this becomes a performance problem—for example what if I run a node app on a server with 128GB of RAM?

If you manage to fill up 128 GB, you get a cookie. The 2 GB that was
reported earlier in this thread is close to the maximum that V8 will
allocate.

If that's still too much, you can limit the heap size with
--max-old-space-size=<x> where <x> is in megabytes.

I like cookies. But let's change the example. Suppose I have a server with 2GB free RAM and I launch a node app. Over time it will occupy all 2GB of free RAM as a consequence of the garbage collection system, even though it's not really using the memory. Now I need to launch another program that needs 1GB of RAM. What will happen?

Nothing. No problem. You even can have 10 programs with 1 terabyte mapped to each and all will be good... until the programs start to *really* use that memory, then, the OS *may* need to start swapping pages to/from disk like crazy, or not, it depends: if the combined *real* usage is > phisical RAM it will swap, if it isn't, it won't.

In case anyone is interested, my memory issue came down to:
https://github.com/LearnBoost/socket.io/issues/438 (i ended up setting a
websocket only instance (no fallbacks except htmlfile) and xhr only
instance, xhr RSS still grows but slower, and websocket will go down when
clients disconnect. all this was happening outside of v8 heap space
(seemed like a Buffer leak when using mtrace).

How did you actually fix the issue, or did you not?

V8 also tries to release a completely emoty memory pages back to the OS.

Pages that have some little garbage on them or have a lot of small
holes are subject to compaction.

So in theory amount of unused memory should not exceed roughly 20% in
stable condition... Heap breathes: becomes larger when you actively
allocate and smaller when you are idle. Memory management is full of
heuristics which require careful tweaking... We have tweaked them a
lot since new GC has been released to the wild to make sure that
memory fragmentation stays on acceptable levels.

Nothing new, even with node 0.8.8.

However it does not feel like a "leak" (where RSS grows until the process crashes), it just fills "most" of the system memory and then stays at this level (even when all the connections are closed). I suspect V8 memory management to be the culprit.

Nico


Am 12.09.2012 um 02:58 schrieb wavded <[email protected]>: