My jvm settings:
-Xmx8192M -Xms8192M -XX:+CMSScavengeBeforeRemark -XX:NewRatio=2
-XX:+CMSParallelRemarkEnabled -XX:+UseParNewGC -XX:+UseConcMarkSweepGC
I turned off IncrementalPacing, and enabled CMSInitiatingOccupancyFraction,
after issues with nodes being reported as down due to large Garbage
collection pauses. The problem with the memory profile was visible before
the drop down to 1.2GB (this was when I reloaded the core), my concern was
that the collection of the old generation didn't seem to free any of the
heap, and we went from occasionally collecting to always collecting the old
Please see the attached gc log.
From: Shawn Heisey
Sent: 28 November 2012 16:48
Subject: Re: Permanently Full Old Generation...
On 11/28/2012 9:06 AM, Annette Newton wrote:
We are seeing strange gc behaviour after running solr cloud under
quite heavy insert load for a period of time. The old generation
becomes full and no amount of garbage collection will free up the
memory. I have attached a memory profile, as you can see it gets
progressively worse as the day goes on to the point where we are
always doing full garbage collections all the time. The only way I
have found to resolve this issue is to reload the core, then
subsequent garbage collections reclaim the used space, that's what
happened at 3pm on the Memory profile. All the nodes eventually
display the same behaviour.
We have multiple threads running adding batches of upto 100 documents
at a time. I have also attached our Schema and Config.
We are running 4 shards each with a single replica, have a 3 node
zookeeper setup and the 8 solr boxes instances are aws High-Memory
Double Extra Large with 34.2 GB Memory, 4 Virtual cores.
Looking at that jconsole graph, I am not seeing a problem. It looks fairly
normal to me, especially for heavy indexing.
Solr has been up for at least 29 hours, just based on the graph I can see,
which may not reflect the full JVM uptime - click on VM Summary to see that.
In that time, you've only spent a total of two minutes in garbage
collection, which does not seem problematic to me. Also, your Old Gen is not
full - it's only using 1.2GB out of 5.5GB available. A full GC
(ConcurrentMarkSweep) would only be automatically triggered when the Old Gen
reaches that 5.5GB mark.
Are you seeing actual performance problems, or are you just concerned about
what you see when you watch the memory?
General note about memory and Solr: I have had very good luck with the
following java memory options. Each machine handles three 22GB index shards
on one JVM.