Running a cluster of small EC2 instances with 1.7GB of memory each.
Using HDFS, MapReduce and Yarn. Cloudera 4 has set the heap size for
hdfs to 52MB, but apparently hdfs is using double that , looking at top..
The following error is appearing in the hdfs log
file hadoop-cmf-hdfs1-DATANODE-hadoop1.log.out , even without mapreduce
Unexpected exception in block pool Block pool
BP-1546357427-10.96.129.120-1344011400942 (storage id DS-1473151029-
10.76.178.155-50010-1344011413394) service to
java.lang.OutOfMemoryError: Java heap space
The datanode is crashing and dying.
Does OutOfMemoryError imply that all physical memory has been consumed on
the machine? Because if I look at these nodes with htop, I see around
500MB physical memory consumed right now, and the other 1.2GB of memory is
showing as being used by cached memory which can be flushed with "sync ;
echo 3 | sudo tee /proc/sys/vm/drop_caches" which means it's not really
being consumed, is it...
What I don't know yet, is if immediately prior to a crash, something
unusual happens, and all physical memory is genuinely consumed somehow.
Any suggestions now for investigation or action?