Grokbase Groups Pig user May 2010
FAQ
sorry to resurrect a old thread. But I am having the same problem here for
Pig 0.7

011-10-19 13:03:22,895 INFO org.apache.pig.impl.util.SpillableMemoryManager:
low memory handler called (Usage threshold exceeded) init =
263323648(257152K) used = 502624976(490844K) committed = 617086976(602624K)
max = 715849728(699072K) 2011-10-19 13:03:58,776 INFO
org.apache.pig.impl.util.SpillableMemoryManager: low memory handler called
(Collection threshold exceeded) init = 263323648(257152K) used =
494572568(482981K) committed = 715849728(699072K) max = 715849728(699072K)
2011-10-19 13:04:01,248 INFO
org.apache.pig.impl.util.SpillableMemoryManager: low memory handler called
(Usage threshold exceeded) init = 263323648(257152K) used =
501339160(489589K) committed = 715849728(699072K) max = 715849728(699072K)
2011-10-19 13:05:01,250 INFO
org.apache.pig.impl.util.SpillableMemoryManager: low memory handler called
(Collection threshold exceeded) init = 263323648(257152K) used =
589147376(575339K) committed = 715849728(699072K) max = 715849728(699072K)
2011-10-19 13:05:42,181 INFO
org.apache.pig.impl.util.SpillableMemoryManager: low memory handler called
(Collection threshold exceeded) init = 263323648(257152K) used =
487209296(475790K) committed = 715849728(699072K) max = 715849728(699072K)
2011-10-19 13:06:07,300 INFO
org.apache.pig.impl.util.SpillableMemoryManager: low memory handler called
(Usage threshold exceeded) init = 263323648(257152K) used =
501963568(490198K) committed = 715849728(699072K) max = 715849728(699072K)
2011-10-19 13:07:11,850 INFO
org.apache.pig.impl.util.SpillableMemoryManager: low memory handler called
(Collection threshold exceeded) init = 263323648(257152K) used =
575026232(561549K) committed = 715849728(699072K) max = 715849728(699072K)
2011-10-19 13:07:53,782 INFO
org.apache.pig.impl.util.SpillableMemoryManager: low memory handler called
(Collection threshold exceeded) init = 263323648(257152K) used =
505687176(493835K) committed = 715849728(699072K) max = 715849728(699072K)
2011-10-19 13:09:02,451 INFO
org.apache.pig.impl.util.SpillableMemoryManager: low memory handler called
(Collection threshold exceeded) init = 263323648(257152K) used =
571977464(558571K) committed = 715849728(699072K) max = 715849728(699072K)
2011-10-19 13:09:44,027 INFO
org.apache.pig.impl.util.SpillableMemoryManager: low memory handler called
(Collection threshold exceeded) init = 263323648(257152K) used =
509238248(497302K) committed = 715849728(699072K) max = 715849728(699072K)
2011-10-19 13:10:55,596 INFO
org.apache.pig.impl.util.SpillableMemoryManager: low memory handler called
(Collection threshold exceeded) init = 263323648(257152K) used =
561933048(548762K) committed = 715849728(699072K) max = 715849728(699072K)
2011-10-19 13:11:40,931 INFO
org.apache.pig.impl.util.SpillableMemoryManager: low memory handler called
(Collection threshold exceeded) init = 263323648(257152K) used =
517425112(505297K) committed = 715849728(699072K) max = 715849728(699072K)
2011-10-19 13:12:48,891 INFO
org.apache.pig.impl.util.SpillableMemoryManager: low memory handler called
(Collection threshold exceeded) init = 263323648(257152K) used =
558586544(545494K) committed = 715849728(699072K) max = 715849728(699072K)

The job took a while to finish and not sure what causes it, it only happens
in the reduce phase and i am wondering how to increase the max memory to be
used by pig. In my mapreduce.reduce.java.opts, I have set the value to 1GB
but it seems the max is only 700MB for whatever reason.

Thanks,

Felix

Search Discussions

Discussion Posts

Previous

Related Discussions

Discussion Navigation
viewthread | post
posts ‹ prev | 9 of 9 | next ›
Discussion Overview
groupuser @
categoriespig, hadoop
postedMay 6, '10 at 9:18p
activeOct 19, '11 at 7:00p
posts9
users5
websitepig.apache.org

People

Translate

site design / logo © 2021 Grokbase