mem_limit and impalad

On Tue, Apr 23, 2013 at 10:51 AM, Michael Allman wrote:

Hello,

I've set the impala daemon memory limit through cloudera manager to 1 GiB
(for testing purposes). However, in top I see that impalad is taking 3.6 gb
of resident memory with a virtual image of 7.7 gb. What does the mem_limit
configuration parameter mean and what does it do?

Cheers,

Michael

On Apr 23, 2013, at 11:56 AM, Udai Kiran Potluri wrote:

Hello Michael,

How did you set the mem_limit? Did you set via the impala-shell or while starting the impalad daemon?
mem_limit is the memory limit in bytes for Impala Daemon. If reached, queries running on the Impalad may be killed.

Thanks,
Udai


On Tue, Apr 23, 2013 at 10:51 AM, Michael Allman wrote:
Hello,

I've set the impala daemon memory limit through cloudera manager to 1 GiB (for testing purposes). However, in top I see that impalad is taking 3.6 gb of resident memory with a virtual image of 7.7 gb. What does the mem_limit configuration parameter mean and what does it do?

Cheers,

Michael

On Tue, Apr 23, 2013 at 12:01 PM, Michael Allman wrote:
Hi Udai,

I set the memory limit in the configuration in cloudera manager. What does
it mean for the impala daemon to "reach" the memory limit? How is memory use
being measured here? Like I said, it seems to be using way more memory than
the limit I set.

Cheers,

Michael

On Apr 23, 2013, at 11:56 AM, Udai Kiran Potluri wrote:

Hello Michael,

How did you set the mem_limit? Did you set via the impala-shell or while
starting the impalad daemon?
mem_limit is the memory limit in bytes for Impala Daemon. If reached,
queries running on the Impalad may be killed.

Thanks,
Udai

On Apr 23, 2013, at 1:28 PM, Marcel Kornacker wrote:

Michael,

the memory limit at the moment only applies to the memory that's
directly consumed by queries. It excludes what the impalad consumes at
startup, such as for cached metadata. Could you do me a favor and do
the following, then report back the resident set size?
- restart your impalad
- do a 'describe <tbl>' on all the tables in your schema; this will
load the metadata

Marcel