FAQ
Hello,

I've set the impala daemon memory limit through cloudera manager to 1 GiB
(for testing purposes). However, in top I see that impalad is taking 3.6 gb
of resident memory with a virtual image of 7.7 gb. What does the mem_limit
configuration parameter mean and what does it do?

Cheers,

Michael

Search Discussions

  • Udai Kiran Potluri at Apr 23, 2013 at 6:56 pm
    Hello Michael,

    How did you set the mem_limit? Did you set via the impala-shell or while
    starting the impalad daemon?
    mem_limit is the memory limit in bytes for Impala Daemon. If reached,
    queries running on the Impalad may be killed.

    Thanks,
    Udai

    On Tue, Apr 23, 2013 at 10:51 AM, Michael Allman wrote:

    Hello,

    I've set the impala daemon memory limit through cloudera manager to 1 GiB
    (for testing purposes). However, in top I see that impalad is taking 3.6 gb
    of resident memory with a virtual image of 7.7 gb. What does the mem_limit
    configuration parameter mean and what does it do?

    Cheers,

    Michael
  • Michael Allman at Apr 23, 2013 at 7:01 pm
    Hi Udai,

    I set the memory limit in the configuration in cloudera manager. What does it mean for the impala daemon to "reach" the memory limit? How is memory use being measured here? Like I said, it seems to be using way more memory than the limit I set.

    Cheers,

    Michael
    On Apr 23, 2013, at 11:56 AM, Udai Kiran Potluri wrote:

    Hello Michael,

    How did you set the mem_limit? Did you set via the impala-shell or while starting the impalad daemon?
    mem_limit is the memory limit in bytes for Impala Daemon. If reached, queries running on the Impalad may be killed.

    Thanks,
    Udai


    On Tue, Apr 23, 2013 at 10:51 AM, Michael Allman wrote:
    Hello,

    I've set the impala daemon memory limit through cloudera manager to 1 GiB (for testing purposes). However, in top I see that impalad is taking 3.6 gb of resident memory with a virtual image of 7.7 gb. What does the mem_limit configuration parameter mean and what does it do?

    Cheers,

    Michael
  • Marcel Kornacker at Apr 23, 2013 at 8:28 pm
    Michael,

    the memory limit at the moment only applies to the memory that's
    directly consumed by queries. It excludes what the impalad consumes at
    startup, such as for cached metadata. Could you do me a favor and do
    the following, then report back the resident set size?
    - restart your impalad
    - do a 'describe <tbl>' on all the tables in your schema; this will
    load the metadata

    Marcel
    On Tue, Apr 23, 2013 at 12:01 PM, Michael Allman wrote:
    Hi Udai,

    I set the memory limit in the configuration in cloudera manager. What does
    it mean for the impala daemon to "reach" the memory limit? How is memory use
    being measured here? Like I said, it seems to be using way more memory than
    the limit I set.

    Cheers,

    Michael

    On Apr 23, 2013, at 11:56 AM, Udai Kiran Potluri wrote:

    Hello Michael,

    How did you set the mem_limit? Did you set via the impala-shell or while
    starting the impalad daemon?
    mem_limit is the memory limit in bytes for Impala Daemon. If reached,
    queries running on the Impalad may be killed.

    Thanks,
    Udai

    On Tue, Apr 23, 2013 at 10:51 AM, Michael Allman wrote:

    Hello,

    I've set the impala daemon memory limit through cloudera manager to 1 GiB
    (for testing purposes). However, in top I see that impalad is taking 3.6 gb
    of resident memory with a virtual image of 7.7 gb. What does the mem_limit
    configuration parameter mean and what does it do?

    Cheers,

    Michael
  • Michael Allman at Apr 23, 2013 at 9:50 pm
    Hi Marcel,

    Thanks for clarifying. I can't restart impala now, but I'll try tonight and write back.

    Cheers,

    Michael
    On Apr 23, 2013, at 1:28 PM, Marcel Kornacker wrote:

    Michael,

    the memory limit at the moment only applies to the memory that's
    directly consumed by queries. It excludes what the impalad consumes at
    startup, such as for cached metadata. Could you do me a favor and do
    the following, then report back the resident set size?
    - restart your impalad
    - do a 'describe <tbl>' on all the tables in your schema; this will
    load the metadata

    Marcel
    On Tue, Apr 23, 2013 at 12:01 PM, Michael Allman wrote:
    Hi Udai,

    I set the memory limit in the configuration in cloudera manager. What does
    it mean for the impala daemon to "reach" the memory limit? How is memory use
    being measured here? Like I said, it seems to be using way more memory than
    the limit I set.

    Cheers,

    Michael

    On Apr 23, 2013, at 11:56 AM, Udai Kiran Potluri wrote:

    Hello Michael,

    How did you set the mem_limit? Did you set via the impala-shell or while
    starting the impalad daemon?
    mem_limit is the memory limit in bytes for Impala Daemon. If reached,
    queries running on the Impalad may be killed.

    Thanks,
    Udai

    On Tue, Apr 23, 2013 at 10:51 AM, Michael Allman wrote:

    Hello,

    I've set the impala daemon memory limit through cloudera manager to 1 GiB
    (for testing purposes). However, in top I see that impalad is taking 3.6 gb
    of resident memory with a virtual image of 7.7 gb. What does the mem_limit
    configuration parameter mean and what does it do?

    Cheers,

    Michael

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupimpala-user @
categorieshadoop
postedApr 23, '13 at 5:51p
activeApr 23, '13 at 9:50p
posts5
users3
websitecloudera.com
irc#hadoop

People

Translate

site design / logo © 2022 Grokbase