FAQ
Hello,
   I've got a new 45 node cluster with CM 4.8 ... I'm getting alerts that
the CM host (which is running CM management services) is swapping, even
though vm.swappiness = 0.

   The box is also running a MySQL for Hive/Hue/Oozie/SCM.

   How would I go about figuring out what's causing the swapping? I have
not seen this happen on a CM box before.

Thanks
-Joe

To unsubscribe from this group and stop receiving emails from it, send an email to scm-users+unsubscribe@cloudera.org.

Search Discussions

  • Philip Zeyliger at Jan 23, 2014 at 3:35 pm
    Hi Joe,

    The first question is how much it's swapping. If it's just a few pages
    here and there, you can tune the alert to ignore it. I doubt a few pages
    times 4KB is going to cause a performance issue.

    If you're curious what process has swapped pages, you can look in
    /proc/pid/smaps, like this guy did:
    http://northernmost.org/blog/find-out-what-is-using-your-swap/

    Is your machine over-subscribed? That'll cause it too. I research that by
    using top/htop and adding things up.

    -- Philip

    On Thu, Jan 23, 2014 at 5:46 AM, Joe Travaglini wrote:

    Hello,
    I've got a new 45 node cluster with CM 4.8 ... I'm getting alerts that
    the CM host (which is running CM management services) is swapping, even
    though vm.swappiness = 0.

    The box is also running a MySQL for Hive/Hue/Oozie/SCM.

    How would I go about figuring out what's causing the swapping? I have
    not seen this happen on a CM box before.

    Thanks
    -Joe

    To unsubscribe from this group and stop receiving emails from it, send an
    email to scm-users+unsubscribe@cloudera.org.
    To unsubscribe from this group and stop receiving emails from it, send an email to scm-users+unsubscribe@cloudera.org.
  • Joe Travaglini at Jan 23, 2014 at 7:26 pm
    Hey Philip,
    Thanks for the info. We're getting alerts pretty frequently and it seems
    that there is no swapping and then it spikes to around 1.4K pages per
    second, according CM.

    It looks like the MySQLD on this box is taking up 91% of the system's RAM.
      That seems high to me. It is serving CM, CM services, Hue, Hive, and
    Oozie. I also notice that the cmf.main java program is taking up about 25%
    of the server's CPU on average.

    Perhaps we need to separate these services to run on different machines....

    Thanks
    -Joe

    On Thu, Jan 23, 2014 at 10:35 AM, Philip Zeyliger wrote:

    Hi Joe,

    The first question is how much it's swapping. If it's just a few pages
    here and there, you can tune the alert to ignore it. I doubt a few pages
    times 4KB is going to cause a performance issue.

    If you're curious what process has swapped pages, you can look in
    /proc/pid/smaps, like this guy did:
    http://northernmost.org/blog/find-out-what-is-using-your-swap/

    Is your machine over-subscribed? That'll cause it too. I research that
    by using top/htop and adding things up.

    -- Philip

    On Thu, Jan 23, 2014 at 5:46 AM, Joe Travaglini wrote:

    Hello,
    I've got a new 45 node cluster with CM 4.8 ... I'm getting alerts that
    the CM host (which is running CM management services) is swapping, even
    though vm.swappiness = 0.

    The box is also running a MySQL for Hive/Hue/Oozie/SCM.

    How would I go about figuring out what's causing the swapping? I have
    not seen this happen on a CM box before.

    Thanks
    -Joe

    To unsubscribe from this group and stop receiving emails from it, send an
    email to scm-users+unsubscribe@cloudera.org.
    To unsubscribe from this group and stop receiving emails from it, send an email to scm-users+unsubscribe@cloudera.org.
  • Joe Travaglini at Jan 23, 2014 at 7:46 pm
    I figured out what was wrong -- one of our DBAs had set the innodb buffer
    far too large for this machine. I think we're good now. Thanks for
    googling things for me :)


    -joe
    On Thu, Jan 23, 2014 at 2:26 PM, Joe Travaglini wrote:

    Hey Philip,
    Thanks for the info. We're getting alerts pretty frequently and it seems
    that there is no swapping and then it spikes to around 1.4K pages per
    second, according CM.

    It looks like the MySQLD on this box is taking up 91% of the system's RAM.
    That seems high to me. It is serving CM, CM services, Hue, Hive, and
    Oozie. I also notice that the cmf.main java program is taking up about 25%
    of the server's CPU on average.

    Perhaps we need to separate these services to run on different machines....

    Thanks
    -Joe

    On Thu, Jan 23, 2014 at 10:35 AM, Philip Zeyliger wrote:

    Hi Joe,

    The first question is how much it's swapping. If it's just a few pages
    here and there, you can tune the alert to ignore it. I doubt a few pages
    times 4KB is going to cause a performance issue.

    If you're curious what process has swapped pages, you can look in
    /proc/pid/smaps, like this guy did:
    http://northernmost.org/blog/find-out-what-is-using-your-swap/

    Is your machine over-subscribed? That'll cause it too. I research that
    by using top/htop and adding things up.

    -- Philip


    On Thu, Jan 23, 2014 at 5:46 AM, Joe Travaglini <
    joe.travaglini@gmail.com> wrote:
    Hello,
    I've got a new 45 node cluster with CM 4.8 ... I'm getting alerts that
    the CM host (which is running CM management services) is swapping, even
    though vm.swappiness = 0.

    The box is also running a MySQL for Hive/Hue/Oozie/SCM.

    How would I go about figuring out what's causing the swapping? I have
    not seen this happen on a CM box before.

    Thanks
    -Joe

    To unsubscribe from this group and stop receiving emails from it, send
    an email to scm-users+unsubscribe@cloudera.org.
    To unsubscribe from this group and stop receiving emails from it, send an email to scm-users+unsubscribe@cloudera.org.

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupscm-users @
categorieshadoop
postedJan 23, '14 at 1:46p
activeJan 23, '14 at 7:46p
posts4
users2
websitecloudera.com
irc#hadoop

2 users in discussion

Joe Travaglini: 3 posts Philip Zeyliger: 1 post

People

Translate

site design / logo © 2022 Grokbase