FAQ
we have a case that one topology somethings uses a lot of CPU. It happens
to run in one worker and it reaching 300%+ CPU causing other topologies to
die in that machine.

I was wondering if someone has tried to limit the amount of CPU any storm
process that use.

The effect of this worker using so much CPU is that the other workers start
timing out and supervisor kills them.

http://www.cyberciti.biz/faq/cpu-usage-limiter-for-linux/

thoughts?

--
You received this message because you are subscribed to the Google Groups "storm-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to storm-user+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Search Discussions

  • Nathan Marz at Feb 22, 2013 at 7:52 pm
    That is a dark rabbit hole that I don't recommend going down. Instead, use
    the isolation scheduler released in 0.8.2:
    http://storm-project.net/2013/01/11/storm082-released.html
    On Fri, Feb 22, 2013 at 11:20 AM, Patricio Echagüe wrote:

    we have a case that one topology somethings uses a lot of CPU. It happens
    to run in one worker and it reaching 300%+ CPU causing other topologies to
    die in that machine.

    I was wondering if someone has tried to limit the amount of CPU any storm
    process that use.

    The effect of this worker using so much CPU is that the other workers
    start timing out and supervisor kills them.

    http://www.cyberciti.biz/faq/cpu-usage-limiter-for-linux/

    thoughts?


    --
    You received this message because you are subscribed to the Google Groups
    "storm-user" group.
    To unsubscribe from this group and stop receiving emails from it, send an
    email to storm-user+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.



    --
    Twitter: @nathanmarz
    http://nathanmarz.com

    --
    You received this message because you are subscribed to the Google Groups "storm-user" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to storm-user+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.
  • Patricio Echagüe at Feb 22, 2013 at 9:48 pm
    Thanks Nathan. How about your storm-mesos project?
    https://github.com/nathanmarz/storm-mesos

    Mesos seems to control the access to CPU and memory which seems to
    accomplish what I want.

    On Fri, Feb 22, 2013 at 11:52 AM, Nathan Marz wrote:

    That is a dark rabbit hole that I don't recommend going down. Instead, use
    the isolation scheduler released in 0.8.2:
    http://storm-project.net/2013/01/11/storm082-released.html
    On Fri, Feb 22, 2013 at 11:20 AM, Patricio Echagüe wrote:

    we have a case that one topology somethings uses a lot of CPU. It
    happens to run in one worker and it reaching 300%+ CPU causing other
    topologies to die in that machine.

    I was wondering if someone has tried to limit the amount of CPU any storm
    process that use.

    The effect of this worker using so much CPU is that the other workers
    start timing out and supervisor kills them.

    http://www.cyberciti.biz/faq/cpu-usage-limiter-for-linux/

    thoughts?


    --
    You received this message because you are subscribed to the Google Groups
    "storm-user" group.
    To unsubscribe from this group and stop receiving emails from it, send an
    email to storm-user+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.



    --
    Twitter: @nathanmarz
    http://nathanmarz.com

    --
    You received this message because you are subscribed to the Google Groups
    "storm-user" group.
    To unsubscribe from this group and stop receiving emails from it, send an
    email to storm-user+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.

    --
    You received this message because you are subscribed to the Google Groups "storm-user" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to storm-user+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.
  • Nathan Marz at Feb 22, 2013 at 9:54 pm
    We spent a long time trying to get resource isolation to work through Mesos
    and like I said, it's a deep dark rabbit hole. Stick with the simple
    solution here, which is the isolation scheduler.
    On Fri, Feb 22, 2013 at 1:48 PM, Patricio Echagüe wrote:

    Thanks Nathan. How about your storm-mesos project?
    https://github.com/nathanmarz/storm-mesos

    Mesos seems to control the access to CPU and memory which seems to
    accomplish what I want.

    On Fri, Feb 22, 2013 at 11:52 AM, Nathan Marz wrote:

    That is a dark rabbit hole that I don't recommend going down. Instead,
    use the isolation scheduler released in 0.8.2:
    http://storm-project.net/2013/01/11/storm082-released.html
    On Fri, Feb 22, 2013 at 11:20 AM, Patricio Echagüe wrote:

    we have a case that one topology somethings uses a lot of CPU. It
    happens to run in one worker and it reaching 300%+ CPU causing other
    topologies to die in that machine.

    I was wondering if someone has tried to limit the amount of CPU any
    storm process that use.

    The effect of this worker using so much CPU is that the other workers
    start timing out and supervisor kills them.

    http://www.cyberciti.biz/faq/cpu-usage-limiter-for-linux/

    thoughts?


    --
    You received this message because you are subscribed to the Google
    Groups "storm-user" group.
    To unsubscribe from this group and stop receiving emails from it, send
    an email to storm-user+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.



    --
    Twitter: @nathanmarz
    http://nathanmarz.com

    --
    You received this message because you are subscribed to the Google Groups
    "storm-user" group.
    To unsubscribe from this group and stop receiving emails from it, send an
    email to storm-user+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.

    --
    You received this message because you are subscribed to the Google Groups
    "storm-user" group.
    To unsubscribe from this group and stop receiving emails from it, send an
    email to storm-user+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.



    --
    Twitter: @nathanmarz
    http://nathanmarz.com

    --
    You received this message because you are subscribed to the Google Groups "storm-user" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to storm-user+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.
  • Patricio Echagüe at Feb 22, 2013 at 10:20 pm
    Sure I will, thanks. But mind sharing what the problem was/is with the
    mesos approach or what problem did you guys ran into?

    On Fri, Feb 22, 2013 at 1:54 PM, Nathan Marz wrote:

    We spent a long time trying to get resource isolation to work through
    Mesos and like I said, it's a deep dark rabbit hole. Stick with the simple
    solution here, which is the isolation scheduler.

    On Fri, Feb 22, 2013 at 1:48 PM, Patricio Echagüe wrote:

    Thanks Nathan. How about your storm-mesos project?
    https://github.com/nathanmarz/storm-mesos

    Mesos seems to control the access to CPU and memory which seems to
    accomplish what I want.

    On Fri, Feb 22, 2013 at 11:52 AM, Nathan Marz wrote:

    That is a dark rabbit hole that I don't recommend going down. Instead,
    use the isolation scheduler released in 0.8.2:
    http://storm-project.net/2013/01/11/storm082-released.html
    On Fri, Feb 22, 2013 at 11:20 AM, Patricio Echagüe wrote:

    we have a case that one topology somethings uses a lot of CPU. It
    happens to run in one worker and it reaching 300%+ CPU causing other
    topologies to die in that machine.

    I was wondering if someone has tried to limit the amount of CPU any
    storm process that use.

    The effect of this worker using so much CPU is that the other workers
    start timing out and supervisor kills them.

    http://www.cyberciti.biz/faq/cpu-usage-limiter-for-linux/

    thoughts?


    --
    You received this message because you are subscribed to the Google
    Groups "storm-user" group.
    To unsubscribe from this group and stop receiving emails from it, send
    an email to storm-user+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.



    --
    Twitter: @nathanmarz
    http://nathanmarz.com

    --
    You received this message because you are subscribed to the Google
    Groups "storm-user" group.
    To unsubscribe from this group and stop receiving emails from it, send
    an email to storm-user+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.

    --
    You received this message because you are subscribed to the Google Groups
    "storm-user" group.
    To unsubscribe from this group and stop receiving emails from it, send an
    email to storm-user+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.



    --
    Twitter: @nathanmarz
    http://nathanmarz.com

    --
    You received this message because you are subscribed to the Google Groups
    "storm-user" group.
    To unsubscribe from this group and stop receiving emails from it, send an
    email to storm-user+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.

    --
    You received this message because you are subscribed to the Google Groups "storm-user" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to storm-user+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.
  • Bobby Evans at Feb 25, 2013 at 5:03 pm
    Yes I am very interested in the problems you faced. We are trying to get
    storm on YARN working well, and the latest versions of YARN want to enforce
    CPU similar to mesos.

    Thanks,

    Bobby

    On Fri, Feb 22, 2013 at 4:19 PM, Patricio Echagüe wrote:

    Sure I will, thanks. But mind sharing what the problem was/is with the
    mesos approach or what problem did you guys ran into?

    On Fri, Feb 22, 2013 at 1:54 PM, Nathan Marz wrote:

    We spent a long time trying to get resource isolation to work through
    Mesos and like I said, it's a deep dark rabbit hole. Stick with the simple
    solution here, which is the isolation scheduler.

    On Fri, Feb 22, 2013 at 1:48 PM, Patricio Echagüe wrote:

    Thanks Nathan. How about your storm-mesos project?
    https://github.com/nathanmarz/storm-mesos

    Mesos seems to control the access to CPU and memory which seems to
    accomplish what I want.

    On Fri, Feb 22, 2013 at 11:52 AM, Nathan Marz wrote:

    That is a dark rabbit hole that I don't recommend going down. Instead,
    use the isolation scheduler released in 0.8.2:
    http://storm-project.net/2013/01/11/storm082-released.html

    On Fri, Feb 22, 2013 at 11:20 AM, Patricio Echagüe <patricioe@gmail.com
    wrote:
    we have a case that one topology somethings uses a lot of CPU. It
    happens to run in one worker and it reaching 300%+ CPU causing other
    topologies to die in that machine.

    I was wondering if someone has tried to limit the amount of CPU any
    storm process that use.

    The effect of this worker using so much CPU is that the other workers
    start timing out and supervisor kills them.

    http://www.cyberciti.biz/faq/cpu-usage-limiter-for-linux/

    thoughts?


    --
    You received this message because you are subscribed to the Google
    Groups "storm-user" group.
    To unsubscribe from this group and stop receiving emails from it, send
    an email to storm-user+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.



    --
    Twitter: @nathanmarz
    http://nathanmarz.com

    --
    You received this message because you are subscribed to the Google
    Groups "storm-user" group.
    To unsubscribe from this group and stop receiving emails from it, send
    an email to storm-user+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.

    --
    You received this message because you are subscribed to the Google
    Groups "storm-user" group.
    To unsubscribe from this group and stop receiving emails from it, send
    an email to storm-user+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.



    --
    Twitter: @nathanmarz
    http://nathanmarz.com

    --
    You received this message because you are subscribed to the Google Groups
    "storm-user" group.
    To unsubscribe from this group and stop receiving emails from it, send an
    email to storm-user+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.

    --
    You received this message because you are subscribed to the Google Groups
    "storm-user" group.
    To unsubscribe from this group and stop receiving emails from it, send an
    email to storm-user+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.

    --
    You received this message because you are subscribed to the Google Groups "storm-user" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to storm-user+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.
  • Nathan Marz at Feb 25, 2013 at 9:03 pm
    Mesos provided an at-least resource model, which sounds good but causes
    complex problems. We would have a topology deployed that's working fine,
    and then suddenly the performance would drop and it would fall behind. This
    is because it actually was using more than it was given, and when another
    topology took away those resources the performance would suffer
    drastically. It's basically impossible to capacity plan under these
    conditions.

    We then started playing around with putting an upper limit on CPU usage,
    but there are all sorts of complexities there like caching and hardware
    threading that could lead to inconsistent performance – which would again
    make it difficult to capacity plan. So we are going to wait until the Mesos
    folks can do large amounts of testing so that we know exactly what kind of
    variance can be expected from a resource isolation scheme before looking at
    that again. Ultimately though, the isolation scheduler solves the same
    problem in a much, much simpler and easier to reason about way.
    On Mon, Feb 25, 2013 at 8:55 AM, Bobby Evans wrote:

    Yes I am very interested in the problems you faced. We are trying to get
    storm on YARN working well, and the latest versions of YARN want to enforce
    CPU similar to mesos.

    Thanks,

    Bobby

    On Fri, Feb 22, 2013 at 4:19 PM, Patricio Echagüe wrote:

    Sure I will, thanks. But mind sharing what the problem was/is with the
    mesos approach or what problem did you guys ran into?

    On Fri, Feb 22, 2013 at 1:54 PM, Nathan Marz wrote:

    We spent a long time trying to get resource isolation to work through
    Mesos and like I said, it's a deep dark rabbit hole. Stick with the simple
    solution here, which is the isolation scheduler.

    On Fri, Feb 22, 2013 at 1:48 PM, Patricio Echagüe wrote:

    Thanks Nathan. How about your storm-mesos project?
    https://github.com/nathanmarz/storm-mesos

    Mesos seems to control the access to CPU and memory which seems to
    accomplish what I want.

    On Fri, Feb 22, 2013 at 11:52 AM, Nathan Marz wrote:

    That is a dark rabbit hole that I don't recommend going down. Instead,
    use the isolation scheduler released in 0.8.2:
    http://storm-project.net/2013/01/11/storm082-released.html

    On Fri, Feb 22, 2013 at 11:20 AM, Patricio Echagüe <
    patricioe@gmail.com> wrote:
    we have a case that one topology somethings uses a lot of CPU. It
    happens to run in one worker and it reaching 300%+ CPU causing other
    topologies to die in that machine.

    I was wondering if someone has tried to limit the amount of CPU any
    storm process that use.

    The effect of this worker using so much CPU is that the other workers
    start timing out and supervisor kills them.

    http://www.cyberciti.biz/faq/cpu-usage-limiter-for-linux/

    thoughts?


    --
    You received this message because you are subscribed to the Google
    Groups "storm-user" group.
    To unsubscribe from this group and stop receiving emails from it,
    send an email to storm-user+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.



    --
    Twitter: @nathanmarz
    http://nathanmarz.com

    --
    You received this message because you are subscribed to the Google
    Groups "storm-user" group.
    To unsubscribe from this group and stop receiving emails from it, send
    an email to storm-user+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.

    --
    You received this message because you are subscribed to the Google
    Groups "storm-user" group.
    To unsubscribe from this group and stop receiving emails from it, send
    an email to storm-user+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.



    --
    Twitter: @nathanmarz
    http://nathanmarz.com

    --
    You received this message because you are subscribed to the Google
    Groups "storm-user" group.
    To unsubscribe from this group and stop receiving emails from it, send
    an email to storm-user+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.

    --
    You received this message because you are subscribed to the Google Groups
    "storm-user" group.
    To unsubscribe from this group and stop receiving emails from it, send an
    email to storm-user+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.

    --
    You received this message because you are subscribed to the Google Groups
    "storm-user" group.
    To unsubscribe from this group and stop receiving emails from it, send an
    email to storm-user+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.



    --
    Twitter: @nathanmarz
    http://nathanmarz.com

    --
    You received this message because you are subscribed to the Google Groups "storm-user" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to storm-user+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.
  • Bobby Evans at Feb 26, 2013 at 3:23 pm
    Thanks for the response. It makes a lot of sense and you most likely just
    saved me a lot of time and headaches. I'll try to see what I can do to get
    the isolation scheduler to work properly for YARN (Grab the entire node).

    Bobby

    On Mon, Feb 25, 2013 at 3:03 PM, Nathan Marz wrote:

    Mesos provided an at-least resource model, which sounds good but causes
    complex problems. We would have a topology deployed that's working fine,
    and then suddenly the performance would drop and it would fall behind. This
    is because it actually was using more than it was given, and when another
    topology took away those resources the performance would suffer
    drastically. It's basically impossible to capacity plan under these
    conditions.

    We then started playing around with putting an upper limit on CPU usage,
    but there are all sorts of complexities there like caching and hardware
    threading that could lead to inconsistent performance – which would again
    make it difficult to capacity plan. So we are going to wait until the Mesos
    folks can do large amounts of testing so that we know exactly what kind of
    variance can be expected from a resource isolation scheme before looking at
    that again. Ultimately though, the isolation scheduler solves the same
    problem in a much, much simpler and easier to reason about way.

    On Mon, Feb 25, 2013 at 8:55 AM, Bobby Evans wrote:

    Yes I am very interested in the problems you faced. We are trying to get
    storm on YARN working well, and the latest versions of YARN want to enforce
    CPU similar to mesos.

    Thanks,

    Bobby

    On Fri, Feb 22, 2013 at 4:19 PM, Patricio Echagüe wrote:

    Sure I will, thanks. But mind sharing what the problem was/is with the
    mesos approach or what problem did you guys ran into?

    On Fri, Feb 22, 2013 at 1:54 PM, Nathan Marz wrote:

    We spent a long time trying to get resource isolation to work through
    Mesos and like I said, it's a deep dark rabbit hole. Stick with the simple
    solution here, which is the isolation scheduler.

    On Fri, Feb 22, 2013 at 1:48 PM, Patricio Echagüe wrote:

    Thanks Nathan. How about your storm-mesos project?
    https://github.com/nathanmarz/storm-mesos

    Mesos seems to control the access to CPU and memory which seems to
    accomplish what I want.

    On Fri, Feb 22, 2013 at 11:52 AM, Nathan Marz wrote:

    That is a dark rabbit hole that I don't recommend going down.
    Instead, use the isolation scheduler released in 0.8.2:
    http://storm-project.net/2013/01/11/storm082-released.html

    On Fri, Feb 22, 2013 at 11:20 AM, Patricio Echagüe <
    patricioe@gmail.com> wrote:
    we have a case that one topology somethings uses a lot of CPU. It
    happens to run in one worker and it reaching 300%+ CPU causing other
    topologies to die in that machine.

    I was wondering if someone has tried to limit the amount of CPU any
    storm process that use.

    The effect of this worker using so much CPU is that the other
    workers start timing out and supervisor kills them.

    http://www.cyberciti.biz/faq/cpu-usage-limiter-for-linux/

    thoughts?


    --
    You received this message because you are subscribed to the Google
    Groups "storm-user" group.
    To unsubscribe from this group and stop receiving emails from it,
    send an email to storm-user+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.



    --
    Twitter: @nathanmarz
    http://nathanmarz.com

    --
    You received this message because you are subscribed to the Google
    Groups "storm-user" group.
    To unsubscribe from this group and stop receiving emails from it,
    send an email to storm-user+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.

    --
    You received this message because you are subscribed to the Google
    Groups "storm-user" group.
    To unsubscribe from this group and stop receiving emails from it, send
    an email to storm-user+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.



    --
    Twitter: @nathanmarz
    http://nathanmarz.com

    --
    You received this message because you are subscribed to the Google
    Groups "storm-user" group.
    To unsubscribe from this group and stop receiving emails from it, send
    an email to storm-user+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.

    --
    You received this message because you are subscribed to the Google
    Groups "storm-user" group.
    To unsubscribe from this group and stop receiving emails from it, send
    an email to storm-user+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.

    --
    You received this message because you are subscribed to the Google Groups
    "storm-user" group.
    To unsubscribe from this group and stop receiving emails from it, send an
    email to storm-user+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.



    --
    Twitter: @nathanmarz
    http://nathanmarz.com

    --
    You received this message because you are subscribed to the Google Groups
    "storm-user" group.
    To unsubscribe from this group and stop receiving emails from it, send an
    email to storm-user+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.

    --
    You received this message because you are subscribed to the Google Groups "storm-user" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to storm-user+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupstorm-user @
postedFeb 22, '13 at 7:27p
activeFeb 26, '13 at 3:23p
posts8
users3
websitestorm-project.net
irc#storm-user

People

Translate

site design / logo © 2022 Grokbase