[storm-user] Did anyone try to install cpulimit along with Storm to limit workers CPU usage?

On Fri, Feb 22, 2013 at 11:20 AM, Patricio Echagüe wrote:

we have a case that one topology somethings uses a lot of CPU. It happens
to run in one worker and it reaching 300%+ CPU causing other topologies to
die in that machine.

I was wondering if someone has tried to limit the amount of CPU any storm
process that use.

The effect of this worker using so much CPU is that the other workers
start timing out and supervisor kills them.

http://www.cyberciti.biz/faq/cpu-usage-limiter-for-linux/

thoughts?


--
You received this message because you are subscribed to the Google Groups
"storm-user" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to storm-user+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

On Fri, Feb 22, 2013 at 11:52 AM, Nathan Marz wrote:

That is a dark rabbit hole that I don't recommend going down. Instead, use
the isolation scheduler released in 0.8.2:
http://storm-project.net/2013/01/11/storm082-released.html


--
Twitter: @nathanmarz
http://nathanmarz.com

--
You received this message because you are subscribed to the Google Groups
"storm-user" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to storm-user+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

On Fri, Feb 22, 2013 at 1:48 PM, Patricio Echagüe wrote:

Thanks Nathan. How about your storm-mesos project?
https://github.com/nathanmarz/storm-mesos

Mesos seems to control the access to CPU and memory which seems to
accomplish what I want.

--
You received this message because you are subscribed to the Google Groups
"storm-user" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to storm-user+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

On Fri, Feb 22, 2013 at 1:54 PM, Nathan Marz wrote:

We spent a long time trying to get resource isolation to work through
Mesos and like I said, it's a deep dark rabbit hole. Stick with the simple
solution here, which is the isolation scheduler.



--
Twitter: @nathanmarz
http://nathanmarz.com

--
You received this message because you are subscribed to the Google Groups
"storm-user" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to storm-user+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

On Fri, Feb 22, 2013 at 4:19 PM, Patricio Echagüe wrote:

Sure I will, thanks. But mind sharing what the problem was/is with the
mesos approach or what problem did you guys ran into?

--
You received this message because you are subscribed to the Google Groups
"storm-user" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to storm-user+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

On Mon, Feb 25, 2013 at 8:55 AM, Bobby Evans wrote:

Yes I am very interested in the problems you faced. We are trying to get
storm on YARN working well, and the latest versions of YARN want to enforce
CPU similar to mesos.

Thanks,

Bobby

--
You received this message because you are subscribed to the Google Groups
"storm-user" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to storm-user+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

On Mon, Feb 25, 2013 at 3:03 PM, Nathan Marz wrote:

Mesos provided an at-least resource model, which sounds good but causes
complex problems. We would have a topology deployed that's working fine,
and then suddenly the performance would drop and it would fall behind. This
is because it actually was using more than it was given, and when another
topology took away those resources the performance would suffer
drastically. It's basically impossible to capacity plan under these
conditions.

We then started playing around with putting an upper limit on CPU usage,
but there are all sorts of complexities there like caching and hardware
threading that could lead to inconsistent performance – which would again
make it difficult to capacity plan. So we are going to wait until the Mesos
folks can do large amounts of testing so that we know exactly what kind of
variance can be expected from a resource isolation scheme before looking at
that again. Ultimately though, the isolation scheduler solves the same
problem in a much, much simpler and easier to reason about way.



--
Twitter: @nathanmarz
http://nathanmarz.com

--
You received this message because you are subscribed to the Google Groups
"storm-user" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to storm-user+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.