|| at Dec 7, 2012 at 11:46 am
Thanks for the reply... well I'm assuming that the hardware is very simple
let's say M1 Medium Instance 3.75 GiB of memory, 2 EC2 from amazon. So
really my question is this then. Let's take the twitter word counter
example and let's imagine the twitter feed gives 50 000 qps. From my
understanding, you can only have 1 spout listening to that socket and can't
be threaded to prevent duplicates otherwise if we have threads this would
create multiple connections. What I would like to know, if a stream that is
coming into Storm is very big what would be the best options to spread the
incoming data across multiple spouts in the same style as a load balancer.
Or you just need to have massive powerful system that can handle the stream
with 1 spout?
hope it makes sense
On Wednesday, December 5, 2012 10:33:25 PM UTC, Michael Rose wrote:
That's all dependent on how tight your code is and how fast your source of
data is, network conditions, etc.
Generally for 20k qps I'd expect your spout would have a parallelism
greater than 1 reading from multiple brokers. A single spout is
horizontally scalable under storm (in most cases).
Without more information about your expected use case, "queries",
hardware, etc. nobody can tell you how Storm will perform for you.
Michael Rose (@Xorlev)
Senior Platform Engineer, FullContact
On Wednesday, December 5, 2012 4:53:14 AM UTC-7, tony wrote:
I have been reading about storm for a bit and I'm trying to search for
its performance. I'm curious if I have a stream of 20k qps I want to send
to storm, can 1 spout handle it ? basically what is the maximum qps a spout