FAQ
Hi,

I have been reading about storm for a bit and I'm trying to search for its
performance. I'm curious if I have a stream of 20k qps I want to send to
storm, can 1 spout handle it ? basically what is the maximum qps a spout
can handle?

Thanks

Search Discussions

  • Michael Rose at Dec 5, 2012 at 10:33 pm
    That's all dependent on how tight your code is and how fast your source of
    data is, network conditions, etc.

    Generally for 20k qps I'd expect your spout would have a parallelism
    greater than 1 reading from multiple brokers. A single spout is
    horizontally scalable under storm (in most cases).

    Without more information about your expected use case, "queries", hardware,
    etc. nobody can tell you how Storm will perform for you.

    --
    Michael Rose (@Xorlev)
    Senior Platform Engineer, FullContact
    michael@fullcontact.com
    On Wednesday, December 5, 2012 4:53:14 AM UTC-7, tony wrote:

    Hi,

    I have been reading about storm for a bit and I'm trying to search for its
    performance. I'm curious if I have a stream of 20k qps I want to send to
    storm, can 1 spout handle it ? basically what is the maximum qps a spout
    can handle?

    Thanks
  • Tony at Dec 7, 2012 at 11:46 am
    Hi Michael,

    Thanks for the reply... well I'm assuming that the hardware is very simple
    let's say M1 Medium Instance 3.75 GiB of memory, 2 EC2 from amazon. So
    really my question is this then. Let's take the twitter word counter
    example and let's imagine the twitter feed gives 50 000 qps. From my
    understanding, you can only have 1 spout listening to that socket and can't
    be threaded to prevent duplicates otherwise if we have threads this would
    create multiple connections. What I would like to know, if a stream that is
    coming into Storm is very big what would be the best options to spread the
    incoming data across multiple spouts in the same style as a load balancer.
    Or you just need to have massive powerful system that can handle the stream
    with 1 spout?

    hope it makes sense

    Anthony

    On Wednesday, December 5, 2012 10:33:25 PM UTC, Michael Rose wrote:

    That's all dependent on how tight your code is and how fast your source of
    data is, network conditions, etc.

    Generally for 20k qps I'd expect your spout would have a parallelism
    greater than 1 reading from multiple brokers. A single spout is
    horizontally scalable under storm (in most cases).

    Without more information about your expected use case, "queries",
    hardware, etc. nobody can tell you how Storm will perform for you.

    --
    Michael Rose (@Xorlev)
    Senior Platform Engineer, FullContact
    mic...@fullcontact.com <javascript:>
    On Wednesday, December 5, 2012 4:53:14 AM UTC-7, tony wrote:

    Hi,

    I have been reading about storm for a bit and I'm trying to search for
    its performance. I'm curious if I have a stream of 20k qps I want to send
    to storm, can 1 spout handle it ? basically what is the maximum qps a spout
    can handle?

    Thanks

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupstorm-user @
postedDec 5, '12 at 11:53a
activeDec 7, '12 at 11:46a
posts3
users2
websitestorm-project.net
irc#storm-user

2 users in discussion

Tony: 2 posts Michael Rose: 1 post

People

Translate

site design / logo © 2022 Grokbase