FAQ
Hi,

I have a following topology deployed in local cluster:

topologyBuilder.setSpout("status-emitter", new AmqpStatusEmitterSpout());
topologyBuilder.setBolt("status-splitter", new StatusSplittingBolt())
     .shuffleGrouping("status-emitter");
topologyBuilder.setBolt("status-segmentator", new StatusSegmentatorBolt(),
5)
.fieldsGrouping("status-splitter", new Fields("word"));
  Config conf = new Config();
conf.setDebug(true);
  conf.put(Config.TOPOLOGY_MAX_SPOUT_PENDING, 1);
  cluster.submitTopology( "social-topology",
                 conf,
                 topologyBuilder.createTopology());

- AmqpStatusEmitterSpout consumes messages from RabbitMQ queue and emits
tuples from consumed message payload.
- StatusSplittingBolt seperates tokens from a large sentence
- StatusSegmentatorBolt count the occurrences of a token

When I increment the parallelism for some components, there is
no performance improvement. In some cases I get the same results for
StatusSegmentatorBolt with parallelism hint 1 and 5.

What I am doing wrong?

Thanks

--
You received this message because you are subscribed to the Google Groups "storm-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to storm-user+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Search Discussions

  • Philip O'Toole at Jun 6, 2013 at 1:29 pm
    What version of storm?
    On Jun 4, 2013, at 2:37 AM, Nedo Sab wrote:

    Hi,

    I have a following topology deployed in local cluster:

    topologyBuilder.setSpout("status-emitter", new AmqpStatusEmitterSpout());
    topologyBuilder.setBolt("status-splitter", new StatusSplittingBolt())
    .shuffleGrouping("status-emitter");
    topologyBuilder.setBolt("status-segmentator", new StatusSegmentatorBolt(), 5)
    .fieldsGrouping("status-splitter", new Fields("word"));

    Config conf = new Config();
    conf.setDebug(true);

    conf.put(Config.TOPOLOGY_MAX_SPOUT_PENDING, 1);

    cluster.submitTopology( "social-topology",
    conf,
    topologyBuilder.createTopology());

    - AmqpStatusEmitterSpout consumes messages from RabbitMQ queue and emits tuples from consumed message payload.
    - StatusSplittingBolt seperates tokens from a large sentence
    - StatusSegmentatorBolt count the occurrences of a token

    When I increment the parallelism for some components, there is no performance improvement. In some cases I get the same results for
    StatusSegmentatorBolt with parallelism hint 1 and 5.

    What I am doing wrong?

    Thanks
    --
    You received this message because you are subscribed to the Google Groups "storm-user" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to storm-user+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.
    --
    You received this message because you are subscribed to the Google Groups "storm-user" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to storm-user+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.
  • P. Taylor Goetz at Jun 6, 2013 at 2:54 pm

    This is most likely your problem:

    conf.put(Config.TOPOLOGY_MAX_SPOUT_PENDING, 1);
    That's telling storm to only allow one outstanding tuple at a time, negating any parallelism (I'm assuming you are anchoring tuples).

    Try setting it to something higher like 1000.

    - Taylor

    --
    P. Taylor Goetz
    Lead Developer, Software Development

    Health Market Science
    The Science of Better Results
    2700 Horizon Drive • King of Prussia, PA • 19406
    P: 610.994.5237 • healthmarketscience.com
    On Jun 4, 2013, at 5:37 AM, Nedo Sab wrote:

    Hi,

    I have a following topology deployed in local cluster:

    topologyBuilder.setSpout("status-emitter", new AmqpStatusEmitterSpout());
    topologyBuilder.setBolt("status-splitter", new StatusSplittingBolt())
    .shuffleGrouping("status-emitter");
    topologyBuilder.setBolt("status-segmentator", new StatusSegmentatorBolt(), 5)
    .fieldsGrouping("status-splitter", new Fields("word"));

    Config conf = new Config();
    conf.setDebug(true);

    conf.put(Config.TOPOLOGY_MAX_SPOUT_PENDING, 1);

    cluster.submitTopology( "social-topology",
    conf,
    topologyBuilder.createTopology());

    - AmqpStatusEmitterSpout consumes messages from RabbitMQ queue and emits tuples from consumed message payload.
    - StatusSplittingBolt seperates tokens from a large sentence
    - StatusSegmentatorBolt count the occurrences of a token

    When I increment the parallelism for some components, there is no performance improvement. In some cases I get the same results for
    StatusSegmentatorBolt with parallelism hint 1 and 5.

    What I am doing wrong?

    Thanks

    --
    You received this message because you are subscribed to the Google Groups "storm-user" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to storm-user+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.
    --
    You received this message because you are subscribed to the Google Groups "storm-user" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to storm-user+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.
  • Nedo Sab at Jun 7, 2013 at 4:55 pm
    Setting TOPLOGY_MAX_SPOUT_PENDING to 1000, didn't make any difference.

    I'm using Storm 0.6.0, but I also tested it with the latest version.

    El jueves, 6 de junio de 2013 16:54:02 UTC+2, P. Taylor Goetz escribió:
    This is most likely your problem:

    conf.put(Config.TOPOLOGY_MAX_SPOUT_PENDING, *1*);


    That's telling storm to only allow one outstanding tuple at a time,
    negating any parallelism (I'm assuming you are anchoring tuples).

    Try setting it to something higher like 1000.

    - Taylor

    --
    P. Taylor Goetz
    Lead Developer, Software Development

    Health Market Science
    The Science of Better Results
    2700 Horizon Drive • King of Prussia, PA • 19406
    P: 610.994.5237 • healthmarketscience.com

    On Jun 4, 2013, at 5:37 AM, Nedo Sab <bhd...@gmail.com <javascript:>>
    wrote:

    Hi,

    I have a following topology deployed in local cluster:

    topologyBuilder.setSpout("status-emitter", new AmqpStatusEmitterSpout());
    topologyBuilder.setBolt("status-splitter", new StatusSplittingBolt())
    .shuffleGrouping("status-emitter");
    topologyBuilder.setBolt("status-segmentator", new StatusSegmentatorBolt(),
    5)
    .fieldsGrouping("status-splitter", new Fields("word"));
    Config conf = new Config();
    conf.setDebug(true);
    conf.put(Config.TOPOLOGY_MAX_SPOUT_PENDING, 1);
    cluster.submitTopology( "social-topology",
    conf,
    topologyBuilder.createTopology());

    - AmqpStatusEmitterSpout consumes messages from RabbitMQ queue and emits
    tuples from consumed message payload.
    - StatusSplittingBolt seperates tokens from a large sentence
    - StatusSegmentatorBolt count the occurrences of a token

    When I increment the parallelism for some components, there is
    no performance improvement. In some cases I get the same results for
    StatusSegmentatorBolt with parallelism hint 1 and 5.

    What I am doing wrong?

    Thanks

    --
    You received this message because you are subscribed to the Google Groups
    "storm-user" group.
    To unsubscribe from this group and stop receiving emails from it, send an
    email to storm-user+...@googlegroups.com <javascript:>.
    For more options, visit https://groups.google.com/groups/opt_out.



    --
    You received this message because you are subscribed to the Google Groups "storm-user" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to storm-user+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.
  • Sean Allen at Jun 8, 2013 at 2:16 pm
    You status segmentator being a field grouping could mean that all tuples
    end up going to the same bolt. That would really depend on the data.

    Or the splitter isn't doing its job correctly and everything is "split"
    into one thing so any parallelism after that is moot.

    What does the message payload look like?
    How is it being split?

    On Fri, Jun 7, 2013 at 12:55 PM, Nedo Sab wrote:

    Setting TOPLOGY_MAX_SPOUT_PENDING to 1000, didn't make any difference.

    I'm using Storm 0.6.0, but I also tested it with the latest version.

    El jueves, 6 de junio de 2013 16:54:02 UTC+2, P. Taylor Goetz escribió:
    This is most likely your problem:

    conf.put(Config.TOPOLOGY_MAX_**SPOUT_PENDING, *1*);


    That's telling storm to only allow one outstanding tuple at a time,
    negating any parallelism (I'm assuming you are anchoring tuples).

    Try setting it to something higher like 1000.

    - Taylor

    --
    P. Taylor Goetz
    Lead Developer, Software Development

    Health Market Science
    The Science of Better Results
    2700 Horizon Drive • King of Prussia, PA • 19406
    P: 610.994.5237 • healthmarketscience.com

    On Jun 4, 2013, at 5:37 AM, Nedo Sab wrote:

    Hi,

    I have a following topology deployed in local cluster:

    topologyBuilder.setSpout("**status-emitter", new
    AmqpStatusEmitterSpout());
    topologyBuilder.setBolt("**status-splitter", new StatusSplittingBolt())
    .shuffleGrouping("status-**emitter");
    topologyBuilder.setBolt("**status-segmentator", new
    StatusSegmentatorBolt(), 5)
    .fieldsGrouping("status-**splitter", new Fields("word"));
    Config conf = new Config();
    conf.setDebug(true);
    conf.put(Config.TOPOLOGY_MAX_**SPOUT_PENDING, 1);
    cluster.submitTopology( "social-topology",
    conf,
    topologyBuilder.**createTopology());

    - AmqpStatusEmitterSpout consumes messages from RabbitMQ queue and emits
    tuples from consumed message payload.
    - StatusSplittingBolt seperates tokens from a large sentence
    - StatusSegmentatorBolt count the occurrences of a token

    When I increment the parallelism for some components, there is
    no performance improvement. In some cases I get the same results for
    StatusSegmentatorBolt with parallelism hint 1 and 5.

    What I am doing wrong?

    Thanks

    --
    You received this message because you are subscribed to the Google Groups
    "storm-user" group.
    To unsubscribe from this group and stop receiving emails from it, send an
    email to storm-user+...@**googlegroups.com.

    For more options, visit https://groups.google.com/**groups/opt_out<https://groups.google.com/groups/opt_out>
    .




    --
    You received this message because you are subscribed to the Google Groups
    "storm-user" group.
    To unsubscribe from this group and stop receiving emails from it, send an
    email to storm-user+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.



    --

    Ce n'est pas une signature

    --
    You received this message because you are subscribed to the Google Groups "storm-user" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to storm-user+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.
  • Nedo Sab at Jun 9, 2013 at 7:33 pm
    Here's the full source code.

    https://github.com/bhnedo/storm-social-analyzer

    Thanks

    El sábado, 8 de junio de 2013 16:16:28 UTC+2, Sean T Allen escribió:
    You status segmentator being a field grouping could mean that all tuples
    end up going to the same bolt. That would really depend on the data.

    Or the splitter isn't doing its job correctly and everything is "split"
    into one thing so any parallelism after that is moot.

    What does the message payload look like?
    How is it being split?


    On Fri, Jun 7, 2013 at 12:55 PM, Nedo Sab <bhd...@gmail.com <javascript:>>wrote:
    Setting TOPLOGY_MAX_SPOUT_PENDING to 1000, didn't make any difference.

    I'm using Storm 0.6.0, but I also tested it with the latest version.

    El jueves, 6 de junio de 2013 16:54:02 UTC+2, P. Taylor Goetz escribió:
    This is most likely your problem:

    conf.put(Config.TOPOLOGY_MAX_**SPOUT_PENDING, *1*);


    That's telling storm to only allow one outstanding tuple at a time,
    negating any parallelism (I'm assuming you are anchoring tuples).

    Try setting it to something higher like 1000.

    - Taylor

    --
    P. Taylor Goetz
    Lead Developer, Software Development

    Health Market Science
    The Science of Better Results
    2700 Horizon Drive • King of Prussia, PA • 19406
    P: 610.994.5237 • healthmarketscience.com

    On Jun 4, 2013, at 5:37 AM, Nedo Sab wrote:

    Hi,

    I have a following topology deployed in local cluster:

    topologyBuilder.setSpout("**status-emitter", new
    AmqpStatusEmitterSpout());
    topologyBuilder.setBolt("**status-splitter", new StatusSplittingBolt())
    .shuffleGrouping("status-**emitter");
    topologyBuilder.setBolt("**status-segmentator", new
    StatusSegmentatorBolt(), 5)
    .fieldsGrouping("status-**splitter", new Fields("word"));
    Config conf = new Config();
    conf.setDebug(true);
    conf.put(Config.TOPOLOGY_MAX_**SPOUT_PENDING, 1);
    cluster.submitTopology( "social-topology",
    conf,
    topologyBuilder.**createTopology());

    - AmqpStatusEmitterSpout consumes messages from RabbitMQ queue and emits
    tuples from consumed message payload.
    - StatusSplittingBolt seperates tokens from a large sentence
    - StatusSegmentatorBolt count the occurrences of a token

    When I increment the parallelism for some components, there is
    no performance improvement. In some cases I get the same results for
    StatusSegmentatorBolt with parallelism hint 1 and 5.

    What I am doing wrong?

    Thanks

    --
    You received this message because you are subscribed to the Google
    Groups "storm-user" group.
    To unsubscribe from this group and stop receiving emails from it, send
    an email to storm-user+...@**googlegroups.com.

    For more options, visit https://groups.google.com/**groups/opt_out<https://groups.google.com/groups/opt_out>
    .




    --
    You received this message because you are subscribed to the Google Groups
    "storm-user" group.
    To unsubscribe from this group and stop receiving emails from it, send an
    email to storm-user+...@googlegroups.com <javascript:>.
    For more options, visit https://groups.google.com/groups/opt_out.



    --

    Ce n'est pas une signature
    --
    You received this message because you are subscribed to the Google Groups "storm-user" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to storm-user+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.
  • Nedo Sab at Jun 17, 2013 at 2:57 pm
    Anyone to shed some light on this?
    Thanks

    El domingo, 9 de junio de 2013 21:33:36 UTC+2, Nedo Sab escribió:
    Here's the full source code.

    https://github.com/bhnedo/storm-social-analyzer

    Thanks

    El sábado, 8 de junio de 2013 16:16:28 UTC+2, Sean T Allen escribió:
    You status segmentator being a field grouping could mean that all tuples
    end up going to the same bolt. That would really depend on the data.

    Or the splitter isn't doing its job correctly and everything is "split"
    into one thing so any parallelism after that is moot.

    What does the message payload look like?
    How is it being split?

    On Fri, Jun 7, 2013 at 12:55 PM, Nedo Sab wrote:

    Setting TOPLOGY_MAX_SPOUT_PENDING to 1000, didn't make any difference.

    I'm using Storm 0.6.0, but I also tested it with the latest version.

    El jueves, 6 de junio de 2013 16:54:02 UTC+2, P. Taylor Goetz escribió:
    This is most likely your problem:

    conf.put(Config.TOPOLOGY_MAX_**SPOUT_PENDING, *1*);


    That's telling storm to only allow one outstanding tuple at a time,
    negating any parallelism (I'm assuming you are anchoring tuples).

    Try setting it to something higher like 1000.

    - Taylor

    --
    P. Taylor Goetz
    Lead Developer, Software Development

    Health Market Science
    The Science of Better Results
    2700 Horizon Drive • King of Prussia, PA • 19406
    P: 610.994.5237 • healthmarketscience.com

    On Jun 4, 2013, at 5:37 AM, Nedo Sab wrote:

    Hi,

    I have a following topology deployed in local cluster:

    topologyBuilder.setSpout("**status-emitter", new
    AmqpStatusEmitterSpout());
    topologyBuilder.setBolt("**status-splitter", new StatusSplittingBolt())
    .shuffleGrouping("status-**emitter");
    topologyBuilder.setBolt("**status-segmentator", new
    StatusSegmentatorBolt(), 5)
    .fieldsGrouping("status-**splitter", new Fields("word"));
    Config conf = new Config();
    conf.setDebug(true);
    conf.put(Config.TOPOLOGY_MAX_**SPOUT_PENDING, 1);
    cluster.submitTopology( "social-topology",
    conf,
    topologyBuilder.**createTopology());

    - AmqpStatusEmitterSpout consumes messages from RabbitMQ queue and
    emits tuples from consumed message payload.
    - StatusSplittingBolt seperates tokens from a large sentence
    - StatusSegmentatorBolt count the occurrences of a token

    When I increment the parallelism for some components, there is
    no performance improvement. In some cases I get the same results for
    StatusSegmentatorBolt with parallelism hint 1 and 5.

    What I am doing wrong?

    Thanks

    --
    You received this message because you are subscribed to the Google
    Groups "storm-user" group.
    To unsubscribe from this group and stop receiving emails from it, send
    an email to storm-user+...@**googlegroups.com.

    For more options, visit https://groups.google.com/**groups/opt_out<https://groups.google.com/groups/opt_out>
    .




    --
    You received this message because you are subscribed to the Google
    Groups "storm-user" group.
    To unsubscribe from this group and stop receiving emails from it, send
    an email to storm-user+...@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.



    --

    Ce n'est pas une signature
    --
    You received this message because you are subscribed to the Google Groups "storm-user" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to storm-user+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupstorm-user @
postedJun 6, '13 at 7:20a
activeJun 17, '13 at 2:57p
posts7
users4
websitestorm-project.net
irc#storm-user

People

Translate

site design / logo © 2021 Grokbase