FAQ
We have sort of a weird case here:

We deploy to EC2, and have our entire app stack in multiple zones/regions.
We are hoping to stand up a dedicated RabbitMQ instance for each zone that
sits in front of Storm (we're using the AMQP spout from Rapportive
(https://github.com/rapportive-oss/storm-amqp-spout). In order to do this
without a bunch of hops through ELBs, we would ideally like one Nimbus
machine and Supervisors across multiple boxes in multiple zones, with each
Supervisor pointed at the RabbitMQ machine dedicated to that zone.

Unfortunately, it seems that the connection configuration for a Spout all
takes place in the Topology main class, which is executed on the Nimbus box
during deploy. Is a spout config per worker a possibility? I know that if
we use something like Kafka, then perhaps this problem doesn't exist as
such, but we're hoping to stick with RabbitMQ for now.

--
You received this message because you are subscribed to the Google Groups "storm-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to storm-user+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Search Discussions

  • Kirk Gray at Feb 21, 2013 at 2:45 am
    Is there any way to have a different Spout configuration per worker
    machine? We are deploying multi zone/region in EC2 and would like workers
    in certain zones to hit our RabbitMQ instances' queues in their respective
    zones, but I don't see a way to have this configuration happen outside the
    Main class that Nimbus executes on topology submission.

    We're trying to avoid the ELB hops by going directly to the Rabbit machines
    inside the zone, if possible.

    --
    You received this message because you are subscribed to the Google Groups "storm-user" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to storm-user+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.
  • Nathan Marz at Feb 21, 2013 at 6:56 am
    You could just have each supervisor machine list in its storm.yaml the
    preferred rabbitmq queues, and then that will be available in the conf map
    provided to each task on startup when the prepare method is called.
    On Wed, Feb 20, 2013 at 8:48 AM, Kirk Gray wrote:

    Is there any way to have a different Spout configuration per worker
    machine? We are deploying multi zone/region in EC2 and would like workers
    in certain zones to hit our RabbitMQ instances' queues in their respective
    zones, but I don't see a way to have this configuration happen outside the
    Main class that Nimbus executes on topology submission.

    We're trying to avoid the ELB hops by going directly to the Rabbit
    machines inside the zone, if possible.

    --
    You received this message because you are subscribed to the Google Groups
    "storm-user" group.
    To unsubscribe from this group and stop receiving emails from it, send an
    email to storm-user+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.



    --
    Twitter: @nathanmarz
    http://nathanmarz.com

    --
    You received this message because you are subscribed to the Google Groups "storm-user" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to storm-user+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.
  • Michael Rose at Feb 21, 2013 at 3:10 am
    Hi Kirk,

    I'm the current maintainer of the storm-amqp-spout (https://github.com/Xorlev/storm-amqp-spout).

    Unfortunately, everything is currently marked as private final which make subclassing the spout a somewhat moot point. I've also had desires to add more interesting behavior to the AMQP Spout connections as well.

    https://github.com/Xorlev/storm-amqp-spout/tree/protected-connection-factory

    This is a small change that moves the creation of the connection factory out into a protected method as well as exposing the AMQP connection information to subclasses.

    By subclassing it, you could pretty easily grab your instance metadata by doing a HTTP request to http://169.254.169.254/latest/meta-data/placement/availability-zone

    None of this is on Clojars or Central currently, so you'd have to build it yourself.

    --
    Michael Rose (@Xorlev (https://twitter.com/xorlev))
    Senior Platform Engineer, FullContact (http://fullcontact.com/)
    michael@fullcontact.com (mailto:michael@fullcontact.com)

    On Wednesday, February 20, 2013 at 9:30 AM, Kirk Gray wrote:

    We have sort of a weird case here:

    We deploy to EC2, and have our entire app stack in multiple zones/regions. We are hoping to stand up a dedicated RabbitMQ instance for each zone that sits in front of Storm (we're using the AMQP spout from Rapportive (https://github.com/rapportive-oss/storm-amqp-spout). In order to do this without a bunch of hops through ELBs, we would ideally like one Nimbus machine and Supervisors across multiple boxes in multiple zones, with each Supervisor pointed at the RabbitMQ machine dedicated to that zone.

    Unfortunately, it seems that the connection configuration for a Spout all takes place in the Topology main class, which is executed on the Nimbus box during deploy. Is a spout config per worker a possibility? I know that if we use something like Kafka, then perhaps this problem doesn't exist as such, but we're hoping to stick with RabbitMQ for now.

    --
    You received this message because you are subscribed to the Google Groups "storm-user" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to storm-user+unsubscribe@googlegroups.com (mailto:storm-user+unsubscribe@googlegroups.com).
    For more options, visit https://groups.google.com/groups/opt_out.
    --
    You received this message because you are subscribed to the Google Groups "storm-user" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to storm-user+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.
  • Michael Vogiatzis at Mar 16, 2013 at 9:24 pm
    Hello,

    I have observed the following behavior when using AMQPSpout (found here
    https://github.com/Xorlev/storm-amqp-spout) in my *trident* topology, local
    cluster.

    when nextDelivery is passed without a timeout, nextTuple() is executed only
    once and then blocks.
    when nextDelivery is passed with a timeout (i.e. nextDelivery(5000) ),
    nextTuple() unblocks every 5 seconds but messages are coming off of the
    queue at a cumulative basis. That is: first message at 00:05, second at
    00:15, third at 00:30, etc.

    As a very basic example, I just read messages from the AMQPSpout and print
    them out from a BaseFunction.

    Is RabbitMQ working with Trident as intended?

    Regards,
    Michael Vogiatzis
    On Thursday, 21 February 2013 03:09:56 UTC, Michael Rose wrote:

    Hi Kirk,

    I'm the current maintainer of the storm-amqp-spout (
    https://github.com/Xorlev/storm-amqp-spout).

    Unfortunately, everything is currently marked as private final which make
    subclassing the spout a somewhat moot point. I've also had desires to add
    more interesting behavior to the AMQP Spout connections as well.


    https://github.com/Xorlev/storm-amqp-spout/tree/protected-connection-factory

    This is a small change that moves the creation of the connection factory
    out into a protected method as well as exposing the AMQP connection
    information to subclasses.

    By subclassing it, you could pretty easily grab your instance metadata by
    doing a HTTP request to
    http://169.254.169.254/latest/meta-data/placement/availability-zone

    None of this is on Clojars or Central currently, so you'd have to build it
    yourself.

    --
    Michael Rose (@Xorlev <https://twitter.com/xorlev>)
    Senior Platform Engineer, FullContact <http://fullcontact.com/>
    mic...@fullcontact.com <javascript:>

    On Wednesday, February 20, 2013 at 9:30 AM, Kirk Gray wrote:

    We have sort of a weird case here:

    We deploy to EC2, and have our entire app stack in multiple zones/regions.
    We are hoping to stand up a dedicated RabbitMQ instance for each zone that
    sits in front of Storm (we're using the AMQP spout from Rapportive (
    https://github.com/rapportive-oss/storm-amqp-spout). In order to do this
    without a bunch of hops through ELBs, we would ideally like one Nimbus
    machine and Supervisors across multiple boxes in multiple zones, with each
    Supervisor pointed at the RabbitMQ machine dedicated to that zone.

    Unfortunately, it seems that the connection configuration for a Spout all
    takes place in the Topology main class, which is executed on the Nimbus box
    during deploy. Is a spout config per worker a possibility? I know that if
    we use something like Kafka, then perhaps this problem doesn't exist as
    such, but we're hoping to stick with RabbitMQ for now.

    --
    You received this message because you are subscribed to the Google Groups
    "storm-user" group.
    To unsubscribe from this group and stop receiving emails from it, send an
    email to storm-user+...@googlegroups.com <javascript:>.
    For more options, visit https://groups.google.com/groups/opt_out.



    --
    You received this message because you are subscribed to the Google Groups "storm-user" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to storm-user+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupstorm-user @
postedFeb 21, '13 at 2:45a
activeMar 16, '13 at 9:24p
posts5
users4
websitestorm-project.net
irc#storm-user

People

Translate

site design / logo © 2021 Grokbase