FAQ
Hello list,

any feedback appreciated about a potential AWS setup for a Storm cluster to integrate with our existent app. There are N-instances of our current app that can accept messages as HTTP POST requests. I'd like those messages to be sent to the Storm cluster on E2C for analysis.

I suppose I can use a AWS SQS to initially collect messages. Spouts can then dequeue from there to initiate analysis. SQS is already available there but I know there are potentially faster alternatives like Kestrel, or Kafka.

While I know how I can send a message to SQS, I don't understand what I should do to send a message to Kestrel queues I deployed myself as part of the EC2 topology. I see there is a kafka-deploy. If I use kafka-deploy for example, in which way I send messages to the queue up there and how spouts know where to fetch messages from? Do I need to extract the IP addresses after deploy? Can storm-deploy be instructed to also deploy Kestrel server nodes? Sorry for the confused questions.

Thanks
Renzo

Search Discussions

  • Serg at May 8, 2013 at 8:06 pm
    I would be very interesting to hear the answer to this question as well.

    Thank you
    On Thursday, December 20, 2012 10:43:52 AM UTC-5, Renzo Borgatti wrote:

    Hello list,

    any feedback appreciated about a potential AWS setup for a Storm cluster
    to integrate with our existent app. There are N-instances of our current
    app that can accept messages as HTTP POST requests. I'd like those messages
    to be sent to the Storm cluster on E2C for analysis.

    I suppose I can use a AWS SQS to initially collect messages. Spouts can
    then dequeue from there to initiate analysis. SQS is already available
    there but I know there are potentially faster alternatives like Kestrel, or
    Kafka.

    While I know how I can send a message to SQS, I don't understand what I
    should do to send a message to Kestrel queues I deployed myself as part of
    the EC2 topology. I see there is a kafka-deploy. If I use kafka-deploy for
    example, in which way I send messages to the queue up there and how spouts
    know where to fetch messages from? Do I need to extract the IP addresses
    after deploy? Can storm-deploy be instructed to also deploy Kestrel server
    nodes? Sorry for the confused questions.

    Thanks
    Renzo
    --
    You received this message because you are subscribed to the Google Groups "storm-user" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to storm-user+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.
  • Viral Bajaria at May 8, 2013 at 8:36 pm

    On Wed, May 8, 2013 at 1:06 PM, Serg wrote:

    I would be very interesting to hear the answer to this question as well.

    Thank you
    It's unclear what kind of answer you are looking for. Do you want to know
    whether kafka-deploy can be used to setup a kafka cluster ? Or do you want
    to know if you can use SQS as your queuing system to process data ?

    -Viral

    --
    You received this message because you are subscribed to the Google Groups "storm-user" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to storm-user+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.
  • Serg at May 8, 2013 at 9:12 pm
    Thank you for your reply. What I would like to hear is basically, how would
    you upload data TO the kafka or SQS? For example, how would you store
    tweets in kafka or sqs for the future processing by storm?
    On Wednesday, May 8, 2013 4:36:53 PM UTC-4, Viral Bajaria wrote:


    On Wed, May 8, 2013 at 1:06 PM, Serg <ser...@openbridge.com <javascript:>>wrote:
    I would be very interesting to hear the answer to this question as well.

    Thank you
    It's unclear what kind of answer you are looking for. Do you want to know
    whether kafka-deploy can be used to setup a kafka cluster ? Or do you want
    to know if you can use SQS as your queuing system to process data ?

    -Viral
    --
    You received this message because you are subscribed to the Google Groups "storm-user" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to storm-user+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.
  • Serg at May 8, 2013 at 9:14 pm
    I already found your answer in another question. Thank you
    On Wednesday, May 8, 2013 5:12:50 PM UTC-4, Serg wrote:

    Thank you for your reply. What I would like to hear is basically, how
    would you upload data TO the kafka or SQS? For example, how would you store
    tweets in kafka or sqs for the future processing by storm?
    On Wednesday, May 8, 2013 4:36:53 PM UTC-4, Viral Bajaria wrote:

    On Wed, May 8, 2013 at 1:06 PM, Serg wrote:

    I would be very interesting to hear the answer to this question as well.

    Thank you
    It's unclear what kind of answer you are looking for. Do you want to know
    whether kafka-deploy can be used to setup a kafka cluster ? Or do you want
    to know if you can use SQS as your queuing system to process data ?

    -Viral
    --
    You received this message because you are subscribed to the Google Groups "storm-user" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to storm-user+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupstorm-user @
postedDec 25, '12 at 6:49a
activeMay 8, '13 at 9:14p
posts5
users3
websitestorm-project.net
irc#storm-user

People

Translate

site design / logo © 2022 Grokbase