FAQ
Hi Shrikar,
I also meet this problem, could you please tell me how you solve this
problem? it was so confused and I have no any clue.
Any help greatly appreciate.

Thanks
Saisai

在 2012年10月14日星期日UTC+8上午3时32分06秒,Shrikar archak写道:
Hi All,
I need some help with setting the number of workers in a cluster. I get a
lot of warning messages when I change the number of workers.

Setup:
1) Nimbus on a seperate host.
2) 2 supervisors on 2 different nodes. ( I am not providing any ports for
supervisor in storm.yaml , so I assume its 4 by default.)

When I dont provide the worker setting. There is only one worker and I
dont see these warnings.

conf.setMaxTaskParallelism(80);

conf.setNumAckers(4);

for the above setting I dont see any warnings.
*Problematic settings.*

conf.setMaxTaskParallelism(80);

conf.setNumAckers(4);

*conf.setNumWorkers(4);*

Lots of Warning messages :
2012-10-12 18:25:16 worker [WARN] Received invalid messages for unknown
tasks. Dropping...
2012-10-12 18:25:16 worker [WARN] Received invalid messages for unknown
tasks. Dropping...
2012-10-12 18:25:16 worker [WARN] Received invalid messages for unknown
tasks. Dropping...
2012-10-12 18:25:17 worker [WARN] Received invalid messages for unknown
tasks. Dropping...
2012-10-12 18:25:17 worker [WARN] Received invalid messages for unknown
tasks. Dropping...

PS : the storm-local-dir is not shared among supervisors since they are on
different nodes.

Am I missing something? Any help greatly appreciated.

Thanks,

Shrikar

Search Discussions

  • Shrikar archak at Jan 5, 2013 at 7:03 am
    Hi Saisai,
    It was a networking problem. Make sure your cluster machine have unique
    host names
    and also have a mapping in you /etc/hosts file as below.

    Naming convention which I used

    TYPE : hostname
    Nimbus : nimbus
    Supervisor : supervisor-1
    Supervisor : supervisor-2
    Supervisor : supervisor-3
    Supervisor : supervisor-4

    example /etc/hosts file
    192.168.8.122 nimbus
    192.168.8.123 supervisor-1
    192.168.8.124 supervisor-2
    192.168.8.125 supervisor-3
    192.168.8.126 supervisor-4

    Hope that helps.
    Thanks,
    Shrikar
    On Tuesday, December 25, 2012 5:07:14 PM UTC-8, 邵赛赛 wrote:

    Hi Shrikar,
    I also meet this problem, could you please tell me how you solve this
    problem? it was so confused and I have no any clue.
    Any help greatly appreciate.

    Thanks
    Saisai

    在 2012年10月14日星期日UTC+8上午3时32分06秒,Shrikar archak写道:
    Hi All,
    I need some help with setting the number of workers in a cluster. I get
    a lot of warning messages when I change the number of workers.

    Setup:
    1) Nimbus on a seperate host.
    2) 2 supervisors on 2 different nodes. ( I am not providing any ports for
    supervisor in storm.yaml , so I assume its 4 by default.)

    When I dont provide the worker setting. There is only one worker and I
    dont see these warnings.

    conf.setMaxTaskParallelism(80);

    conf.setNumAckers(4);

    for the above setting I dont see any warnings.
    *Problematic settings.*

    conf.setMaxTaskParallelism(80);

    conf.setNumAckers(4);

    *conf.setNumWorkers(4);*

    Lots of Warning messages :
    2012-10-12 18:25:16 worker [WARN] Received invalid messages for unknown
    tasks. Dropping...
    2012-10-12 18:25:16 worker [WARN] Received invalid messages for unknown
    tasks. Dropping...
    2012-10-12 18:25:16 worker [WARN] Received invalid messages for unknown
    tasks. Dropping...
    2012-10-12 18:25:17 worker [WARN] Received invalid messages for unknown
    tasks. Dropping...
    2012-10-12 18:25:17 worker [WARN] Received invalid messages for unknown
    tasks. Dropping...

    PS : the storm-local-dir is not shared among supervisors since they are
    on different nodes.

    Am I missing something? Any help greatly appreciated.

    Thanks,

    Shrikar
  • Rich Schumacher at May 1, 2013 at 10:48 pm
    I have a similar setup and am experiencing the same problem.

    I have three AWS EC2 instances (provisioned by Chef/OpsWorks), one Nimbus
    and two Supervisor nodes. When I execute the ExclamationTopology from the
    tutorial on a single Supervisor it works as expected. However, if this
    topology executes on both Supervisors I see a lot of the dropped messages
    error, mixed in with stdout from the PrintingBolt I'm using.

    [2013-05-01 16:47:03,651] worker [WARN] Received invalid messages for
    unknown tasks. Dropping...
    [2013-05-01 16:47:03,652] worker [WARN] Received invalid messages for
    unknown tasks. Dropping...
    [2013-05-01 16:47:03,655] worker [WARN] Received invalid messages for
    unknown tasks. Dropping...
    [2013-05-01 16:47:03,658] STDIO [INFO] source: exclaim2:6, stream: default,
    id: {}, [jackson!!!!!!]
    [2013-05-01 16:47:03,756] STDIO [INFO] source: exclaim2:9, stream: default,
    id: {}, [mike!!!!!!]
    [2013-05-01 16:47:03,758] worker [WARN] Received invalid messages for
    unknown tasks. Dropping...

    I've read the Troubleshooting wiki as well as the above recommendations for
    setting /etc/hosts properly but to no avail. AWS OpsWorks automatically
    adds records to /etc/hosts, so each machine has something very similar to
    the following:

    # This file was generated by OpsWorks
    # any manual changes will be removed on the next update.

    127.0.0.1 localhost localhost.localdomain

    # The following lines are desirable for IPv6 capable hosts
    ::1 ip6-localhost ip6-loopback
    fe00::0 ip6-localnet
    ff00::0 ip6-mcastprefix
    ff02::1 ip6-allnodes
    ff02::2 ip6-allrouters
    ff02::3 ip6-allhosts

    # OpsWorks Layer State
    127.0.0.1 storm-nimbus.localdomain storm-nimbus
    10.141.143.158 storm-nimbus
    <public IP> storm-nimbus-ext
    10.137.30.236 storm-supervisor-1
    <public IP> storm-supervisor-1-ext
    10.142.132.134 storm-supervisor-2
    <public IP> storm-supervisor-2-ext
    10.209.138.3 zookeeper-1
    <public IP> zookeeper-1-ext
    10.254.226.114 zookeeper-2
    <public IP> zookeeper-2-ext
    10.209.135.84 zookeeper-3
    <public IP> zookeeper-3-ext

    As far as I can tell, everything in here is sane and works correctly.

    The one thing that sticks out in my mind is that the Storm UI indicates
    both Supervisor nodes report themselves as "localhost"
    (http://cl.ly/image/1I1D2D0W3T46). Likewise, I notice that whenever any of
    the Storm components (Nimbus and Supervisor) start up the log output from
    the ZooKeeper client lists their hostname as localhost, as can be seen here:

    [2013-05-01 17:36:30,742] ZooKeeper [INFO] Client
    environment:host.name=localhost

    Does anyone have any ideas about what is happening here?

    Thanks for listening!
    On Saturday, January 5, 2013 2:03:41 AM UTC-5, Shrikar archak wrote:

    Hi Saisai,
    It was a networking problem. Make sure your cluster machine have unique
    host names
    and also have a mapping in you /etc/hosts file as below.

    Naming convention which I used

    TYPE : hostname
    Nimbus : nimbus
    Supervisor : supervisor-1
    Supervisor : supervisor-2
    Supervisor : supervisor-3
    Supervisor : supervisor-4

    example /etc/hosts file
    192.168.8.122 nimbus
    192.168.8.123 supervisor-1
    192.168.8.124 supervisor-2
    192.168.8.125 supervisor-3
    192.168.8.126 supervisor-4

    Hope that helps.
    Thanks,
    Shrikar
    On Tuesday, December 25, 2012 5:07:14 PM UTC-8, 邵赛赛 wrote:

    Hi Shrikar,
    I also meet this problem, could you please tell me how you solve this
    problem? it was so confused and I have no any clue.
    Any help greatly appreciate.

    Thanks
    Saisai

    在 2012年10月14日星期日UTC+8上午3时32分06秒,Shrikar archak写道:
    Hi All,
    I need some help with setting the number of workers in a cluster. I get
    a lot of warning messages when I change the number of workers.

    Setup:
    1) Nimbus on a seperate host.
    2) 2 supervisors on 2 different nodes. ( I am not providing any ports
    for supervisor in storm.yaml , so I assume its 4 by default.)

    When I dont provide the worker setting. There is only one worker and I
    dont see these warnings.

    conf.setMaxTaskParallelism(80);

    conf.setNumAckers(4);

    for the above setting I dont see any warnings.
    *Problematic settings.*

    conf.setMaxTaskParallelism(80);

    conf.setNumAckers(4);

    *conf.setNumWorkers(4);*

    Lots of Warning messages :
    2012-10-12 18:25:16 worker [WARN] Received invalid messages for unknown
    tasks. Dropping...
    2012-10-12 18:25:16 worker [WARN] Received invalid messages for unknown
    tasks. Dropping...
    2012-10-12 18:25:16 worker [WARN] Received invalid messages for unknown
    tasks. Dropping...
    2012-10-12 18:25:17 worker [WARN] Received invalid messages for unknown
    tasks. Dropping...
    2012-10-12 18:25:17 worker [WARN] Received invalid messages for unknown
    tasks. Dropping...

    PS : the storm-local-dir is not shared among supervisors since they are
    on different nodes.

    Am I missing something? Any help greatly appreciated.

    Thanks,

    Shrikar
    --
    You received this message because you are subscribed to the Google Groups "storm-user" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to storm-user+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.
  • Shrikar archak at May 1, 2013 at 11:07 pm
    I remember fixing this problem by doing this.

    1) Make sure the hostname of all the supervisor and nimbus are different.
    In your case I see both the supervisors are localhost
    2) The above /etc/host should have similar entries on the supervisors too.

    Hope that helps.

    Thanks,
    Shrikar

    On Wed, May 1, 2013 at 11:55 AM, Rich Schumacher wrote:

    I have a similar setup and am experiencing the same problem.

    I have three AWS EC2 instances (provisioned by Chef/OpsWorks), one Nimbus
    and two Supervisor nodes. When I execute the ExclamationTopology from the
    tutorial on a single Supervisor it works as expected. However, if this
    topology executes on both Supervisors I see a lot of the dropped messages
    error, mixed in with stdout from the PrintingBolt I'm using.

    [2013-05-01 16:47:03,651] worker [WARN] Received invalid messages for
    unknown tasks. Dropping...
    [2013-05-01 16:47:03,652] worker [WARN] Received invalid messages for
    unknown tasks. Dropping...
    [2013-05-01 16:47:03,655] worker [WARN] Received invalid messages for
    unknown tasks. Dropping...
    [2013-05-01 16:47:03,658] STDIO [INFO] source: exclaim2:6, stream:
    default, id: {}, [jackson!!!!!!]
    [2013-05-01 16:47:03,756] STDIO [INFO] source: exclaim2:9, stream:
    default, id: {}, [mike!!!!!!]
    [2013-05-01 16:47:03,758] worker [WARN] Received invalid messages for
    unknown tasks. Dropping...

    I've read the Troubleshooting wiki as well as the above recommendations
    for setting /etc/hosts properly but to no avail. AWS OpsWorks automatically
    adds records to /etc/hosts, so each machine has something very similar to
    the following:

    # This file was generated by OpsWorks
    # any manual changes will be removed on the next update.

    127.0.0.1 localhost localhost.localdomain

    # The following lines are desirable for IPv6 capable hosts
    ::1 ip6-localhost ip6-loopback
    fe00::0 ip6-localnet
    ff00::0 ip6-mcastprefix
    ff02::1 ip6-allnodes
    ff02::2 ip6-allrouters
    ff02::3 ip6-allhosts

    # OpsWorks Layer State
    127.0.0.1 storm-nimbus.localdomain storm-nimbus
    10.141.143.158 storm-nimbus
    <public IP> storm-nimbus-ext
    10.137.30.236 storm-supervisor-1
    <public IP> storm-supervisor-1-ext
    10.142.132.134 storm-supervisor-2
    <public IP> storm-supervisor-2-ext
    10.209.138.3 zookeeper-1
    <public IP> zookeeper-1-ext
    10.254.226.114 zookeeper-2
    <public IP> zookeeper-2-ext
    10.209.135.84 zookeeper-3
    <public IP> zookeeper-3-ext

    As far as I can tell, everything in here is sane and works correctly.

    The one thing that sticks out in my mind is that the Storm UI indicates
    both Supervisor nodes report themselves as "localhost" (
    http://cl.ly/image/1I1D2D0W3T46). Likewise, I notice that whenever any of
    the Storm components (Nimbus and Supervisor) start up the log output from
    the ZooKeeper client lists their hostname as localhost, as can be seen here:

    [2013-05-01 17:36:30,742] ZooKeeper [INFO] Client environment:host.name
    =localhost

    Does anyone have any ideas about what is happening here?

    Thanks for listening!
    On Saturday, January 5, 2013 2:03:41 AM UTC-5, Shrikar archak wrote:

    Hi Saisai,
    It was a networking problem. Make sure your cluster machine have unique
    host names
    and also have a mapping in you /etc/hosts file as below.

    Naming convention which I used

    TYPE : hostname
    Nimbus : nimbus
    Supervisor : supervisor-1
    Supervisor : supervisor-2
    Supervisor : supervisor-3
    Supervisor : supervisor-4

    example /etc/hosts file
    192.168.8.122 nimbus
    192.168.8.123 supervisor-1
    192.168.8.124 supervisor-2
    192.168.8.125 supervisor-3
    192.168.8.126 supervisor-4

    Hope that helps.
    Thanks,
    Shrikar
    On Tuesday, December 25, 2012 5:07:14 PM UTC-8, 邵赛赛 wrote:

    Hi Shrikar,
    I also meet this problem, could you please tell me how you solve this
    problem? it was so confused and I have no any clue.
    Any help greatly appreciate.

    Thanks
    Saisai

    在 2012年10月14日星期日UTC+8上午3时32分06秒,**Shrikar archak写道:
    Hi All,
    I need some help with setting the number of workers in a cluster. I
    get a lot of warning messages when I change the number of workers.

    Setup:
    1) Nimbus on a seperate host.
    2) 2 supervisors on 2 different nodes. ( I am not providing any ports
    for supervisor in storm.yaml , so I assume its 4 by default.)

    When I dont provide the worker setting. There is only one worker and I
    dont see these warnings.

    conf.setMaxTaskParallelism(80)**;

    conf.setNumAckers(4);

    for the above setting I dont see any warnings.
    *Problematic settings.*

    conf.setMaxTaskParallelism(80)**;

    conf.setNumAckers(4);

    *conf.setNumWorkers(4);*

    Lots of Warning messages :
    2012-10-12 18:25:16 worker [WARN] Received invalid messages for unknown
    tasks. Dropping...
    2012-10-12 18:25:16 worker [WARN] Received invalid messages for unknown
    tasks. Dropping...
    2012-10-12 18:25:16 worker [WARN] Received invalid messages for unknown
    tasks. Dropping...
    2012-10-12 18:25:17 worker [WARN] Received invalid messages for unknown
    tasks. Dropping...
    2012-10-12 18:25:17 worker [WARN] Received invalid messages for unknown
    tasks. Dropping...

    PS : the storm-local-dir is not shared among supervisors since they are
    on different nodes.

    Am I missing something? Any help greatly appreciated.

    Thanks,

    Shrikar
    --
    You received this message because you are subscribed to a topic in the
    Google Groups "storm-user" group.
    To unsubscribe from this topic, visit
    https://groups.google.com/d/topic/storm-user/DgIDxawBczg/unsubscribe?hl=en
    .
    To unsubscribe from this group and all its topics, send an email to
    storm-user+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.

    --
    You received this message because you are subscribed to the Google Groups "storm-user" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to storm-user+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.
  • Rich Schumacher at May 2, 2013 at 1:29 am
    I managed to track this down not long after I posted and it was definitely
    the localhost issue. I stumbled across the STORM_LOCAL_HOSTNAME property in
    backtype.storm.Config which states that the default value is the result
    of InetAddress.getLocalHost().getCanonicalHostName(). Sure enough, on
    instances launched via OpsWorks in AWS this value is always localhost. I'm
    not entirely sure why this is but instances launched via EC2 directly have
    the appropriate value.

    In any event, the fix was to explicitly set the storm.local.hostname in
    storm.yaml. After that everything worked fine.

    Thanks for the help and I hope this will help someone else out in the
    future.
    On Wednesday, May 1, 2013 7:07:10 PM UTC-4, Shrikar archak wrote:

    I remember fixing this problem by doing this.

    1) Make sure the hostname of all the supervisor and nimbus are different.
    In your case I see both the supervisors are localhost
    2) The above /etc/host should have similar entries on the supervisors too.

    Hope that helps.

    Thanks,
    Shrikar


    On Wed, May 1, 2013 at 11:55 AM, Rich Schumacher <rich...@gmail.com<javascript:>
    wrote:
    I have a similar setup and am experiencing the same problem.

    I have three AWS EC2 instances (provisioned by Chef/OpsWorks), one Nimbus
    and two Supervisor nodes. When I execute the ExclamationTopology from the
    tutorial on a single Supervisor it works as expected. However, if this
    topology executes on both Supervisors I see a lot of the dropped messages
    error, mixed in with stdout from the PrintingBolt I'm using.

    [2013-05-01 16:47:03,651] worker [WARN] Received invalid messages for
    unknown tasks. Dropping...
    [2013-05-01 16:47:03,652] worker [WARN] Received invalid messages for
    unknown tasks. Dropping...
    [2013-05-01 16:47:03,655] worker [WARN] Received invalid messages for
    unknown tasks. Dropping...
    [2013-05-01 16:47:03,658] STDIO [INFO] source: exclaim2:6, stream:
    default, id: {}, [jackson!!!!!!]
    [2013-05-01 16:47:03,756] STDIO [INFO] source: exclaim2:9, stream:
    default, id: {}, [mike!!!!!!]
    [2013-05-01 16:47:03,758] worker [WARN] Received invalid messages for
    unknown tasks. Dropping...

    I've read the Troubleshooting wiki as well as the above recommendations
    for setting /etc/hosts properly but to no avail. AWS OpsWorks automatically
    adds records to /etc/hosts, so each machine has something very similar to
    the following:

    # This file was generated by OpsWorks
    # any manual changes will be removed on the next update.

    127.0.0.1 localhost localhost.localdomain

    # The following lines are desirable for IPv6 capable hosts
    ::1 ip6-localhost ip6-loopback
    fe00::0 ip6-localnet
    ff00::0 ip6-mcastprefix
    ff02::1 ip6-allnodes
    ff02::2 ip6-allrouters
    ff02::3 ip6-allhosts

    # OpsWorks Layer State
    127.0.0.1 storm-nimbus.localdomain storm-nimbus
    10.141.143.158 storm-nimbus
    <public IP> storm-nimbus-ext
    10.137.30.236 storm-supervisor-1
    <public IP> storm-supervisor-1-ext
    10.142.132.134 storm-supervisor-2
    <public IP> storm-supervisor-2-ext
    10.209.138.3 zookeeper-1
    <public IP> zookeeper-1-ext
    10.254.226.114 zookeeper-2
    <public IP> zookeeper-2-ext
    10.209.135.84 zookeeper-3
    <public IP> zookeeper-3-ext

    As far as I can tell, everything in here is sane and works correctly.

    The one thing that sticks out in my mind is that the Storm UI indicates
    both Supervisor nodes report themselves as "localhost" (
    http://cl.ly/image/1I1D2D0W3T46). Likewise, I notice that whenever any
    of the Storm components (Nimbus and Supervisor) start up the log output
    from the ZooKeeper client lists their hostname as localhost, as can be seen
    here:

    [2013-05-01 17:36:30,742] ZooKeeper [INFO] Client environment:host.name
    =localhost

    Does anyone have any ideas about what is happening here?

    Thanks for listening!
    On Saturday, January 5, 2013 2:03:41 AM UTC-5, Shrikar archak wrote:

    Hi Saisai,
    It was a networking problem. Make sure your cluster machine have unique
    host names
    and also have a mapping in you /etc/hosts file as below.

    Naming convention which I used

    TYPE : hostname
    Nimbus : nimbus
    Supervisor : supervisor-1
    Supervisor : supervisor-2
    Supervisor : supervisor-3
    Supervisor : supervisor-4

    example /etc/hosts file
    192.168.8.122 nimbus
    192.168.8.123 supervisor-1
    192.168.8.124 supervisor-2
    192.168.8.125 supervisor-3
    192.168.8.126 supervisor-4

    Hope that helps.
    Thanks,
    Shrikar
    On Tuesday, December 25, 2012 5:07:14 PM UTC-8, 邵赛赛 wrote:

    Hi Shrikar,
    I also meet this problem, could you please tell me how you solve this
    problem? it was so confused and I have no any clue.
    Any help greatly appreciate.

    Thanks
    Saisai

    在 2012年10月14日星期日UTC+8上午3时32分06秒,**Shrikar archak写道:
    Hi All,
    I need some help with setting the number of workers in a cluster. I
    get a lot of warning messages when I change the number of workers.

    Setup:
    1) Nimbus on a seperate host.
    2) 2 supervisors on 2 different nodes. ( I am not providing any ports
    for supervisor in storm.yaml , so I assume its 4 by default.)

    When I dont provide the worker setting. There is only one worker and I
    dont see these warnings.

    conf.setMaxTaskParallelism(80)**;

    conf.setNumAckers(4);

    for the above setting I dont see any warnings.
    *Problematic settings.*

    conf.setMaxTaskParallelism(80)**;

    conf.setNumAckers(4);

    *conf.setNumWorkers(4);*

    Lots of Warning messages :
    2012-10-12 18:25:16 worker [WARN] Received invalid messages for
    unknown tasks. Dropping...
    2012-10-12 18:25:16 worker [WARN] Received invalid messages for
    unknown tasks. Dropping...
    2012-10-12 18:25:16 worker [WARN] Received invalid messages for
    unknown tasks. Dropping...
    2012-10-12 18:25:17 worker [WARN] Received invalid messages for
    unknown tasks. Dropping...
    2012-10-12 18:25:17 worker [WARN] Received invalid messages for
    unknown tasks. Dropping...

    PS : the storm-local-dir is not shared among supervisors since they
    are on different nodes.

    Am I missing something? Any help greatly appreciated.

    Thanks,

    Shrikar
    --
    You received this message because you are subscribed to a topic in the
    Google Groups "storm-user" group.
    To unsubscribe from this topic, visit
    https://groups.google.com/d/topic/storm-user/DgIDxawBczg/unsubscribe?hl=en
    .
    To unsubscribe from this group and all its topics, send an email to
    storm-user+...@googlegroups.com <javascript:>.
    For more options, visit https://groups.google.com/groups/opt_out.

    --
    You received this message because you are subscribed to the Google Groups "storm-user" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to storm-user+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.
  • Aimee at May 10, 2013 at 7:23 am
    " set the storm.local.hostname in storm.yaml." That's really helpful!
    Thanks very much.

    Thanks
    Aimee
    On Thursday, May 2, 2013 9:29:13 AM UTC+8, Rich Schumacher wrote:

    I managed to track this down not long after I posted and it was definitely
    the localhost issue. I stumbled across the STORM_LOCAL_HOSTNAME property in
    backtype.storm.Config which states that the default value is the result
    of InetAddress.getLocalHost().getCanonicalHostName(). Sure enough, on
    instances launched via OpsWorks in AWS this value is always localhost. I'm
    not entirely sure why this is but instances launched via EC2 directly have
    the appropriate value.

    In any event, the fix was to explicitly set the storm.local.hostname in
    storm.yaml. After that everything worked fine.

    Thanks for the help and I hope this will help someone else out in the
    future.
    On Wednesday, May 1, 2013 7:07:10 PM UTC-4, Shrikar archak wrote:

    I remember fixing this problem by doing this.

    1) Make sure the hostname of all the supervisor and nimbus are different.
    In your case I see both the supervisors are localhost
    2) The above /etc/host should have similar entries on the supervisors too.

    Hope that helps.

    Thanks,
    Shrikar

    On Wed, May 1, 2013 at 11:55 AM, Rich Schumacher wrote:

    I have a similar setup and am experiencing the same problem.

    I have three AWS EC2 instances (provisioned by Chef/OpsWorks), one
    Nimbus and two Supervisor nodes. When I execute the ExclamationTopology
    from the tutorial on a single Supervisor it works as expected. However, if
    this topology executes on both Supervisors I see a lot of the dropped
    messages error, mixed in with stdout from the PrintingBolt I'm using.

    [2013-05-01 16:47:03,651] worker [WARN] Received invalid messages for
    unknown tasks. Dropping...
    [2013-05-01 16:47:03,652] worker [WARN] Received invalid messages for
    unknown tasks. Dropping...
    [2013-05-01 16:47:03,655] worker [WARN] Received invalid messages for
    unknown tasks. Dropping...
    [2013-05-01 16:47:03,658] STDIO [INFO] source: exclaim2:6, stream:
    default, id: {}, [jackson!!!!!!]
    [2013-05-01 16:47:03,756] STDIO [INFO] source: exclaim2:9, stream:
    default, id: {}, [mike!!!!!!]
    [2013-05-01 16:47:03,758] worker [WARN] Received invalid messages for
    unknown tasks. Dropping...

    I've read the Troubleshooting wiki as well as the above recommendations
    for setting /etc/hosts properly but to no avail. AWS OpsWorks automatically
    adds records to /etc/hosts, so each machine has something very similar to
    the following:

    # This file was generated by OpsWorks
    # any manual changes will be removed on the next update.

    127.0.0.1 localhost localhost.localdomain

    # The following lines are desirable for IPv6 capable hosts
    ::1 ip6-localhost ip6-loopback
    fe00::0 ip6-localnet
    ff00::0 ip6-mcastprefix
    ff02::1 ip6-allnodes
    ff02::2 ip6-allrouters
    ff02::3 ip6-allhosts

    # OpsWorks Layer State
    127.0.0.1 storm-nimbus.localdomain storm-nimbus
    10.141.143.158 storm-nimbus
    <public IP> storm-nimbus-ext
    10.137.30.236 storm-supervisor-1
    <public IP> storm-supervisor-1-ext
    10.142.132.134 storm-supervisor-2
    <public IP> storm-supervisor-2-ext
    10.209.138.3 zookeeper-1
    <public IP> zookeeper-1-ext
    10.254.226.114 zookeeper-2
    <public IP> zookeeper-2-ext
    10.209.135.84 zookeeper-3
    <public IP> zookeeper-3-ext

    As far as I can tell, everything in here is sane and works correctly.

    The one thing that sticks out in my mind is that the Storm UI indicates
    both Supervisor nodes report themselves as "localhost" (
    http://cl.ly/image/1I1D2D0W3T46). Likewise, I notice that whenever any
    of the Storm components (Nimbus and Supervisor) start up the log output
    from the ZooKeeper client lists their hostname as localhost, as can be seen
    here:

    [2013-05-01 17:36:30,742] ZooKeeper [INFO] Client environment:host.name
    =localhost

    Does anyone have any ideas about what is happening here?

    Thanks for listening!
    On Saturday, January 5, 2013 2:03:41 AM UTC-5, Shrikar archak wrote:

    Hi Saisai,
    It was a networking problem. Make sure your cluster machine have unique
    host names
    and also have a mapping in you /etc/hosts file as below.

    Naming convention which I used

    TYPE : hostname
    Nimbus : nimbus
    Supervisor : supervisor-1
    Supervisor : supervisor-2
    Supervisor : supervisor-3
    Supervisor : supervisor-4

    example /etc/hosts file
    192.168.8.122 nimbus
    192.168.8.123 supervisor-1
    192.168.8.124 supervisor-2
    192.168.8.125 supervisor-3
    192.168.8.126 supervisor-4

    Hope that helps.
    Thanks,
    Shrikar
    On Tuesday, December 25, 2012 5:07:14 PM UTC-8, 邵赛赛 wrote:

    Hi Shrikar,
    I also meet this problem, could you please tell me how you solve this
    problem? it was so confused and I have no any clue.
    Any help greatly appreciate.

    Thanks
    Saisai

    在 2012年10月14日星期日UTC+8上午3时32分06秒,**Shrikar archak写道:
    Hi All,
    I need some help with setting the number of workers in a cluster. I
    get a lot of warning messages when I change the number of workers.

    Setup:
    1) Nimbus on a seperate host.
    2) 2 supervisors on 2 different nodes. ( I am not providing any ports
    for supervisor in storm.yaml , so I assume its 4 by default.)

    When I dont provide the worker setting. There is only one worker and
    I dont see these warnings.

    conf.setMaxTaskParallelism(80)**;

    conf.setNumAckers(4);

    for the above setting I dont see any warnings.
    *Problematic settings.*

    conf.setMaxTaskParallelism(80)**;

    conf.setNumAckers(4);

    *conf.setNumWorkers(4);*

    Lots of Warning messages :
    2012-10-12 18:25:16 worker [WARN] Received invalid messages for
    unknown tasks. Dropping...
    2012-10-12 18:25:16 worker [WARN] Received invalid messages for
    unknown tasks. Dropping...
    2012-10-12 18:25:16 worker [WARN] Received invalid messages for
    unknown tasks. Dropping...
    2012-10-12 18:25:17 worker [WARN] Received invalid messages for
    unknown tasks. Dropping...
    2012-10-12 18:25:17 worker [WARN] Received invalid messages for
    unknown tasks. Dropping...

    PS : the storm-local-dir is not shared among supervisors since they
    are on different nodes.

    Am I missing something? Any help greatly appreciated.

    Thanks,

    Shrikar
    --
    You received this message because you are subscribed to a topic in the
    Google Groups "storm-user" group.
    To unsubscribe from this topic, visit
    https://groups.google.com/d/topic/storm-user/DgIDxawBczg/unsubscribe?hl=en
    .
    To unsubscribe from this group and all its topics, send an email to
    storm-user+...@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.

    --
    You received this message because you are subscribed to the Google Groups "storm-user" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to storm-user+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.
  • Mj at Jun 9, 2013 at 1:44 am
    Sweet, "explicitly set the storm.local.hostname in storm.yaml" seems to
    work for me as well. I pulled out some hair on this one, this should be in
    the troubleshooting section.
    On Thursday, May 2, 2013 3:29:13 AM UTC+2, Rich Schumacher wrote:

    I managed to track this down not long after I posted and it was definitely
    the localhost issue. I stumbled across the STORM_LOCAL_HOSTNAME property in
    backtype.storm.Config which states that the default value is the result
    of InetAddress.getLocalHost().getCanonicalHostName(). Sure enough, on
    instances launched via OpsWorks in AWS this value is always localhost. I'm
    not entirely sure why this is but instances launched via EC2 directly have
    the appropriate value.

    In any event, the fix was to explicitly set the storm.local.hostname in
    storm.yaml. After that everything worked fine.

    Thanks for the help and I hope this will help someone else out in the
    future.
    On Wednesday, May 1, 2013 7:07:10 PM UTC-4, Shrikar archak wrote:

    I remember fixing this problem by doing this.

    1) Make sure the hostname of all the supervisor and nimbus are different.
    In your case I see both the supervisors are localhost
    2) The above /etc/host should have similar entries on the supervisors too.

    Hope that helps.

    Thanks,
    Shrikar

    On Wed, May 1, 2013 at 11:55 AM, Rich Schumacher wrote:

    I have a similar setup and am experiencing the same problem.

    I have three AWS EC2 instances (provisioned by Chef/OpsWorks), one
    Nimbus and two Supervisor nodes. When I execute the ExclamationTopology
    from the tutorial on a single Supervisor it works as expected. However, if
    this topology executes on both Supervisors I see a lot of the dropped
    messages error, mixed in with stdout from the PrintingBolt I'm using.

    [2013-05-01 16:47:03,651] worker [WARN] Received invalid messages for
    unknown tasks. Dropping...
    [2013-05-01 16:47:03,652] worker [WARN] Received invalid messages for
    unknown tasks. Dropping...
    [2013-05-01 16:47:03,655] worker [WARN] Received invalid messages for
    unknown tasks. Dropping...
    [2013-05-01 16:47:03,658] STDIO [INFO] source: exclaim2:6, stream:
    default, id: {}, [jackson!!!!!!]
    [2013-05-01 16:47:03,756] STDIO [INFO] source: exclaim2:9, stream:
    default, id: {}, [mike!!!!!!]
    [2013-05-01 16:47:03,758] worker [WARN] Received invalid messages for
    unknown tasks. Dropping...

    I've read the Troubleshooting wiki as well as the above recommendations
    for setting /etc/hosts properly but to no avail. AWS OpsWorks automatically
    adds records to /etc/hosts, so each machine has something very similar to
    the following:

    # This file was generated by OpsWorks
    # any manual changes will be removed on the next update.

    127.0.0.1 localhost localhost.localdomain

    # The following lines are desirable for IPv6 capable hosts
    ::1 ip6-localhost ip6-loopback
    fe00::0 ip6-localnet
    ff00::0 ip6-mcastprefix
    ff02::1 ip6-allnodes
    ff02::2 ip6-allrouters
    ff02::3 ip6-allhosts

    # OpsWorks Layer State
    127.0.0.1 storm-nimbus.localdomain storm-nimbus
    10.141.143.158 storm-nimbus
    <public IP> storm-nimbus-ext
    10.137.30.236 storm-supervisor-1
    <public IP> storm-supervisor-1-ext
    10.142.132.134 storm-supervisor-2
    <public IP> storm-supervisor-2-ext
    10.209.138.3 zookeeper-1
    <public IP> zookeeper-1-ext
    10.254.226.114 zookeeper-2
    <public IP> zookeeper-2-ext
    10.209.135.84 zookeeper-3
    <public IP> zookeeper-3-ext

    As far as I can tell, everything in here is sane and works correctly.

    The one thing that sticks out in my mind is that the Storm UI indicates
    both Supervisor nodes report themselves as "localhost" (
    http://cl.ly/image/1I1D2D0W3T46). Likewise, I notice that whenever any
    of the Storm components (Nimbus and Supervisor) start up the log output
    from the ZooKeeper client lists their hostname as localhost, as can be seen
    here:

    [2013-05-01 17:36:30,742] ZooKeeper [INFO] Client environment:host.name
    =localhost

    Does anyone have any ideas about what is happening here?

    Thanks for listening!
    On Saturday, January 5, 2013 2:03:41 AM UTC-5, Shrikar archak wrote:

    Hi Saisai,
    It was a networking problem. Make sure your cluster machine have unique
    host names
    and also have a mapping in you /etc/hosts file as below.

    Naming convention which I used

    TYPE : hostname
    Nimbus : nimbus
    Supervisor : supervisor-1
    Supervisor : supervisor-2
    Supervisor : supervisor-3
    Supervisor : supervisor-4

    example /etc/hosts file
    192.168.8.122 nimbus
    192.168.8.123 supervisor-1
    192.168.8.124 supervisor-2
    192.168.8.125 supervisor-3
    192.168.8.126 supervisor-4

    Hope that helps.
    Thanks,
    Shrikar
    On Tuesday, December 25, 2012 5:07:14 PM UTC-8, 邵赛赛 wrote:

    Hi Shrikar,
    I also meet this problem, could you please tell me how you solve this
    problem? it was so confused and I have no any clue.
    Any help greatly appreciate.

    Thanks
    Saisai

    在 2012年10月14日星期日UTC+8上午3时32分06秒,**Shrikar archak写道:
    Hi All,
    I need some help with setting the number of workers in a cluster. I
    get a lot of warning messages when I change the number of workers.

    Setup:
    1) Nimbus on a seperate host.
    2) 2 supervisors on 2 different nodes. ( I am not providing any ports
    for supervisor in storm.yaml , so I assume its 4 by default.)

    When I dont provide the worker setting. There is only one worker and
    I dont see these warnings.

    conf.setMaxTaskParallelism(80)**;

    conf.setNumAckers(4);

    for the above setting I dont see any warnings.
    *Problematic settings.*

    conf.setMaxTaskParallelism(80)**;

    conf.setNumAckers(4);

    *conf.setNumWorkers(4);*

    Lots of Warning messages :
    2012-10-12 18:25:16 worker [WARN] Received invalid messages for
    unknown tasks. Dropping...
    2012-10-12 18:25:16 worker [WARN] Received invalid messages for
    unknown tasks. Dropping...
    2012-10-12 18:25:16 worker [WARN] Received invalid messages for
    unknown tasks. Dropping...
    2012-10-12 18:25:17 worker [WARN] Received invalid messages for
    unknown tasks. Dropping...
    2012-10-12 18:25:17 worker [WARN] Received invalid messages for
    unknown tasks. Dropping...

    PS : the storm-local-dir is not shared among supervisors since they
    are on different nodes.

    Am I missing something? Any help greatly appreciated.

    Thanks,

    Shrikar
    --
    You received this message because you are subscribed to a topic in the
    Google Groups "storm-user" group.
    To unsubscribe from this topic, visit
    https://groups.google.com/d/topic/storm-user/DgIDxawBczg/unsubscribe?hl=en
    .
    To unsubscribe from this group and all its topics, send an email to
    storm-user+...@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.

    --
    You received this message because you are subscribed to the Google Groups "storm-user" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to storm-user+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupstorm-user @
postedJan 5, '13 at 6:37a
activeJun 9, '13 at 1:44a
posts7
users5
websitestorm-project.net
irc#storm-user

People

Translate

site design / logo © 2022 Grokbase