start seeing error messages like
09-01-2013 17:33:01.385 {main-EventThread} [INFO ]
com.netflix.curator.framework.state.ConnectionStateManager - State change:
SUSPENDED
09-01-2013 17:33:01.386 {main-EventThread} [WARN ] backtype.storm.cluster -
Received event :disconnected::none: with disconnected Zookeeper.
09-01-2013 17:33:01.387 {Thread-13-EventThread} [INFO ]
com.netflix.curator.framework.state.ConnectionStateManager - State change:
SUSPENDED
09-01-2013 17:33:01.387 {ConnectionStateManager-0} [WARN ]
com.netflix.curator.framework.state.ConnectionStateManager - There are no
ConnectionStateListeners registered.
09-01-2013 17:33:04.946 {ConnectionStateManager-0} [WARN ]
com.netflix.curator.framework.state.ConnectionStateManager - There are no
ConnectionStateListeners registered.
The spout does seem to continue to emit tuples. In my case, the tuples seem
to be processed by the first couple of stages of my (Trident) topology but
then they are mysteriously dropped, and
no further processing occurs on those tuples. I'm using 0.8.2-wip8 (which I
moved to in an effort to overcome this problem, which also manifested
itself in 0.8.1).
Denis
On Friday, 2 November 2012 20:24:04 UTC, Naresh Kosgi wrote:
Brad,
Were you able to solve this problem. I am having the same issue
currently.
Thanks,
Naresh
appears as if nextTuple is indeed getting called but nothing is getting
emitted by the spout and bolts are not emitting anything.
to read additional data from client sessionid 0x139936a474c0007, likely
client has closed socket
to read additional data from client sessionid 0x139936a474c0003, likely
client has closed socket
change: SUSPENDED
change: SUSPENDED
ConnectionStateListeners registered.
ConnectionStateListeners registered.
to read additional data from client sessionid 0x139936a474c0005, likely
client has closed socket
change: SUSPENDED
ConnectionStateListeners registered.
change: SUSPENDED
session 0x139936a474c0001 due to java.io.IOException: Connection reset by
peer
Brad,
Were you able to solve this problem. I am having the same issue
currently.
Thanks,
Naresh
On Sep 4, 6:51 pm, Brad Heller wrote:
Hello list,
This problem has rather suddenly manifested itself when running in local
mode. Storm will be chugging along then…suddenly…all processing stops. ItHello list,
This problem has rather suddenly manifested itself when running in local
appears as if nextTuple is indeed getting called but nothing is getting
emitted by the spout and bolts are not emitting anything.
Then, suddenly, ZooKeeper exceptions appear!
577557 [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2000] WARN
org.apache.zookeeper.server.NIOServerCnxn - EndOfStreamException: Unable577557 [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2000] WARN
to read additional data from client sessionid 0x139936a474c0007, likely
client has closed socket
578673 [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2000] WARN
org.apache.zookeeper.server.NIOServerCnxn - EndOfStreamException: Unableto read additional data from client sessionid 0x139936a474c0003, likely
client has closed socket
579304 [main-EventThread] INFO
com.netflix.curator.framework.state.ConnectionStateManager - Statechange: SUSPENDED
579304 [main-EventThread] INFO
com.netflix.curator.framework.state.ConnectionStateManager - Statechange: SUSPENDED
579327 [ConnectionStateManager-0] WARN
com.netflix.curator.framework.state.ConnectionStateManager - There are noConnectionStateListeners registered.
579327 [ConnectionStateManager-0] WARN
com.netflix.curator.framework.state.ConnectionStateManager - There are noConnectionStateListeners registered.
579962 [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2000] WARN
org.apache.zookeeper.server.NIOServerCnxn - EndOfStreamException: Unableto read additional data from client sessionid 0x139936a474c0005, likely
client has closed socket
579962 [main-EventThread] INFO
com.netflix.curator.framework.state.ConnectionStateManager - Statechange: SUSPENDED
579962 [ConnectionStateManager-0] WARN
com.netflix.curator.framework.state.ConnectionStateManager - There are noConnectionStateListeners registered.
580496 [SyncThread:0] ERROR org.apache.zookeeper.server.NIOServerCnxn -
Unexpected Exception:.
java.nio.channels.CancelledKeyException
at sun.nio.ch.SelectionKeyImpl.ensureValid(SelectionKeyImpl.java:55)
at sun.nio.ch.SelectionKeyImpl.interestOps(SelectionKeyImpl.java:59)
at
org.apache.zookeeper.server.NIOServerCnxn.sendBuffer(NIOServerCnxn.java:418)
at
org.apache.zookeeper.server.NIOServerCnxn.sendResponse(NIOServerCnxn.java:1509)
at
org.apache.zookeeper.server.FinalRequestProcessor.processRequest(FinalRequestProcessor.java:171)
at
org.apache.zookeeper.server.SyncRequestProcessor.flush(SyncRequestProcessor.java:161)
at
org.apache.zookeeper.server.SyncRequestProcessor.run(SyncRequestProcessor.java:98)
581129 [main-EventThread] INFO
com.netflix.curator.framework.state.ConnectionStateManager - StateUnexpected Exception:.
java.nio.channels.CancelledKeyException
at sun.nio.ch.SelectionKeyImpl.ensureValid(SelectionKeyImpl.java:55)
at sun.nio.ch.SelectionKeyImpl.interestOps(SelectionKeyImpl.java:59)
at
org.apache.zookeeper.server.NIOServerCnxn.sendBuffer(NIOServerCnxn.java:418)
at
org.apache.zookeeper.server.NIOServerCnxn.sendResponse(NIOServerCnxn.java:1509)
at
org.apache.zookeeper.server.FinalRequestProcessor.processRequest(FinalRequestProcessor.java:171)
at
org.apache.zookeeper.server.SyncRequestProcessor.flush(SyncRequestProcessor.java:161)
at
org.apache.zookeeper.server.SyncRequestProcessor.run(SyncRequestProcessor.java:98)
581129 [main-EventThread] INFO
change: SUSPENDED
580497 [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2000] WARN
org.apache.zookeeper.server.NIOServerCnxn - Exception causing close ofsession 0x139936a474c0001 due to java.io.IOException: Connection reset by
peer
Is it possible that storm can overwhelm ZooKeeper? I *did* adjust
parallelism on some bolts…but I wouldn't think this could break anything.
Thanks,
Brad Heller | Engineering Lead | Cloudability.com | 541-231-1514 |
Skype: brad.heller | @bradhe | @cloudability
We're hiring!http://cloudability.com/jobs
parallelism on some bolts…but I wouldn't think this could break anything.
Thanks,
Brad Heller | Engineering Lead | Cloudability.com | 541-231-1514 |
Skype: brad.heller | @bradhe | @cloudability
We're hiring!http://cloudability.com/jobs