FAQ
Hi,

I'm using Storm 0.8.1 and I'm trying to deploy a topology to a
pre-production cluster (with 10 workers slots).
It seems that my spout is failing to start if deployed on a separate worker
(ie. works fine if using only one worker slot - no serialization).

I'm getting the following exception :
2012-12-06 11:32:15 executor [INFO] Loading executor twitterCollector:[6 6]
2012-12-06 11:32:15 executor [INFO] Loaded executor tasks
twitterCollector:[6 6]
2012-12-06 11:32:15 executor [INFO] Opening spout twitterCollector:(6)
[...]
2012-12-06 11:32:15 executor [INFO] Opened spout twitterCollector:(6)
2012-12-06 11:32:15 executor [INFO] Finished loading executor
twitterCollector:[6 6]
2012-12-06 11:32:15 executor [INFO] Activating spout twitterCollector:(6)
[...]
2012-12-06 11:32:15 util [ERROR] Async loop died!
java.lang.RuntimeException: java.lang.RuntimeException:
java.io.NotSerializableException: backtype.storm.spout.SpoutOutputCollector
at
backtype.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:84)
at
backtype.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:55)
at
backtype.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:56)
at
backtype.storm.disruptor$consume_loop_STAR_$fn__1597.invoke(disruptor.clj:67)
at backtype.storm.util$async_loop$fn__465.invoke(util.clj:377)
at clojure.lang.AFn.run(AFn.java:24)
at java.lang.Thread.run(Thread.java:679)
Caused by: java.lang.RuntimeException: java.io.NotSerializableException:
backtype.storm.spout.SpoutOutputCollector
at
backtype.storm.serialization.SerializableSerializer.write(SerializableSerializer.java:24)
at com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:554)
at
com.esotericsoftware.kryo.serializers.CollectionSerializer.write(CollectionSerializer.java:77)
at
com.esotericsoftware.kryo.serializers.CollectionSerializer.write(CollectionSerializer.java:18)
at com.esotericsoftware.kryo.Kryo.writeObject(Kryo.java:472)
at
backtype.storm.serialization.KryoValuesSerializer.serializeInto(KryoValuesSerializer.java:27)
at
backtype.storm.serialization.KryoTupleSerializer.serialize(KryoTupleSerializer.java:27)
at
backtype.storm.daemon.worker$mk_transfer_fn$fn__4128$fn__4132.invoke(worker.clj:99)
at backtype.storm.util$fast_list_map.invoke(util.clj:771)
at
backtype.storm.daemon.worker$mk_transfer_fn$fn__4128.invoke(worker.clj:99)
at
backtype.storm.daemon.executor$start_batch_transfer__GT_worker_handler_BANG_$fn__3905.invoke(executor.clj:208)
at
backtype.storm.disruptor$clojure_handler$reify__1585.onEvent(disruptor.clj:43)
at
backtype.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:81)
... 6 more
Caused by: java.io.NotSerializableException:
backtype.storm.spout.SpoutOutputCollector
at
java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1180)
at
java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1528)
at
java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1493)
at
java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1416)
at
java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1174)
at
java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1528)
at
java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1493)
at
java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1416)
at
java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1174)
at
java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:346)
at
backtype.storm.serialization.SerializableSerializer.write(SerializableSerializer.java:21)
... 18 more

I'm using the following code :
-------------
private SpoutOutputCollector collector;

@Override
public void open(
@SuppressWarnings("rawtypes") Map conf, TopologyContext
context, SpoutOutputCollector collector) {

this.collector = collector;
i = 0;

this.activate();
}
-------------

Should I set my "collector" attribute as transient ?

Best regards,
--
Damien

Search Discussions

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupstorm-user @
postedDec 6, '12 at 9:47p
activeDec 6, '12 at 9:47p
posts1
users1
websitestorm-project.net
irc#storm-user

1 user in discussion

Damien Raude-Morvan: 1 post

People

Translate

site design / logo © 2022 Grokbase