FAQ
Hi all,

Lately we found this issue in our performance set up. The following is the
stack trace that we get in worker logs:

java.lang.OutOfMemoryError: Java heap space
at java.util.Arrays.copyOf(Arrays.java:2798)
at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:111)
at
com.esotericsoftware.kryo.ObjectBuffer.writeToStream(ObjectBuffer.java:196)
at
com.esotericsoftware.kryo.ObjectBuffer.writeObject(ObjectBuffer.java:170)
at
backtype.storm.serialization.KryoValuesSerializer.serializeInto(KryoValuesSerializer.java:26)
at
backtype.storm.serialization.KryoTupleSerializer$Worker.serialize(KryoTupleSerializer.java:65)
at
backtype.storm.serialization.KryoTupleSerializer.serialize(KryoTupleSerializer.java:29)
at
backtype.storm.daemon.worker$mk_transfer_fn$fn__3535.invoke(worker.clj:86)
at
backtype.storm.daemon.task$fn__3417$bolt_emit__3418.invoke(task.clj:459)
at backtype.storm.daemon.task$fn$reify__3463.emit(task.clj:463)
at backtype.storm.task.OutputCollector.emit(OutputCollector.java:186)
at backtype.storm.task.OutputCollector.emit(OutputCollector.java:32)
at backtype.storm.task.OutputCollector.emit(OutputCollector.java:71)
at com.tkl.ITBolt.execute(ITBolt.java:85)
at
backtype.storm.daemon.task$fn__3417$tuple_action_fn__3498.invoke(task.clj:514)
at
backtype.storm.daemon.task$mk_task_receiver$fn__3344.invoke(task.clj:317)
at
backtype.storm.daemon.task$mk_task$iter__3281__3285$fn__3286$fn__3287$fn__3288.invoke(task.clj:262)
at clojure.lang.AFn.applyToHelper(AFn.java:159)
at clojure.lang.AFn.applyTo(AFn.java:151)
at clojure.core$apply.invoke(core.clj:601)
at backtype.storm.util$async_loop$fn__451.invoke(util.clj:369)
at clojure.lang.AFn.run(AFn.java:24)
at java.lang.Thread.run(Thread.java:679

We are using storm 0.7.4 and normal topologies. We are facing out of memory
during high traffic. We have 3 classes registered for kryo serialization.
These classes are used in the tuples that are passed between bolts.

1. Is this happening due to the delay in processing the tuples in this
bolt? Will increasing the parallelism help us in solving this issue?
2. Are we missing some parameters which are supposed to be configured (like
buffer size)
3. Will we face this if tuples get piled up and there is a low rate of
consumption?

Thanks,
Richards Peter.

--
You received this message because you are subscribed to the Google Groups "storm-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to storm-user+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Search Discussions

  • Bobby Evans at Mar 5, 2013 at 3:28 pm
    Do you have acking enabled in your performance tests, or is it turned off?
    If you do not have acking enabled there is no flow control and you will see
    OOMs unless you have over provisioned your hardware enough to handle the
    highest load. If you do have acking enabled you need to set
    topology.max.spout.pending which controls the maximum number of tuples each
    spout can have in flight at any point in time.

    --
    Bobby

    On Tue, Mar 5, 2013 at 8:21 AM, Richards Peter wrote:


    Hi all,

    Lately we found this issue in our performance set up. The following is the
    stack trace that we get in worker logs:

    java.lang.OutOfMemoryError: Java heap space
    at java.util.Arrays.copyOf(Arrays.java:2798)
    at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:111)
    at
    com.esotericsoftware.kryo.ObjectBuffer.writeToStream(ObjectBuffer.java:196)
    at
    com.esotericsoftware.kryo.ObjectBuffer.writeObject(ObjectBuffer.java:170)
    at
    backtype.storm.serialization.KryoValuesSerializer.serializeInto(KryoValuesSerializer.java:26)
    at
    backtype.storm.serialization.KryoTupleSerializer$Worker.serialize(KryoTupleSerializer.java:65)
    at
    backtype.storm.serialization.KryoTupleSerializer.serialize(KryoTupleSerializer.java:29)
    at
    backtype.storm.daemon.worker$mk_transfer_fn$fn__3535.invoke(worker.clj:86)
    at
    backtype.storm.daemon.task$fn__3417$bolt_emit__3418.invoke(task.clj:459)
    at backtype.storm.daemon.task$fn$reify__3463.emit(task.clj:463)
    at backtype.storm.task.OutputCollector.emit(OutputCollector.java:186)
    at backtype.storm.task.OutputCollector.emit(OutputCollector.java:32)
    at backtype.storm.task.OutputCollector.emit(OutputCollector.java:71)
    at com.tkl.ITBolt.execute(ITBolt.java:85)
    at
    backtype.storm.daemon.task$fn__3417$tuple_action_fn__3498.invoke(task.clj:514)
    at
    backtype.storm.daemon.task$mk_task_receiver$fn__3344.invoke(task.clj:317)
    at
    backtype.storm.daemon.task$mk_task$iter__3281__3285$fn__3286$fn__3287$fn__3288.invoke(task.clj:262)
    at clojure.lang.AFn.applyToHelper(AFn.java:159)
    at clojure.lang.AFn.applyTo(AFn.java:151)
    at clojure.core$apply.invoke(core.clj:601)
    at backtype.storm.util$async_loop$fn__451.invoke(util.clj:369)
    at clojure.lang.AFn.run(AFn.java:24)
    at java.lang.Thread.run(Thread.java:679

    We are using storm 0.7.4 and normal topologies. We are facing out of
    memory during high traffic. We have 3 classes registered for kryo
    serialization. These classes are used in the tuples that are passed between
    bolts.

    1. Is this happening due to the delay in processing the tuples in this
    bolt? Will increasing the parallelism help us in solving this issue?
    2. Are we missing some parameters which are supposed to be configured
    (like buffer size)
    3. Will we face this if tuples get piled up and there is a low rate of
    consumption?

    Thanks,
    Richards Peter.

    --
    You received this message because you are subscribed to the Google Groups
    "storm-user" group.
    To unsubscribe from this group and stop receiving emails from it, send an
    email to storm-user+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.

    --
    You received this message because you are subscribed to the Google Groups "storm-user" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to storm-user+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.
  • Richards Peter at Mar 6, 2013 at 4:15 am
    Hi Bobby,

    We have ack enabled application. We have configured
    topology.max.spout.pending also. Our topology structure is like this:

    S1 -> B1 -> B2 -> B3 -> B4

    For each tuple emitted from S1, B2 could generate thousands of tuples.
    These tuples propagate to B3 and B4 after some transformation at each
    layer. We get this exception in B3.

    Thanks,
    Richards Peter.

    --
    You received this message because you are subscribed to the Google Groups "storm-user" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to storm-user+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.
  • Richards Peter at Mar 6, 2013 at 3:28 pm
    Hi,

    We are still stuck with this issue. Has anyone faced this issue?

    Thanks,
    Richards Peter.

    --
    You received this message because you are subscribed to the Google Groups "storm-user" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to storm-user+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.
  • Patricio Echagüe at Mar 8, 2013 at 8:03 pm
    Any chance you need more memory to support that high traffic? What's the
    Xmx ?

    On Wed, Mar 6, 2013 at 7:28 AM, Richards Peter wrote:

    Hi,

    We are still stuck with this issue. Has anyone faced this issue?

    Thanks,
    Richards Peter.

    --
    You received this message because you are subscribed to the Google Groups
    "storm-user" group.
    To unsubscribe from this group and stop receiving emails from it, send an
    email to storm-user+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.

    --
    You received this message because you are subscribed to the Google Groups "storm-user" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to storm-user+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.
  • Richards Peter at Mar 9, 2013 at 3:14 am
    Hi,

    We have fixed the issue. It was a configuration problem in our system due
    to which some bolts were receiving data from the producers at a faster rate
    eventually leading to out of memory.

    Thanks for the help,
    Richards Peter.

    --
    You received this message because you are subscribed to the Google Groups "storm-user" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to storm-user+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.
  • Richards Peter at Mar 13, 2013 at 4:57 am
    Hi,

    I just wanted to ask one more doubt related to this exception. How is Out
    of memory exception handled by storm? Will the worker facing out of memory
    be restarted automatically? If it is restarted, is it just that worker
    which is restated or will other workers be restarted?

    Thanks,
    Richards Peter.

    --
    You received this message because you are subscribed to the Google Groups "storm-user" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to storm-user+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupstorm-user @
postedMar 5, '13 at 2:23p
activeMar 13, '13 at 4:57a
posts7
users3
websitestorm-project.net
irc#storm-user

People

Translate

site design / logo © 2021 Grokbase