FAQ
We consistently see this exception in one of our bolts:

java.lang.OutOfMemoryError: Java heap space

at com.esotericsoftware.kryo.io.Output.toBytes(Output.java:103)

at
backtype.storm.serialization.KryoTupleSerializer.serialize(KryoTupleSerializer.java:28)

at
backtype.storm.daemon.worker$mk_transfer_fn$fn__4128$fn__4132.invoke(worker.clj:99)

at backtype.storm.util$fast_list_map.invoke(util.clj:771)

at
backtype.storm.daemon.worker$mk_transfer_fn$fn__4128.invoke(worker.clj:99)

at
backtype.storm.daemon.executor$start_batch_transfer__GT_worker_handler_BANG_$fn__3905.invoke(executor.clj:208)

at
backtype.storm.disruptor$clojure_handler$reify__1585.onEvent(disruptor.clj:43)

at
backtype.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:81)

at
backtype.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:55)

at
backtype.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:56)

at
backtype.storm.disruptor$consume_loop_STAR_$fn__1597.invoke(disruptor.clj:67)

at backtype.storm.util$async_loop$fn__465.invoke(util.clj:377)

at clojure.lang.AFn.run(AFn.java:24)

at java.lang.Thread.run(Thread.java:636)


We're using Trident topology. We see this even when the worker jvm is set
to 7Gb. Our tuple data, meaning the size of individual objects our code
emits is not really that large, <1Mb. But some bolts will emit 1k tuples
in a single call (not sure if that is relevant). I realize there is
probably not enough info in this post to identify the problem. But I
suspect this is somehow an issue with topology configuration. Could we be
emitting too many tuples on a single call? Are we perhaps running too
many executors in a single worker (we speced it to three trying to make
sure each executor had it's own core)? Any clues as to where to look would
be great. Thanks!


-mrf

Search Discussions

  • Nathan Marz at Dec 10, 2012 at 7:24 am
    The place to start is to do a heap dump and figure out what's taking up the
    memory
    On Fri, Dec 7, 2012 at 9:25 AM, Michael Frost wrote:

    We consistently see this exception in one of our bolts:

    java.lang.OutOfMemoryError: Java heap space

    at com.esotericsoftware.kryo.io.Output.toBytes(Output.java:103)

    at
    backtype.storm.serialization.KryoTupleSerializer.serialize(KryoTupleSerializer.java:28)

    at
    backtype.storm.daemon.worker$mk_transfer_fn$fn__4128$fn__4132.invoke(worker.clj:99)

    at backtype.storm.util$fast_list_map.invoke(util.clj:771)

    at
    backtype.storm.daemon.worker$mk_transfer_fn$fn__4128.invoke(worker.clj:99)

    at
    backtype.storm.daemon.executor$start_batch_transfer__GT_worker_handler_BANG_$fn__3905.invoke(executor.clj:208)

    at
    backtype.storm.disruptor$clojure_handler$reify__1585.onEvent(disruptor.clj:43)

    at
    backtype.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:81)

    at
    backtype.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:55)

    at
    backtype.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:56)

    at
    backtype.storm.disruptor$consume_loop_STAR_$fn__1597.invoke(disruptor.clj:67)

    at backtype.storm.util$async_loop$fn__465.invoke(util.clj:377)

    at clojure.lang.AFn.run(AFn.java:24)

    at java.lang.Thread.run(Thread.java:636)


    We're using Trident topology. We see this even when the worker jvm is set
    to 7Gb. Our tuple data, meaning the size of individual objects our code
    emits is not really that large, <1Mb. But some bolts will emit 1k tuples
    in a single call (not sure if that is relevant). I realize there is
    probably not enough info in this post to identify the problem. But I
    suspect this is somehow an issue with topology configuration. Could we be
    emitting too many tuples on a single call? Are we perhaps running too
    many executors in a single worker (we speced it to three trying to make
    sure each executor had it's own core)? Any clues as to where to look would
    be great. Thanks!


    -mrf


    --
    Twitter: @nathanmarz
    http://nathanmarz.com
  • Michael Rose at Dec 10, 2012 at 9:30 pm
    If you run your supervisors with -XX:+HeapDumpOnOutOfMemoryError you can go
    ahead and use jhat<http://docs.oracle.com/javase/6/docs/technotes/tools/share/jhat.html>to explore what's taking so much memory (after it dumps to disk)

    --
    Michael Rose (@Xorlev <https://twitter.com/xorlev>)
    Senior Platform Engineer, FullContact <http://www.fullcontact.com>
    michael@fullcontact.com
    On Monday, December 10, 2012 12:24:34 AM UTC-7, nathanmarz wrote:

    The place to start is to do a heap dump and figure out what's taking up
    the memory

    On Fri, Dec 7, 2012 at 9:25 AM, Michael Frost <mfro...@gmail.com<javascript:>
    wrote:
    We consistently see this exception in one of our bolts:

    java.lang.OutOfMemoryError: Java heap space

    at com.esotericsoftware.kryo.io.Output.toBytes(Output.java:103)

    at
    backtype.storm.serialization.KryoTupleSerializer.serialize(KryoTupleSerializer.java:28)

    at
    backtype.storm.daemon.worker$mk_transfer_fn$fn__4128$fn__4132.invoke(worker.clj:99)

    at backtype.storm.util$fast_list_map.invoke(util.clj:771)

    at
    backtype.storm.daemon.worker$mk_transfer_fn$fn__4128.invoke(worker.clj:99)

    at
    backtype.storm.daemon.executor$start_batch_transfer__GT_worker_handler_BANG_$fn__3905.invoke(executor.clj:208)

    at
    backtype.storm.disruptor$clojure_handler$reify__1585.onEvent(disruptor.clj:43)

    at
    backtype.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:81)

    at
    backtype.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:55)

    at
    backtype.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:56)

    at
    backtype.storm.disruptor$consume_loop_STAR_$fn__1597.invoke(disruptor.clj:67)

    at backtype.storm.util$async_loop$fn__465.invoke(util.clj:377)

    at clojure.lang.AFn.run(AFn.java:24)

    at java.lang.Thread.run(Thread.java:636)


    We're using Trident topology. We see this even when the worker jvm is
    set to 7Gb. Our tuple data, meaning the size of individual objects our
    code emits is not really that large, <1Mb. But some bolts will emit 1k
    tuples in a single call (not sure if that is relevant). I realize there is
    probably not enough info in this post to identify the problem. But I
    suspect this is somehow an issue with topology configuration. Could we be
    emitting too many tuples on a single call? Are we perhaps running too
    many executors in a single worker (we speced it to three trying to make
    sure each executor had it's own core)? Any clues as to where to look would
    be great. Thanks!


    -mrf


    --
    Twitter: @nathanmarz
    http://nathanmarz.com

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupstorm-user @
postedDec 10, '12 at 7:23a
activeDec 10, '12 at 9:30p
posts3
users3
websitestorm-project.net
irc#storm-user

People

Translate

site design / logo © 2022 Grokbase