FAQ
Hi,

I'm trying to attach VisualVM to a storm worker process to profile where we
are consuming the most CPU. However, every time I attach the worker process
immediately dies. I believe this might be due to the heartbeat timing out
during the overhead of attaching to the process.

Any suggestions for which of the heartbeat timeout values are best to
extend in this case? This is what I see in the supervisor.log:

2013-02-21 17:06:06 supervisor [INFO] Shutting down and clearing state for
id 43f4f778-e3ab-46e6-ba99-0ea208f934bd. Current supervisor time:
1361466366. State: :disallowed, Heartbeat:
#backtype.storm.daemon.common.WorkerHeartbeat{:time-secs 1361466366,
:storm-id "scommits-6-1359050691", :executors #{[3 3] [5 5] [7 7] [9 9] [11
11] [1 1]}, :port 6701}

Is that the supervisor -> worker heartbeat timeout failing?

Anyone have luck profiling a running topology -- what tools/procedures work
best?


Cheers,

Mike

--

Mike Heffner <mike@librato.com>
Librato, Inc.

--
You received this message because you are subscribed to the Google Groups "storm-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to storm-user+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Search Discussions

  • Mike Heffner at Feb 25, 2013 at 5:38 pm
    After increasing the worker heartbeat timeouts I'm still seeing the worker
    processes disappear shortly after VisualVM attaches to them. It's possible
    that the VM is crashing or exiting it's attached to, but I don't see
    anything in the worker logs. Is there a way to enable logging of the exit
    conditions and/or backtraces?

    Fomr the supervisor logs, it detects that the worker is in a "disallowed"
    heartbeat state and restarts it immediately.


    Mike

    On Thu, Feb 21, 2013 at 1:25 PM, Mike Heffner wrote:

    Hi,

    I'm trying to attach VisualVM to a storm worker process to profile where
    we are consuming the most CPU. However, every time I attach the worker
    process immediately dies. I believe this might be due to the heartbeat
    timing out during the overhead of attaching to the process.

    Any suggestions for which of the heartbeat timeout values are best to
    extend in this case? This is what I see in the supervisor.log:

    2013-02-21 17:06:06 supervisor [INFO] Shutting down and clearing state for
    id 43f4f778-e3ab-46e6-ba99-0ea208f934bd. Current supervisor time:
    1361466366. State: :disallowed, Heartbeat:
    #backtype.storm.daemon.common.WorkerHeartbeat{:time-secs 1361466366,
    :storm-id "scommits-6-1359050691", :executors #{[3 3] [5 5] [7 7] [9 9] [11
    11] [1 1]}, :port 6701}

    Is that the supervisor -> worker heartbeat timeout failing?

    Anyone have luck profiling a running topology -- what tools/procedures
    work best?


    Cheers,

    Mike

    --

    Mike Heffner <mike@librato.com>
    Librato, Inc.

    --

    Mike Heffner <mike@librato.com>
    Librato, Inc.

    --
    You received this message because you are subscribed to the Google Groups "storm-user" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to storm-user+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.
  • Nathan Marz at Feb 25, 2013 at 9:04 pm
    I use YourKit. When you do trace profiling with YourKit it freezes the
    process for awhile, which can lead to these issues. Sample profiling
    doesn't have any of these issues though.

    On Mon, Feb 25, 2013 at 9:38 AM, Mike Heffner wrote:

    After increasing the worker heartbeat timeouts I'm still seeing the worker
    processes disappear shortly after VisualVM attaches to them. It's possible
    that the VM is crashing or exiting it's attached to, but I don't see
    anything in the worker logs. Is there a way to enable logging of the exit
    conditions and/or backtraces?

    Fomr the supervisor logs, it detects that the worker is in a "disallowed"
    heartbeat state and restarts it immediately.


    Mike

    On Thu, Feb 21, 2013 at 1:25 PM, Mike Heffner wrote:

    Hi,

    I'm trying to attach VisualVM to a storm worker process to profile where
    we are consuming the most CPU. However, every time I attach the worker
    process immediately dies. I believe this might be due to the heartbeat
    timing out during the overhead of attaching to the process.

    Any suggestions for which of the heartbeat timeout values are best to
    extend in this case? This is what I see in the supervisor.log:

    2013-02-21 17:06:06 supervisor [INFO] Shutting down and clearing state
    for id 43f4f778-e3ab-46e6-ba99-0ea208f934bd. Current supervisor time:
    1361466366. State: :disallowed, Heartbeat:
    #backtype.storm.daemon.common.WorkerHeartbeat{:time-secs 1361466366,
    :storm-id "scommits-6-1359050691", :executors #{[3 3] [5 5] [7 7] [9 9] [11
    11] [1 1]}, :port 6701}

    Is that the supervisor -> worker heartbeat timeout failing?

    Anyone have luck profiling a running topology -- what tools/procedures
    work best?


    Cheers,

    Mike

    --

    Mike Heffner <mike@librato.com>
    Librato, Inc.

    --

    Mike Heffner <mike@librato.com>
    Librato, Inc.

    --
    You received this message because you are subscribed to the Google Groups
    "storm-user" group.
    To unsubscribe from this group and stop receiving emails from it, send an
    email to storm-user+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.



    --
    Twitter: @nathanmarz
    http://nathanmarz.com

    --
    You received this message because you are subscribed to the Google Groups "storm-user" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to storm-user+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.
  • Ilya Kaplun at Feb 26, 2013 at 2:26 am
    I've attached with VisualVM to our production workers over JMX without any
    issues or disconnects, though you only get sampling with that approach.
    On Monday, February 25, 2013 12:38:15 PM UTC-5, Mike Heffner wrote:

    After increasing the worker heartbeat timeouts I'm still seeing the worker
    processes disappear shortly after VisualVM attaches to them. It's possible
    that the VM is crashing or exiting it's attached to, but I don't see
    anything in the worker logs. Is there a way to enable logging of the exit
    conditions and/or backtraces?

    Fomr the supervisor logs, it detects that the worker is in a "disallowed"
    heartbeat state and restarts it immediately.


    Mike


    On Thu, Feb 21, 2013 at 1:25 PM, Mike Heffner <mi...@librato.com<javascript:>
    wrote:
    Hi,

    I'm trying to attach VisualVM to a storm worker process to profile where
    we are consuming the most CPU. However, every time I attach the worker
    process immediately dies. I believe this might be due to the heartbeat
    timing out during the overhead of attaching to the process.

    Any suggestions for which of the heartbeat timeout values are best to
    extend in this case? This is what I see in the supervisor.log:

    2013-02-21 17:06:06 supervisor [INFO] Shutting down and clearing state
    for id 43f4f778-e3ab-46e6-ba99-0ea208f934bd. Current supervisor time:
    1361466366. State: :disallowed, Heartbeat:
    #backtype.storm.daemon.common.WorkerHeartbeat{:time-secs 1361466366,
    :storm-id "scommits-6-1359050691", :executors #{[3 3] [5 5] [7 7] [9 9] [11
    11] [1 1]}, :port 6701}

    Is that the supervisor -> worker heartbeat timeout failing?

    Anyone have luck profiling a running topology -- what tools/procedures
    work best?


    Cheers,

    Mike

    --

    Mike Heffner <mi...@librato.com <javascript:>>
    Librato, Inc.

    --

    Mike Heffner <mi...@librato.com <javascript:>>
    Librato, Inc.
    --
    You received this message because you are subscribed to the Google Groups "storm-user" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to storm-user+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupstorm-user @
postedFeb 21, '13 at 6:31p
activeFeb 26, '13 at 2:26a
posts4
users3
websitestorm-project.net
irc#storm-user

People

Translate

site design / logo © 2022 Grokbase