FAQ
Hi,

I'm trying to add a bit of lightweight instrumentation to my akka-based
project, like for instance mailbox size, processing time, etc, based on
Codahale's Metrics library (and sent to graphite if that matters).

All this can be done, either by implementing a special Mailbox or a
special Receive block (ala LoggingReceive).

I'm now trying to monitor things like actor restart (or resume). I fail
to see a simple way (ie without modifying the akka code) to do that.
I can of course override preRestart or postRestart, but that wouldn't
catch for instance resuming.

I know that the Typesafe Console provides all this in a very effective
and nice way (you guys really rock!). Unfortunately at this time this
doesn't easily blend into both our budget and our current monitoring
infrastructure (hence my attempt).

So I'm open for any suggestions :)

Thanks!
--
Brice Figureau
My Blog: http://www.masterzen.fr/

--
---
You received this message because you are subscribed to the Google Groups "Akka User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+unsubscribe@googlegroups.com.
To post to this group, send email to akka-user@googlegroups.com.
Visit this group at http://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/groups/opt_out.

Search Discussions

  • Brice Figureau at Jul 26, 2013 at 9:19 am

    On Thu, 2013-07-25 at 16:36 +0200, Brice Figureau wrote:
    Hi,

    I'm trying to add a bit of lightweight instrumentation to my akka-based
    project, like for instance mailbox size, processing time, etc, based on
    Codahale's Metrics library (and sent to graphite if that matters).

    All this can be done, either by implementing a special Mailbox or a
    special Receive block (ala LoggingReceive).

    I'm now trying to monitor things like actor restart (or resume). I fail
    to see a simple way (ie without modifying the akka code) to do that.
    I can of course override preRestart or postRestart, but that wouldn't
    catch for instance resuming.

    I know that the Typesafe Console provides all this in a very effective
    and nice way (you guys really rock!). Unfortunately at this time this
    doesn't easily blend into both our budget and our current monitoring
    infrastructure (hence my attempt).

    So I'm open for any suggestions :)
    I finally found a solution (not very elegant, though). I created a
    specific SupervisorStrategy that delegates to the original used one.
    By overriding the logFailure method, I can increase my metrics on the
    various restart/resume events.

    Unfortunately SupervisorStrategy.logFailure is protected which means I
    can't delegate back to the original behavior (does it really need to be
    that strict?).

    Here's the pseudocode if anyone is interested:

    class MonitoringSupervisorStrategy(strategy: SupervisorStrategy) extends SupervisorStrategy {
       def decider: SupervisorStrategy.Decider = strategy.decider

       def handleChildTerminated(context: ActorContext, child: ActorRef, children: Iterable[ActorRef]) = strategy.handleChildTerminated(context, child, children)

       def processFailure(context: ActorContext, restart: Boolean, child: ActorRef, cause: Throwable, stats: ChildRestartStats, children: Iterable[ChildRestartStats]) = {
         strategy.processFailure(context, restart, child, cause, stats, children)
       }

       override protected def logFailure(context: ActorContext, child: ActorRef, cause: Throwable, decision: Directive): Unit = {
         // Here increment counter for this `child` and this `decision`
         MetricsExtension(context.system).meter(actorToMetricName(child), decisionName(decision)).mark()

         if (loggingEnabled) {
           // log here, I'd love to be able to call strategy.logFailure directly but it is proteceted
         }
       }

       private def decisionName(directive: Directive): String = directive match {
         case Resume => "resumed"
         case Restart => "restarted"
         case Stop => "stopped"
         case Escalate => "escalated"
       }
    }

    And with this companion class:
    trait ActorMonitoring {
       this: Actor =>

       final override def supervisorStrategy: SupervisorStrategy = new MonitoringSupervisorStrategy(monitoredSupervisorStrategy)
       def monitoredSupervisorStrategy: SupervisorStrategy = SupervisorStrategy.defaultStrategy
    }

    We can have a supervisor actor that keeps supervision metrics on it's
    childs with the minimum amount of actor modification:

       class Supervisor extends Actor with ActorMonitoring {
         override val monitoredSupervisorStrategy = {
           OneForOneStrategy(maxNrOfRetries = 10, withinTimeRange = 1 minute) {
             case _: ArithmeticException => Resume
             case _: NullPointerException => Restart
             case _: IllegalArgumentException => Stop
             case _: Exception => Escalate
           }
         }
         ...
       }

    Hope that would help anyone
    Thanks,
    --
    Brice Figureau
    My Blog: http://www.masterzen.fr/

    --
    ---
    You received this message because you are subscribed to the Google Groups "Akka User List" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+unsubscribe@googlegroups.com.
    To post to this group, send email to akka-user@googlegroups.com.
    Visit this group at http://groups.google.com/group/akka-user.
    For more options, visit https://groups.google.com/groups/opt_out.
  • Roland Kuhn at Jul 30, 2013 at 6:26 pm
    Hi Brice,

    yes, this looks useful: I have created a ticket to make logFailure public because I think it makes sense to compose SupervisorStrategies.

    Regards,

    Roland

    26 jul 2013 kl. 11:19 skrev Brice Figureau:
    On Thu, 2013-07-25 at 16:36 +0200, Brice Figureau wrote:
    Hi,

    I'm trying to add a bit of lightweight instrumentation to my akka-based
    project, like for instance mailbox size, processing time, etc, based on
    Codahale's Metrics library (and sent to graphite if that matters).

    All this can be done, either by implementing a special Mailbox or a
    special Receive block (ala LoggingReceive).

    I'm now trying to monitor things like actor restart (or resume). I fail
    to see a simple way (ie without modifying the akka code) to do that.
    I can of course override preRestart or postRestart, but that wouldn't
    catch for instance resuming.

    I know that the Typesafe Console provides all this in a very effective
    and nice way (you guys really rock!). Unfortunately at this time this
    doesn't easily blend into both our budget and our current monitoring
    infrastructure (hence my attempt).

    So I'm open for any suggestions :)
    I finally found a solution (not very elegant, though). I created a
    specific SupervisorStrategy that delegates to the original used one.
    By overriding the logFailure method, I can increase my metrics on the
    various restart/resume events.

    Unfortunately SupervisorStrategy.logFailure is protected which means I
    can't delegate back to the original behavior (does it really need to be
    that strict?).

    Here's the pseudocode if anyone is interested:

    class MonitoringSupervisorStrategy(strategy: SupervisorStrategy) extends SupervisorStrategy {
    def decider: SupervisorStrategy.Decider = strategy.decider

    def handleChildTerminated(context: ActorContext, child: ActorRef, children: Iterable[ActorRef]) = strategy.handleChildTerminated(context, child, children)

    def processFailure(context: ActorContext, restart: Boolean, child: ActorRef, cause: Throwable, stats: ChildRestartStats, children: Iterable[ChildRestartStats]) = {
    strategy.processFailure(context, restart, child, cause, stats, children)
    }

    override protected def logFailure(context: ActorContext, child: ActorRef, cause: Throwable, decision: Directive): Unit = {
    // Here increment counter for this `child` and this `decision`
    MetricsExtension(context.system).meter(actorToMetricName(child), decisionName(decision)).mark()

    if (loggingEnabled) {
    // log here, I'd love to be able to call strategy.logFailure directly but it is proteceted
    }
    }

    private def decisionName(directive: Directive): String = directive match {
    case Resume => "resumed"
    case Restart => "restarted"
    case Stop => "stopped"
    case Escalate => "escalated"
    }
    }

    And with this companion class:
    trait ActorMonitoring {
    this: Actor =>

    final override def supervisorStrategy: SupervisorStrategy = new MonitoringSupervisorStrategy(monitoredSupervisorStrategy)
    def monitoredSupervisorStrategy: SupervisorStrategy = SupervisorStrategy.defaultStrategy
    }

    We can have a supervisor actor that keeps supervision metrics on it's
    childs with the minimum amount of actor modification:

    class Supervisor extends Actor with ActorMonitoring {
    override val monitoredSupervisorStrategy = {
    OneForOneStrategy(maxNrOfRetries = 10, withinTimeRange = 1 minute) {
    case _: ArithmeticException => Resume
    case _: NullPointerException => Restart
    case _: IllegalArgumentException => Stop
    case _: Exception => Escalate
    }
    }
    ...
    }

    Hope that would help anyone
    Thanks,
    --
    Brice Figureau
    My Blog: http://www.masterzen.fr/

    --
    ---
    You received this message because you are subscribed to the Google Groups "Akka User List" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+unsubscribe@googlegroups.com.
    To post to this group, send email to akka-user@googlegroups.com.
    Visit this group at http://groups.google.com/group/akka-user.
    For more options, visit https://groups.google.com/groups/opt_out.


    Dr. Roland Kuhn
    Akka Tech Lead
    Typesafe – Reactive apps on the JVM.
    twitter: @rolandkuhn


    --
    ---
    You received this message because you are subscribed to the Google Groups "Akka User List" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+unsubscribe@googlegroups.com.
    To post to this group, send email to akka-user@googlegroups.com.
    Visit this group at http://groups.google.com/group/akka-user.
    For more options, visit https://groups.google.com/groups/opt_out.

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupakka-user @
categoriesscala
postedJul 25, '13 at 2:36p
activeJul 30, '13 at 6:26p
posts3
users2
websiteakka.io
irc#akka

2 users in discussion

Brice Figureau: 2 posts Roland Kuhn: 1 post

People

Translate

site design / logo © 2022 Grokbase