Grokbase › Groups › Node.js › nodejs › November 2014
FAQ

Search Discussions

  • Ben Noordhuis at Nov 11, 2014 at 12:13 am ⇧

  • Andreas Marschke at Nov 11, 2014 at 2:05 pm ⇧

  • Bruno Jouhier at Nov 12, 2014 at 12:10 am ⇧

  • Ben Noordhuis at Nov 12, 2014 at 12:11 am ⇧

  • // ravi at Nov 12, 2014 at 12:31 am ⇧

  • Stephen Belanger at Nov 12, 2014 at 1:31 am ⇧

  • Bruno Jouhier at Nov 12, 2014 at 2:58 pm ⇧

    TJ Fontaine did a bit of experimentation with node-tracing, and Forrest
    Norvell made CLS. Trevor Norris had also created AsyncListener to track
    changes between async contexts. But all the approaches so far have been too
    naive or simplistic and ultimately flawed.
      Isn't this a bit of a red herring?

    If you extend the language with an async/await construct (which is what
    streamline.js does), then the continuations are captured at the "language"
    level and you can instrument the language runtime to propagate a context
    across continuations and to track performance counters with "long
    stacktrace" semantics. This is completely transparent. The instrumentation
    service is provided by the "language" runtime and does not require any
    special support from the specific libraries that you are using (although I
    haven't tried it this way, streamline-flamegraph recording should work in
    the browser).

    If you don't extend the language with async/await then you need ad-hoc
    mechanisms. You need special mechanisms in node.js to propagate a context
    across continuations. This can be done in APIs like fs that are aligned on
    the continuation callback pattern but it won't work with other APIs that
    use events for continuation (for example a connect call that continues with
    connect/error events, or a read call that continues with readable/error
    events).

    I see a lot of effort being put in trying to provide context propagation
    features around raw callbacks (domains, long-stacktrace, async listeners).
    These APIs are of no use for people like me who are relying on a
    "language-level" construct and who get these features for free from their
    language runtime. They add complexity and they don't seem to hit their
    target (at least so far).

    IMO, raw callbacks should just be raw callbacks. They should be as fast as
    possible and they should not try to deal with context propagation. If
    developers want context propagation they should use a language extension (a
    la async/await) that captures the "continuation threads". They will also
    get other benefits (like robust exception handling).

    Better be explicit about what your program means (express your
    continuations with async/await) than rely on ad-hoc mechanisms that will
    try to reconstruct the continuation semantics from callbacks and special
    APIs.

    My 2 cents.

    Bruno

    --
    Job board: http://jobs.nodejs.org/
    New group rules: https://gist.github.com/othiym23/9886289#file-moderation-policy-md
    Old group rules: https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
    ---
    You received this message because you are subscribed to the Google Groups "nodejs" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to nodejs+unsubscribe@googlegroups.com.
    To post to this group, send email to nodejs@googlegroups.com.
    To view this discussion on the web visit https://groups.google.com/d/msgid/nodejs/7a021f0f-0f7f-444a-b6e3-e30c27a4871f%40googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
  • Andreas Marschke at Nov 12, 2014 at 7:39 pm ⇧
    +1

    The important question here I guess is : How do you get these library
    neutral instrumentations into the node runtime? Which is also what I
    eluded to asking if would make sense to have people from v8 & node core
    team together with people from the node-community at the same table and
    start working out a standardize solution without stepping on each others
    toes and make node and subsequently v8 a better engine to work with.
    Imagine the benefits the browsers using v8 could foster from a central
    orchestration of changes.

    Just saying.
    On 12 November 2014 13:54, Bruno Jouhier wrote:

    TJ Fontaine did a bit of experimentation with node-tracing, and Forrest
    Norvell made CLS. Trevor Norris had also created AsyncListener to track
    changes between async contexts. But all the approaches so far have been too
    naive or simplistic and ultimately flawed.
    Isn't this a bit of a red herring?

    If you extend the language with an async/await construct (which is what
    streamline.js does), then the continuations are captured at the "language"
    level and you can instrument the language runtime to propagate a context
    across continuations and to track performance counters with "long
    stacktrace" semantics. This is completely transparent. The instrumentation
    service is provided by the "language" runtime and does not require any
    special support from the specific libraries that you are using (although I
    haven't tried it this way, streamline-flamegraph recording should work in
    the browser).

    If you don't extend the language with async/await then you need ad-hoc
    mechanisms. You need special mechanisms in node.js to propagate a context
    across continuations. This can be done in APIs like fs that are aligned on
    the continuation callback pattern but it won't work with other APIs that
    use events for continuation (for example a connect call that continues with
    connect/error events, or a read call that continues with readable/error
    events).

    I see a lot of effort being put in trying to provide context propagation
    features around raw callbacks (domains, long-stacktrace, async listeners).
    These APIs are of no use for people like me who are relying on a
    "language-level" construct and who get these features for free from their
    language runtime. They add complexity and they don't seem to hit their
    target (at least so far).

    IMO, raw callbacks should just be raw callbacks. They should be as fast as
    possible and they should not try to deal with context propagation. If
    developers want context propagation they should use a language extension (a
    la async/await) that captures the "continuation threads". They will also
    get other benefits (like robust exception handling).

    Better be explicit about what your program means (express your
    continuations with async/await) than rely on ad-hoc mechanisms that will
    try to reconstruct the continuation semantics from callbacks and special
    APIs.

    My 2 cents.

    Bruno

    --
    Job board: http://jobs.nodejs.org/
    New group rules:
    https://gist.github.com/othiym23/9886289#file-moderation-policy-md
    Old group rules:
    https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
    ---
    You received this message because you are subscribed to a topic in the
    Google Groups "nodejs" group.
    To unsubscribe from this topic, visit
    https://groups.google.com/d/topic/nodejs/XYJih9Jv_40/unsubscribe.
    To unsubscribe from this group and all its topics, send an email to
    nodejs+unsubscribe@googlegroups.com.
    To post to this group, send email to nodejs@googlegroups.com.
    To view this discussion on the web visit
    https://groups.google.com/d/msgid/nodejs/7a021f0f-0f7f-444a-b6e3-e30c27a4871f%40googlegroups.com
    <https://groups.google.com/d/msgid/nodejs/7a021f0f-0f7f-444a-b6e3-e30c27a4871f%40googlegroups.com?utm_medium=email&utm_source=footer>
    .

    For more options, visit https://groups.google.com/d/optout.


    --
    Mit freundlichen Grüßen,

    Andreas Marschke.
    _________________________________________________________
    Web: http://www.andreas-marschke.name
    Blog: http://andreasmarschke.wordpress.com

    --
    Job board: http://jobs.nodejs.org/
    New group rules: https://gist.github.com/othiym23/9886289#file-moderation-policy-md
    Old group rules: https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
    ---
    You received this message because you are subscribed to the Google Groups "nodejs" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to nodejs+unsubscribe@googlegroups.com.
    To post to this group, send email to nodejs@googlegroups.com.
    To view this discussion on the web visit https://groups.google.com/d/msgid/nodejs/CADq4APtj7MW6uJ%2BqipqT_1bBpddYU6ohgNJF1cw2V%3DrebtZbsQ%40mail.gmail.com.
    For more options, visit https://groups.google.com/d/optout.
  • Stephen Belanger at Nov 12, 2014 at 8:36 pm ⇧
    Unfortunately, Google/V8 doesn't really care about nodejs. They only care
    about the browser, for which they deem the WebKit inspector to be
    sufficient. StrongLoop put so much effort into their node-inspector port of
    those tools for that exact reason.
    On Nov 12, 2014 11:39 AM, "Andreas Marschke" wrote:

    +1

    The important question here I guess is : How do you get these library
    neutral instrumentations into the node runtime? Which is also what I
    eluded to asking if would make sense to have people from v8 & node core
    team together with people from the node-community at the same table and
    start working out a standardize solution without stepping on each others
    toes and make node and subsequently v8 a better engine to work with.
    Imagine the benefits the browsers using v8 could foster from a central
    orchestration of changes.

    Just saying.
    On 12 November 2014 13:54, Bruno Jouhier wrote:

    TJ Fontaine did a bit of experimentation with node-tracing, and Forrest
    Norvell made CLS. Trevor Norris had also created AsyncListener to track
    changes between async contexts. But all the approaches so far have been too
    naive or simplistic and ultimately flawed.
    Isn't this a bit of a red herring?

    If you extend the language with an async/await construct (which is what
    streamline.js does), then the continuations are captured at the "language"
    level and you can instrument the language runtime to propagate a context
    across continuations and to track performance counters with "long
    stacktrace" semantics. This is completely transparent. The instrumentation
    service is provided by the "language" runtime and does not require any
    special support from the specific libraries that you are using (although I
    haven't tried it this way, streamline-flamegraph recording should work in
    the browser).

    If you don't extend the language with async/await then you need ad-hoc
    mechanisms. You need special mechanisms in node.js to propagate a context
    across continuations. This can be done in APIs like fs that are aligned on
    the continuation callback pattern but it won't work with other APIs that
    use events for continuation (for example a connect call that continues with
    connect/error events, or a read call that continues with readable/error
    events).

    I see a lot of effort being put in trying to provide context propagation
    features around raw callbacks (domains, long-stacktrace, async listeners).
    These APIs are of no use for people like me who are relying on a
    "language-level" construct and who get these features for free from their
    language runtime. They add complexity and they don't seem to hit their
    target (at least so far).

    IMO, raw callbacks should just be raw callbacks. They should be as fast
    as possible and they should not try to deal with context propagation. If
    developers want context propagation they should use a language extension (a
    la async/await) that captures the "continuation threads". They will also
    get other benefits (like robust exception handling).

    Better be explicit about what your program means (express your
    continuations with async/await) than rely on ad-hoc mechanisms that will
    try to reconstruct the continuation semantics from callbacks and special
    APIs.

    My 2 cents.

    Bruno

    --
    Job board: http://jobs.nodejs.org/
    New group rules:
    https://gist.github.com/othiym23/9886289#file-moderation-policy-md
    Old group rules:
    https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
    ---
    You received this message because you are subscribed to a topic in the
    Google Groups "nodejs" group.
    To unsubscribe from this topic, visit
    https://groups.google.com/d/topic/nodejs/XYJih9Jv_40/unsubscribe.
    To unsubscribe from this group and all its topics, send an email to
    nodejs+unsubscribe@googlegroups.com.
    To post to this group, send email to nodejs@googlegroups.com.
    To view this discussion on the web visit
    https://groups.google.com/d/msgid/nodejs/7a021f0f-0f7f-444a-b6e3-e30c27a4871f%40googlegroups.com
    <https://groups.google.com/d/msgid/nodejs/7a021f0f-0f7f-444a-b6e3-e30c27a4871f%40googlegroups.com?utm_medium=email&utm_source=footer>
    .

    For more options, visit https://groups.google.com/d/optout.


    --
    Mit freundlichen Grüßen,

    Andreas Marschke.
    _________________________________________________________
    Web: http://www.andreas-marschke.name
    Blog: http://andreasmarschke.wordpress.com

    --
    Job board: http://jobs.nodejs.org/
    New group rules:
    https://gist.github.com/othiym23/9886289#file-moderation-policy-md
    Old group rules:
    https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
    ---
    You received this message because you are subscribed to the Google Groups
    "nodejs" group.
    To unsubscribe from this group and stop receiving emails from it, send an
    email to nodejs+unsubscribe@googlegroups.com.
    To post to this group, send email to nodejs@googlegroups.com.
    To view this discussion on the web visit
    https://groups.google.com/d/msgid/nodejs/CADq4APtj7MW6uJ%2BqipqT_1bBpddYU6ohgNJF1cw2V%3DrebtZbsQ%40mail.gmail.com
    <https://groups.google.com/d/msgid/nodejs/CADq4APtj7MW6uJ%2BqipqT_1bBpddYU6ohgNJF1cw2V%3DrebtZbsQ%40mail.gmail.com?utm_medium=email&utm_source=footer>
    .
    For more options, visit https://groups.google.com/d/optout.
    --
    Job board: http://jobs.nodejs.org/
    New group rules: https://gist.github.com/othiym23/9886289#file-moderation-policy-md
    Old group rules: https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
    ---
    You received this message because you are subscribed to the Google Groups "nodejs" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to nodejs+unsubscribe@googlegroups.com.
    To post to this group, send email to nodejs@googlegroups.com.
    To view this discussion on the web visit https://groups.google.com/d/msgid/nodejs/CABcFeBW7PaS9jp_HzF5D%2BrQLAkvUK%3DOQQag9Z%3DcwLaTnKGfifw%40mail.gmail.com.
    For more options, visit https://groups.google.com/d/optout.
  • Andreas Marschke at Nov 12, 2014 at 2:59 pm ⇧

    On Wednesday, 12 November 2014 01:11:54 UTC+1, Ben Noordhuis wrote:
    On Tue, Nov 11, 2014 at 12:04 PM, Andreas Marschke
    <andreas....@gmail.com <javascript:>> wrote:
    Hi Ben,

    thanks for your pointers.

    The fact that I need an API-key and have to (If I understand you
    correctly)
    still send my data to the service provider (in worst case
    even through the same network interface as my data I need to process comes
    in from) means I still have no control over my data
    and sorry but some more security conscious potential customers may take an
    issue with that. Besides the fact that I'll have to
    doubly saturate my existing bandwidth to get it somehwere else is slightly
    questionable. What I actually meant by "master of
    my own data" (granted the terminology is a bit messy here) is 1) I keep my
    data on my premises and get to validate and evaluate
    it myself. A solution (and still use StrongLoop) would be providing an
    appliance or installation package for "on-premise" deployment
    for StrongLoop.
    Maybe I didn't explain it well but in metrics-only mode, no data is
    sent to our collector. Basically, the agent collects metrics
    in-process and your agent.use() callback is the data sink. We have
    more advanced options for clustered applications but that's the basic
    mode of operation.
    Ah my bad. But thanks for the information I will check it out. Thanks.

    Since you said that the landscape is "scattered" regarding performance and
    profiling and the like. Would it be an improvement to
    commence something like a roundtable of the elders to start "Working Group"
    or (in theory more agile than a committee) that would
    define, support and create standards as to what to profile? Maybe even ask
    people from the Companies in the community (ie.
    yourself and breatheren) with blink/v8/node developers together and think of
    a proper way to guide community as a whole towards
    a common goal of a good standard and development of good tools. I've seen it
    work for years in the networking community (see
    RIPE working groups) sure there will be the occasional vitriol and
    unnecessary bikeshedding but if it helps more than a handful of
    developers to better understand the node environment from a performance
    point of view. This could be a sister project to what has
    been instantiated next to the recently unveiled nodejs group.
    There was a "birds of a feather" session in Vancouver in August; my
    co-worker Sam represented StrongLoop.

    To the best of my knowledge, not much came of it and I suspect that's
    because there is not much overlap in the wants and needs of the
    stakeholders. About the only thing everyone could agree on is that it
    would be great if we don't have to monkey-patch everything when
    instrumenting applications and that is what Trevor Norris's
    async-listener work is trying to do.

    The fact that you came to no consensus during the discussion makes me
    wonder how the BoF worked and/or how you went about finding consensus.
    Sure existing tools that could be improved are always there. But the
    question is can you with an acceptable amount of effort improve them to
    service the
    whole or most parts of the node-community.

    The much more (to me at least) frustrating thing about this is that node
    as a scripting language is not the first to experience these painpoints and
    surely
    will not be the last. And I cannot imagine that 99% of the able-minded
    community around NodeJS came from wrinting Jquery to writing Node all day.
    (Not to undermine or devalue those who HAVE come from a pure
    Web-Development Background).

    Most of us have (IME) either a CS-Degree or some form of former knowledge
    towards these things and have seen other projects/languages struggle
    under similar conditions in other areas.

    More to the point: Given that Node/Javascript -- and v8 for that matter --
    came from a Frontend World where performance is king especially when
    improving on a
    byte-level what has been (sometimes) ruined on the BL-level, it bogels my
    mind that there are not much else except the Webtoolkit from Blink/Webkit
    to understand
    what was wrong.

    The question is too how much of what is to be analysed and understood is
    actually Node and not v8 and how have v8-developers been standing towards
    this issue?

    The situation is different from node-forward. With that project, it's
    clear what the problems are that need to be addressed, and there is
    broad consensus on how to make it happen.
    I'll have a look at node-forward. Granted it is not necessarily performance
    focused but being a sort of self-help/support-group makes it kind of
    worthwhile to be checked out,


    --
    Job board: http://jobs.nodejs.org/
    New group rules: https://gist.github.com/othiym23/9886289#file-moderation-policy-md
    Old group rules: https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
    ---
    You received this message because you are subscribed to the Google Groups "nodejs" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to nodejs+unsubscribe@googlegroups.com.
    To post to this group, send email to nodejs@googlegroups.com.
    To view this discussion on the web visit https://groups.google.com/d/msgid/nodejs/c7f6a529-64d1-4cca-aecf-5833897a8e33%40googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
  • Alex Gounares at Nov 14, 2014 at 10:52 pm ⇧
    Hi Andreas

    Another option to consider for performance monitoring is Concurix
    (http://concurix.com). Full disclosure--I am the founder and CEO there!

    We have a fully on premise profiling and monitoring solution for Node.js.
      There is no API key or signup needed, and it runs on stock Node.js on any
    OS.

    Concurix works by injecting performance instrumentation at load time at a
    source level versus using the v8 profiler. This has the advantage of
    providing monitoring/profiling across the entire app, not just a limited
    set of libraries. It also means you can continuously run the profiler,
    even in production. However, it does mean the profiling is at a higher
    semantic level (the function/module level) versus say the v8 profiler.

    Give it a whirl and we'll be happy to help you out!

    regards,
    alex

    alex@concurix.com

    On Tuesday, November 11, 2014 3:04:36 AM UTC-8, Andreas Marschke wrote:

    Hi Ben,

    thanks for your pointers.

    The fact that I need an API-key and have to (If I understand you
    correctly) still send my data to the service provider (in worst case
    even through the same network interface as my data I need to process comes
    in from) means I still have no control over my data
    and sorry but some more security conscious potential customers may take an
    issue with that. Besides the fact that I'll have to
    doubly saturate my existing bandwidth to get it somehwere else is slightly
    questionable. What I actually meant by "master of
    my own data" (granted the terminology is a bit messy here) is 1) I keep my
    data on my premises and get to validate and evaluate
    it myself. A solution (and still use StrongLoop) would be providing an
    appliance or installation package for "on-premise" deployment
    for StrongLoop.

    Since you said that the landscape is "scattered" regarding performance and
    profiling and the like. Would it be an improvement to
    commence something like a roundtable of the elders to start "Working
    Group" or (in theory more agile than a committee) that would
    define, support and create standards as to what to profile? Maybe even ask
    people from the Companies in the community (ie.
    yourself and breatheren) with blink/v8/node developers together and think
    of a proper way to guide community as a whole towards
    a common goal of a good standard and development of good tools. I've seen
    it work for years in the networking community (see
    RIPE working groups) sure there will be the occasional vitriol and
    unnecessary bikeshedding but if it helps more than a handful of
    developers to better understand the node environment from a performance
    point of view. This could be a sister project to what has
    been instantiated next to the recently unveiled nodejs group.

    Just throwing it out there may it'll stick.

    Either way, thanks for the input I will review the sources.

    Cheers,

    Andreas Marschke.
    On Tuesday, 11 November 2014 01:13:28 UTC+1, Ben Noordhuis wrote:

    On Mon, Nov 10, 2014 at 9:03 PM, Andreas Marschke
    wrote:
    Hi,

    first off: Don't get me wrong I like the fact, that there are companys out
    ther supporting nodejs with
    extra services and "consulting". I would however prefer to be the master
    over my data and learn from it
    myself without a frontend that takes away most of the insights they
    (probably) learned the hard way.

    So, my question is this: What do you currently prefer to monitor your
    applications and profile their performance?

    As far as I've seen there is only a limited amount of still viable
    solutions. I'm about to checkout tracegl. But I feel
    that it would be lacking any helpfull understanding as seeing that the
    project itself - in full Mozilla Fashion (see
    i18n-abide for another example) - is barely documented and seems to support
    a limited amount of platforms
    currently(Mostly those that support WebGL and have it enabled, which is not
    the case for me most of the time).

    Other Solutions (ie. node-profiler ) have been gutted to an extend as their
    upstream dependency (v8) has
    removed most of the functionality they so heavily depend on.

    Another pain point as I see it in profiling my nodecode is, that I don't run
    on illu(o)mos which also makes
    30% - 40% of the documentation in the Joyent ("patron" of nodejs) best
    practices for debugging/optimizing
    profiling worthless to me.

    Is there something I'm missing at this point? I would LOVE to
    understand
    where my code may go wrong or
    where it is excessively brittle performance-wise, but at this stage it is
    very hard for me to see much in the
    way of a proper toolchain here.

    Have there been any noteworthy projects around Node performance that I have
    simply been missing?
    Is it worth reading the Node C/C++ code to get a full grasp of how it works?
    Is it simply an issue of p
    promotion in the sense that not all parts of the community have
    embraced it?
    I am happy for any and every suggestions you may write underneath this. Feel
    free also to lambast me on my
    maybe ignorant views towards services like nodetime and StrongLoop.

    Thanks in advance,

    Andreas Marschke,
    Full disclosure: I'm the author of node-profiler and node-heapdump and
    a StrongLoop founder.

    Thorsten Lorenz maintains a curated list of profiling and debugging
    tools, see [0]. Be warned that the free software landscape is still
    pretty scattered and that won't change anytime soon. Writing good,
    integrated performance/debugging tools takes time and is not very
    rewarding in itself; most of the existing tools scratch a particular
    itch and stop there.

    If I can circle back to a comment you made at the start of your post -
    being the master of your own data; the StrongLoop agent is moving to a
    model where it simply collects the raw or aggregated metrics and
    leaves it up to you to process or store it. We provide integration
    for popular services like statsd[1], Splunk[2], Datadog[3], etc. Here
    is an example of our statsd integration[4]:

    var agent = require('strong-agent');
    var statsd = require('strong-agent-statsd')();
    agent.use(statsd); // that's all, metrics are now reported to statsd

    You can find more details in the README[5]; scroll down to the section
    on the metrics API. Questions or suggestions welcome.

    [0] https://github.com/thlorenz/v8-perf

    [1] https://github.com/etsy/statsd

    [2] http://www.splunk.com/

    [3] https://www.datadoghq.com/

    [4] https://github.com/strongloop/strong-agent-statsd

    [5] https://github.com/strongloop/strong-agent/blob/master/README.md
    --
    Job board: http://jobs.nodejs.org/
    New group rules: https://gist.github.com/othiym23/9886289#file-moderation-policy-md
    Old group rules: https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
    ---
    You received this message because you are subscribed to the Google Groups "nodejs" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to nodejs+unsubscribe@googlegroups.com.
    To post to this group, send email to nodejs@googlegroups.com.
    To view this discussion on the web visit https://groups.google.com/d/msgid/nodejs/ed74412d-d65c-4937-9afb-c3e60da51bce%40googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
  • Andreas Marschke at Nov 15, 2014 at 5:58 am ⇧
    Hi Alex,

    thank you very much. Has any of the work that you and your company have
    made filtered back to the community in any way? Just interested.
    Because as we've already realized in this Thread. The tools in the
    community is very barebones or not entirely interoperable or anything.

    At best -- to my mind -- there should a better (maybe even standardized?)
    toolset or set of libraries to better understand the layers and what is
    going on in the application.
    Sure Node being Node means some optimizations are already there (v8 coming
    from Chrom(e/ium)) but you still have javascript as your "FootGun" (see
    Crockford).

    Like I suggested there could/should be a concerted effort towards that.

    Thanks!

    <!-- please stop me if I've become rambly about this... -->
    On Friday, 14 November 2014 23:48:28 UTC+1, Alex Gounares wrote:

    Hi Andreas

    Another option to consider for performance monitoring is Concurix (
    http://concurix.com
    <http://www.google.com/url?q=http%3A%2F%2Fconcurix.com&sa=D&sntz=1&usg=AFQjCNGPxiJSn_vx9yQseOZiyKqy3u0InA>).
    Full disclosure--I am the founder and CEO there!

    We have a fully on premise profiling and monitoring solution for Node.js.
    There is no API key or signup needed, and it runs on stock Node.js on any
    OS.

    Concurix works by injecting performance instrumentation at load time at a
    source level versus using the v8 profiler. This has the advantage of
    providing monitoring/profiling across the entire app, not just a limited
    set of libraries. It also means you can continuously run the profiler,
    even in production. However, it does mean the profiling is at a higher
    semantic level (the function/module level) versus say the v8 profiler.

    Give it a whirl and we'll be happy to help you out!

    regards,
    alex

    al...@concurix.com <javascript:>

    On Tuesday, November 11, 2014 3:04:36 AM UTC-8, Andreas Marschke wrote:

    Hi Ben,

    thanks for your pointers.

    The fact that I need an API-key and have to (If I understand you
    correctly) still send my data to the service provider (in worst case
    even through the same network interface as my data I need to process
    comes in from) means I still have no control over my data
    and sorry but some more security conscious potential customers may take
    an issue with that. Besides the fact that I'll have to
    doubly saturate my existing bandwidth to get it somehwere else is
    slightly questionable. What I actually meant by "master of
    my own data" (granted the terminology is a bit messy here) is 1) I keep
    my data on my premises and get to validate and evaluate
    it myself. A solution (and still use StrongLoop) would be providing an
    appliance or installation package for "on-premise" deployment
    for StrongLoop.

    Since you said that the landscape is "scattered" regarding performance
    and profiling and the like. Would it be an improvement to
    commence something like a roundtable of the elders to start "Working
    Group" or (in theory more agile than a committee) that would
    define, support and create standards as to what to profile? Maybe even
    ask people from the Companies in the community (ie.
    yourself and breatheren) with blink/v8/node developers together and think
    of a proper way to guide community as a whole towards
    a common goal of a good standard and development of good tools. I've seen
    it work for years in the networking community (see
    RIPE working groups) sure there will be the occasional vitriol and
    unnecessary bikeshedding but if it helps more than a handful of
    developers to better understand the node environment from a performance
    point of view. This could be a sister project to what has
    been instantiated next to the recently unveiled nodejs group.

    Just throwing it out there may it'll stick.

    Either way, thanks for the input I will review the sources.

    Cheers,

    Andreas Marschke.
    On Tuesday, 11 November 2014 01:13:28 UTC+1, Ben Noordhuis wrote:

    On Mon, Nov 10, 2014 at 9:03 PM, Andreas Marschke
    wrote:
    Hi,

    first off: Don't get me wrong I like the fact, that there are companys out
    ther supporting nodejs with
    extra services and "consulting". I would however prefer to be the master
    over my data and learn from it
    myself without a frontend that takes away most of the insights they
    (probably) learned the hard way.

    So, my question is this: What do you currently prefer to monitor your
    applications and profile their performance?

    As far as I've seen there is only a limited amount of still viable
    solutions. I'm about to checkout tracegl. But I feel
    that it would be lacking any helpfull understanding as seeing that the
    project itself - in full Mozilla Fashion (see
    i18n-abide for another example) - is barely documented and seems to support
    a limited amount of platforms
    currently(Mostly those that support WebGL and have it enabled, which is not
    the case for me most of the time).

    Other Solutions (ie. node-profiler ) have been gutted to an extend as their
    upstream dependency (v8) has
    removed most of the functionality they so heavily depend on.

    Another pain point as I see it in profiling my nodecode is, that I don't run
    on illu(o)mos which also makes
    30% - 40% of the documentation in the Joyent ("patron" of nodejs) best
    practices for debugging/optimizing
    profiling worthless to me.

    Is there something I'm missing at this point? I would LOVE to
    understand
    where my code may go wrong or
    where it is excessively brittle performance-wise, but at this stage it is
    very hard for me to see much in the
    way of a proper toolchain here.

    Have there been any noteworthy projects around Node performance that I have
    simply been missing?
    Is it worth reading the Node C/C++ code to get a full grasp of how it works?
    Is it simply an issue of p
    promotion in the sense that not all parts of the community have
    embraced it?
    I am happy for any and every suggestions you may write underneath
    this. Feel
    free also to lambast me on my
    maybe ignorant views towards services like nodetime and StrongLoop.

    Thanks in advance,

    Andreas Marschke,
    Full disclosure: I'm the author of node-profiler and node-heapdump and
    a StrongLoop founder.

    Thorsten Lorenz maintains a curated list of profiling and debugging
    tools, see [0]. Be warned that the free software landscape is still
    pretty scattered and that won't change anytime soon. Writing good,
    integrated performance/debugging tools takes time and is not very
    rewarding in itself; most of the existing tools scratch a particular
    itch and stop there.

    If I can circle back to a comment you made at the start of your post -
    being the master of your own data; the StrongLoop agent is moving to a
    model where it simply collects the raw or aggregated metrics and
    leaves it up to you to process or store it. We provide integration
    for popular services like statsd[1], Splunk[2], Datadog[3], etc. Here
    is an example of our statsd integration[4]:

    var agent = require('strong-agent');
    var statsd = require('strong-agent-statsd')();
    agent.use(statsd); // that's all, metrics are now reported to statsd

    You can find more details in the README[5]; scroll down to the section
    on the metrics API. Questions or suggestions welcome.

    [0] https://github.com/thlorenz/v8-perf

    [1] https://github.com/etsy/statsd

    [2] http://www.splunk.com/

    [3] https://www.datadoghq.com/

    [4] https://github.com/strongloop/strong-agent-statsd

    [5] https://github.com/strongloop/strong-agent/blob/master/README.md
    --
    Job board: http://jobs.nodejs.org/
    New group rules: https://gist.github.com/othiym23/9886289#file-moderation-policy-md
    Old group rules: https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
    ---
    You received this message because you are subscribed to the Google Groups "nodejs" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to nodejs+unsubscribe@googlegroups.com.
    To post to this group, send email to nodejs@googlegroups.com.
    To view this discussion on the web visit https://groups.google.com/d/msgid/nodejs/0ff79bd8-ce16-4a11-952e-1a2a81ebc95b%40googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
  • Nodar Chkuaselidze at Nov 15, 2014 at 3:05 pm ⇧
    Is there the SET of tools that you can work while being offline ?

    so you don't need to communicate to other server, login to other system...
    Just like node-inspector, but for profiling and monitoring ?
    On Saturday, November 15, 2014 7:05:33 AM UTC+4, Andreas Marschke wrote:

    Hi Alex,

    thank you very much. Has any of the work that you and your company have
    made filtered back to the community in any way? Just interested.
    Because as we've already realized in this Thread. The tools in the
    community is very barebones or not entirely interoperable or anything.

    At best -- to my mind -- there should a better (maybe even standardized?)
    toolset or set of libraries to better understand the layers and what is
    going on in the application.
    Sure Node being Node means some optimizations are already there (v8 coming
    from Chrom(e/ium)) but you still have javascript as your "FootGun" (see
    Crockford).

    Like I suggested there could/should be a concerted effort towards that.

    Thanks!

    <!-- please stop me if I've become rambly about this... -->
    On Friday, 14 November 2014 23:48:28 UTC+1, Alex Gounares wrote:

    Hi Andreas

    Another option to consider for performance monitoring is Concurix (
    http://concurix.com
    <http://www.google.com/url?q=http%3A%2F%2Fconcurix.com&sa=D&sntz=1&usg=AFQjCNGPxiJSn_vx9yQseOZiyKqy3u0InA>).
    Full disclosure--I am the founder and CEO there!

    We have a fully on premise profiling and monitoring solution for Node.js.
    There is no API key or signup needed, and it runs on stock Node.js on any
    OS.

    Concurix works by injecting performance instrumentation at load time at a
    source level versus using the v8 profiler. This has the advantage of
    providing monitoring/profiling across the entire app, not just a limited
    set of libraries. It also means you can continuously run the profiler,
    even in production. However, it does mean the profiling is at a higher
    semantic level (the function/module level) versus say the v8 profiler.

    Give it a whirl and we'll be happy to help you out!

    regards,
    alex

    al...@concurix.com

    On Tuesday, November 11, 2014 3:04:36 AM UTC-8, Andreas Marschke wrote:

    Hi Ben,

    thanks for your pointers.

    The fact that I need an API-key and have to (If I understand you
    correctly) still send my data to the service provider (in worst case
    even through the same network interface as my data I need to process
    comes in from) means I still have no control over my data
    and sorry but some more security conscious potential customers may take
    an issue with that. Besides the fact that I'll have to
    doubly saturate my existing bandwidth to get it somehwere else is
    slightly questionable. What I actually meant by "master of
    my own data" (granted the terminology is a bit messy here) is 1) I keep
    my data on my premises and get to validate and evaluate
    it myself. A solution (and still use StrongLoop) would be providing an
    appliance or installation package for "on-premise" deployment
    for StrongLoop.

    Since you said that the landscape is "scattered" regarding performance
    and profiling and the like. Would it be an improvement to
    commence something like a roundtable of the elders to start "Working
    Group" or (in theory more agile than a committee) that would
    define, support and create standards as to what to profile? Maybe even
    ask people from the Companies in the community (ie.
    yourself and breatheren) with blink/v8/node developers together and
    think of a proper way to guide community as a whole towards
    a common goal of a good standard and development of good tools. I've
    seen it work for years in the networking community (see
    RIPE working groups) sure there will be the occasional vitriol and
    unnecessary bikeshedding but if it helps more than a handful of
    developers to better understand the node environment from a performance
    point of view. This could be a sister project to what has
    been instantiated next to the recently unveiled nodejs group.

    Just throwing it out there may it'll stick.

    Either way, thanks for the input I will review the sources.

    Cheers,

    Andreas Marschke.
    On Tuesday, 11 November 2014 01:13:28 UTC+1, Ben Noordhuis wrote:

    On Mon, Nov 10, 2014 at 9:03 PM, Andreas Marschke
    wrote:
    Hi,

    first off: Don't get me wrong I like the fact, that there are
    companys out
    ther supporting nodejs with
    extra services and "consulting". I would however prefer to be the master
    over my data and learn from it
    myself without a frontend that takes away most of the insights they
    (probably) learned the hard way.

    So, my question is this: What do you currently prefer to monitor your
    applications and profile their performance?

    As far as I've seen there is only a limited amount of still viable
    solutions. I'm about to checkout tracegl. But I feel
    that it would be lacking any helpfull understanding as seeing that the
    project itself - in full Mozilla Fashion (see
    i18n-abide for another example) - is barely documented and seems to support
    a limited amount of platforms
    currently(Mostly those that support WebGL and have it enabled, which is not
    the case for me most of the time).

    Other Solutions (ie. node-profiler ) have been gutted to an extend as their
    upstream dependency (v8) has
    removed most of the functionality they so heavily depend on.

    Another pain point as I see it in profiling my nodecode is, that I don't run
    on illu(o)mos which also makes
    30% - 40% of the documentation in the Joyent ("patron" of nodejs) best
    practices for debugging/optimizing
    profiling worthless to me.

    Is there something I'm missing at this point? I would LOVE to
    understand
    where my code may go wrong or
    where it is excessively brittle performance-wise, but at this stage it is
    very hard for me to see much in the
    way of a proper toolchain here.

    Have there been any noteworthy projects around Node performance that I have
    simply been missing?
    Is it worth reading the Node C/C++ code to get a full grasp of how it works?
    Is it simply an issue of p
    promotion in the sense that not all parts of the community have
    embraced it?
    I am happy for any and every suggestions you may write underneath
    this. Feel
    free also to lambast me on my
    maybe ignorant views towards services like nodetime and StrongLoop.

    Thanks in advance,

    Andreas Marschke,
    Full disclosure: I'm the author of node-profiler and node-heapdump and
    a StrongLoop founder.

    Thorsten Lorenz maintains a curated list of profiling and debugging
    tools, see [0]. Be warned that the free software landscape is still
    pretty scattered and that won't change anytime soon. Writing good,
    integrated performance/debugging tools takes time and is not very
    rewarding in itself; most of the existing tools scratch a particular
    itch and stop there.

    If I can circle back to a comment you made at the start of your post -
    being the master of your own data; the StrongLoop agent is moving to a
    model where it simply collects the raw or aggregated metrics and
    leaves it up to you to process or store it. We provide integration
    for popular services like statsd[1], Splunk[2], Datadog[3], etc. Here
    is an example of our statsd integration[4]:

    var agent = require('strong-agent');
    var statsd = require('strong-agent-statsd')();
    agent.use(statsd); // that's all, metrics are now reported to statsd

    You can find more details in the README[5]; scroll down to the section
    on the metrics API. Questions or suggestions welcome.

    [0] https://github.com/thlorenz/v8-perf

    [1] https://github.com/etsy/statsd

    [2] http://www.splunk.com/

    [3] https://www.datadoghq.com/

    [4] https://github.com/strongloop/strong-agent-statsd

    [5] https://github.com/strongloop/strong-agent/blob/master/README.md
    --
    Job board: http://jobs.nodejs.org/
    New group rules: https://gist.github.com/othiym23/9886289#file-moderation-policy-md
    Old group rules: https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
    ---
    You received this message because you are subscribed to the Google Groups "nodejs" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to nodejs+unsubscribe@googlegroups.com.
    To post to this group, send email to nodejs@googlegroups.com.
    To view this discussion on the web visit https://groups.google.com/d/msgid/nodejs/93017219-01e7-40ec-9738-0cfe6ff8dc75%40googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
  • Mark Getz at Nov 17, 2014 at 4:53 pm ⇧
    Alex,
    Does your code work well with Streamlined code yet? Liked your software but
    in earlier tests the overhead brought our system down because we used
    streamline which adds a lot of extra functions when it rewrites the code.

    We are currently writing our own flavor of performance monitoring using the
    hooks provided in streamline. It adds a 10% overhead at the moment but
    looks like it will give us good timing and code path metrics for our
    system. Most of the overhead time when using the streamline hooks is in
    saving the data. Collecting the data is minimal overhead.

    Thanks.

    --
    Job board: http://jobs.nodejs.org/
    New group rules: https://gist.github.com/othiym23/9886289#file-moderation-policy-md
    Old group rules: https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
    ---
    You received this message because you are subscribed to the Google Groups "nodejs" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to nodejs+unsubscribe@googlegroups.com.
    To post to this group, send email to nodejs@googlegroups.com.
    To view this discussion on the web visit https://groups.google.com/d/msgid/nodejs/b4f22356-112d-4da5-b594-b98530fde0bc%40googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupnodejs @
categoriesnodejs
postedNov 10, '14 at 8:37p
activeNov 17, '14 at 4:53p
posts15
users8
websitenodejs.org
irc#node.js

8 users in discussion

Andreas Marschke: 5 posts Ben Noordhuis: 2 posts Stephen Belanger: 2 posts Bruno Jouhier: 2 posts Alex Gounares: 1 post // ravi: 1 post Mark Getz: 1 post Nodar Chkuaselidze: 1 post

Content

People

Support

Translate

site design / logo © 2022 Grokbase