I'm really close to having a fully functional Loggregator client in Java
for both emitting logs as well as reading logs over Web Sockets.

There a few things that are bugging me so I thought I'd provide some
feedback:

- I'm not very fond of the DIY padding. PKCS#7 is really really easy to
implement (since Go doesn't appear to support it.) Why not go with this
standard padding mechanism? Regardless, it works but it was a pain to
figure out and reimplement.
- The signature in the LogEnvelope only covers the LogMessage's message
field and not the entire LogMessage. This probably isn't a huge problem
since it would still be rather difficult to intercept a LogEnvelope but if
you did, you could emit the same message to any application masquerading as
coming from any source. It seems more secure, to apply the digital
signature to the entire LogMessage but again, not a big deal.
- The nonsecond time resolution in the timestamp seems a little over the
top but I don't anticipate being around on April 11, 2262 when it overflows
so I'm not too concerned. :)

I'll post a link to my client once it's ready.

Thanks,

-Mike

To unsubscribe from this group and stop receiving emails from it, send an email to vcap-dev+unsubscribe@cloudfoundry.org.

Search Discussions

  • Dr Nic Williams at Jan 18, 2014 at 1:28 am
    Nice work Mike!
    On Fri, Jan 17, 2014 at 5:21 PM, Mike Heath wrote:

    I'm really close to having a fully functional Loggregator client in Java
    for both emitting logs as well as reading logs over Web Sockets.
    There a few things that are bugging me so I thought I'd provide some
    feedback:
    - I'm not very fond of the DIY padding. PKCS#7 is really really easy to
    implement (since Go doesn't appear to support it.) Why not go with this
    standard padding mechanism? Regardless, it works but it was a pain to
    figure out and reimplement.
    - The signature in the LogEnvelope only covers the LogMessage's message
    field and not the entire LogMessage. This probably isn't a huge problem
    since it would still be rather difficult to intercept a LogEnvelope but if
    you did, you could emit the same message to any application masquerading as
    coming from any source. It seems more secure, to apply the digital
    signature to the entire LogMessage but again, not a big deal.
    - The nonsecond time resolution in the timestamp seems a little over the
    top but I don't anticipate being around on April 11, 2262 when it overflows
    so I'm not too concerned. :)
    I'll post a link to my client once it's ready.
    Thanks,
    -Mike
    To unsubscribe from this group and stop receiving emails from it, send an email to vcap-dev+unsubscribe@cloudfoundry.org.
    To unsubscribe from this group and stop receiving emails from it, send an email to vcap-dev+unsubscribe@cloudfoundry.org.
  • Mike Heath at Jan 22, 2014 at 1:04 am
    Has the loggregator team considered using Gary's Burd's WebSocket
    implementation
    (http://godoc.org/github.com/garyburd/go-websocket/websocket) instead of
    Go's built-in WebSocket library? Go's built-in library doesn't appear to
    support ping/pong frames (which seems weird that a Google invented protocol
    would have a half-baked implementation in a Google runtime...)

    It seems kind of silly to do stuff like this
    https://github.com/cloudfoundry/cli/blob/master/src/cf/api/logs.go#L125-L130
    rather than using the WebSocket protocol's built-in mechanism for detecting
    stale connections.

    Forgive me if I sound like a whiner. I love Loggregator. I just keep
    finding things in Loggregator that are done in a way that's easy with Go
    but non-standard which causes more effort for those of us on more
    established platforms.

    -Mike
    On Friday, January 17, 2014 6:21:24 PM UTC-7, Mike Heath wrote:

    I'm really close to having a fully functional Loggregator client in Java
    for both emitting logs as well as reading logs over Web Sockets.

    There a few things that are bugging me so I thought I'd provide some
    feedback:

    - I'm not very fond of the DIY padding. PKCS#7 is really really easy to
    implement (since Go doesn't appear to support it.) Why not go with this
    standard padding mechanism? Regardless, it works but it was a pain to
    figure out and reimplement.
    - The signature in the LogEnvelope only covers the LogMessage's message
    field and not the entire LogMessage. This probably isn't a huge problem
    since it would still be rather difficult to intercept a LogEnvelope but if
    you did, you could emit the same message to any application masquerading as
    coming from any source. It seems more secure, to apply the digital
    signature to the entire LogMessage but again, not a big deal.
    - The nonsecond time resolution in the timestamp seems a little over the
    top but I don't anticipate being around on April 11, 2262 when it overflows
    so I'm not too concerned. :)

    I'll post a link to my client once it's ready.

    Thanks,

    -Mike
    To unsubscribe from this group and stop receiving emails from it, send an email to vcap-dev+unsubscribe@cloudfoundry.org.
  • Johannes Tuchscherer at Jan 22, 2014 at 4:13 pm
    Hi Mike,

    Thanks for digging through the code and coming up with improvements. We
    really appreciate that.

    That WebSocket implementation looks interesting and powerful. Indeed, we
    could rip out some code from the loggregator and the CLI by switching over
    to that implementation. BTW, it looks like this library got merged into the
    Gorilla project and is now available here:
    https://github.com/gorilla/websocket

    Switching to that implementation, though, comes at the cost of increasing
    the number of our third party dependencies. Considering we already wrote
    our keep-alive logic, we don't get an immediate benefit from switching
    right now. The other difficulty comes from the coupling between loggregator
    an CLI. If we use a new WebSocket implementation on the server, we have to
    update the CLI at the same time and we would need to handle both, the
    ping-pong frames and the old implementation for backwards compatibility
    with existing clients.

    Nevertheless, we see the benefit for the community in switching over to a
    more standard conform implementation. I will put in a story in our Pivotal
    Tracker Icebox and talk to our Product Owner about it.

    Thanks again for coming up with all these suggestions,
    Johannes
    On Tuesday, January 21, 2014 6:04:16 PM UTC-7, Mike Heath wrote:

    Has the loggregator team considered using Gary's Burd's WebSocket
    implementation (
    http://godoc.org/github.com/garyburd/go-websocket/websocket) instead of
    Go's built-in WebSocket library? Go's built-in library doesn't appear to
    support ping/pong frames (which seems weird that a Google invented protocol
    would have a half-baked implementation in a Google runtime...)

    It seems kind of silly to do stuff like this
    https://github.com/cloudfoundry/cli/blob/master/src/cf/api/logs.go#L125-L130rather than using the WebSocket protocol's built-in mechanism for detecting
    stale connections.

    Forgive me if I sound like a whiner. I love Loggregator. I just keep
    finding things in Loggregator that are done in a way that's easy with Go
    but non-standard which causes more effort for those of us on more
    established platforms.

    -Mike
    On Friday, January 17, 2014 6:21:24 PM UTC-7, Mike Heath wrote:

    I'm really close to having a fully functional Loggregator client in Java
    for both emitting logs as well as reading logs over Web Sockets.

    There a few things that are bugging me so I thought I'd provide some
    feedback:

    - I'm not very fond of the DIY padding. PKCS#7 is really really easy to
    implement (since Go doesn't appear to support it.) Why not go with this
    standard padding mechanism? Regardless, it works but it was a pain to
    figure out and reimplement.
    - The signature in the LogEnvelope only covers the LogMessage's message
    field and not the entire LogMessage. This probably isn't a huge problem
    since it would still be rather difficult to intercept a LogEnvelope but if
    you did, you could emit the same message to any application masquerading as
    coming from any source. It seems more secure, to apply the digital
    signature to the entire LogMessage but again, not a big deal.
    - The nonsecond time resolution in the timestamp seems a little over the
    top but I don't anticipate being around on April 11, 2262 when it overflows
    so I'm not too concerned. :)

    I'll post a link to my client once it's ready.

    Thanks,

    -Mike
    To unsubscribe from this group and stop receiving emails from it, send an email to vcap-dev+unsubscribe@cloudfoundry.org.
  • Mike Heath at Jan 22, 2014 at 4:37 pm
    Hey Johannes,

    Thanks for taking a look at this.

    You could modify the server without modifying the client. The server could
    periodically send ping frames. If the server does not receive any frames (a
    pong frame or otherwise) after a certain threshold, it assume the
    connection is stale and closes it. This will work with the existing the
    clients that send an "I'm still here!" text frame and will work with newer
    clients that use ping/pong. Clients going forward can then close their
    connection (or try to ping the server) if they don't receive any frames
    after a certain amount of time.

    Just a suggestion. Thanks again for the response.

    -Mike
    On Wed, Jan 22, 2014 at 9:13 AM, Johannes Tuchscherer wrote:

    Hi Mike,

    Thanks for digging through the code and coming up with improvements. We
    really appreciate that.

    That WebSocket implementation looks interesting and powerful. Indeed, we
    could rip out some code from the loggregator and the CLI by switching over
    to that implementation. BTW, it looks like this library got merged into the
    Gorilla project and is now available here: https://github.com/gorilla/
    websocket

    Switching to that implementation, though, comes at the cost of increasing
    the number of our third party dependencies. Considering we already wrote
    our keep-alive logic, we don't get an immediate benefit from switching
    right now. The other difficulty comes from the coupling between loggregator
    an CLI. If we use a new WebSocket implementation on the server, we have to
    update the CLI at the same time and we would need to handle both, the
    ping-pong frames and the old implementation for backwards compatibility
    with existing clients.

    Nevertheless, we see the benefit for the community in switching over to a
    more standard conform implementation. I will put in a story in our Pivotal
    Tracker Icebox and talk to our Product Owner about it.

    Thanks again for coming up with all these suggestions,
    Johannes
    On Tuesday, January 21, 2014 6:04:16 PM UTC-7, Mike Heath wrote:

    Has the loggregator team considered using Gary's Burd's WebSocket
    implementation (http://godoc.org/github.com/garyburd/go-websocket/
    websocket) instead of Go's built-in WebSocket library? Go's built-in
    library doesn't appear to support ping/pong frames (which seems weird that
    a Google invented protocol would have a half-baked implementation in a
    Google runtime...)

    It seems kind of silly to do stuff like this https://github.com/
    cloudfoundry/cli/blob/master/src/cf/api/logs.go#L125-L130 rather than
    using the WebSocket protocol's built-in mechanism for detecting stale
    connections.

    Forgive me if I sound like a whiner. I love Loggregator. I just keep
    finding things in Loggregator that are done in a way that's easy with Go
    but non-standard which causes more effort for those of us on more
    established platforms.

    -Mike
    On Friday, January 17, 2014 6:21:24 PM UTC-7, Mike Heath wrote:

    I'm really close to having a fully functional Loggregator client in Java
    for both emitting logs as well as reading logs over Web Sockets.

    There a few things that are bugging me so I thought I'd provide some
    feedback:

    - I'm not very fond of the DIY padding. PKCS#7 is really really easy to
    implement (since Go doesn't appear to support it.) Why not go with this
    standard padding mechanism? Regardless, it works but it was a pain to
    figure out and reimplement.
    - The signature in the LogEnvelope only covers the LogMessage's message
    field and not the entire LogMessage. This probably isn't a huge problem
    since it would still be rather difficult to intercept a LogEnvelope but if
    you did, you could emit the same message to any application masquerading as
    coming from any source. It seems more secure, to apply the digital
    signature to the entire LogMessage but again, not a big deal.
    - The nonsecond time resolution in the timestamp seems a little over the
    top but I don't anticipate being around on April 11, 2262 when it overflows
    so I'm not too concerned. :)

    I'll post a link to my client once it's ready.

    Thanks,

    -Mike
    To unsubscribe from this group and stop receiving emails from it, send an email to vcap-dev+unsubscribe@cloudfoundry.org.
  • Patrick Mueller at Jan 22, 2014 at 7:40 pm
    Has any thought been given to also making log data available via
    server-sent events [1]?

    I've only used these once, with Netflix's Hystrix dashboard, and found it
    SUPER EASY to generate an event stream to feed the Hystrix dashboard. I
    assume it's just as easy for code to consume it.

    I'm not sure what all loggregator can do - server-sent events are an
    output-only thing - you only interact with it by reading the stream, no
    writing once it's started. Of course, you could use URL patterns, query
    parameters on the URL, etc to select streams, filter, etc.

    [1] http://www.w3.org/TR/eventsource/

    On Wed, Jan 22, 2014 at 11:37 AM, Mike Heath wrote:

    Hey Johannes,

    Thanks for taking a look at this.

    You could modify the server without modifying the client. The server could
    periodically send ping frames. If the server does not receive any frames (a
    pong frame or otherwise) after a certain threshold, it assume the
    connection is stale and closes it. This will work with the existing the
    clients that send an "I'm still here!" text frame and will work with newer
    clients that use ping/pong. Clients going forward can then close their
    connection (or try to ping the server) if they don't receive any frames
    after a certain amount of time.

    Just a suggestion. Thanks again for the response.

    -Mike

    On Wed, Jan 22, 2014 at 9:13 AM, Johannes Tuchscherer <
    jtuchscherer@pivotallabs.com> wrote:
    Hi Mike,

    Thanks for digging through the code and coming up with improvements. We
    really appreciate that.

    That WebSocket implementation looks interesting and powerful. Indeed, we
    could rip out some code from the loggregator and the CLI by switching over
    to that implementation. BTW, it looks like this library got merged into the
    Gorilla project and is now available here: https://github.com/gorilla/
    websocket

    Switching to that implementation, though, comes at the cost of increasing
    the number of our third party dependencies. Considering we already wrote
    our keep-alive logic, we don't get an immediate benefit from switching
    right now. The other difficulty comes from the coupling between loggregator
    an CLI. If we use a new WebSocket implementation on the server, we have to
    update the CLI at the same time and we would need to handle both, the
    ping-pong frames and the old implementation for backwards compatibility
    with existing clients.

    Nevertheless, we see the benefit for the community in switching over to a
    more standard conform implementation. I will put in a story in our Pivotal
    Tracker Icebox and talk to our Product Owner about it.

    Thanks again for coming up with all these suggestions,
    Johannes
    On Tuesday, January 21, 2014 6:04:16 PM UTC-7, Mike Heath wrote:

    Has the loggregator team considered using Gary's Burd's WebSocket
    implementation (http://godoc.org/github.com/garyburd/go-websocket/
    websocket) instead of Go's built-in WebSocket library? Go's built-in
    library doesn't appear to support ping/pong frames (which seems weird that
    a Google invented protocol would have a half-baked implementation in a
    Google runtime...)

    It seems kind of silly to do stuff like this https://github.com/
    cloudfoundry/cli/blob/master/src/cf/api/logs.go#L125-L130 rather than
    using the WebSocket protocol's built-in mechanism for detecting stale
    connections.

    Forgive me if I sound like a whiner. I love Loggregator. I just keep
    finding things in Loggregator that are done in a way that's easy with Go
    but non-standard which causes more effort for those of us on more
    established platforms.

    -Mike
    On Friday, January 17, 2014 6:21:24 PM UTC-7, Mike Heath wrote:

    I'm really close to having a fully functional Loggregator client in
    Java for both emitting logs as well as reading logs over Web Sockets.

    There a few things that are bugging me so I thought I'd provide some
    feedback:

    - I'm not very fond of the DIY padding. PKCS#7 is really really easy to
    implement (since Go doesn't appear to support it.) Why not go with this
    standard padding mechanism? Regardless, it works but it was a pain to
    figure out and reimplement.
    - The signature in the LogEnvelope only covers the LogMessage's message
    field and not the entire LogMessage. This probably isn't a huge problem
    since it would still be rather difficult to intercept a LogEnvelope but if
    you did, you could emit the same message to any application masquerading as
    coming from any source. It seems more secure, to apply the digital
    signature to the entire LogMessage but again, not a big deal.
    - The nonsecond time resolution in the timestamp seems a little over
    the top but I don't anticipate being around on April 11, 2262 when it
    overflows so I'm not too concerned. :)

    I'll post a link to my client once it's ready.

    Thanks,

    -Mike
    To unsubscribe from this group and stop receiving emails from it, send an
    email to vcap-dev+unsubscribe@cloudfoundry.org.


    --
    Patrick Mueller
    http://muellerware.org

    To unsubscribe from this group and stop receiving emails from it, send an email to vcap-dev+unsubscribe@cloudfoundry.org.
  • David Lee at Jan 22, 2014 at 8:15 pm
    Patrick,

    A goal of this websocket work is to enable a large number of clients to
    access the log information. Websockets has better support across browsers
    (SSE is missing in IE).

    Other than Hysterix (which doesn't seem to be a logging consumer really),
    do you know of any other clients that primarily accept SSE?

    -Dave


    On Wed, Jan 22, 2014 at 11:40 AM, Patrick Mueller wrote:

    Has any thought been given to also making log data available via
    server-sent events [1]?

    I've only used these once, with Netflix's Hystrix dashboard, and found it
    SUPER EASY to generate an event stream to feed the Hystrix dashboard. I
    assume it's just as easy for code to consume it.

    I'm not sure what all loggregator can do - server-sent events are an
    output-only thing - you only interact with it by reading the stream, no
    writing once it's started. Of course, you could use URL patterns, query
    parameters on the URL, etc to select streams, filter, etc.

    [1] http://www.w3.org/TR/eventsource/

    On Wed, Jan 22, 2014 at 11:37 AM, Mike Heath wrote:

    Hey Johannes,

    Thanks for taking a look at this.

    You could modify the server without modifying the client. The server
    could periodically send ping frames. If the server does not receive any
    frames (a pong frame or otherwise) after a certain threshold, it assume the
    connection is stale and closes it. This will work with the existing the
    clients that send an "I'm still here!" text frame and will work with newer
    clients that use ping/pong. Clients going forward can then close their
    connection (or try to ping the server) if they don't receive any frames
    after a certain amount of time.

    Just a suggestion. Thanks again for the response.

    -Mike

    On Wed, Jan 22, 2014 at 9:13 AM, Johannes Tuchscherer <
    jtuchscherer@pivotallabs.com> wrote:
    Hi Mike,

    Thanks for digging through the code and coming up with improvements. We
    really appreciate that.

    That WebSocket implementation looks interesting and powerful. Indeed, we
    could rip out some code from the loggregator and the CLI by switching over
    to that implementation. BTW, it looks like this library got merged into the
    Gorilla project and is now available here: https://github.com/gorilla/
    websocket

    Switching to that implementation, though, comes at the cost of
    increasing the number of our third party dependencies. Considering we
    already wrote our keep-alive logic, we don't get an immediate benefit from
    switching right now. The other difficulty comes from the coupling between
    loggregator an CLI. If we use a new WebSocket implementation on the server,
    we have to update the CLI at the same time and we would need to handle
    both, the ping-pong frames and the old implementation for backwards
    compatibility with existing clients.

    Nevertheless, we see the benefit for the community in switching over to
    a more standard conform implementation. I will put in a story in our
    Pivotal Tracker Icebox and talk to our Product Owner about it.

    Thanks again for coming up with all these suggestions,
    Johannes
    On Tuesday, January 21, 2014 6:04:16 PM UTC-7, Mike Heath wrote:

    Has the loggregator team considered using Gary's Burd's WebSocket
    implementation (http://godoc.org/github.com/garyburd/go-websocket/
    websocket) instead of Go's built-in WebSocket library? Go's built-in
    library doesn't appear to support ping/pong frames (which seems weird that
    a Google invented protocol would have a half-baked implementation in a
    Google runtime...)

    It seems kind of silly to do stuff like this https://github.com/
    cloudfoundry/cli/blob/master/src/cf/api/logs.go#L125-L130 rather than
    using the WebSocket protocol's built-in mechanism for detecting stale
    connections.

    Forgive me if I sound like a whiner. I love Loggregator. I just keep
    finding things in Loggregator that are done in a way that's easy with Go
    but non-standard which causes more effort for those of us on more
    established platforms.

    -Mike
    On Friday, January 17, 2014 6:21:24 PM UTC-7, Mike Heath wrote:

    I'm really close to having a fully functional Loggregator client in
    Java for both emitting logs as well as reading logs over Web Sockets.

    There a few things that are bugging me so I thought I'd provide some
    feedback:

    - I'm not very fond of the DIY padding. PKCS#7 is really really easy
    to implement (since Go doesn't appear to support it.) Why not go with this
    standard padding mechanism? Regardless, it works but it was a pain to
    figure out and reimplement.
    - The signature in the LogEnvelope only covers the LogMessage's
    message field and not the entire LogMessage. This probably isn't a huge
    problem since it would still be rather difficult to intercept a LogEnvelope
    but if you did, you could emit the same message to any application
    masquerading as coming from any source. It seems more secure, to apply the
    digital signature to the entire LogMessage but again, not a big deal.
    - The nonsecond time resolution in the timestamp seems a little over
    the top but I don't anticipate being around on April 11, 2262 when it
    overflows so I'm not too concerned. :)

    I'll post a link to my client once it's ready.

    Thanks,

    -Mike
    To unsubscribe from this group and stop receiving emails from it, send
    an email to vcap-dev+unsubscribe@cloudfoundry.org.


    --
    Patrick Mueller
    http://muellerware.org

    To unsubscribe from this group and stop receiving emails from it, send an
    email to vcap-dev+unsubscribe@cloudfoundry.org.
    To unsubscribe from this group and stop receiving emails from it, send an email to vcap-dev+unsubscribe@cloudfoundry.org.
  • Patrick Mueller at Jan 22, 2014 at 9:50 pm
    There's no reason you can't do both :-)

    No, unaware of other apps that consume or spit-out SSE. I'm sure I'll end
    up running across a need for it myself, someday. And right, Hystrix isn't
    a logging consumer; it's a dashboard and consumes performance-related data
    that streams from servers. Of course, you could imagine a situation where
    you "log" performance data in such a way that it could read by Hystrix, and
    voila! Hystrix dashboard for your servers!

    To me, the advantage of SSE over WebSockets is ease of consumption. I
    could just curl it and get some interesting info. Consuming in a browser
    is also easy, assuming your browser supports it, and most do [1]. And
    there are polyfills [2].

    The big downside for me is wondering if I can use it over various gateways,
    proxys, etc. Whether things like CORS will work with it. Etc. So many
    times new tech like this ends up running into unexpected problems in our
    cloudy world. "unknown unknowns"

    [1] http://caniuse.com/#feat=eventsource
    [2]
    https://github.com/Modernizr/Modernizr/wiki/HTML5-Cross-browser-Polyfills#eventsource


    On Wed, Jan 22, 2014 at 3:15 PM, David Lee wrote:

    Patrick,

    A goal of this websocket work is to enable a large number of clients to
    access the log information. Websockets has better support across browsers
    (SSE is missing in IE).

    Other than Hysterix (which doesn't seem to be a logging consumer really),
    do you know of any other clients that primarily accept SSE?

    -Dave


    On Wed, Jan 22, 2014 at 11:40 AM, Patrick Mueller wrote:

    Has any thought been given to also making log data available via
    server-sent events [1]?

    I've only used these once, with Netflix's Hystrix dashboard, and found it
    SUPER EASY to generate an event stream to feed the Hystrix dashboard. I
    assume it's just as easy for code to consume it.

    I'm not sure what all loggregator can do - server-sent events are an
    output-only thing - you only interact with it by reading the stream, no
    writing once it's started. Of course, you could use URL patterns, query
    parameters on the URL, etc to select streams, filter, etc.

    [1] http://www.w3.org/TR/eventsource/

    On Wed, Jan 22, 2014 at 11:37 AM, Mike Heath wrote:

    Hey Johannes,

    Thanks for taking a look at this.

    You could modify the server without modifying the client. The server
    could periodically send ping frames. If the server does not receive any
    frames (a pong frame or otherwise) after a certain threshold, it assume the
    connection is stale and closes it. This will work with the existing the
    clients that send an "I'm still here!" text frame and will work with newer
    clients that use ping/pong. Clients going forward can then close their
    connection (or try to ping the server) if they don't receive any frames
    after a certain amount of time.

    Just a suggestion. Thanks again for the response.

    -Mike

    On Wed, Jan 22, 2014 at 9:13 AM, Johannes Tuchscherer <
    jtuchscherer@pivotallabs.com> wrote:
    Hi Mike,

    Thanks for digging through the code and coming up with improvements. We
    really appreciate that.

    That WebSocket implementation looks interesting and powerful. Indeed,
    we could rip out some code from the loggregator and the CLI by switching
    over to that implementation. BTW, it looks like this library got merged
    into the Gorilla project and is now available here:
    https://github.com/gorilla/websocket

    Switching to that implementation, though, comes at the cost of
    increasing the number of our third party dependencies. Considering we
    already wrote our keep-alive logic, we don't get an immediate benefit from
    switching right now. The other difficulty comes from the coupling between
    loggregator an CLI. If we use a new WebSocket implementation on the server,
    we have to update the CLI at the same time and we would need to handle
    both, the ping-pong frames and the old implementation for backwards
    compatibility with existing clients.

    Nevertheless, we see the benefit for the community in switching over to
    a more standard conform implementation. I will put in a story in our
    Pivotal Tracker Icebox and talk to our Product Owner about it.

    Thanks again for coming up with all these suggestions,
    Johannes
    On Tuesday, January 21, 2014 6:04:16 PM UTC-7, Mike Heath wrote:

    Has the loggregator team considered using Gary's Burd's WebSocket
    implementation (http://godoc.org/github.com/garyburd/go-websocket/
    websocket) instead of Go's built-in WebSocket library? Go's built-in
    library doesn't appear to support ping/pong frames (which seems weird that
    a Google invented protocol would have a half-baked implementation in a
    Google runtime...)

    It seems kind of silly to do stuff like this https://github.com/
    cloudfoundry/cli/blob/master/src/cf/api/logs.go#L125-L130 rather than
    using the WebSocket protocol's built-in mechanism for detecting stale
    connections.

    Forgive me if I sound like a whiner. I love Loggregator. I just keep
    finding things in Loggregator that are done in a way that's easy with Go
    but non-standard which causes more effort for those of us on more
    established platforms.

    -Mike
    On Friday, January 17, 2014 6:21:24 PM UTC-7, Mike Heath wrote:

    I'm really close to having a fully functional Loggregator client in
    Java for both emitting logs as well as reading logs over Web Sockets.

    There a few things that are bugging me so I thought I'd provide some
    feedback:

    - I'm not very fond of the DIY padding. PKCS#7 is really really easy
    to implement (since Go doesn't appear to support it.) Why not go with this
    standard padding mechanism? Regardless, it works but it was a pain to
    figure out and reimplement.
    - The signature in the LogEnvelope only covers the LogMessage's
    message field and not the entire LogMessage. This probably isn't a huge
    problem since it would still be rather difficult to intercept a LogEnvelope
    but if you did, you could emit the same message to any application
    masquerading as coming from any source. It seems more secure, to apply the
    digital signature to the entire LogMessage but again, not a big deal.
    - The nonsecond time resolution in the timestamp seems a little over
    the top but I don't anticipate being around on April 11, 2262 when it
    overflows so I'm not too concerned. :)

    I'll post a link to my client once it's ready.

    Thanks,

    -Mike
    To unsubscribe from this group and stop receiving emails from it, send
    an email to vcap-dev+unsubscribe@cloudfoundry.org.


    --
    Patrick Mueller
    http://muellerware.org

    To unsubscribe from this group and stop receiving emails from it, send an
    email to vcap-dev+unsubscribe@cloudfoundry.org.
    To unsubscribe from this group and stop receiving emails from it, send an
    email to vcap-dev+unsubscribe@cloudfoundry.org.


    --
    Patrick Mueller
    http://muellerware.org

    To unsubscribe from this group and stop receiving emails from it, send an email to vcap-dev+unsubscribe@cloudfoundry.org.
  • Mike Heath at Jan 22, 2014 at 10:07 pm
    Dave,

    The Loggregator protocol will have to be modified to support consuming the
    Log stream in browsers since custom headers (for authorization) cannot be
    set in the browser. [1,2] It's also a bit of a pain to consume protocol
    buffers in the browser. Using JSON would be preferred, IMO.

    -Mike

    1 - http://www.w3.org/TR/2011/WD-websockets-20110419/
    2 -
    http://stackoverflow.com/questions/4361173/http-headers-in-websockets-client-api

    On Wed, Jan 22, 2014 at 1:15 PM, David Lee wrote:

    Patrick,

    A goal of this websocket work is to enable a large number of clients to
    access the log information. Websockets has better support across browsers
    (SSE is missing in IE).

    Other than Hysterix (which doesn't seem to be a logging consumer really),
    do you know of any other clients that primarily accept SSE?

    -Dave


    On Wed, Jan 22, 2014 at 11:40 AM, Patrick Mueller wrote:

    Has any thought been given to also making log data available via
    server-sent events [1]?

    I've only used these once, with Netflix's Hystrix dashboard, and found it
    SUPER EASY to generate an event stream to feed the Hystrix dashboard. I
    assume it's just as easy for code to consume it.

    I'm not sure what all loggregator can do - server-sent events are an
    output-only thing - you only interact with it by reading the stream, no
    writing once it's started. Of course, you could use URL patterns, query
    parameters on the URL, etc to select streams, filter, etc.

    [1] http://www.w3.org/TR/eventsource/

    On Wed, Jan 22, 2014 at 11:37 AM, Mike Heath wrote:

    Hey Johannes,

    Thanks for taking a look at this.

    You could modify the server without modifying the client. The server
    could periodically send ping frames. If the server does not receive any
    frames (a pong frame or otherwise) after a certain threshold, it assume the
    connection is stale and closes it. This will work with the existing the
    clients that send an "I'm still here!" text frame and will work with newer
    clients that use ping/pong. Clients going forward can then close their
    connection (or try to ping the server) if they don't receive any frames
    after a certain amount of time.

    Just a suggestion. Thanks again for the response.

    -Mike

    On Wed, Jan 22, 2014 at 9:13 AM, Johannes Tuchscherer <
    jtuchscherer@pivotallabs.com> wrote:
    Hi Mike,

    Thanks for digging through the code and coming up with improvements. We
    really appreciate that.

    That WebSocket implementation looks interesting and powerful. Indeed,
    we could rip out some code from the loggregator and the CLI by switching
    over to that implementation. BTW, it looks like this library got merged
    into the Gorilla project and is now available here:
    https://github.com/gorilla/websocket

    Switching to that implementation, though, comes at the cost of
    increasing the number of our third party dependencies. Considering we
    already wrote our keep-alive logic, we don't get an immediate benefit from
    switching right now. The other difficulty comes from the coupling between
    loggregator an CLI. If we use a new WebSocket implementation on the server,
    we have to update the CLI at the same time and we would need to handle
    both, the ping-pong frames and the old implementation for backwards
    compatibility with existing clients.

    Nevertheless, we see the benefit for the community in switching over to
    a more standard conform implementation. I will put in a story in our
    Pivotal Tracker Icebox and talk to our Product Owner about it.

    Thanks again for coming up with all these suggestions,
    Johannes
    On Tuesday, January 21, 2014 6:04:16 PM UTC-7, Mike Heath wrote:

    Has the loggregator team considered using Gary's Burd's WebSocket
    implementation (http://godoc.org/github.com/garyburd/go-websocket/
    websocket) instead of Go's built-in WebSocket library? Go's built-in
    library doesn't appear to support ping/pong frames (which seems weird that
    a Google invented protocol would have a half-baked implementation in a
    Google runtime...)

    It seems kind of silly to do stuff like this https://github.com/
    cloudfoundry/cli/blob/master/src/cf/api/logs.go#L125-L130 rather than
    using the WebSocket protocol's built-in mechanism for detecting stale
    connections.

    Forgive me if I sound like a whiner. I love Loggregator. I just keep
    finding things in Loggregator that are done in a way that's easy with Go
    but non-standard which causes more effort for those of us on more
    established platforms.

    -Mike
    On Friday, January 17, 2014 6:21:24 PM UTC-7, Mike Heath wrote:

    I'm really close to having a fully functional Loggregator client in
    Java for both emitting logs as well as reading logs over Web Sockets.

    There a few things that are bugging me so I thought I'd provide some
    feedback:

    - I'm not very fond of the DIY padding. PKCS#7 is really really easy
    to implement (since Go doesn't appear to support it.) Why not go with this
    standard padding mechanism? Regardless, it works but it was a pain to
    figure out and reimplement.
    - The signature in the LogEnvelope only covers the LogMessage's
    message field and not the entire LogMessage. This probably isn't a huge
    problem since it would still be rather difficult to intercept a LogEnvelope
    but if you did, you could emit the same message to any application
    masquerading as coming from any source. It seems more secure, to apply the
    digital signature to the entire LogMessage but again, not a big deal.
    - The nonsecond time resolution in the timestamp seems a little over
    the top but I don't anticipate being around on April 11, 2262 when it
    overflows so I'm not too concerned. :)

    I'll post a link to my client once it's ready.

    Thanks,

    -Mike
    To unsubscribe from this group and stop receiving emails from it, send
    an email to vcap-dev+unsubscribe@cloudfoundry.org.


    --
    Patrick Mueller
    http://muellerware.org

    To unsubscribe from this group and stop receiving emails from it, send an
    email to vcap-dev+unsubscribe@cloudfoundry.org.
    To unsubscribe from this group and stop receiving emails from it, send an email to vcap-dev+unsubscribe@cloudfoundry.org.
  • David Lee at Jan 24, 2014 at 5:53 am
    Yes, we're are of the header issue and have something in the works to
    address that.

    I understand your point about JSON format. We'll think about some options
    there.

    On Wed, Jan 22, 2014 at 2:07 PM, Mike Heath wrote:

    Dave,

    The Loggregator protocol will have to be modified to support consuming the
    Log stream in browsers since custom headers (for authorization) cannot be
    set in the browser. [1,2] It's also a bit of a pain to consume protocol
    buffers in the browser. Using JSON would be preferred, IMO.

    -Mike

    1 - http://www.w3.org/TR/2011/WD-websockets-20110419/
    2 -
    http://stackoverflow.com/questions/4361173/http-headers-in-websockets-client-api

    On Wed, Jan 22, 2014 at 1:15 PM, David Lee wrote:

    Patrick,

    A goal of this websocket work is to enable a large number of clients to
    access the log information. Websockets has better support across browsers
    (SSE is missing in IE).

    Other than Hysterix (which doesn't seem to be a logging consumer really),
    do you know of any other clients that primarily accept SSE?

    -Dave


    On Wed, Jan 22, 2014 at 11:40 AM, Patrick Mueller wrote:

    Has any thought been given to also making log data available via
    server-sent events [1]?

    I've only used these once, with Netflix's Hystrix dashboard, and found
    it SUPER EASY to generate an event stream to feed the Hystrix dashboard. I
    assume it's just as easy for code to consume it.

    I'm not sure what all loggregator can do - server-sent events are an
    output-only thing - you only interact with it by reading the stream, no
    writing once it's started. Of course, you could use URL patterns, query
    parameters on the URL, etc to select streams, filter, etc.

    [1] http://www.w3.org/TR/eventsource/

    On Wed, Jan 22, 2014 at 11:37 AM, Mike Heath wrote:

    Hey Johannes,

    Thanks for taking a look at this.

    You could modify the server without modifying the client. The server
    could periodically send ping frames. If the server does not receive any
    frames (a pong frame or otherwise) after a certain threshold, it assume the
    connection is stale and closes it. This will work with the existing the
    clients that send an "I'm still here!" text frame and will work with newer
    clients that use ping/pong. Clients going forward can then close their
    connection (or try to ping the server) if they don't receive any frames
    after a certain amount of time.

    Just a suggestion. Thanks again for the response.

    -Mike

    On Wed, Jan 22, 2014 at 9:13 AM, Johannes Tuchscherer <
    jtuchscherer@pivotallabs.com> wrote:
    Hi Mike,

    Thanks for digging through the code and coming up with improvements.
    We really appreciate that.

    That WebSocket implementation looks interesting and powerful. Indeed,
    we could rip out some code from the loggregator and the CLI by switching
    over to that implementation. BTW, it looks like this library got merged
    into the Gorilla project and is now available here:
    https://github.com/gorilla/websocket

    Switching to that implementation, though, comes at the cost of
    increasing the number of our third party dependencies. Considering we
    already wrote our keep-alive logic, we don't get an immediate benefit from
    switching right now. The other difficulty comes from the coupling between
    loggregator an CLI. If we use a new WebSocket implementation on the server,
    we have to update the CLI at the same time and we would need to handle
    both, the ping-pong frames and the old implementation for backwards
    compatibility with existing clients.

    Nevertheless, we see the benefit for the community in switching over
    to a more standard conform implementation. I will put in a story in our
    Pivotal Tracker Icebox and talk to our Product Owner about it.

    Thanks again for coming up with all these suggestions,
    Johannes
    On Tuesday, January 21, 2014 6:04:16 PM UTC-7, Mike Heath wrote:

    Has the loggregator team considered using Gary's Burd's WebSocket
    implementation (http://godoc.org/github.com/garyburd/go-websocket/
    websocket) instead of Go's built-in WebSocket library? Go's built-in
    library doesn't appear to support ping/pong frames (which seems weird that
    a Google invented protocol would have a half-baked implementation in a
    Google runtime...)

    It seems kind of silly to do stuff like this https://github.com/
    cloudfoundry/cli/blob/master/src/cf/api/logs.go#L125-L130 rather
    than using the WebSocket protocol's built-in mechanism for detecting stale
    connections.

    Forgive me if I sound like a whiner. I love Loggregator. I just keep
    finding things in Loggregator that are done in a way that's easy with Go
    but non-standard which causes more effort for those of us on more
    established platforms.

    -Mike
    On Friday, January 17, 2014 6:21:24 PM UTC-7, Mike Heath wrote:

    I'm really close to having a fully functional Loggregator client in
    Java for both emitting logs as well as reading logs over Web Sockets.

    There a few things that are bugging me so I thought I'd provide some
    feedback:

    - I'm not very fond of the DIY padding. PKCS#7 is really really easy
    to implement (since Go doesn't appear to support it.) Why not go with this
    standard padding mechanism? Regardless, it works but it was a pain to
    figure out and reimplement.
    - The signature in the LogEnvelope only covers the LogMessage's
    message field and not the entire LogMessage. This probably isn't a huge
    problem since it would still be rather difficult to intercept a LogEnvelope
    but if you did, you could emit the same message to any application
    masquerading as coming from any source. It seems more secure, to apply the
    digital signature to the entire LogMessage but again, not a big deal.
    - The nonsecond time resolution in the timestamp seems a little over
    the top but I don't anticipate being around on April 11, 2262 when it
    overflows so I'm not too concerned. :)

    I'll post a link to my client once it's ready.

    Thanks,

    -Mike
    To unsubscribe from this group and stop receiving emails from it, send
    an email to vcap-dev+unsubscribe@cloudfoundry.org.


    --
    Patrick Mueller
    http://muellerware.org

    To unsubscribe from this group and stop receiving emails from it, send
    an email to vcap-dev+unsubscribe@cloudfoundry.org.
    To unsubscribe from this group and stop receiving emails from it, send an
    email to vcap-dev+unsubscribe@cloudfoundry.org.
    To unsubscribe from this group and stop receiving emails from it, send an email to vcap-dev+unsubscribe@cloudfoundry.org.

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupvcap-dev @
postedJan 18, '14 at 1:21a
activeJan 24, '14 at 5:53a
posts10
users5

People

Translate

site design / logo © 2017 Grokbase