FAQ
Is it possible to read HTTP response body chunk by chunk like in node in Go?

In node, the could would be:

request.on('response', function (response) {
   response.on('data', function (chunk) {
     doSomething(chunk)
   });
   response.on('end', function () {
     done()
   });});

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Search Discussions

  • Matt Harden at Jul 13, 2015 at 2:05 pm
    Go's net/http already handles the chunks for you and returns them to you as
    you read the response Body.
    On Mon, Jul 13, 2015 at 8:59 AM wrote:

    Is it possible to read HTTP response body chunk by chunk like in node in
    Go?

    In node, the could would be:

    request.on('response', function (response) {
    response.on('data', function (chunk) {
    doSomething(chunk)
    });
    response.on('end', function () {
    done()
    });});

    --
    You received this message because you are subscribed to the Google Groups
    "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an
    email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
  • Sacheendra T at Jul 13, 2015 at 9:10 pm
    I actually need the chunks and do not want them to be handled. It is useful
    in cases where you want keep the connection open to send the data for a
    long time.
    On Monday, July 13, 2015 at 9:05:52 AM UTC-5, Matt Harden wrote:

    Go's net/http already handles the chunks for you and returns them to you
    as you read the response Body.
    On Mon, Jul 13, 2015 at 8:59 AM <sachee...@gmail.com <javascript:>> wrote:

    Is it possible to read HTTP response body chunk by chunk like in node in
    Go?

    In node, the could would be:

    request.on('response', function (response) {
    response.on('data', function (chunk) {
    doSomething(chunk)
    });
    response.on('end', function () {
    done()
    });});

    --
    You received this message because you are subscribed to the Google Groups
    "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an
    email to golang-nuts...@googlegroups.com <javascript:>.
    For more options, visit https://groups.google.com/d/optout.
    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
  • Brad Fitzpatrick at Jul 13, 2015 at 9:17 pm
    I don't understand what you mean.

    You can read the request body incrementally. You can read an HTTP response
    body incrementally. You write an HTTP response incrementally. You can
    generate a request body incrementally.

    Everybody about Go's http client & server support supports keeping things
    open and doing things incrementally.

    What are you trying to do?


    On Mon, Jul 13, 2015 at 3:10 PM, wrote:

    I actually need the chunks and do not want them to be handled. It is
    useful in cases where you want keep the connection open to send the data
    for a long time.
    On Monday, July 13, 2015 at 9:05:52 AM UTC-5, Matt Harden wrote:

    Go's net/http already handles the chunks for you and returns them to you
    as you read the response Body.
    On Mon, Jul 13, 2015 at 8:59 AM wrote:

    Is it possible to read HTTP response body chunk by chunk like in node in
    Go?

    In node, the could would be:

    request.on('response', function (response) {
    response.on('data', function (chunk) {
    doSomething(chunk)
    });
    response.on('end', function () {
    done()
    });});

    --
    You received this message because you are subscribed to the Google
    Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send
    an email to golang-nuts...@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
    --
    You received this message because you are subscribed to the Google Groups
    "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an
    email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
  • Sacheendra T at Jul 13, 2015 at 9:59 pm
    I am trying to use an HTTP request like a socket and pushing data from the
    server. The HTTP connection is kept open and the server sends events as
    they happen to the client. The transfer encoding is chunked and each event
    would be received as a chunk.

    One application of this is Server Sent Events. (
    http://www.html5rocks.com/en/tutorials/eventsource/basics/)
    There is no easy way in go that i know of to process an event as it is
    received.

    I have posted an example of javascript code where something can be done as
    the event is received.

    request.on('response', function (response) {
       response.on('data', function (chunk) {
         doSomething(chunk)
       });});

    The function doSomething is called every time a chunk is received.

    I am currently looking at this library (https://github.com/chrhlnd/gochunky)
    and the author seems to be using the net package and parsing the packets
    themselves.
    On Monday, July 13, 2015 at 4:18:12 PM UTC-5, bradfitz wrote:

    I don't understand what you mean.

    You can read the request body incrementally. You can read an HTTP response
    body incrementally. You write an HTTP response incrementally. You can
    generate a request body incrementally.

    Everybody about Go's http client & server support supports keeping things
    open and doing things incrementally.

    What are you trying to do?



    On Mon, Jul 13, 2015 at 3:10 PM, <sachee...@gmail.com <javascript:>>
    wrote:
    I actually need the chunks and do not want them to be handled. It is
    useful in cases where you want keep the connection open to send the data
    for a long time.
    On Monday, July 13, 2015 at 9:05:52 AM UTC-5, Matt Harden wrote:

    Go's net/http already handles the chunks for you and returns them to you
    as you read the response Body.
    On Mon, Jul 13, 2015 at 8:59 AM wrote:

    Is it possible to read HTTP response body chunk by chunk like in node
    in Go?

    In node, the could would be:

    request.on('response', function (response) {
    response.on('data', function (chunk) {
    doSomething(chunk)
    });
    response.on('end', function () {
    done()
    });});

    --
    You received this message because you are subscribed to the Google
    Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send
    an email to golang-nuts...@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
    --
    You received this message because you are subscribed to the Google Groups
    "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an
    email to golang-nuts...@googlegroups.com <javascript:>.
    For more options, visit https://groups.google.com/d/optout.
    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
  • Matt Harden at Jul 13, 2015 at 11:44 pm
    Server sent events include their own chunking of the data. You don't need
    to worry about the HTTP chunks. Consider those an implementation detail
    like TCP packets.

    As for response.on('data', ...) I would expect that to be roughly
    equivalent to just calling Read on req.Body. Read will return as soon as
    there is any data available. I do NOT expect that response.on('data', ...)
    will return complete "chunks". I expect both to return data as it becomes
    available.
    On Mon, Jul 13, 2015 at 4:59 PM wrote:

    I am trying to use an HTTP request like a socket and pushing data from the
    server. The HTTP connection is kept open and the server sends events as
    they happen to the client. The transfer encoding is chunked and each event
    would be received as a chunk.

    One application of this is Server Sent Events. (
    http://www.html5rocks.com/en/tutorials/eventsource/basics/)
    There is no easy way in go that i know of to process an event as it is
    received.

    I have posted an example of javascript code where something can be done as
    the event is received.

    request.on('response', function (response) {
    response.on('data', function (chunk) {
    doSomething(chunk)
    });

    });

    The function doSomething is called every time a chunk is received.

    I am currently looking at this library (
    https://github.com/chrhlnd/gochunky) and the author seems to be using the
    net package and parsing the packets themselves.
    On Monday, July 13, 2015 at 4:18:12 PM UTC-5, bradfitz wrote:

    I don't understand what you mean.

    You can read the request body incrementally. You can read an HTTP
    response body incrementally. You write an HTTP response incrementally. You
    can generate a request body incrementally.

    Everybody about Go's http client & server support supports keeping things
    open and doing things incrementally.

    What are you trying to do?


    On Mon, Jul 13, 2015 at 3:10 PM, wrote:

    I actually need the chunks and do not want them to be handled. It is
    useful in cases where you want keep the connection open to send the data
    for a long time.
    On Monday, July 13, 2015 at 9:05:52 AM UTC-5, Matt Harden wrote:

    Go's net/http already handles the chunks for you and returns them to
    you as you read the response Body.
    On Mon, Jul 13, 2015 at 8:59 AM wrote:

    Is it possible to read HTTP response body chunk by chunk like in node
    in Go?

    In node, the could would be:

    request.on('response', function (response) {
    response.on('data', function (chunk) {
    doSomething(chunk)
    });
    response.on('end', function () {
    done()
    });});

    --
    You received this message because you are subscribed to the Google
    Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send
    an email to golang-nuts...@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
    --
    You received this message because you are subscribed to the Google
    Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send
    an email to golang-nuts...@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
    --
    You received this message because you are subscribed to the Google Groups
    "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an
    email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
  • Matt Harden at Jul 13, 2015 at 11:51 pm
    For example, in this Go Server-Sent-Events library, the decoder just reads
    directly from the resp.Body.
    https://github.com/donovanhide/eventsource/blob/master/stream.go#L77.
    On Mon, Jul 13, 2015 at 6:43 PM Matt Harden wrote:

    Server sent events include their own chunking of the data. You don't need
    to worry about the HTTP chunks. Consider those an implementation detail
    like TCP packets.

    As for response.on('data', ...) I would expect that to be roughly
    equivalent to just calling Read on req.Body. Read will return as soon as
    there is any data available. I do NOT expect that response.on('data', ...)
    will return complete "chunks". I expect both to return data as it becomes
    available.
    On Mon, Jul 13, 2015 at 4:59 PM wrote:

    I am trying to use an HTTP request like a socket and pushing data from
    the server. The HTTP connection is kept open and the server sends events as
    they happen to the client. The transfer encoding is chunked and each event
    would be received as a chunk.

    One application of this is Server Sent Events. (
    http://www.html5rocks.com/en/tutorials/eventsource/basics/)
    There is no easy way in go that i know of to process an event as it is
    received.

    I have posted an example of javascript code where something can be done
    as the event is received.

    request.on('response', function (response) {
    response.on('data', function (chunk) {
    doSomething(chunk)
    });

    });

    The function doSomething is called every time a chunk is received.

    I am currently looking at this library (
    https://github.com/chrhlnd/gochunky) and the author seems to be using
    the net package and parsing the packets themselves.
    On Monday, July 13, 2015 at 4:18:12 PM UTC-5, bradfitz wrote:

    I don't understand what you mean.

    You can read the request body incrementally. You can read an HTTP
    response body incrementally. You write an HTTP response incrementally. You
    can generate a request body incrementally.

    Everybody about Go's http client & server support supports keeping
    things open and doing things incrementally.

    What are you trying to do?


    On Mon, Jul 13, 2015 at 3:10 PM, wrote:

    I actually need the chunks and do not want them to be handled. It is
    useful in cases where you want keep the connection open to send the data
    for a long time.
    On Monday, July 13, 2015 at 9:05:52 AM UTC-5, Matt Harden wrote:

    Go's net/http already handles the chunks for you and returns them to
    you as you read the response Body.
    On Mon, Jul 13, 2015 at 8:59 AM wrote:

    Is it possible to read HTTP response body chunk by chunk like in node
    in Go?

    In node, the could would be:

    request.on('response', function (response) {
    response.on('data', function (chunk) {
    doSomething(chunk)
    });
    response.on('end', function () {
    done()
    });});

    --
    You received this message because you are subscribed to the Google
    Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it,
    send an email to golang-nuts...@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
    --
    You received this message because you are subscribed to the Google
    Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send
    an email to golang-nuts...@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
    --
    You received this message because you are subscribed to the Google Groups
    "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an
    email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
  • Sacheendra T at Jul 14, 2015 at 12:34 am
    Server Sent Events were easily parsed in the library because they come with
    delimiters in the form of '\n\n'. Just have to Read till unescaped \n.

    I apologize for not mentioning my use case before, it is as follows:
    The server sends events as JSON objects and each object is a chunk. I need
    to be able to identify individual events and act on them.
    For example, 2 chunks would be 1) {"foo":"bar"} and 2) {"hello":"world"}.
    As there are no special delimiters, i cannot do what was done with SSE

    If both of them are in the reader, like this
    `{"foo":"bar"}{"hello":"world"}`. The way to identify them as 2 events
    would be to read characters one by one from the reader, add it to an
    existing buffer and check if the buffer has valid JSON.

    I am looking for a way to separate the events more efficiently than doing
    that. I was interested in finding out if i can be informed when a chunk
    arrives because that is information that is available at the network level
    and directly useful in identifying events.
    On Monday, July 13, 2015 at 6:51:06 PM UTC-5, Matt Harden wrote:

    For example, in this Go Server-Sent-Events library, the decoder just reads
    directly from the resp.Body.
    https://github.com/donovanhide/eventsource/blob/master/stream.go#L77.

    On Mon, Jul 13, 2015 at 6:43 PM Matt Harden <matt....@gmail.com
    <javascript:>> wrote:
    Server sent events include their own chunking of the data. You don't need
    to worry about the HTTP chunks. Consider those an implementation detail
    like TCP packets.

    As for response.on('data', ...) I would expect that to be roughly
    equivalent to just calling Read on req.Body. Read will return as soon as
    there is any data available. I do NOT expect that response.on('data', ...)
    will return complete "chunks". I expect both to return data as it becomes
    available.

    On Mon, Jul 13, 2015 at 4:59 PM <sachee...@gmail.com <javascript:>>
    wrote:
    I am trying to use an HTTP request like a socket and pushing data from
    the server. The HTTP connection is kept open and the server sends events as
    they happen to the client. The transfer encoding is chunked and each event
    would be received as a chunk.

    One application of this is Server Sent Events. (
    http://www.html5rocks.com/en/tutorials/eventsource/basics/)
    There is no easy way in go that i know of to process an event as it is
    received.

    I have posted an example of javascript code where something can be done
    as the event is received.

    request.on('response', function (response) {
    response.on('data', function (chunk) {
    doSomething(chunk)
    });

    });

    The function doSomething is called every time a chunk is received.

    I am currently looking at this library (
    https://github.com/chrhlnd/gochunky) and the author seems to be using
    the net package and parsing the packets themselves.
    On Monday, July 13, 2015 at 4:18:12 PM UTC-5, bradfitz wrote:

    I don't understand what you mean.

    You can read the request body incrementally. You can read an HTTP
    response body incrementally. You write an HTTP response incrementally. You
    can generate a request body incrementally.

    Everybody about Go's http client & server support supports keeping
    things open and doing things incrementally.

    What are you trying to do?


    On Mon, Jul 13, 2015 at 3:10 PM, wrote:

    I actually need the chunks and do not want them to be handled. It is
    useful in cases where you want keep the connection open to send the data
    for a long time.
    On Monday, July 13, 2015 at 9:05:52 AM UTC-5, Matt Harden wrote:

    Go's net/http already handles the chunks for you and returns them to
    you as you read the response Body.
    On Mon, Jul 13, 2015 at 8:59 AM wrote:

    Is it possible to read HTTP response body chunk by chunk like in
    node in Go?

    In node, the could would be:

    request.on('response', function (response) {
    response.on('data', function (chunk) {
    doSomething(chunk)
    });
    response.on('end', function () {
    done()
    });});

    --
    You received this message because you are subscribed to the Google
    Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it,
    send an email to golang-nuts...@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
    --
    You received this message because you are subscribed to the Google
    Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send
    an email to golang-nuts...@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
    --
    You received this message because you are subscribed to the Google
    Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send
    an email to golang-nuts...@googlegroups.com <javascript:>.
    For more options, visit https://groups.google.com/d/optout.
    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
  • Brad Fitzpatrick at Jul 14, 2015 at 12:39 am
    You can't trust HTTP chunk boundaries. Anything can and does change them.

    Your protocol (SSE, etc) needs to have its own framing.

    In practice, the http.Response.Body's Read calls will return quickly
    chunk-by-chunk. But don't depend on that either.


    On Mon, Jul 13, 2015 at 6:34 PM, wrote:

    Server Sent Events were easily parsed in the library because they come
    with delimiters in the form of '\n\n'. Just have to Read till unescaped \n.

    I apologize for not mentioning my use case before, it is as follows:
    The server sends events as JSON objects and each object is a chunk. I need
    to be able to identify individual events and act on them.
    For example, 2 chunks would be 1) {"foo":"bar"} and 2) {"hello":"world"}.
    As there are no special delimiters, i cannot do what was done with SSE

    If both of them are in the reader, like this
    `{"foo":"bar"}{"hello":"world"}`. The way to identify them as 2 events
    would be to read characters one by one from the reader, add it to an
    existing buffer and check if the buffer has valid JSON.

    I am looking for a way to separate the events more efficiently than doing
    that. I was interested in finding out if i can be informed when a chunk
    arrives because that is information that is available at the network level
    and directly useful in identifying events.
    On Monday, July 13, 2015 at 6:51:06 PM UTC-5, Matt Harden wrote:

    For example, in this Go Server-Sent-Events library, the decoder just
    reads directly from the resp.Body.
    https://github.com/donovanhide/eventsource/blob/master/stream.go#L77.
    On Mon, Jul 13, 2015 at 6:43 PM Matt Harden wrote:

    Server sent events include their own chunking of the data. You don't
    need to worry about the HTTP chunks. Consider those an implementation
    detail like TCP packets.

    As for response.on('data', ...) I would expect that to be roughly
    equivalent to just calling Read on req.Body. Read will return as soon as
    there is any data available. I do NOT expect that response.on('data', ...)
    will return complete "chunks". I expect both to return data as it becomes
    available.
    On Mon, Jul 13, 2015 at 4:59 PM wrote:

    I am trying to use an HTTP request like a socket and pushing data from
    the server. The HTTP connection is kept open and the server sends events as
    they happen to the client. The transfer encoding is chunked and each event
    would be received as a chunk.

    One application of this is Server Sent Events. (
    http://www.html5rocks.com/en/tutorials/eventsource/basics/)
    There is no easy way in go that i know of to process an event as it is
    received.

    I have posted an example of javascript code where something can be done
    as the event is received.

    request.on('response', function (response) {
    response.on('data', function (chunk) {
    doSomething(chunk)
    });

    });

    The function doSomething is called every time a chunk is received.

    I am currently looking at this library (
    https://github.com/chrhlnd/gochunky) and the author seems to be using
    the net package and parsing the packets themselves.
    On Monday, July 13, 2015 at 4:18:12 PM UTC-5, bradfitz wrote:

    I don't understand what you mean.

    You can read the request body incrementally. You can read an HTTP
    response body incrementally. You write an HTTP response incrementally. You
    can generate a request body incrementally.

    Everybody about Go's http client & server support supports keeping
    things open and doing things incrementally.

    What are you trying to do?


    On Mon, Jul 13, 2015 at 3:10 PM, wrote:

    I actually need the chunks and do not want them to be handled. It is
    useful in cases where you want keep the connection open to send the data
    for a long time.
    On Monday, July 13, 2015 at 9:05:52 AM UTC-5, Matt Harden wrote:

    Go's net/http already handles the chunks for you and returns them to
    you as you read the response Body.
    On Mon, Jul 13, 2015 at 8:59 AM wrote:

    Is it possible to read HTTP response body chunk by chunk like in
    node in Go?

    In node, the could would be:

    request.on('response', function (response) {
    response.on('data', function (chunk) {
    doSomething(chunk)
    });
    response.on('end', function () {
    done()
    });});

    --
    You received this message because you are subscribed to the Google
    Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it,
    send an email to golang-nuts...@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
    --
    You received this message because you are subscribed to the Google
    Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it,
    send an email to golang-nuts...@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
    --
    You received this message because you are subscribed to the Google
    Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send
    an email to golang-nuts...@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
    --
    You received this message because you are subscribed to the Google Groups
    "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an
    email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
  • Matt Harden at Jul 14, 2015 at 2:07 am
    Just use a streaming JSON decoder:
    http://golang.org/pkg/encoding/json/#Decoder
    On Mon, Jul 13, 2015 at 7:40 PM Brad Fitzpatrick wrote:

    You can't trust HTTP chunk boundaries. Anything can and does change them.

    Your protocol (SSE, etc) needs to have its own framing.

    In practice, the http.Response.Body's Read calls will return quickly
    chunk-by-chunk. But don't depend on that either.


    On Mon, Jul 13, 2015 at 6:34 PM, wrote:

    Server Sent Events were easily parsed in the library because they come
    with delimiters in the form of '\n\n'. Just have to Read till unescaped \n.

    I apologize for not mentioning my use case before, it is as follows:
    The server sends events as JSON objects and each object is a chunk. I
    need to be able to identify individual events and act on them.
    For example, 2 chunks would be 1) {"foo":"bar"} and 2) {"hello":"world"}.
    As there are no special delimiters, i cannot do what was done with SSE

    If both of them are in the reader, like this
    `{"foo":"bar"}{"hello":"world"}`. The way to identify them as 2 events
    would be to read characters one by one from the reader, add it to an
    existing buffer and check if the buffer has valid JSON.

    I am looking for a way to separate the events more efficiently than doing
    that. I was interested in finding out if i can be informed when a chunk
    arrives because that is information that is available at the network level
    and directly useful in identifying events.
    On Monday, July 13, 2015 at 6:51:06 PM UTC-5, Matt Harden wrote:

    For example, in this Go Server-Sent-Events library, the decoder just
    reads directly from the resp.Body.
    https://github.com/donovanhide/eventsource/blob/master/stream.go#L77.
    On Mon, Jul 13, 2015 at 6:43 PM Matt Harden wrote:

    Server sent events include their own chunking of the data. You don't
    need to worry about the HTTP chunks. Consider those an implementation
    detail like TCP packets.

    As for response.on('data', ...) I would expect that to be roughly
    equivalent to just calling Read on req.Body. Read will return as soon as
    there is any data available. I do NOT expect that response.on('data', ...)
    will return complete "chunks". I expect both to return data as it becomes
    available.
    On Mon, Jul 13, 2015 at 4:59 PM wrote:

    I am trying to use an HTTP request like a socket and pushing data from
    the server. The HTTP connection is kept open and the server sends events as
    they happen to the client. The transfer encoding is chunked and each event
    would be received as a chunk.

    One application of this is Server Sent Events. (
    http://www.html5rocks.com/en/tutorials/eventsource/basics/)
    There is no easy way in go that i know of to process an event as it is
    received.

    I have posted an example of javascript code where something can be
    done as the event is received.

    request.on('response', function (response) {
    response.on('data', function (chunk) {
    doSomething(chunk)
    });

    });

    The function doSomething is called every time a chunk is received.

    I am currently looking at this library (
    https://github.com/chrhlnd/gochunky) and the author seems to be using
    the net package and parsing the packets themselves.
    On Monday, July 13, 2015 at 4:18:12 PM UTC-5, bradfitz wrote:

    I don't understand what you mean.

    You can read the request body incrementally. You can read an HTTP
    response body incrementally. You write an HTTP response incrementally. You
    can generate a request body incrementally.

    Everybody about Go's http client & server support supports keeping
    things open and doing things incrementally.

    What are you trying to do?


    On Mon, Jul 13, 2015 at 3:10 PM, wrote:

    I actually need the chunks and do not want them to be handled. It is
    useful in cases where you want keep the connection open to send the data
    for a long time.
    On Monday, July 13, 2015 at 9:05:52 AM UTC-5, Matt Harden wrote:

    Go's net/http already handles the chunks for you and returns them
    to you as you read the response Body.
    On Mon, Jul 13, 2015 at 8:59 AM wrote:

    Is it possible to read HTTP response body chunk by chunk like in
    node in Go?

    In node, the could would be:

    request.on('response', function (response) {
    response.on('data', function (chunk) {
    doSomething(chunk)
    });
    response.on('end', function () {
    done()
    });});

    --
    You received this message because you are subscribed to the Google
    Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it,
    send an email to golang-nuts...@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
    --
    You received this message because you are subscribed to the Google
    Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it,
    send an email to golang-nuts...@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
    --
    You received this message because you are subscribed to the Google
    Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send
    an email to golang-nuts...@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
    --
    You received this message because you are subscribed to the Google Groups
    "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an
    email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
    --
    You received this message because you are subscribed to the Google Groups
    "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an
    email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
  • Sacheendra T at Jul 14, 2015 at 2:15 am
    I was just thinking of stringifying the responses on the server and using
    '\n' as the delimiter, but this simplifies everything. Don't know how i
    missed this.

    Thank You!
    On Monday, July 13, 2015 at 9:08:02 PM UTC-5, Matt Harden wrote:

    Just use a streaming JSON decoder:
    http://golang.org/pkg/encoding/json/#Decoder

    On Mon, Jul 13, 2015 at 7:40 PM Brad Fitzpatrick <brad...@golang.org
    <javascript:>> wrote:
    You can't trust HTTP chunk boundaries. Anything can and does change them.

    Your protocol (SSE, etc) needs to have its own framing.

    In practice, the http.Response.Body's Read calls will return quickly
    chunk-by-chunk. But don't depend on that either.



    On Mon, Jul 13, 2015 at 6:34 PM, <sachee...@gmail.com <javascript:>>
    wrote:
    Server Sent Events were easily parsed in the library because they come
    with delimiters in the form of '\n\n'. Just have to Read till unescaped \n.

    I apologize for not mentioning my use case before, it is as follows:
    The server sends events as JSON objects and each object is a chunk. I
    need to be able to identify individual events and act on them.
    For example, 2 chunks would be 1) {"foo":"bar"} and 2)
    {"hello":"world"}. As there are no special delimiters, i cannot do what was
    done with SSE

    If both of them are in the reader, like this
    `{"foo":"bar"}{"hello":"world"}`. The way to identify them as 2 events
    would be to read characters one by one from the reader, add it to an
    existing buffer and check if the buffer has valid JSON.

    I am looking for a way to separate the events more efficiently than
    doing that. I was interested in finding out if i can be informed when a
    chunk arrives because that is information that is available at the network
    level and directly useful in identifying events.
    On Monday, July 13, 2015 at 6:51:06 PM UTC-5, Matt Harden wrote:

    For example, in this Go Server-Sent-Events library, the decoder just
    reads directly from the resp.Body.
    https://github.com/donovanhide/eventsource/blob/master/stream.go#L77.
    On Mon, Jul 13, 2015 at 6:43 PM Matt Harden wrote:

    Server sent events include their own chunking of the data. You don't
    need to worry about the HTTP chunks. Consider those an implementation
    detail like TCP packets.

    As for response.on('data', ...) I would expect that to be roughly
    equivalent to just calling Read on req.Body. Read will return as soon as
    there is any data available. I do NOT expect that response.on('data', ...)
    will return complete "chunks". I expect both to return data as it becomes
    available.
    On Mon, Jul 13, 2015 at 4:59 PM wrote:

    I am trying to use an HTTP request like a socket and pushing data
    from the server. The HTTP connection is kept open and the server sends
    events as they happen to the client. The transfer encoding is chunked and
    each event would be received as a chunk.

    One application of this is Server Sent Events. (
    http://www.html5rocks.com/en/tutorials/eventsource/basics/)
    There is no easy way in go that i know of to process an event as it
    is received.

    I have posted an example of javascript code where something can be
    done as the event is received.

    request.on('response', function (response) {
    response.on('data', function (chunk) {
    doSomething(chunk)
    });

    });

    The function doSomething is called every time a chunk is received.

    I am currently looking at this library (
    https://github.com/chrhlnd/gochunky) and the author seems to be
    using the net package and parsing the packets themselves.
    On Monday, July 13, 2015 at 4:18:12 PM UTC-5, bradfitz wrote:

    I don't understand what you mean.

    You can read the request body incrementally. You can read an HTTP
    response body incrementally. You write an HTTP response incrementally. You
    can generate a request body incrementally.

    Everybody about Go's http client & server support supports keeping
    things open and doing things incrementally.

    What are you trying to do?


    On Mon, Jul 13, 2015 at 3:10 PM, wrote:

    I actually need the chunks and do not want them to be handled. It
    is useful in cases where you want keep the connection open to send the data
    for a long time.
    On Monday, July 13, 2015 at 9:05:52 AM UTC-5, Matt Harden wrote:

    Go's net/http already handles the chunks for you and returns them
    to you as you read the response Body.
    On Mon, Jul 13, 2015 at 8:59 AM wrote:

    Is it possible to read HTTP response body chunk by chunk like in
    node in Go?

    In node, the could would be:

    request.on('response', function (response) {
    response.on('data', function (chunk) {
    doSomething(chunk)
    });
    response.on('end', function () {
    done()
    });});

    --
    You received this message because you are subscribed to the
    Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it,
    send an email to golang-nuts...@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
    --
    You received this message because you are subscribed to the Google
    Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it,
    send an email to golang-nuts...@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
    --
    You received this message because you are subscribed to the Google
    Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it,
    send an email to golang-nuts...@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
    --
    You received this message because you are subscribed to the Google
    Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send
    an email to golang-nuts...@googlegroups.com <javascript:>.
    For more options, visit https://groups.google.com/d/optout.
    --
    You received this message because you are subscribed to the Google Groups
    "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an
    email to golang-nuts...@googlegroups.com <javascript:>.
    For more options, visit https://groups.google.com/d/optout.
    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
  • Chris Holland at Jul 13, 2015 at 4:19 pm
    Wrote some code to do this along time ago.

    https://github.com/chrhlnd/gochunky

    Was going to use it like a websocket from a custom client never got around to it though.

    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
  • Sacheendra T at Jul 13, 2015 at 9:09 pm
    This is exactly what I needed and for the same purpose too.
    On Monday, July 13, 2015 at 11:19:01 AM UTC-5, Chris Holland wrote:

    Wrote some code to do this along time ago.

    https://github.com/chrhlnd/gochunky

    Was going to use it like a websocket from a custom client never got around
    to it though.
    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupgolang-nuts @
categoriesgo
postedJul 13, '15 at 1:59p
activeJul 14, '15 at 2:15a
posts13
users4
websitegolang.org

People

Translate

site design / logo © 2021 Grokbase