FAQ
I have a socket connection to a remote server. When that server posts a
message on the socket, it contains a 4 byte header with the length of the
message.

Stream coming from the socket would look something like....

0015[15 byte message]0035[35byte message]... etc.

var msglen [4]bytes
count, err := bufio.NewReader(MyConnection).Read(msglen[:])
if err != nil {
fmt.Println("err reading socket", err.Error(), count)
os.Exit(1)
}
mlen, err := strconv.Atoi(string(msglen[:]))
count, err = bufio.NewReader(MyConnection).Read(buffer[0:mlen]))

Unfortunately, the second Read() from the socket hangs, as if to say, "Hey,
I already read from that, if you didn't get it the first time, too bad!"

I've tried a variety of other things, including Peek(), with about the same
results.

Is there an idiomatic way to read this stream one message-worth at a time?
I don't necessarily want to just post a single huge Read() (messages can be
up to 9999 bytes) and
risk pulling in multiple messages at a time, although I could do that if
necessary.
Ken

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Search Discussions

  • Chris Kastorff at Mar 20, 2015 at 4:58 pm
    Save the bufio reader and reuse it to keep using its buffer. Also,
    you'll probably want to use io.ReadFull to do repeated Read calls to
    fill your buffer, so that if the first Read call returns a short result
    you'll keep going.

    For example:

    r := bufio.NewReader(conn)
    ...save r somewhere where it will be reused for repeated message reads...

    var msglen [4]byte
    _, err := io.ReadFull(r, msglen[:])
    if err != nil { ... }
    mlen, err := strconv.Atoi(string(msglen[:]))
    if err != nil { ... }
    buf := make([]byte, mlen)
    _, err = io.ReadFull(r, buf)
    return buf
    On 03/20/2015 09:49 AM, Ken MacDonald wrote:
    I have a socket connection to a remote server. When that server posts a
    message on the socket, it contains a 4 byte header with the length of
    the message.

    Stream coming from the socket would look something like....

    0015[15 byte message]0035[35byte message]... etc.

    var msglen [4]bytes
    count, err := bufio.NewReader(MyConnection).Read(msglen[:])
    if err != nil {
    fmt.Println("err reading socket", err.Error(), count)
    os.Exit(1)
    }
    mlen, err := strconv.Atoi(string(msglen[:]))
    count, err = bufio.NewReader(MyConnection).Read(buffer[0:mlen]))

    Unfortunately, the second Read() from the socket hangs, as if to say,
    "Hey, I already read from that, if you didn't get it the first time, too
    bad!"

    I've tried a variety of other things, including Peek(), with about the
    same results.

    Is there an idiomatic way to read this stream one message-worth at a
    time? I don't necessarily want to just post a single huge Read()
    (messages can be up to 9999 bytes) and
    risk pulling in multiple messages at a time, although I could do that if
    necessary.
    Ken

    --
    You received this message because you are subscribed to the Google
    Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send
    an email to golang-nuts+unsubscribe@googlegroups.com
    For more options, visit https://groups.google.com/d/optout.
    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
  • Péter Szilágyi at Mar 20, 2015 at 4:58 pm
    LimitedReader ;)
    On Mar 20, 2015 6:49 PM, "Ken MacDonald" wrote:

    I have a socket connection to a remote server. When that server posts a
    message on the socket, it contains a 4 byte header with the length of the
    message.

    Stream coming from the socket would look something like....

    0015[15 byte message]0035[35byte message]... etc.

    var msglen [4]bytes
    count, err := bufio.NewReader(MyConnection).Read(msglen[:])
    if err != nil {
    fmt.Println("err reading socket", err.Error(), count)
    os.Exit(1)
    }
    mlen, err := strconv.Atoi(string(msglen[:]))
    count, err = bufio.NewReader(MyConnection).Read(buffer[0:mlen]))

    Unfortunately, the second Read() from the socket hangs, as if to say,
    "Hey, I already read from that, if you didn't get it the first time, too
    bad!"

    I've tried a variety of other things, including Peek(), with about the
    same results.

    Is there an idiomatic way to read this stream one message-worth at a time?
    I don't necessarily want to just post a single huge Read() (messages can be
    up to 9999 bytes) and
    risk pulling in multiple messages at a time, although I could do that if
    necessary.
    Ken

    --
    You received this message because you are subscribed to the Google Groups
    "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an
    email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
  • James Bardin at Mar 20, 2015 at 5:01 pm

    On Friday, March 20, 2015 at 12:49:45 PM UTC-4, Ken MacDonald wrote:
    I have a socket connection to a remote server. When that server posts a
    message on the socket, it contains a 4 byte header with the length of the
    message.

    Stream coming from the socket would look something like....

    0015[15 byte message]0035[35byte message]... etc.

    var msglen [4]bytes
    count, err := bufio.NewReader(MyConnection).Read(msglen[:])
    You[re throwing away your bufio Reader each time, along with it's buffer.
    Keep the same Reader and use it for each Read.

    Is there an idiomatic way to read this stream one message-worth at a time?
    I don't necessarily want to just post a single huge Read() (messages can be
    up to 9999 bytes) and
    risk pulling in multiple messages at a time, although I could do that if
    necessary.

    There's no way to read individual messages directly from a (TCP I assume)
    socket, since there's no inherent message framing at the TCP level. You
    have to parse and read the messages according to whatever application-level
    protocol you're using.

    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
  • Konstantin Khomoutov at Mar 20, 2015 at 5:11 pm

    On Fri, 20 Mar 2015 09:49:45 -0700 (PDT) Ken MacDonald wrote:

    I have a socket connection to a remote server. When that server posts
    a message on the socket, it contains a 4 byte header with the length
    of the message.

    Stream coming from the socket would look something like....

    0015[15 byte message]0035[35byte message]... etc.

    var msglen [4]bytes
    count, err := bufio.NewReader(MyConnection).Read(msglen[:])
    if err != nil {
    fmt.Println("err reading socket", err.Error(), count)
    os.Exit(1)
    }
    mlen, err := strconv.Atoi(string(msglen[:]))
    count, err = bufio.NewReader(MyConnection).Read(buffer[0:mlen]))

    Unfortunately, the second Read() from the socket hangs, as if to say,
    "Hey, I already read from that, if you didn't get it the first time,
    too bad!"

    I've tried a variety of other things, including Peek(), with about
    the same results.

    Is there an idiomatic way to read this stream one message-worth at a
    time? I don't necessarily want to just post a single huge Read()
    (messages can be up to 9999 bytes) and
    risk pulling in multiple messages at a time, although I could do that
    if necessary.
    The problem with your code is that you're using buffering (bufio)
    when there's not really needed. Moreover, while using buffering is not
    really bad, you're making a mistake of using two buffered readers on
    the same underlying socket.
    Now think what happens:
    1) The first read of 4 bytes might trigger reading more data as buffered
        reader is free to ask the underlying reader for more data (its task
        is to buffer -- that is, make more data at hand, if possible, than
        you've asked. Say, the kernel had 4k worth of data for that socket
        connection, and the reader buffered it all. But you've just took
        the first 4 bytes of it.
    2) Now you create the second buffering reader. The next time you ask
        it for data, you have your 4k - 4 bytes lost in the buffers of the
        first reader. And if your protocol is message oriented (you ask for
        data, the server supplies it and waits for another request), any
        reads using the second reader might block forever if the read using
        the first reader managed to get all the data the server sent in
        response to your request.
    Do you get that?

    The correct approach is to use io.ReadFull() supplying it a slice
    with the length just matching the number of bytes you want to read,
    like this:

       // First, read the header:
       var hlen [4]byte
       _, err := io.ReadFull(sock, hlen[:])
       // If err == nil, bail out;
       // if not, process these 4 bytes.

       // Let's say, the length is now in dataLen variable.
       // Get the payload:
       data := make([dataLen]byte)
       _, err := io.ReadFull(sock, data[:])

    Note that you might use buffered reader over your socket -- that's fine
    until you use the same reader for all reads of the data from that
    same socket (see above for explanations).

    And you might decide not to use io.ReadFull() to read payloads and
    instead read them piecemeal -- that really depens on your use case
    (for your <10k-sized messages, I think, reading them fully is just OK,
    for multi-gigabyte payloads it's probably not).

    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
  • Konstantin Khomoutov at Mar 20, 2015 at 5:33 pm
    On Fri, 20 Mar 2015 20:11:22 +0300 Konstantin Khomoutov wrote:
    Sorry, minor corrections are due:

    [...]
    1) The first read of 4 bytes might trigger reading more data as
    buffered reader is free to ask the underlying reader for more data
    (its task is to buffer -- that is, make more data at hand,
    I meant "make more data available at hand".

    To reiterate: a buffered reader can actually read more data from its
    underlying reader than you've requested, and keep it around so the next
    requests would not need to hit the underlying reader. The idea is that
    each Read() on a bare socket (file or any other kernel-level object)
    triggers a syscall (a call into the kernel) which implies crossing the
    userspace/kernelspace boundary which is not cheap. So reading 1000
    times of 4 bytes worth of data is way more resource-inefficient than
    reading 4000 bytes worth of data using a single syscall and then
    reading that data -- now buffered in userspace -- by short chunks.
    Basically that's the reason for the bufio facility.
    The same applies to writing the data.

    [...]
    // Let's say, the length is now in dataLen variable.
    // Get the payload:
    data := make([dataLen]byte)
    _, err := io.ReadFull(sock, data[:])
        _, err := io.ReadFull(sock, data)

    Is way more sensible -- the "data" variable is already a slice so
    there's no need to reslice that slice.

    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
  • Ken MacDonald at Mar 20, 2015 at 5:47 pm
    Thanks all,
    That explains a lot more about how readers, sockets, etc. interact; should
    be good now. After next meeting of the day, will see about getting these
    suggestions into the code.

    Jan - completely agree that the length shoulda been an int32, but it's
    apparently a standard the guys on the other end of the socket (large
    multinational bank) have been using for a decade or two.
    I fear my protests will mean little :-)
    Ken
    On Fri, Mar 20, 2015 at 1:32 PM, Konstantin Khomoutov wrote:

    On Fri, 20 Mar 2015 20:11:22 +0300
    Konstantin Khomoutov wrote:

    Sorry, minor corrections are due:

    [...]
    1) The first read of 4 bytes might trigger reading more data as
    buffered reader is free to ask the underlying reader for more data
    (its task is to buffer -- that is, make more data at hand,
    I meant "make more data available at hand".

    To reiterate: a buffered reader can actually read more data from its
    underlying reader than you've requested, and keep it around so the next
    requests would not need to hit the underlying reader. The idea is that
    each Read() on a bare socket (file or any other kernel-level object)
    triggers a syscall (a call into the kernel) which implies crossing the
    userspace/kernelspace boundary which is not cheap. So reading 1000
    times of 4 bytes worth of data is way more resource-inefficient than
    reading 4000 bytes worth of data using a single syscall and then
    reading that data -- now buffered in userspace -- by short chunks.
    Basically that's the reason for the bufio facility.
    The same applies to writing the data.

    [...]
    // Let's say, the length is now in dataLen variable.
    // Get the payload:
    data := make([dataLen]byte)
    _, err := io.ReadFull(sock, data[:])
    _, err := io.ReadFull(sock, data)

    Is way more sensible -- the "data" variable is already a slice so
    there's no need to reslice that slice.
    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupgolang-nuts @
categoriesgo
postedMar 20, '15 at 4:49p
activeMar 20, '15 at 5:47p
posts7
users5
websitegolang.org

People

Translate

site design / logo © 2021 Grokbase