FAQ
I've spent a couple of days trying to solve a problem. Now that I've solved
it, I'm wondering if anyone here might be able to explain what was wrong.

Short version:

- b := bytes.Buffer{}- decoder := gob.NewDecoder(&b) for {

- b.Reset()

+ var b bytes.Buffer+ decoder := gob.NewDecoder(&b)

The old version, declaring the buffer and decoder outside the loop,
actually worked fine -- I was able to bring up dozens of peers and
they were able to communicate.

The problem occurred when I added additional servers to the cluster
after the others had already been communicating. Then I'd get "gob:
unknown type id or corrupted data" for every message the newly-added
servers received (the "old" servers had no errors). Declaring the
buffer and decoder inside the loop solves the problem.

More details:


    - This is a clustered chat server
    - The problem occurred even with a very simple gob (a struct
containing only one string)
    - The problem occurred both in Docker containers and Digital Ocean
droplets, so I don't think it's a networking issue
    - I'm using Go 1.6.2 on 64-bit Ubuntu
    - I made a simplified version of my code and was able to replicate
the problem
    - Servers are communicating with one another using the gorilla
websocket library
    - The problem happens regardless of whether or not the servers all
start at once -- the trigger is if "new" server A connects to existing
server B, and server B has already been communicating with other
servers, then server A gets "corrupted" messages -- despite the fact
that literally the same bytes are being sent by server B to the other
servers server B has been talking to and they're fine
    - Checking the length of the buffer on the sender and the length of
the raw bytes from the receiver shows that the size matches at both
ends

Thanks for looking! I'd really like to understand what was going on --
especially if the "fix" could be a performance issue.

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Search Discussions

  • Voidlogic at Jun 1, 2016 at 7:00 pm
    I think we need more context, to me this sounds like the same thing that
    goes wrong when people try to use GOB over UDP. And existing GOB decoder
    has to always be connected to the same GOB sender. If you include more
    context we can see if this the case. If you feed input from multiple GOB
    encoders to the same decoder you will see this problem. My guess is this
    change fixes the issue because now you create a new decoder for every new
    data source?
    On Wednesday, June 1, 2016 at 1:36:12 PM UTC-5, Shawn Milochik wrote:

    I've spent a couple of days trying to solve a problem. Now that I've
    solved it, I'm wondering if anyone here might be able to explain what was
    wrong.

    Short version:

    - b := bytes.Buffer{}- decoder := gob.NewDecoder(&b) for {

    - b.Reset()

    + var b bytes.Buffer+ decoder := gob.NewDecoder(&b)

    The old version, declaring the buffer and decoder outside the loop, actually worked fine -- I was able to bring up dozens of peers and they were able to communicate.

    The problem occurred when I added additional servers to the cluster after the others had already been communicating. Then I'd get "gob: unknown type id or corrupted data" for every message the newly-added servers received (the "old" servers had no errors). Declaring the buffer and decoder inside the loop solves the problem.

    More details:


    - This is a clustered chat server
    - The problem occurred even with a very simple gob (a struct containing only one string)
    - The problem occurred both in Docker containers and Digital Ocean droplets, so I don't think it's a networking issue
    - I'm using Go 1.6.2 on 64-bit Ubuntu
    - I made a simplified version of my code and was able to replicate the problem
    - Servers are communicating with one another using the gorilla websocket library
    - The problem happens regardless of whether or not the servers all start at once -- the trigger is if "new" server A connects to existing server B, and server B has already been communicating with other servers, then server A gets "corrupted" messages -- despite the fact that literally the same bytes are being sent by server B to the other servers server B has been talking to and they're fine
    - Checking the length of the buffer on the sender and the length of the raw bytes from the receiver shows that the size matches at both ends

    Thanks for looking! I'd really like to understand what was going on -- especially if the "fix" could be a performance issue.
    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
  • Shawn Milochik at Jun 1, 2016 at 7:25 pm
    Thanks, that explains it, if I understand it correctly.

    If I'm connected to 24 peers, I need 24 decoders. Is that correct? I hadn't
    heard about that before. Also, it works fine with one decoder for 24 peers
    if they all connect before sending gobs.

    Why is encoding/gob different from encoding/json and encoding/xml in this
    way?

    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
  • Egon at Jun 1, 2016 at 7:47 pm

    On Wednesday, 1 June 2016 22:25:35 UTC+3, Shawn Milochik wrote:
    Thanks, that explains it, if I understand it correctly.

    If I'm connected to 24 peers, I need 24 decoders. Is that correct? I
    hadn't heard about that before. Also, it works fine with one decoder for 24
    peers if they all connect before sending gobs.
    I guess the more correct way would be to say 1 decoder can only read 1 gob
    stream and only from start to finish. e.g. If you have a broadcasting
    server and all clients are connected from the start then the broadcasting
    server can write a single stream and broadcast it to clients.

    Why is encoding/gob different from encoding/json and encoding/xml in this
    way?
    Different uses -- gob is more efficient in encoding data and can
    dynamically create types (this is also why you need to call gob.Register).
    Since you might be connecting different versions, you need those
    type-descriptions to describe the binary content.

    I'm not sure what you are building, but you might be better off using some
    other encoding e.g. protobuf, capnproto or something custom (each having
    their own trade-offs, see http://ithare.com/marshalling-and-encodings/).
    Based on the for loop you seem to be always sending the type-descriptors,
    which can be avoided by specifying the protocol.

    + Egon

    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
  • Egon at Jun 1, 2016 at 7:21 pm

    On Wednesday, 1 June 2016 21:36:12 UTC+3, Shawn Milochik wrote:
    I've spent a couple of days trying to solve a problem. Now that I've
    solved it, I'm wondering if anyone here might be able to explain what was
    wrong.

    Short version:

    - b := bytes.Buffer{}- decoder := gob.NewDecoder(&b) for {

    - b.Reset()

    + var b bytes.Buffer+ decoder := gob.NewDecoder(&b)

    The old version, declaring the buffer and decoder outside the loop, actually worked fine -- I was able to bring up dozens of peers and they were able to communicate.

    The problem occurred when I added additional servers to the cluster after the others had already been communicating. Then I'd get "gob: unknown type id or corrupted data" for every message the newly-added servers received (the "old" servers had no errors). Declaring the buffer and decoder inside the loop solves the problem.

    More details:


    - This is a clustered chat server
    - The problem occurred even with a very simple gob (a struct containing only one string)
    - The problem occurred both in Docker containers and Digital Ocean droplets, so I don't think it's a networking issue
    - I'm using Go 1.6.2 on 64-bit Ubuntu
    - I made a simplified version of my code and was able to replicate the problem
    - Servers are communicating with one another using the gorilla websocket library
    - The problem happens regardless of whether or not the servers all start at once -- the trigger is if "new" server A connects to existing server B, and server B has already been communicating with other servers, then server A gets "corrupted" messages -- despite the fact that literally the same bytes are being sent by server B to the other servers server B has been talking to and they're fine
    - Checking the length of the buffer on the sender and the length of the raw bytes from the receiver shows that the size matches at both ends

    Thanks for looking! I'd really like to understand what was going on -- especially if the "fix" could be a performance issue.
    Read https://blog.golang.org/gobs-of-data
    and https://golang.org/pkg/encoding/gob/.

    tl;dr; gob data stream contains the type definitions. When you reset the
    buffer, but not the decoder you could be confusing the decoder about what
    things have been received and hence read some things differently.

    tl;dr; one gob.Encoder must always be paired with one gob.Decoder.

    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
  • Shawn Milochik at Jun 1, 2016 at 7:26 pm
    Thanks for the references, Egon.

    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupgolang-nuts @
categoriesgo
postedJun 1, '16 at 6:36p
activeJun 1, '16 at 7:47p
posts6
users3
websitegolang.org

3 users in discussion

Shawn Milochik: 3 posts Egon: 2 posts Voidlogic: 1 post

People

Translate

site design / logo © 2021 Grokbase