FAQ
Hello,

I'm interested in implementing binary ops (i.e., those on
encoding/binary.ByteOrder) on the underlying byte slice in a
bytes.Buffer, such that encoding/decoding binary data to intermediate
buffers could be avoided.

I haven't found a good way to access the underlying byte slice on
bytes.Buffer, so was considering creating a package that carries a copy
of bytes.Buffer (and maybe bytes.Reader), and adds routines like these:
http://pastie.org/10659323

Is there an existing package that provides this functionality, or a
better way to do this?

Thanks, Liam

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Search Discussions

  • Thebrokentoaster at Jan 5, 2016 at 10:30 am
    Have you tried using the Buffer.Bytes method
    <https://golang.org/pkg/bytes/#Buffer.Bytes>?
    On Tuesday, January 5, 2016 at 2:16:55 AM UTC-8, Liam Staskawicz wrote:

    Hello,

    I'm interested in implementing binary ops (i.e., those on
    encoding/binary.ByteOrder) on the underlying byte slice in a bytes.Buffer,
    such that encoding/decoding binary data to intermediate buffers could be
    avoided.

    I haven't found a good way to access the underlying byte slice on
    bytes.Buffer, so was considering creating a package that carries a copy of
    bytes.Buffer (and maybe bytes.Reader), and adds routines like these:
    http://pastie.org/10659323

    Is there an existing package that provides this functionality, or a better
    way to do this?

    Thanks,
    Liam
    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
  • Liam Staskawicz at Jan 5, 2016 at 5:47 pm
    Yes. Buffer.Bytes() is not ideal in this case, since it's not possible to
    grow the underlying byte slice when writing to it, or move the read/write
    pointer of the buffer.

    The idea with this api is that it would support encoding/decoding data
    directly to/from the buffer:

    var b bytes.Buffer
    order := binary.BigEndian
    b.WriteUint32(order, 0x1234)
    // b.Len() == 4
    v, err := b.ReadUint32(order)
    // v == 0x1234, b.Len() == 0

    On Tuesday, January 5, 2016 at 2:30:14 AM UTC-8, thebroke...@gmail.com
    wrote:
    Have you tried using the Buffer.Bytes method
    <https://golang.org/pkg/bytes/#Buffer.Bytes>?
    On Tuesday, January 5, 2016 at 2:16:55 AM UTC-8, Liam Staskawicz wrote:

    Hello,

    I'm interested in implementing binary ops (i.e., those on
    encoding/binary.ByteOrder) on the underlying byte slice in a bytes.Buffer,
    such that encoding/decoding binary data to intermediate buffers could be
    avoided.

    I haven't found a good way to access the underlying byte slice on
    bytes.Buffer, so was considering creating a package that carries a copy of
    bytes.Buffer (and maybe bytes.Reader), and adds routines like these:
    http://pastie.org/10659323
    <http://www.google.com/url?q=http%3A%2F%2Fpastie.org%2F10659323&sa=D&sntz=1&usg=AFQjCNHzfWYc6SyxtbiXYIqB-JOkVwjJnA>

    Is there an existing package that provides this functionality, or a
    better way to do this?

    Thanks,
    Liam
    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
  • Brad Fitzpatrick at Jan 5, 2016 at 6:27 pm
    Use:
    https://golang.org/pkg/bytes/#Buffer.Grow
    https://golang.org/pkg/bytes/#Buffer.Cap
    https://golang.org/pkg/bytes/#Buffer.Bytes

    .... along with binary.BigEndian, etc.

    On Tue, Jan 5, 2016 at 8:20 AM, Liam Staskawicz wrote:

    Yes. Buffer.Bytes() is not ideal in this case, since it's not possible to
    grow the underlying byte slice when writing to it, or move the read/write
    pointer of the buffer.

    The idea with this api is that it would support encoding/decoding data
    directly to/from the buffer:

    var b bytes.Buffer
    order := binary.BigEndian
    b.WriteUint32(order, 0x1234)
    // b.Len() == 4
    v, err := b.ReadUint32(order)
    // v == 0x1234, b.Len() == 0


    On Tuesday, January 5, 2016 at 2:30:14 AM UTC-8, thebroke...@gmail.com
    wrote:
    Have you tried using the Buffer.Bytes method
    <https://golang.org/pkg/bytes/#Buffer.Bytes>?
    On Tuesday, January 5, 2016 at 2:16:55 AM UTC-8, Liam Staskawicz wrote:

    Hello,

    I'm interested in implementing binary ops (i.e., those on
    encoding/binary.ByteOrder) on the underlying byte slice in a bytes.Buffer,
    such that encoding/decoding binary data to intermediate buffers could be
    avoided.

    I haven't found a good way to access the underlying byte slice on
    bytes.Buffer, so was considering creating a package that carries a copy of
    bytes.Buffer (and maybe bytes.Reader), and adds routines like these:
    http://pastie.org/10659323
    <http://www.google.com/url?q=http%3A%2F%2Fpastie.org%2F10659323&sa=D&sntz=1&usg=AFQjCNHzfWYc6SyxtbiXYIqB-JOkVwjJnA>

    Is there an existing package that provides this functionality, or a
    better way to do this?

    Thanks,
    Liam
    --
    You received this message because you are subscribed to the Google Groups
    "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an
    email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
  • Liam Staskawicz at Jan 5, 2016 at 6:57 pm
    Thanks. I think these don't provide the functionality I'm looking for,
    since (unless I'm mistaken) this still implies encoding the binary data
    to an intermediate byte slice before writing to the bytes.Buffer.

    i.e., the following does not work

    var b bytes.Buffer
    b.Grow(4) // len(b.Bytes()) == 0 here, b.Cap() == 64
       binary.BigEndian.PutUint32(b.Bytes(), 0x1234) // fails with index out
       of range, since len(b.Bytes())  is 0

    even if this worked, we'd still need a way to tell b that we've written
    those bytes there (i.e., b.Len() would still return 0 at this point
    since we haven't modified the unexported b.off field).

    so we're left with doing something like:

    buf := make([]byte, 4) binary.BigEndian.PutUint32(buf, 0x1234)
    b.Write(buf)

    which, in addition to being noisier and less direct, involves an extra
    copy step (and possibly allocation?).


    On Tue, Jan 5, 2016, at 10:26 AM, Brad Fitzpatrick wrote:
    Use: https://golang.org/pkg/bytes/#Buffer.Grow
    https://golang.org/pkg/bytes/#Buffer.Cap
    https://golang.org/pkg/bytes/#Buffer.Bytes

    .... along with binary.BigEndian, etc.


    On Tue, Jan 5, 2016 at 8:20 AM, Liam Staskawicz
    wrote:
    Yes. Buffer.Bytes() is not ideal in this case, since it's not
    possible to grow the underlying byte slice when writing to it, or
    move the read/write pointer of the buffer.

    The idea with this api is that it would support encoding/decoding
    data directly to/from the buffer:

    var b bytes.Buffer order := binary.BigEndian
    b.WriteUint32(order, 0x1234) // b.Len() == 4 v, err :=
    b.ReadUint32(order) // v == 0x1234, b.Len() == 0


    On Tuesday, January 5, 2016 at 2:30:14 AM UTC-8,
    thebroke...@gmail.com wrote:
    Have you tried using the Buffer.Bytes method[1]?

    On Tuesday, January 5, 2016 at 2:16:55 AM UTC-8, Liam Staskawicz
    wrote:
    Hello,

    I'm interested in implementing binary ops (i.e., those on
    encoding/binary.ByteOrder) on the underlying byte slice in a
    bytes.Buffer, such that encoding/decoding binary data to
    intermediate buffers could be avoided.

    I haven't found a good way to access the underlying byte slice on
    bytes.Buffer, so was considering creating a package that carries a
    copy of bytes.Buffer (and maybe bytes.Reader), and adds routines
    like these: http://pastie.org/10659323[2]

    Is there an existing package that provides this functionality, or a
    better way to do this?

    Thanks, Liam
    --
    You received this message because you are subscribed to the Google
    Groups "golang-nuts" group.
    >>
    To unsubscribe from this group and stop receiving emails from it, send
    an email to golang-nuts+unsubscribe@googlegroups.com.
    >>
    For more options, visit https://groups.google.com/d/optout.



    Links:

       1. https://golang.org/pkg/bytes/#Buffer.Bytes
       2. http://www.google.com/url?q=http%3A%2F%2Fpastie.org%2F10659323&sa=D&sntz=1&usg=AFQjCNHzfWYc6SyxtbiXYIqB-JOkVwjJnA

    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
  • Brad Fitzpatrick at Jan 5, 2016 at 6:54 pm
    Your way allocates with those temporary 4-byte buf buffers.

    This way doesn't: http://play.golang.org/p/lBw-Kj7QZ5
    or without Grow: http://play.golang.org/p/eUNWnzobKV

    On Tue, Jan 5, 2016 at 10:39 AM, Liam Staskawicz wrote:

    Thanks. I think these don't provide the functionality I'm looking for,
    since (unless I'm mistaken) this still implies encoding the binary data to
    an intermediate byte slice before writing to the bytes.Buffer.

    i.e., the following does not work

    var b bytes.Buffer
    b.Grow(4)
    // len(b.Bytes()) == 0 here, b.Cap() == 64
    binary.BigEndian.PutUint32(b.Bytes(), 0x1234) // fails with index out of
    range, since len(b.Bytes()) is 0

    even if this worked, we'd still need a way to tell b that we've written
    those bytes there (i.e., b.Len() would still return 0 at this point since
    we haven't modified the unexported b.off field).

    so we're left with doing something like:

    buf := make([]byte, 4)
    binary.BigEndian.PutUint32(buf, 0x1234)
    b.Write(buf)

    which, in addition to being noisier and less direct, involves an extra
    copy step (and possibly allocation?).



    On Tue, Jan 5, 2016, at 10:26 AM, Brad Fitzpatrick wrote:

    Use:
    https://golang.org/pkg/bytes/#Buffer.Grow
    https://golang.org/pkg/bytes/#Buffer.Cap
    https://golang.org/pkg/bytes/#Buffer.Bytes

    .... along with binary.BigEndian, etc.


    On Tue, Jan 5, 2016 at 8:20 AM, Liam Staskawicz wrote:

    Yes. Buffer.Bytes() is not ideal in this case, since it's not possible to
    grow the underlying byte slice when writing to it, or move the read/write
    pointer of the buffer.

    The idea with this api is that it would support encoding/decoding data
    directly to/from the buffer:

    var b bytes.Buffer
    order := binary.BigEndian
    b.WriteUint32(order, 0x1234)
    // b.Len() == 4
    v, err := b.ReadUint32(order)
    // v == 0x1234, b.Len() == 0


    On Tuesday, January 5, 2016 at 2:30:14 AM UTC-8, thebroke...@gmail.com
    wrote:

    Have you tried using the Buffer.Bytes method
    <https://golang.org/pkg/bytes/#Buffer.Bytes>?

    On Tuesday, January 5, 2016 at 2:16:55 AM UTC-8, Liam Staskawicz wrote:

    Hello,

    I'm interested in implementing binary ops (i.e., those on
    encoding/binary.ByteOrder) on the underlying byte slice in a bytes.Buffer,
    such that encoding/decoding binary data to intermediate buffers could be
    avoided.

    I haven't found a good way to access the underlying byte slice on
    bytes.Buffer, so was considering creating a package that carries a copy of
    bytes.Buffer (and maybe bytes.Reader), and adds routines like these:
    http://pastie.org/10659323
    <http://www.google.com/url?q=http%3A%2F%2Fpastie.org%2F10659323&sa=D&sntz=1&usg=AFQjCNHzfWYc6SyxtbiXYIqB-JOkVwjJnA>

    Is there an existing package that provides this functionality, or a better
    way to do this?

    Thanks,
    Liam


    --
    You received this message because you are subscribed to the Google Groups
    "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an
    email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.

    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
  • Liam Staskawicz at Jan 5, 2016 at 8:41 pm
    Right - those temp buffers are the thing I'm trying to avoid :)

    Those ways avoid the extra allocation, but trade the copy step from the
    temp buffer for a copy step writing the temp "xxxx", which is still a
    bit suboptimal but seems unavoidable.

    Sounds like I can either live with that, or create a separate
    package. Thanks!

    Liam

    On Tue, Jan 5, 2016, at 10:53 AM, Brad Fitzpatrick wrote:
    Your way allocates with those temporary 4-byte buf buffers.

    This way doesn't: http://play.golang.org/p/lBw-Kj7QZ5 or without Grow:
    http://play.golang.org/p/eUNWnzobKV


    On Tue, Jan 5, 2016 at 10:39 AM, Liam Staskawicz
    wrote:
    __
    Thanks. I think these don't provide the functionality I'm looking
    for, since (unless I'm mistaken) this still implies encoding the
    binary data to an intermediate byte slice before writing to the
    bytes.Buffer.

    i.e., the following does not work

    var b bytes.Buffer
    b.Grow(4) // len(b.Bytes()) == 0 here, b.Cap() == 64
    binary.BigEndian.PutUint32(b.Bytes(), 0x1234) // fails with index
    out of range, since len(b.Bytes())  is 0

    even if this worked, we'd still need a way to tell b that we've
    written those bytes there (i.e., b.Len() would still return 0 at this
    point since we haven't modified the unexported b.off field).

    so we're left with doing something like:

    buf := make([]byte, 4) binary.BigEndian.PutUint32(buf, 0x1234)
    b.Write(buf)

    which, in addition to being noisier and less direct, involves an
    extra copy step (and possibly allocation?).


    On Tue, Jan 5, 2016, at 10:26 AM, Brad Fitzpatrick wrote:
    Use: https://golang.org/pkg/bytes/#Buffer.Grow
    https://golang.org/pkg/bytes/#Buffer.Cap
    https://golang.org/pkg/bytes/#Buffer.Bytes

    .... along with binary.BigEndian, etc.


    On Tue, Jan 5, 2016 at 8:20 AM, Liam Staskawicz <liam@stask.net>
    wrote:
    Yes. Buffer.Bytes() is not ideal in this case, since it's not
    possible to grow the underlying byte slice when writing to it, or
    move the read/write pointer of the buffer.

    The idea with this api is that it would support encoding/decoding
    data directly to/from the buffer:

    var b bytes.Buffer order := binary.BigEndian
    b.WriteUint32(order, 0x1234) // b.Len() == 4 v, err :=
    b.ReadUint32(order) // v == 0x1234, b.Len() == 0


    On Tuesday, January 5, 2016 at 2:30:14 AM UTC-8,
    thebroke...@gmail.com wrote:
    Have you tried using the Buffer.Bytes method[1]?

    On Tuesday, January 5, 2016 at 2:16:55 AM UTC-8, Liam Staskawicz
    wrote:
    Hello,

    I'm interested in implementing binary ops (i.e., those on
    encoding/binary.ByteOrder) on the underlying byte slice in a
    bytes.Buffer, such that encoding/decoding binary data to
    intermediate buffers could be avoided.

    I haven't found a good way to access the underlying byte slice on
    bytes.Buffer, so was considering creating a package that carries
    a copy of bytes.Buffer (and maybe bytes.Reader), and adds
    routines like these: http://pastie.org/10659323[2]

    Is there an existing package that provides this functionality, or
    a better way to do this?

    Thanks, Liam
    --
    You received this message because you are subscribed to the Google
    Groups "golang-nuts" group. To unsubscribe from this group and stop
    receiving emails from it, send an email to golang-
    nuts+unsubscribe@googlegroups.com. For more options, visit
    https://groups.google.com/d/optout.


    Links:

       1. https://golang.org/pkg/bytes/#Buffer.Bytes
       2. http://www.google.com/url?q=http%3A%2F%2Fpastie.org%2F10659323&sa=D&sntz=1&usg=AFQjCNHzfWYc6SyxtbiXYIqB-JOkVwjJnA

    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
  • Brad Fitzpatrick at Jan 5, 2016 at 7:39 pm

    On Tue, Jan 5, 2016 at 11:26 AM, Liam Staskawicz wrote:

    Right - those temp buffers are the thing I'm trying to avoid :)

    Those ways avoid the extra allocation, but trade the copy step from the
    temp buffer for a copy step writing the temp "xxxx", which is still a bit
    suboptimal but seems unavoidable.
    It's basically free, especially compared to an allocation.


    Sounds like I can either live with that, or create a separate package.
    Thanks!

    Liam


    On Tue, Jan 5, 2016, at 10:53 AM, Brad Fitzpatrick wrote:

    Your way allocates with those temporary 4-byte buf buffers.

    This way doesn't: http://play.golang.org/p/lBw-Kj7QZ5
    or without Grow: http://play.golang.org/p/eUNWnzobKV


    On Tue, Jan 5, 2016 at 10:39 AM, Liam Staskawicz wrote:


    Thanks. I think these don't provide the functionality I'm looking for,
    since (unless I'm mistaken) this still implies encoding the binary data to
    an intermediate byte slice before writing to the bytes.Buffer.

    i.e., the following does not work

    var b bytes.Buffer
    b.Grow(4)
    // len(b.Bytes()) == 0 here, b.Cap() == 64
    binary.BigEndian.PutUint32(b.Bytes(), 0x1234) // fails with index out of
    range, since len(b.Bytes()) is 0

    even if this worked, we'd still need a way to tell b that we've written
    those bytes there (i.e., b.Len() would still return 0 at this point since
    we haven't modified the unexported b.off field).

    so we're left with doing something like:

    buf := make([]byte, 4)
    binary.BigEndian.PutUint32(buf, 0x1234)
    b.Write(buf)

    which, in addition to being noisier and less direct, involves an extra
    copy step (and possibly allocation?).



    On Tue, Jan 5, 2016, at 10:26 AM, Brad Fitzpatrick wrote:

    Use:
    https://golang.org/pkg/bytes/#Buffer.Grow
    https://golang.org/pkg/bytes/#Buffer.Cap
    https://golang.org/pkg/bytes/#Buffer.Bytes

    .... along with binary.BigEndian, etc.


    On Tue, Jan 5, 2016 at 8:20 AM, Liam Staskawicz wrote:

    Yes. Buffer.Bytes() is not ideal in this case, since it's not possible to
    grow the underlying byte slice when writing to it, or move the read/write
    pointer of the buffer.

    The idea with this api is that it would support encoding/decoding data
    directly to/from the buffer:

    var b bytes.Buffer
    order := binary.BigEndian
    b.WriteUint32(order, 0x1234)
    // b.Len() == 4
    v, err := b.ReadUint32(order)
    // v == 0x1234, b.Len() == 0


    On Tuesday, January 5, 2016 at 2:30:14 AM UTC-8, thebroke...@gmail.com
    wrote:

    Have you tried using the Buffer.Bytes method
    <https://golang.org/pkg/bytes/#Buffer.Bytes>?

    On Tuesday, January 5, 2016 at 2:16:55 AM UTC-8, Liam Staskawicz wrote:

    Hello,

    I'm interested in implementing binary ops (i.e., those on
    encoding/binary.ByteOrder) on the underlying byte slice in a bytes.Buffer,
    such that encoding/decoding binary data to intermediate buffers could be
    avoided.

    I haven't found a good way to access the underlying byte slice on
    bytes.Buffer, so was considering creating a package that carries a copy of
    bytes.Buffer (and maybe bytes.Reader), and adds routines like these:
    http://pastie.org/10659323
    <http://www.google.com/url?q=http%3A%2F%2Fpastie.org%2F10659323&sa=D&sntz=1&usg=AFQjCNHzfWYc6SyxtbiXYIqB-JOkVwjJnA>

    Is there an existing package that provides this functionality, or a better
    way to do this?

    Thanks,
    Liam


    --
    You received this message because you are subscribed to the Google Groups
    "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an
    email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.



    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupgolang-nuts @
categoriesgo
postedJan 5, '16 at 10:16a
activeJan 5, '16 at 8:41p
posts8
users3
websitegolang.org

People

Translate

site design / logo © 2021 Grokbase