[go-nuts] binary ops on bytes.Buffer

On Tuesday, January 5, 2016 at 2:16:55 AM UTC-8, Liam Staskawicz wrote:

Hello,

I'm interested in implementing binary ops (i.e., those on
encoding/binary.ByteOrder) on the underlying byte slice in a bytes.Buffer,
such that encoding/decoding binary data to intermediate buffers could be
avoided.

I haven't found a good way to access the underlying byte slice on
bytes.Buffer, so was considering creating a package that carries a copy of
bytes.Buffer (and maybe bytes.Reader), and adds routines like these:
http://pastie.org/10659323

Is there an existing package that provides this functionality, or a better
way to do this?

Thanks,
Liam

Have you tried using the Buffer.Bytes method
<https://golang.org/pkg/bytes/#Buffer.Bytes>?

On Tue, Jan 5, 2016 at 8:20 AM, Liam Staskawicz wrote:

Yes. Buffer.Bytes() is not ideal in this case, since it's not possible to
grow the underlying byte slice when writing to it, or move the read/write
pointer of the buffer.

The idea with this api is that it would support encoding/decoding data
directly to/from the buffer:

var b bytes.Buffer
order := binary.BigEndian
b.WriteUint32(order, 0x1234)
// b.Len() == 4
v, err := b.ReadUint32(order)
// v == 0x1234, b.Len() == 0


On Tuesday, January 5, 2016 at 2:30:14 AM UTC-8, thebroke...@gmail.com
wrote:
You received this message because you are subscribed to the Google Groups
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to golang-nuts+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

On Tue, Jan 5, 2016 at 10:39 AM, Liam Staskawicz wrote:

Thanks. I think these don't provide the functionality I'm looking for,
since (unless I'm mistaken) this still implies encoding the binary data to
an intermediate byte slice before writing to the bytes.Buffer.

i.e., the following does not work

var b bytes.Buffer
b.Grow(4)
// len(b.Bytes()) == 0 here, b.Cap() == 64
binary.BigEndian.PutUint32(b.Bytes(), 0x1234) // fails with index out of
range, since len(b.Bytes()) is 0

even if this worked, we'd still need a way to tell b that we've written
those bytes there (i.e., b.Len() would still return 0 at this point since
we haven't modified the unexported b.off field).

so we're left with doing something like:

buf := make([]byte, 4)
binary.BigEndian.PutUint32(buf, 0x1234)
b.Write(buf)

which, in addition to being noisier and less direct, involves an extra
copy step (and possibly allocation?).



On Tue, Jan 5, 2016, at 10:26 AM, Brad Fitzpatrick wrote:

Use:
https://golang.org/pkg/bytes/#Buffer.Grow
https://golang.org/pkg/bytes/#Buffer.Cap
https://golang.org/pkg/bytes/#Buffer.Bytes

.... along with binary.BigEndian, etc.


On Tue, Jan 5, 2016 at 8:20 AM, Liam Staskawicz wrote:

Yes. Buffer.Bytes() is not ideal in this case, since it's not possible to
grow the underlying byte slice when writing to it, or move the read/write
pointer of the buffer.

The idea with this api is that it would support encoding/decoding data
directly to/from the buffer:

var b bytes.Buffer
order := binary.BigEndian
b.WriteUint32(order, 0x1234)
// b.Len() == 4
v, err := b.ReadUint32(order)
// v == 0x1234, b.Len() == 0


On Tuesday, January 5, 2016 at 2:30:14 AM UTC-8, thebroke...@gmail.com
wrote:

Have you tried using the Buffer.Bytes method
<https://golang.org/pkg/bytes/#Buffer.Bytes>?

On Tuesday, January 5, 2016 at 2:16:55 AM UTC-8, Liam Staskawicz wrote:

Hello,

I'm interested in implementing binary ops (i.e., those on
encoding/binary.ByteOrder) on the underlying byte slice in a bytes.Buffer,
such that encoding/decoding binary data to intermediate buffers could be
avoided.

I haven't found a good way to access the underlying byte slice on
bytes.Buffer, so was considering creating a package that carries a copy of
bytes.Buffer (and maybe bytes.Reader), and adds routines like these:
http://pastie.org/10659323
<http://www.google.com/url?q=http%3A%2F%2Fpastie.org%2F10659323&sa=D&sntz=1&usg=AFQjCNHzfWYc6SyxtbiXYIqB-JOkVwjJnA>

Is there an existing package that provides this functionality, or a better
way to do this?

Thanks,
Liam


--
You received this message because you are subscribed to the Google Groups
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to golang-nuts+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

On Tue, Jan 5, 2016 at 11:26 AM, Liam Staskawicz wrote:

Right - those temp buffers are the thing I'm trying to avoid :)

Those ways avoid the extra allocation, but trade the copy step from the
temp buffer for a copy step writing the temp "xxxx", which is still a bit
suboptimal but seems unavoidable.

Sounds like I can either live with that, or create a separate package.
Thanks!

Liam


On Tue, Jan 5, 2016, at 10:53 AM, Brad Fitzpatrick wrote:

Your way allocates with those temporary 4-byte buf buffers.

This way doesn't: http://play.golang.org/p/lBw-Kj7QZ5
or without Grow: http://play.golang.org/p/eUNWnzobKV


On Tue, Jan 5, 2016 at 10:39 AM, Liam Staskawicz wrote:


Thanks. I think these don't provide the functionality I'm looking for,
since (unless I'm mistaken) this still implies encoding the binary data to
an intermediate byte slice before writing to the bytes.Buffer.

i.e., the following does not work

var b bytes.Buffer
b.Grow(4)
// len(b.Bytes()) == 0 here, b.Cap() == 64
binary.BigEndian.PutUint32(b.Bytes(), 0x1234) // fails with index out of
range, since len(b.Bytes()) is 0

even if this worked, we'd still need a way to tell b that we've written
those bytes there (i.e., b.Len() would still return 0 at this point since
we haven't modified the unexported b.off field).

so we're left with doing something like:

buf := make([]byte, 4)
binary.BigEndian.PutUint32(buf, 0x1234)
b.Write(buf)

which, in addition to being noisier and less direct, involves an extra
copy step (and possibly allocation?).



On Tue, Jan 5, 2016, at 10:26 AM, Brad Fitzpatrick wrote:

Use:
https://golang.org/pkg/bytes/#Buffer.Grow
https://golang.org/pkg/bytes/#Buffer.Cap
https://golang.org/pkg/bytes/#Buffer.Bytes

.... along with binary.BigEndian, etc.


On Tue, Jan 5, 2016 at 8:20 AM, Liam Staskawicz wrote:

Yes. Buffer.Bytes() is not ideal in this case, since it's not possible to
grow the underlying byte slice when writing to it, or move the read/write
pointer of the buffer.

The idea with this api is that it would support encoding/decoding data
directly to/from the buffer:

var b bytes.Buffer
order := binary.BigEndian
b.WriteUint32(order, 0x1234)
// b.Len() == 4
v, err := b.ReadUint32(order)
// v == 0x1234, b.Len() == 0


On Tuesday, January 5, 2016 at 2:30:14 AM UTC-8, thebroke...@gmail.com
wrote:

Have you tried using the Buffer.Bytes method
<https://golang.org/pkg/bytes/#Buffer.Bytes>?

On Tuesday, January 5, 2016 at 2:16:55 AM UTC-8, Liam Staskawicz wrote:

Hello,

I'm interested in implementing binary ops (i.e., those on
encoding/binary.ByteOrder) on the underlying byte slice in a bytes.Buffer,
such that encoding/decoding binary data to intermediate buffers could be
avoided.

I haven't found a good way to access the underlying byte slice on
bytes.Buffer, so was considering creating a package that carries a copy of
bytes.Buffer (and maybe bytes.Reader), and adds routines like these:
http://pastie.org/10659323
<http://www.google.com/url?q=http%3A%2F%2Fpastie.org%2F10659323&sa=D&sntz=1&usg=AFQjCNHzfWYc6SyxtbiXYIqB-JOkVwjJnA>

Is there an existing package that provides this functionality, or a better
way to do this?

Thanks,
Liam


--
You received this message because you are subscribed to the Google Groups
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to golang-nuts+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.