FAQ
Up until now I used buffer to append data. Unfortunately a buffer
(bytes.Buffer) copies the contents on Write

* copy(b.buf[m:], p)*

so it's not very memory efficient and will consume double the memory until
the next GC.
How can I achieve this without a copy?

--

Search Discussions

  • Bryanturley at Dec 4, 2012 at 5:23 am

    On Monday, December 3, 2012 11:19:15 PM UTC-6, Erik Aigner wrote:
    Up until now I used buffer to append data. Unfortunately a buffer
    (bytes.Buffer) copies the contents on Write

    * copy(b.buf[m:], p)*

    so it's not very memory efficient and will consume double the memory until
    the next GC.
    How can I achieve this without a copy?
    Are you asking how to copy without copying?
    Can you append with the append() builtin?
    http://golang.org/ref/spec#Appending_and_copying_slices

    You could take bytes.Buffer and rewrite it to your own ends.



    --
  • Erik Aigner at Dec 4, 2012 at 5:29 am
    No I'm not asking to copy without copying, I'm asking how to append without
    copying,
    but if append() already does that, than that's what I was looking for.

    I just wasn't sure what the builtin append() exactly does underneath (if it
    allocates memory)
    On Tuesday, December 4, 2012 6:23:19 AM UTC+1, bryanturley wrote:
    On Monday, December 3, 2012 11:19:15 PM UTC-6, Erik Aigner wrote:

    Up until now I used buffer to append data. Unfortunately a buffer
    (bytes.Buffer) copies the contents on Write

    * copy(b.buf[m:], p)*

    so it's not very memory efficient and will consume double the memory
    until the next GC.
    How can I achieve this without a copy?
    Are you asking how to copy without copying?
    Can you append with the append() builtin?
    http://golang.org/ref/spec#Appending_and_copying_slices

    You could take bytes.Buffer and rewrite it to your own ends.

    --
  • Bryanturley at Dec 4, 2012 at 5:46 am

    On Monday, December 3, 2012 11:29:39 PM UTC-6, Erik Aigner wrote:
    No I'm not asking to copy without copying, I'm asking how to append
    without copying,
    but if append() already does that, than that's what I was looking for.

    I just wasn't sure what the builtin append() exactly does underneath (if
    it allocates memory)
    Well if it *needs* to allocate new memory to append it does, which I have
    always assumed does a full copy.
    len := 10
    cap := 20 // capacity
    x := make([]byte, len, cap)

    you can append 10 bytes onto that without it doing an alloc.
    but if you never do it has 10 unused bytes...

    Slices have a backing array. You could try a linked list or other data
    structure if the copying is slowing down your program.

    --
  • David Leimbach at Dec 4, 2012 at 8:49 pm

    On Monday, December 3, 2012 9:46:19 PM UTC-8, bryanturley wrote:
    On Monday, December 3, 2012 11:29:39 PM UTC-6, Erik Aigner wrote:

    No I'm not asking to copy without copying, I'm asking how to append
    without copying,
    but if append() already does that, than that's what I was looking for.

    I just wasn't sure what the builtin append() exactly does underneath (if
    it allocates memory)
    Well if it *needs* to allocate new memory to append it does, which I have
    always assumed does a full copy.
    len := 10
    cap := 20 // capacity
    x := make([]byte, len, cap)

    you can append 10 bytes onto that without it doing an alloc.
    but if you never do it has 10 unused bytes...

    Slices have a backing array. You could try a linked list or other data
    structure if the copying is slowing down your program.
    When you make a slice you can set it's length and the backing array's
    capacity if you want to fine tune this.

    http://golang.org/ref/spec#Slice_types

    And yes a linked structure may work better if you need constant insertion
    time. Arrays are often the best way to get hardware level performance (due
    to caches and such).

    --
  • Sugu Sougoumarane at Dec 4, 2012 at 11:03 pm
    If your buffers are too large, you may want to use a chunked implementation
    like this one:
    http://code.google.com/p/vitess/source/browse/go/bytes2/chunked_writer.go

    But there are tradeoffs. So, you have to choose what you're willing to give
    up.
    On Tuesday, December 4, 2012 12:49:25 PM UTC-8, David Leimbach wrote:


    On Monday, December 3, 2012 9:46:19 PM UTC-8, bryanturley wrote:
    On Monday, December 3, 2012 11:29:39 PM UTC-6, Erik Aigner wrote:

    No I'm not asking to copy without copying, I'm asking how to append
    without copying,
    but if append() already does that, than that's what I was looking for.

    I just wasn't sure what the builtin append() exactly does underneath (if
    it allocates memory)
    Well if it *needs* to allocate new memory to append it does, which I have
    always assumed does a full copy.
    len := 10
    cap := 20 // capacity
    x := make([]byte, len, cap)

    you can append 10 bytes onto that without it doing an alloc.
    but if you never do it has 10 unused bytes...

    Slices have a backing array. You could try a linked list or other data
    structure if the copying is slowing down your program.
    When you make a slice you can set it's length and the backing array's
    capacity if you want to fine tune this.

    http://golang.org/ref/spec#Slice_types

    And yes a linked structure may work better if you need constant insertion
    time. Arrays are often the best way to get hardware level performance (due
    to caches and such).
    --
  • Job van der Zwan at Dec 4, 2012 at 11:54 pm
    Somebody just blogged about custom allocation the other day, so that's
    another source of inspiration to look at:

    http://blog.kowalczyk.info/article/u5o7/Speeding-up-Go-and-C-with-custom-allocators.html

    As I noted in the comments on Reddit: there's a subtle difference between
    arrays and slices, and he could have used an array instead of a slice like
    so <http://play.golang.org/p/5smf9XdmbD>, which might be even faster.
    On Wednesday, 5 December 2012 00:03:42 UTC+1, Sugu Sougoumarane wrote:

    If your buffers are too large, you may want to use a chunked
    implementation like this one:
    http://code.google.com/p/vitess/source/browse/go/bytes2/chunked_writer.go

    But there are tradeoffs. So, you have to choose what you're willing to
    give up.
    On Tuesday, December 4, 2012 12:49:25 PM UTC-8, David Leimbach wrote:


    On Monday, December 3, 2012 9:46:19 PM UTC-8, bryanturley wrote:
    On Monday, December 3, 2012 11:29:39 PM UTC-6, Erik Aigner wrote:

    No I'm not asking to copy without copying, I'm asking how to append
    without copying,
    but if append() already does that, than that's what I was looking for.

    I just wasn't sure what the builtin append() exactly does underneath
    (if it allocates memory)
    Well if it *needs* to allocate new memory to append it does, which I
    have always assumed does a full copy.
    len := 10
    cap := 20 // capacity
    x := make([]byte, len, cap)

    you can append 10 bytes onto that without it doing an alloc.
    but if you never do it has 10 unused bytes...

    Slices have a backing array. You could try a linked list or other data
    structure if the copying is slowing down your program.
    When you make a slice you can set it's length and the backing array's
    capacity if you want to fine tune this.

    http://golang.org/ref/spec#Slice_types

    And yes a linked structure may work better if you need constant insertion
    time. Arrays are often the best way to get hardware level performance (due
    to caches and such).
    --

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupgolang-nuts @
categoriesgo
postedDec 4, '12 at 5:19a
activeDec 4, '12 at 11:54p
posts7
users5
websitegolang.org

People

Translate

site design / logo © 2021 Grokbase