FAQ

On Saturday, October 27, 2012 6:18:20 AM UTC+1, darkgray wrote:
I was staring at my Windows task manager when I noticed my Go program was
taking up lots of RAM, so I wanted to see how much space Go gobbles up when
allocating a bunch of data in appended slices.

This is the code: http://play.golang.org/p/ZkmuFAUfiq

Output:
Size: 495
Mem: 1922
Heap: 1750
Size: 495
Mem: 2303
Heap: 2107

My understanding of the above is that the size of my slice is 495MB (+
overhead for slices themselves), but the code is using 1.9GB of RAM to keep
this in the air, even after forcing a GC when the slices should no longer
be reachable. I also tried running it twice, so see if it's simply
reserving RAM "softly" in case it will need it again, but allocating 495MB
again now appears to grow the RAM use further up to 2.3GB. Again, the
Windows Task Manager confirms that the Go program is "using" this much
private memory.

I have a pretty poor grasp of how Windows sees RAM taken or available, so
I would be grateful if someone could explain this.
Those calculations look a little off to me. Consider you have a slice of
sliices length 2e7. Assuming a pointer and an int are 4 bytes each that
means each of those slices are 12 bytes + the data. Now multiply that by
2e7 and add a bit for various overheads. Still sound unreasonable? Consider
that that parent slice has to be resized and a resizes means a copy of that
data to a large slice which means you will need at least two times the
memory of your old slice plus N. where N is the memory required for the
extra space allocated in the new slice



--

Search Discussions

  • Darkgray at Oct 27, 2012 at 6:52 pm
    I was staring at my Windows task manager when I noticed my Go program was
    taking up lots of RAM, so I wanted to see how much space Go gobbles up when
    allocating a bunch of data in appended slices.

    This is the code: http://play.golang.org/p/ZkmuFAUfiq

    Output:
    Size: 495
    Mem: 1922
    Heap: 1750
    Size: 495
    Mem: 2303
    Heap: 2107

    My understanding of the above is that the size of my slice is 495MB (+
    overhead for slices themselves), but the code is using 1.9GB of RAM to keep
    this in the air, even after forcing a GC when the slices should no longer
    be reachable. I also tried running it twice, so see if it's simply
    reserving RAM "softly" in case it will need it again, but allocating 495MB
    again now appears to grow the RAM use further up to 2.3GB. Again, the
    Windows Task Manager confirms that the Go program is "using" this much
    private memory.

    I have a pretty poor grasp of how Windows sees RAM taken or available, so I
    would be grateful if someone could explain this.

    --
  • Andrey mirtchovski at Oct 27, 2012 at 9:44 am
    with a few slight modifications to your code I'm seeing a strange
    divergence between linux and osx (and I assume windows sides with OSX here,
    from the runs you showed earlier). all platforms are amd64:

    http://play.golang.org/p/YJvafCiz4g

    On Linux, the first iteration already frees the whole array (6 milion frees
    plus a few extra):

    starting : numgc = 1 bytes = 261832 frees = 9 total = 264176 footprint =
    2625536 released = 0
    before GC: numgc = 18 bytes = 458819312 frees =84 total = 931905440
    footprint = 794239224 released = 0
    after GC : numgc = 19 bytes = 714592 frees =6000087 total = 931905600
    footprint = 794239224 released = 0

    On OSX, however, the first iteration doesn't free anything, instead
    accumulating the data:

    starting : numgc = 1 bytes = 267144 frees = 9 total = 269488 footprint =
    2625536 released = 0
    before GC: numgc = 17 bytes = 738128560 frees =80 total = 931906912
    footprint = 794239224 released = 0
    after GC : numgc = 18 bytes = 567821088 frees =83 total = 931907072
    footprint = 794239224 released = 0

    This continues for a while until the OSX GC ramps up and starts releasing
    data, starting with the second iteration and ramping up to full 6 million
    frees (and some more) on the third:

    starting : numgc = 18 bytes = 567825888 frees = 83 total = 931911840
    footprint = 794239224 released = 0
    before GC: numgc = 19 bytes = 1245122288 frees =132 total = 1863533344
    footprint = 1510375160 released = 0
    after GC : numgc = 20 bytes = 681298640 frees =3981731 total = 1863533504
    footprint = 1510375160 released = 0
    starting : numgc = 20 bytes = 681298752 frees = 3981732 total = 1863533664
    footprint = 1510375160 released = 0
    before GC: numgc = 21 bytes = 1250770000 frees =3981781 total = 2794890976
    footprint = 1637768952 released = 0
    after GC : numgc = 22 bytes = 718515472 frees =9320397 total = 2794891136
    footprint = 1637768952 released = 0

    uncommenting the memstats print in the alloc loop shows that the two GCs
    diverge around the time of the 16-th garbage collection run (not triggered
    by our code but by the runtime). Up until that point the two different
    traces are almost identical and they still track closely in footprint and
    total allocations...

    Linux, bytes allocated decrease:
    inside : numgc = 16 bytes = 284395632 frees = 132 total = 478179344
    footprint = 408283384 released = 0
    inside : numgc = 17 bytes = 273220880 frees = 146 total = 576004528
    footprint = 508831992 released = 0

    OSX:
    inside : numgc = 16 bytes = 284404016 frees = 132 total = 478186192
    footprint = 408283384 released = 0
    inside : numgc = 17 bytes = 382225616 frees = 144 total = 576009072
    footprint = 508831992 released = 0

    The end result of all this is that Linux never has a footprint above 2x the
    amount of data allocated in the tight loop, while OSX (and Windows,
    presumably) go way above that.

    I thought I had an easy answer for you, but now there's an actual mystery :)

    --
  • ⚛ at Oct 27, 2012 at 3:15 pm
    The size printed by your program is incorrect. You should use unsafe.Sizeof:

    http://play.golang.org/p/7gOBej2PIx

    Also, in my opinion, it is better to print HeapAlloc instead of HeapIdle
    and to use "HeapAlloc=%v" instead of "Heap=%v".

    After runtime.GC() the program should print "HeapAlloc=0". Is this the case
    when you run it on your machine?
    On Saturday, October 27, 2012 7:18:20 AM UTC+2, darkgray wrote:

    I was staring at my Windows task manager when I noticed my Go program was
    taking up lots of RAM, so I wanted to see how much space Go gobbles up when
    allocating a bunch of data in appended slices.

    This is the code: http://play.golang.org/p/ZkmuFAUfiq

    Output:
    Size: 495
    Mem: 1922
    Heap: 1750
    Size: 495
    Mem: 2303
    Heap: 2107

    My understanding of the above is that the size of my slice is 495MB (+
    overhead for slices themselves), but the code is using 1.9GB of RAM to keep
    this in the air, even after forcing a GC when the slices should no longer
    be reachable. I also tried running it twice, so see if it's simply
    reserving RAM "softly" in case it will need it again, but allocating 495MB
    again now appears to grow the RAM use further up to 2.3GB. Again, the
    Windows Task Manager confirms that the Go program is "using" this much
    private memory.

    I have a pretty poor grasp of how Windows sees RAM taken or available, so
    I would be grateful if someone could explain this.
    --
  • Darkgray at Oct 27, 2012 at 3:35 pm
    This version prints:
    Size: 801 MiB
    Sys: 1922
    HeapAlloc: 43
    Size: 801 MiB
    Sys: 2303
    HeapAlloc: 45

    I don't know what "heap idle" means, though, and there isn't much of an
    explanation in the docs. I presumed "memory allocated by not currently
    referenced by anything". Still confused as to why Sys is 2-3x larger than
    what my data seemingly requires.

    On Saturday, 27 October 2012 11:48:58 UTC+2, ⚛ wrote:

    The size printed by your program is incorrect. You should use
    unsafe.Sizeof:

    http://play.golang.org/p/7gOBej2PIx

    Also, in my opinion, it is better to print HeapAlloc instead of HeapIdle
    and to use "HeapAlloc=%v" instead of "Heap=%v".

    After runtime.GC() the program should print "HeapAlloc=0". Is this the
    case when you run it on your machine?
    On Saturday, October 27, 2012 7:18:20 AM UTC+2, darkgray wrote:

    I was staring at my Windows task manager when I noticed my Go program was
    taking up lots of RAM, so I wanted to see how much space Go gobbles up when
    allocating a bunch of data in appended slices.

    This is the code: http://play.golang.org/p/ZkmuFAUfiq

    Output:
    Size: 495
    Mem: 1922
    Heap: 1750
    Size: 495
    Mem: 2303
    Heap: 2107

    My understanding of the above is that the size of my slice is 495MB (+
    overhead for slices themselves), but the code is using 1.9GB of RAM to keep
    this in the air, even after forcing a GC when the slices should no longer
    be reachable. I also tried running it twice, so see if it's simply
    reserving RAM "softly" in case it will need it again, but allocating 495MB
    again now appears to grow the RAM use further up to 2.3GB. Again, the
    Windows Task Manager confirms that the Go program is "using" this much
    private memory.

    I have a pretty poor grasp of how Windows sees RAM taken or available, so
    I would be grateful if someone could explain this.
    --
  • ⚛ at Oct 27, 2012 at 5:20 pm

    On Saturday, October 27, 2012 4:00:08 PM UTC+2, darkgray wrote:

    This version prints:
    Size: 801 MiB
    Sys: 1922
    HeapAlloc: 43
    Size: 801 MiB
    Sys: 2303
    HeapAlloc: 45

    I don't know what "heap idle" means,

    Basically, it corresponds to the amount of free memory.

    though, and there isn't much of an explanation in the docs. I presumed
    "memory allocated by not currently referenced by anything". Still confused
    as to why Sys is 2-3x larger than what my data seemingly requires.
    Sys is 2x-3x larger than your expectations because:

    1. The append() function needs more than 2 times the memory when it is
    reallocating the array. For example, if the array has 100KiB then append()
    will probably allocate 100*1.25=125KiB of new memory (the multiplier
    depends on the total number of elements in the original array). In total,
    in your code append() needs 2.25 times the memory of the original array.
    Your array has 2e7 elements and each element has 3*8=24 bytes, so a higher
    bound for memory consumed by append() is 2.25*480MiB. We do not know when
    append() will decide to do reallocation, but we can assume that it may be
    close to 2.25*400=900 MiB. This is 900 MiB just for the outer array.

    2. There are 2e7 instances of the inner array "abc....xyz". Each instance
    has 26 bytes, but the size of the memory region needs to be aligned. So the
    actual size of an instance may be 32 bytes. 2e7*32=640MiB.

    3. The garbage collector needs 4 bits per machine word. This means that the
    total so far is (900+640)*(1+1/8*0.5) = 1636 MiB.

    These results may seem counter-intuitive because we can imagine the Go
    compiler and run-time to behave more rationally in the case of your code
    and allow the program's memory consumption to be closer to the expected 800
    MiB. However, the problem is that such a style of thinking is assuming
    magical (unreal) optimizations in the compiler. Some advanced compiler
    optimizations may happen in the future (maybe by a different Go compiler),
    but right now they aren't implemented.
    On Saturday, 27 October 2012 11:48:58 UTC+2, ⚛ wrote:

    The size printed by your program is incorrect. You should use
    unsafe.Sizeof:

    http://play.golang.org/p/7gOBej2PIx

    Also, in my opinion, it is better to print HeapAlloc instead of HeapIdle
    and to use "HeapAlloc=%v" instead of "Heap=%v".

    After runtime.GC() the program should print "HeapAlloc=0". Is this the
    case when you run it on your machine?
    On Saturday, October 27, 2012 7:18:20 AM UTC+2, darkgray wrote:

    I was staring at my Windows task manager when I noticed my Go program
    was taking up lots of RAM, so I wanted to see how much space Go gobbles up
    when allocating a bunch of data in appended slices.

    This is the code: http://play.golang.org/p/ZkmuFAUfiq

    Output:
    Size: 495
    Mem: 1922
    Heap: 1750
    Size: 495
    Mem: 2303
    Heap: 2107

    My understanding of the above is that the size of my slice is 495MB (+
    overhead for slices themselves), but the code is using 1.9GB of RAM to keep
    this in the air, even after forcing a GC when the slices should no longer
    be reachable. I also tried running it twice, so see if it's simply
    reserving RAM "softly" in case it will need it again, but allocating 495MB
    again now appears to grow the RAM use further up to 2.3GB. Again, the
    Windows Task Manager confirms that the Go program is "using" this much
    private memory.

    I have a pretty poor grasp of how Windows sees RAM taken or available,
    so I would be grateful if someone could explain this.
    --
  • Patrik Roos at Oct 27, 2012 at 5:30 pm
    Thanks! What about releasing memory back to the OS? Is it unnecessary?
    I tried a version that just sleeps for 5 minutes in main after
    dostuff(), forces GC, sleeps another 5 mins, forces GC, etc for ~20
    minutes, and the memory use in Task Manager remains at 2GB.

    Pardon the ignorance, if there's a good resource link to understand
    this I would be happy to read it instead of bothering you.
    On Sat, Oct 27, 2012 at 6:19 PM, ⚛ wrote:
    On Saturday, October 27, 2012 4:00:08 PM UTC+2, darkgray wrote:

    This version prints:
    Size: 801 MiB
    Sys: 1922
    HeapAlloc: 43
    Size: 801 MiB
    Sys: 2303
    HeapAlloc: 45

    I don't know what "heap idle" means,

    Basically, it corresponds to the amount of free memory.
    though, and there isn't much of an explanation in the docs. I presumed
    "memory allocated by not currently referenced by anything". Still confused
    as to why Sys is 2-3x larger than what my data seemingly requires.

    Sys is 2x-3x larger than your expectations because:

    1. The append() function needs more than 2 times the memory when it is
    reallocating the array. For example, if the array has 100KiB then append()
    will probably allocate 100*1.25=125KiB of new memory (the multiplier depends
    on the total number of elements in the original array). In total, in your
    code append() needs 2.25 times the memory of the original array. Your array
    has 2e7 elements and each element has 3*8=24 bytes, so a higher bound for
    memory consumed by append() is 2.25*480MiB. We do not know when append()
    will decide to do reallocation, but we can assume that it may be close to
    2.25*400=900 MiB. This is 900 MiB just for the outer array.

    2. There are 2e7 instances of the inner array "abc....xyz". Each instance
    has 26 bytes, but the size of the memory region needs to be aligned. So the
    actual size of an instance may be 32 bytes. 2e7*32=640MiB.

    3. The garbage collector needs 4 bits per machine word. This means that the
    total so far is (900+640)*(1+1/8*0.5) = 1636 MiB.

    These results may seem counter-intuitive because we can imagine the Go
    compiler and run-time to behave more rationally in the case of your code and
    allow the program's memory consumption to be closer to the expected 800 MiB.
    However, the problem is that such a style of thinking is assuming magical
    (unreal) optimizations in the compiler. Some advanced compiler optimizations
    may happen in the future (maybe by a different Go compiler), but right now
    they aren't implemented.
    On Saturday, 27 October 2012 11:48:58 UTC+2, ⚛ wrote:

    The size printed by your program is incorrect. You should use
    unsafe.Sizeof:

    http://play.golang.org/p/7gOBej2PIx

    Also, in my opinion, it is better to print HeapAlloc instead of HeapIdle
    and to use "HeapAlloc=%v" instead of "Heap=%v".

    After runtime.GC() the program should print "HeapAlloc=0". Is this the
    case when you run it on your machine?
    On Saturday, October 27, 2012 7:18:20 AM UTC+2, darkgray wrote:

    I was staring at my Windows task manager when I noticed my Go program
    was taking up lots of RAM, so I wanted to see how much space Go gobbles up
    when allocating a bunch of data in appended slices.

    This is the code: http://play.golang.org/p/ZkmuFAUfiq

    Output:
    Size: 495
    Mem: 1922
    Heap: 1750
    Size: 495
    Mem: 2303
    Heap: 2107

    My understanding of the above is that the size of my slice is 495MB (+
    overhead for slices themselves), but the code is using 1.9GB of RAM to keep
    this in the air, even after forcing a GC when the slices should no longer be
    reachable. I also tried running it twice, so see if it's simply reserving
    RAM "softly" in case it will need it again, but allocating 495MB again now
    appears to grow the RAM use further up to 2.3GB. Again, the Windows Task
    Manager confirms that the Go program is "using" this much private memory.

    I have a pretty poor grasp of how Windows sees RAM taken or available,
    so I would be grateful if someone could explain this.
    --
    --

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupgolang-nuts @
categoriesgo
postedOct 27, '12 at 9:00a
activeOct 27, '12 at 6:52p
posts7
users5
websitegolang.org

People

Translate

site design / logo © 2021 Grokbase