Re: [golang-dev] knob proposal: tweaking MHeap_Scavenger period

http://code.google.com/p/go/issues/detail?id=4171<http://code.google.com/p/go/issues/detail?id=4171&thanks=4171&ts=1348840249>

I'm working on an interactive application for visualising many gigabytes of
data. At the moment, I'm trying to implement a sort of "deep zoom" by
rendering a particular subset of the data to many textures, which needs
spare OS memory.

The problem I'm having is that for the application to be responsive I want
to keep as much of the data in (go's) RAM as possible for as long as
possible, then rapidly free up that memory so that the data can instead be
represented as textures which get drawn on screen. Go does turn HeapIdle
memory into free system memory, but it's currently hard coded how often it
will do it:

http://code.google.com/p/go/source/browse/src/pkg/runtime/mheap.c?spec=svn0a3866d6cc6b16e44ba6e8fb631a471ce81ca412&name=b96e722aad5a&r=0a3866d6cc6b16e44ba6e8fb631a471ce81ca412#349

The existing hard coded five minutes is too slow, since I'm running out of
system memory in a matter of ~10s. I've verified that memory has been freed
by the GC (it's HeapIdle), but not to the OS.

Ideally, I would be able to trigger freeing to the OS on demand, but I
would also settle for the `limit` variable in runtime·MHeap_Scavenger
becoming controllable through `pkg/runtime`-controlled variable.

If I submitted a patch implementing either of these modifications to the
runtime, would it be accepted? Alternatively should an item be made on the
issue tracker?

Are there other ways forward which I might have missed? I could try and
hack around this in my application, but it might ultimately require
sacrificing some of the responsiveness, in addition to making things more
complicated. I also doubt I'll be the last person to find myself wanting to
use lots of memory for a short time, then make it available for some other
process.

Thanks in advance.

Hello

You'd have to do some work to make it portable, but have you considered
allocating your many gigabytes of memory with syscall.Mmap instead of on
the Go heap?

Regards

Albert

This is obviously one possibility, but it's not a nice route to go down
with respect to maintainability.

There are lots of ways of hacking around the problem, but if I didn't want
to use Go's nice features (which come with garbage collection as a sort of
side effect/useful feature), I would have written C instead! :-)

With regards to the issue, I was lead to believe that go did actually free
the memory to the system after some time; but I haven't seen it for myself.
I interpreted this feature as introducing this feature:

http://code.google.com/p/go/source/detail?spec=svnb96e722aad5a97990f754895bc8831875c65488e&name=b96e722aad5a&r=b96e722aad5a97990f754895bc8831875c65488e

However, I tried reducing the `limit` down to 1s and this doesn't actually
appear to do anything. mortdeus pointed out on IRC:

http://code.google.com/p/go/source/browse/src/pkg/runtime/malloc.h?spec=svnb96e722aad5a97990f754895bc8831875c65488e&name=b96e722aad5a&r=b96e722aad5a97990f754895bc8831875c65488e#59

The code mentions that freeing memory to the OS isn't implemented
("TODO(rsc): Step 4 is not implemented.").

So this is possibly quite a bit less trivial than I initially thought on
the go runtime side.

The comment is not totally accurate. Freeing memory to the OS is
indeed implemented, but maybe not for all operating systems. Which one
are you using?

Rémy.

Linux x86_64.

Then it is implemented, but mind that it is an "advisory" release: the
kernel is just hinted about the situation, and he's free to recycle pages
whenever he wants taking into consideration all the other information he
has (memory pressure mainly).

For performance reasons, pages also require to be idle for more than 5 mn
(iirc) before being freed in the current implementation.

I'm not sure however that tweaking scavenger's frequency alone is the best
way to go given the very short time frame you have to release and transfer
all the memory to your other process (<10s). A standalone
"ReleaseAllUnusedMemNow()" call might be more appropriate.. provided that
you can kick it at the right moment.

Do you have any small reproducer of the situation you're trying to solve
(incl. resources info)?

Sebastien

Hi Sebastian,

I just whipped something up to illustrate the problem: it's attached. I did
my best to satisfy the "small" requirement but mostly I pasted things in
from other projects without modifying them too much. Output is pasted below.

Quick explanation: The program will consume some number of GB of memory
(with the -n) parameter. It writes one byte into every page to force the
memory to be resident. It will wait some time, then use the memory to
evaluate something stupid, then try to free it by assigning an empty byte
slice and calling runtime.GC(). In order to simulate "memory outside of the
go process", the program runs itself again, if this is the first time it is
running.

What you see is that `Inuse` increases during allocation, and decreases as
`Idle` goes up when you "free" (& GC) the memory. Free system memory is the
far left number, and never goes down (which is what we would like to fix).

Note that if "-n" is to high, the second execution fails because exec/fork
fails due to lack of memory.

Oddity which I don't understand: The GC actually somehow "misses" 1GB. I
haven't figured out why yet, I guess it is some feature of my program.

Thanks for taking a look,

- Peter

$ go build && time ./go-consumememory -n=3
[parent] 18:42:01.564907 --------- Program start ----------
[parent] 18:42:01.564954 SysFree Alloc Total Sys |
Heap-> | Alloc Sys Idle Inuse Relsd N
[parent] 18:42:01.565299 6.72GB 345.26kB 359.66kB 2.63MB
345.26kB 1.05MB 585.73kB 462.85kB 0.00B 148
[parent] 18:42:01.739056 --------- After allocation 0 ----------
[parent] 18:42:01.739124 SysFree Alloc Total Sys |
Heap-> | Alloc Sys Idle Inuse Relsd N
[parent] 18:42:01.739511 5.72GB 1.00GB 1.00GB 1.07GB
1.00GB 1.00GB 512.00kB 1.00GB 0.00B 177
[parent] 18:42:01.912637 --------- After allocation 1 ----------
[parent] 18:42:01.912672 SysFree Alloc Total Sys |
Heap-> | Alloc Sys Idle Inuse Relsd N
[parent] 18:42:01.913047 4.71GB 2.00GB 2.00GB 2.13GB
2.00GB 2.00GB 475.14kB 2.00GB 0.00B 575
[parent] 18:42:02.086908 --------- After allocation 2 ----------
[parent] 18:42:02.086944 SysFree Alloc Total Sys |
Heap-> | Alloc Sys Idle Inuse Relsd N
[parent] 18:42:02.087332 3.71GB 3.00GB 3.00GB 3.19GB
3.00GB 3.00GB 487.42kB 3.00GB 0.00B 196
[parent] 18:42:02.087341 --------- Sleeping for 1s ----------
[parent] 18:42:03.099030 Your lucky bit is: 1
[parent] 18:42:03.099049 --------- Before Forced GC ----------
[parent] 18:42:03.099059 SysFree Alloc Total Sys |
Heap-> | Alloc Sys Idle Inuse Relsd N
[parent] 18:42:03.099409 3.71GB 3.00GB 3.00GB 3.19GB
3.00GB 3.00GB 483.33kB 3.00GB 0.00B 419
[parent] 18:42:03.099666 --------- After Forced GC ----------
[parent] 18:42:03.099699 SysFree Alloc Total Sys |
Heap-> | Alloc Sys Idle Inuse Relsd N
[parent] 18:42:03.100026 3.71GB 1.00GB 3.00GB 3.19GB
1.00GB 3.00GB 2.00GB 1.00GB 0.00B 194

[parent] 18:42:04.100254 -- Running child : ./go-consumememory --

[!child] 18:42:04.142488 --------- Program start ----------
[!child] 18:42:04.142545 SysFree Alloc Total Sys |
Heap-> | Alloc Sys Idle Inuse Relsd N
[!child] 18:42:04.142885 3.71GB 345.27kB 359.68kB 2.63MB
345.27kB 1.05MB 585.73kB 462.85kB 0.00B 148
[!child] 18:42:04.318438 --------- After allocation 0 ----------
[!child] 18:42:04.318506 SysFree Alloc Total Sys |
Heap-> | Alloc Sys Idle Inuse Relsd N
[!child] 18:42:04.318879 2.70GB 1.00GB 1.00GB 1.07GB
1.00GB 1.00GB 512.00kB 1.00GB 0.00B 177
[!child] 18:42:04.494097 --------- After allocation 1 ----------
[!child] 18:42:04.494137 SysFree Alloc Total Sys |
Heap-> | Alloc Sys Idle Inuse Relsd N
[!child] 18:42:04.494513 1.70GB 2.00GB 2.00GB 2.13GB
2.00GB 2.00GB 475.14kB 2.00GB 0.00B 573
[!child] 18:42:04.670291 --------- After allocation 2 ----------
[!child] 18:42:04.670318 SysFree Alloc Total Sys |
Heap-> | Alloc Sys Idle Inuse Relsd N
[!child] 18:42:04.670682 695.81MB 3.00GB 3.00GB 3.19GB
3.00GB 3.00GB 487.42kB 3.00GB 0.00B 192
[!child] 18:42:04.671242 Ran out of memory; 695.81MB < 1.00GB
panic: exit status 1

goroutine 1 [running]:
main.main()
/home/pwaller/Projects/go-consumememory/main.go:106 +0x954

goroutine 2 [syscall]:
created by runtime.main
.../go/src/pkg/runtime/proc.c:220

goroutine 3 [timer goroutine (idle)]:
created by addtimer
.../go/src/pkg/runtime/ztime_linux_amd64.c:72

real 0m3.239s
user 0m0.148s
sys 0m1.082s