FAQ

On Fri, Sep 28, 2012 at 6:36 AM, Peter Waller wrote:
Ideally, I would be able to trigger freeing to the OS on demand, but I would
also settle for the `limit` variable in runtime·MHeap_Scavenger becoming
controllable through `pkg/runtime`-controlled variable.
It's hard for me to get excited about a GC tuning parameter, but
personally I don't see anything wrong with a runtime function that
runs the heap scavenger, just as we already have a runtime function
that runs the garbage collector. Please open an issue for that.
Thanks.

Ian

Search Discussions

  • Peter Waller at Sep 28, 2012 at 1:51 pm

    On Friday, 28 September 2012 14:44:40 UTC+1, Ian Lance Taylor wrote:
    I don't see anything wrong with a runtime function that
    runs the heap scavenger, just as we already have a runtime function
    that runs the garbage collector. Please open an issue for that.
    http://code.google.com/p/go/issues/detail?id=4171<http://code.google.com/p/go/issues/detail?id=4171&thanks=4171&ts=1348840249>
  • Peter Waller at Sep 28, 2012 at 2:39 pm
    I'm working on an interactive application for visualising many gigabytes of
    data. At the moment, I'm trying to implement a sort of "deep zoom" by
    rendering a particular subset of the data to many textures, which needs
    spare OS memory.

    The problem I'm having is that for the application to be responsive I want
    to keep as much of the data in (go's) RAM as possible for as long as
    possible, then rapidly free up that memory so that the data can instead be
    represented as textures which get drawn on screen. Go does turn HeapIdle
    memory into free system memory, but it's currently hard coded how often it
    will do it:

    http://code.google.com/p/go/source/browse/src/pkg/runtime/mheap.c?spec=svn0a3866d6cc6b16e44ba6e8fb631a471ce81ca412&name=b96e722aad5a&r=0a3866d6cc6b16e44ba6e8fb631a471ce81ca412#349

    The existing hard coded five minutes is too slow, since I'm running out of
    system memory in a matter of ~10s. I've verified that memory has been freed
    by the GC (it's HeapIdle), but not to the OS.

    Ideally, I would be able to trigger freeing to the OS on demand, but I
    would also settle for the `limit` variable in runtime·MHeap_Scavenger
    becoming controllable through `pkg/runtime`-controlled variable.

    If I submitted a patch implementing either of these modifications to the
    runtime, would it be accepted? Alternatively should an item be made on the
    issue tracker?

    Are there other ways forward which I might have missed? I could try and
    hack around this in my application, but it might ultimately require
    sacrificing some of the responsiveness, in addition to making things more
    complicated. I also doubt I'll be the last person to find myself wanting to
    use lots of memory for a short time, then make it available for some other
    process.

    Thanks in advance.
  • Albert Strasheim at Sep 28, 2012 at 5:08 pm
    Hello

    You'd have to do some work to make it portable, but have you considered
    allocating your many gigabytes of memory with syscall.Mmap instead of on
    the Go heap?

    Regards

    Albert
  • Peter Waller at Sep 29, 2012 at 11:21 pm

    On Friday, 28 September 2012 18:08:46 UTC+1, Albert Strasheim wrote:
    You'd have to do some work to make it portable, but have you considered
    allocating your many gigabytes of memory with syscall.Mmap instead of on
    the Go heap?

    This is obviously one possibility, but it's not a nice route to go down
    with respect to maintainability.

    There are lots of ways of hacking around the problem, but if I didn't want
    to use Go's nice features (which come with garbage collection as a sort of
    side effect/useful feature), I would have written C instead! :-)

    With regards to the issue, I was lead to believe that go did actually free
    the memory to the system after some time; but I haven't seen it for myself.
    I interpreted this feature as introducing this feature:

    http://code.google.com/p/go/source/detail?spec=svnb96e722aad5a97990f754895bc8831875c65488e&name=b96e722aad5a&r=b96e722aad5a97990f754895bc8831875c65488e

    However, I tried reducing the `limit` down to 1s and this doesn't actually
    appear to do anything. mortdeus pointed out on IRC:

    http://code.google.com/p/go/source/browse/src/pkg/runtime/malloc.h?spec=svnb96e722aad5a97990f754895bc8831875c65488e&name=b96e722aad5a&r=b96e722aad5a97990f754895bc8831875c65488e#59

    The code mentions that freeing memory to the OS isn't implemented
    ("TODO(rsc): Step 4 is not implemented.").

    So this is possibly quite a bit less trivial than I initially thought on
    the go runtime side.
  • Rémy Oudompheng at Sep 30, 2012 at 7:33 am

    On 2012/9/30 Peter Waller wrote:
    On Friday, 28 September 2012 18:08:46 UTC+1, Albert Strasheim wrote:

    You'd have to do some work to make it portable, but have you considered
    allocating your many gigabytes of memory with syscall.Mmap instead of on the
    Go heap?

    This is obviously one possibility, but it's not a nice route to go down with
    respect to maintainability.

    There are lots of ways of hacking around the problem, but if I didn't want
    to use Go's nice features (which come with garbage collection as a sort of
    side effect/useful feature), I would have written C instead! :-)

    With regards to the issue, I was lead to believe that go did actually free
    the memory to the system after some time; but I haven't seen it for myself.
    I interpreted this feature as introducing this feature:

    http://code.google.com/p/go/source/detail?spec=svnb96e722aad5a97990f754895bc8831875c65488e&name=b96e722aad5a&r=b96e722aad5a97990f754895bc8831875c65488e

    However, I tried reducing the `limit` down to 1s and this doesn't actually
    appear to do anything. mortdeus pointed out on IRC:

    http://code.google.com/p/go/source/browse/src/pkg/runtime/malloc.h?spec=svnb96e722aad5a97990f754895bc8831875c65488e&name=b96e722aad5a&r=b96e722aad5a97990f754895bc8831875c65488e#59

    The code mentions that freeing memory to the OS isn't implemented
    ("TODO(rsc): Step 4 is not implemented.").
    The comment is not totally accurate. Freeing memory to the OS is
    indeed implemented, but maybe not for all operating systems. Which one
    are you using?

    Rémy.
  • Peter Waller at Sep 30, 2012 at 10:54 am

    On 30 September 2012 08:33, Rémy Oudompheng wrote:
    The code mentions that freeing memory to the OS isn't implemented
    ("TODO(rsc): Step 4 is not implemented.").
    The comment is not totally accurate. Freeing memory to the OS is
    indeed implemented, but maybe not for all operating systems. Which one
    are you using?

    Linux x86_64.
  • Sébastien Paolacci at Sep 30, 2012 at 1:52 pm
    Then it is implemented, but mind that it is an "advisory" release: the
    kernel is just hinted about the situation, and he's free to recycle pages
    whenever he wants taking into consideration all the other information he
    has (memory pressure mainly).

    For performance reasons, pages also require to be idle for more than 5 mn
    (iirc) before being freed in the current implementation.

    I'm not sure however that tweaking scavenger's frequency alone is the best
    way to go given the very short time frame you have to release and transfer
    all the memory to your other process (<10s). A standalone
    "ReleaseAllUnusedMemNow()" call might be more appropriate.. provided that
    you can kick it at the right moment.

    Do you have any small reproducer of the situation you're trying to solve
    (incl. resources info)?

    Sebastien
    On Sep 30, 2012 12:54 PM, "Peter Waller" wrote:
    On 30 September 2012 08:33, Rémy Oudompheng wrote:

    The code mentions that freeing memory to the OS isn't implemented
    ("TODO(rsc): Step 4 is not implemented.").
    The comment is not totally accurate. Freeing memory to the OS is
    indeed implemented, but maybe not for all operating systems. Which one
    are you using?

    Linux x86_64.
  • Peter Waller at Sep 30, 2012 at 5:54 pm
    Hi Sebastian,
    On 30 September 2012 14:52, Sébastien Paolacci wrote:

    Do you have any small reproducer of the situation you're trying to solve
    (incl. resources info)?

    I just whipped something up to illustrate the problem: it's attached. I did
    my best to satisfy the "small" requirement but mostly I pasted things in
    from other projects without modifying them too much. Output is pasted below.

    Quick explanation: The program will consume some number of GB of memory
    (with the -n) parameter. It writes one byte into every page to force the
    memory to be resident. It will wait some time, then use the memory to
    evaluate something stupid, then try to free it by assigning an empty byte
    slice and calling runtime.GC(). In order to simulate "memory outside of the
    go process", the program runs itself again, if this is the first time it is
    running.

    What you see is that `Inuse` increases during allocation, and decreases as
    `Idle` goes up when you "free" (& GC) the memory. Free system memory is the
    far left number, and never goes down (which is what we would like to fix).

    Note that if "-n" is to high, the second execution fails because exec/fork
    fails due to lack of memory.

    Oddity which I don't understand: The GC actually somehow "misses" 1GB. I
    haven't figured out why yet, I guess it is some feature of my program.

    Thanks for taking a look,

    - Peter

    $ go build && time ./go-consumememory -n=3
    [parent] 18:42:01.564907 --------- Program start ----------
    [parent] 18:42:01.564954 SysFree Alloc Total Sys |
    Heap-> | Alloc Sys Idle Inuse Relsd N
    [parent] 18:42:01.565299 6.72GB 345.26kB 359.66kB 2.63MB
    345.26kB 1.05MB 585.73kB 462.85kB 0.00B 148
    [parent] 18:42:01.739056 --------- After allocation 0 ----------
    [parent] 18:42:01.739124 SysFree Alloc Total Sys |
    Heap-> | Alloc Sys Idle Inuse Relsd N
    [parent] 18:42:01.739511 5.72GB 1.00GB 1.00GB 1.07GB
    1.00GB 1.00GB 512.00kB 1.00GB 0.00B 177
    [parent] 18:42:01.912637 --------- After allocation 1 ----------
    [parent] 18:42:01.912672 SysFree Alloc Total Sys |
    Heap-> | Alloc Sys Idle Inuse Relsd N
    [parent] 18:42:01.913047 4.71GB 2.00GB 2.00GB 2.13GB
    2.00GB 2.00GB 475.14kB 2.00GB 0.00B 575
    [parent] 18:42:02.086908 --------- After allocation 2 ----------
    [parent] 18:42:02.086944 SysFree Alloc Total Sys |
    Heap-> | Alloc Sys Idle Inuse Relsd N
    [parent] 18:42:02.087332 3.71GB 3.00GB 3.00GB 3.19GB
    3.00GB 3.00GB 487.42kB 3.00GB 0.00B 196
    [parent] 18:42:02.087341 --------- Sleeping for 1s ----------
    [parent] 18:42:03.099030 Your lucky bit is: 1
    [parent] 18:42:03.099049 --------- Before Forced GC ----------
    [parent] 18:42:03.099059 SysFree Alloc Total Sys |
    Heap-> | Alloc Sys Idle Inuse Relsd N
    [parent] 18:42:03.099409 3.71GB 3.00GB 3.00GB 3.19GB
    3.00GB 3.00GB 483.33kB 3.00GB 0.00B 419
    [parent] 18:42:03.099666 --------- After Forced GC ----------
    [parent] 18:42:03.099699 SysFree Alloc Total Sys |
    Heap-> | Alloc Sys Idle Inuse Relsd N
    [parent] 18:42:03.100026 3.71GB 1.00GB 3.00GB 3.19GB
    1.00GB 3.00GB 2.00GB 1.00GB 0.00B 194

    [parent] 18:42:04.100254 -- Running child : ./go-consumememory --

    [!child] 18:42:04.142488 --------- Program start ----------
    [!child] 18:42:04.142545 SysFree Alloc Total Sys |
    Heap-> | Alloc Sys Idle Inuse Relsd N
    [!child] 18:42:04.142885 3.71GB 345.27kB 359.68kB 2.63MB
    345.27kB 1.05MB 585.73kB 462.85kB 0.00B 148
    [!child] 18:42:04.318438 --------- After allocation 0 ----------
    [!child] 18:42:04.318506 SysFree Alloc Total Sys |
    Heap-> | Alloc Sys Idle Inuse Relsd N
    [!child] 18:42:04.318879 2.70GB 1.00GB 1.00GB 1.07GB
    1.00GB 1.00GB 512.00kB 1.00GB 0.00B 177
    [!child] 18:42:04.494097 --------- After allocation 1 ----------
    [!child] 18:42:04.494137 SysFree Alloc Total Sys |
    Heap-> | Alloc Sys Idle Inuse Relsd N
    [!child] 18:42:04.494513 1.70GB 2.00GB 2.00GB 2.13GB
    2.00GB 2.00GB 475.14kB 2.00GB 0.00B 573
    [!child] 18:42:04.670291 --------- After allocation 2 ----------
    [!child] 18:42:04.670318 SysFree Alloc Total Sys |
    Heap-> | Alloc Sys Idle Inuse Relsd N
    [!child] 18:42:04.670682 695.81MB 3.00GB 3.00GB 3.19GB
    3.00GB 3.00GB 487.42kB 3.00GB 0.00B 192
    [!child] 18:42:04.671242 Ran out of memory; 695.81MB < 1.00GB
    panic: exit status 1

    goroutine 1 [running]:
    main.main()
    /home/pwaller/Projects/go-consumememory/main.go:106 +0x954

    goroutine 2 [syscall]:
    created by runtime.main
    .../go/src/pkg/runtime/proc.c:220

    goroutine 3 [timer goroutine (idle)]:
    created by addtimer
    .../go/src/pkg/runtime/ztime_linux_amd64.c:72

    real 0m3.239s
    user 0m0.148s
    sys 0m1.082s

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupgolang-dev @
categoriesgo
postedSep 28, '12 at 1:44p
activeSep 30, '12 at 5:54p
posts9
users5
websitegolang.org

People

Translate

site design / logo © 2022 Grokbase