FAQ
I have a long running Go program that allocates a large amount of
memory over time. On FreeBSD 64 bit I am seeing a problem where once I
am done with the memory the RES size as reported by top never
decreases.

For example, after my program has finished processing requests I stop
all requests and let CPU drop to 0% and then leave it so that GC and
scavenging has time to work. Then I look at the RES size in top and
dump out a memory profile for pprof. In one example, I have RES of
15666M (i.e. 15GB) and pprof showing me:

(pprof) top10
Total: 32.7 MB

This is using Go version: devel +40ba4d4e4672 Tue Nov 13 10:45:30 2012
-0800

Are there any known problem with Go not returning memory to the OS on
that version (or the bleeding edge in general)? I do not see memory
problems like this on Linux. Although the nature of my test
environment means I can't do precisely the same test on Linux and
FreeBSD because FreeBSD as it a client site.

John.

--

Search Discussions

  • John Graham-Cumming at Nov 19, 2012 at 11:52 pm

    On Nov 19, 9:20 pm, John Graham-Cumming wrote:
    I have a long running Go program that allocates a large amount of
    memory over time. On FreeBSD 64 bit I am seeing a problem where once I
    am done with the memory the RES size as reported by top never
    decreases.
    Hmm. I think I might have found the answer to my question in
    mem_freebsd.c:

    void
    runtime·SysUnused(void *v, uintptr n)
    {
    USED(v);
    USED(n);
    // TODO(rsc): call madvise MADV_DONTNEED
    }

    It appears that it does not tell the OS that the memory is no longer
    needed. Am I correct?

    John.

    --
  • Dave Cheney at Nov 19, 2012 at 11:47 pm
    Hi John,

    I was just chasing that down. You are correct that implementing this
    method should activate the scavenger. I'll prepare a CL.

    Cheers

    Dave

    On Tue, Nov 20, 2012 at 10:45 AM, John Graham-Cumming
    wrote:
    On Nov 19, 9:20 pm, John Graham-Cumming wrote:
    I have a long running Go program that allocates a large amount of
    memory over time. On FreeBSD 64 bit I am seeing a problem where once I
    am done with the memory the RES size as reported by top never
    decreases.
    Hmm. I think I might have found the answer to my question in
    mem_freebsd.c:

    void
    runtime·SysUnused(void *v, uintptr n)
    {
    USED(v);
    USED(n);
    // TODO(rsc): call madvise MADV_DONTNEED
    }

    It appears that it does not tell the OS that the memory is no longer
    needed. Am I correct?

    John.

    --
    --
  • John Graham-Cumming at Nov 19, 2012 at 11:50 pm

    On Nov 19, 11:47 pm, Dave Cheney wrote:
    I was just chasing that down. You are correct that implementing this
    method should activate the scavenger. I'll prepare a CL.
    It looks like for xBSD (where x is Free, Net and Open) runtime.madvise
    is not implemented at all.

    John.

    --
  • John Graham-Cumming at Nov 20, 2012 at 3:47 am

    On Monday, November 19, 2012 11:51:04 PM UTC, John Graham-Cumming wrote:
    On Nov 19, 11:47 pm, Dave Cheney wrote:
    I was just chasing that down. You are correct that implementing this
    method should activate the scavenger. I'll prepare a CL.
    It looks like for xBSD (where x is Free, Net and Open) runtime.madvise
    is not implemented at all.

    I made an experimental patch to Go on FreeBSD amd64 by adding the following
    to sys_freebsd_amd64.s:

    TEXT runtime?madvise(SB),7,$0

    MOVQ 8(SP), DI

    MOVQ 16(SP), SI

    MOVQ 24(SP), DX

    MOVQ $75, AX // madvise

    SYSCALL

    CMPQ AX, $0xfffffffffffff001

    JLS 2(PC)

    MOVL $0xf1, 0xf1 // crash

    RET

    And changing mem_freebsd.c so that it actually gets called. That worked.

    John.

    --
  • Dave Cheney at Nov 20, 2012 at 3:46 am
    Yup, that is all that is needed. Do you want to propose a CL ?

    Cheers

    Dave
    On Tue, Nov 20, 2012 at 2:40 PM, John Graham-Cumming wrote:
    On Monday, November 19, 2012 11:51:04 PM UTC, John Graham-Cumming wrote:
    On Nov 19, 11:47 pm, Dave Cheney wrote:
    I was just chasing that down. You are correct that implementing this
    method should activate the scavenger. I'll prepare a CL.
    It looks like for xBSD (where x is Free, Net and Open) runtime.madvise
    is not implemented at all.

    I made an experimental patch to Go on FreeBSD amd64 by adding the following
    to sys_freebsd_amd64.s:

    TEXT runtime?madvise(SB),7,$0
    MOVQ 8(SP), DI
    MOVQ 16(SP), SI
    MOVQ 24(SP), DX
    MOVQ $75, AX // madvise
    SYSCALL
    CMPQ AX, $0xfffffffffffff001
    JLS 2(PC)
    MOVL $0xf1, 0xf1 // crash
    RET

    And changing mem_freebsd.c so that it actually gets called. That worked.

    John.

    --
    --
  • John Graham-Cumming at Nov 20, 2012 at 3:48 am

    On Tuesday, November 20, 2012 3:46:37 AM UTC, Dave Cheney wrote:
    Yup, that is all that is needed. Do you want to propose a CL ?
    What's needed is a CL for all the BSDs and both 32 and 64 bit. I don't have
    all those systems to test against.

    John.


    --
  • Devon H. O'Dell at Nov 20, 2012 at 3:50 am

    2012/11/19 John Graham-Cumming <jgrahamc@gmail.com>:
    On Tuesday, November 20, 2012 3:46:37 AM UTC, Dave Cheney wrote:

    Yup, that is all that is needed. Do you want to propose a CL ?

    What's needed is a CL for all the BSDs and both 32 and 64 bit. I don't have
    all those systems to test against.
    Just for reference (in case this isn't what you've done) I'm pretty
    sure that MADV_FREE is what we want on FreeBSD. (I don't know about
    OpenBSD or NetBSD). MADV_DONTNEED doesn't mean the same thing on Linux
    and FreeBSD.

    --dho
    John.


    --
    --
  • Dave Cheney at Nov 20, 2012 at 3:51 am
    Yup, that matches my research,

    see: https://codereview.appspot.com/6856066/
    On Tue, Nov 20, 2012 at 2:50 PM, Devon H. O'Dell wrote:
    2012/11/19 John Graham-Cumming <jgrahamc@gmail.com>:
    On Tuesday, November 20, 2012 3:46:37 AM UTC, Dave Cheney wrote:

    Yup, that is all that is needed. Do you want to propose a CL ?

    What's needed is a CL for all the BSDs and both 32 and 64 bit. I don't have
    all those systems to test against.
    Just for reference (in case this isn't what you've done) I'm pretty
    sure that MADV_FREE is what we want on FreeBSD. (I don't know about
    OpenBSD or NetBSD). MADV_DONTNEED doesn't mean the same thing on Linux
    and FreeBSD.

    --dho
    John.


    --
    --
    --
  • John Graham-Cumming at Nov 20, 2012 at 6:55 pm

    On Tuesday, November 20, 2012 3:51:47 AM UTC, Dave Cheney wrote:
    Yup, that matches my research,

    see: https://codereview.appspot.com/6856066/
    Are you planning a CL to fix this problem? My experiments with FreeBSD
    indicate that MADV_FREE is the way to go, but I am not enough of an expert
    on the BSD family to do a CL and feel confident that I have fully
    understood the solution.

    John.


    --
  • Devon H. O'Dell at Nov 20, 2012 at 8:03 pm

    2012/11/20 John Graham-Cumming <jgrahamc@gmail.com>:
    On Tuesday, November 20, 2012 3:51:47 AM UTC, Dave Cheney wrote:

    Yup, that matches my research,

    see: https://codereview.appspot.com/6856066/

    Are you planning a CL to fix this problem? My experiments with FreeBSD
    indicate that MADV_FREE is the way to go, but I am not enough of an expert
    on the BSD family to do a CL and feel confident that I have fully understood
    the solution.
    I think Dave was going to let you do it. We can review and make
    suggestions if you'd like to continue, otherwise I'm sure Dave can
    continue?

    --dho
    John.
    --
  • Dave Cheney at Nov 20, 2012 at 8:08 pm
    Yup. That was the plan, but if you would rather have me do it, please let
    me know.
    On 21 Nov 2012 07:03, "Devon H. O'Dell" wrote:

    2012/11/20 John Graham-Cumming <jgrahamc@gmail.com>:
    On Tuesday, November 20, 2012 3:51:47 AM UTC, Dave Cheney wrote:

    Yup, that matches my research,

    see: https://codereview.appspot.com/6856066/

    Are you planning a CL to fix this problem? My experiments with FreeBSD
    indicate that MADV_FREE is the way to go, but I am not enough of an expert
    on the BSD family to do a CL and feel confident that I have fully
    understood
    the solution.
    I think Dave was going to let you do it. We can review and make
    suggestions if you'd like to continue, otherwise I'm sure Dave can
    continue?

    --dho
    John.
    --

    --
  • John Graham-Cumming at Nov 20, 2012 at 8:09 pm

    On Tuesday, November 20, 2012 8:08:10 PM UTC, Dave Cheney wrote:
    Yup. That was the plan, but if you would rather have me do it, please let
    me know.
    I'll prepare a patch and submit for FreeBSD amd64 which I have tested.
    Others can expand to the other platforms.

    John.


    --
  • Dave Cheney at Nov 20, 2012 at 8:14 pm
    SGTM.
    On 21 Nov 2012 07:09, "John Graham-Cumming" wrote:
    On Tuesday, November 20, 2012 8:08:10 PM UTC, Dave Cheney wrote:

    Yup. That was the plan, but if you would rather have me do it, please let
    me know.
    I'll prepare a patch and submit for FreeBSD amd64 which I have tested.
    Others can expand to the other platforms.

    John.


    --

    --
  • John Graham-Cumming at Nov 20, 2012 at 8:52 pm
    Done. Patch is here: http://codereview.appspot.com/6850081

    John.

    --
  • Devon H. O'Dell at Nov 20, 2012 at 9:44 pm
    FWIW i386 should basically just be the same code; the assembler should
    be simple enough to crib from elsewhere.

    --dho

    2012/11/20 Dave Cheney <dave@cheney.net>:
    SGTM.
    On 21 Nov 2012 07:09, "John Graham-Cumming" wrote:
    On Tuesday, November 20, 2012 8:08:10 PM UTC, Dave Cheney wrote:

    Yup. That was the plan, but if you would rather have me do it, please let
    me know.
    I'll prepare a patch and submit for FreeBSD amd64 which I have tested.
    Others can expand to the other platforms.

    John.


    --
    --
  • John Graham-Cumming at Nov 20, 2012 at 9:56 pm

    On Tuesday, November 20, 2012 9:20:06 PM UTC, Devon H. O'Dell wrote:
    FWIW i386 should basically just be the same code; the assembler should
    be simple enough to crib from elsewhere.
    Agreed. Similarly for NetBSD and OpenBSD. I'm sure they are all very
    similar. I just wasn't comfortable submitting code that I had not tested.

    John.


    --
  • Dave Cheney at Nov 20, 2012 at 3:57 am
    I'd start with FreeBSD, then do the others in a later CL, generally
    there is someone on the list who has hardware available to test.
    On Tue, Nov 20, 2012 at 2:48 PM, John Graham-Cumming wrote:
    On Tuesday, November 20, 2012 3:46:37 AM UTC, Dave Cheney wrote:

    Yup, that is all that is needed. Do you want to propose a CL ?

    What's needed is a CL for all the BSDs and both 32 and 64 bit. I don't have
    all those systems to test against.

    John.


    --
    --
  • Steven Hartland at Nov 20, 2012 at 2:04 pm
    It may be unrelated but there was a kernel memory leak in fadvise which was fixed by:-
    http://svnweb.freebsd.org/base?view=revision&revision=234661

    Regards
    Steve
    ----- Original Message -----
    From: John Graham-Cumming
    To: golang-nuts@googlegroups.com
    Sent: Tuesday, November 20, 2012 3:40 AM
    Subject: [go-nuts] Re: Memory use problem on FreeBSD


    On Monday, November 19, 2012 11:51:04 PM UTC, John Graham-Cumming wrote:
    On Nov 19, 11:47 pm, Dave Cheney wrote:
    I was just chasing that down. You are correct that implementing this
    method should activate the scavenger. I'll prepare a CL.
    It looks like for xBSD (where x is Free, Net and Open) runtime.madvise
    is not implemented at all.


    I made an experimental patch to Go on FreeBSD amd64 by adding the following to sys_freebsd_amd64.s:


    TEXT runtime?madvise(SB),7,$0
    MOVQ 8(SP), DI
    MOVQ 16(SP), SI
    MOVQ 24(SP), DX
    MOVQ $75, AX // madvise
    SYSCALL
    CMPQ AX, $0xfffffffffffff001
    JLS 2(PC)
    MOVL $0xf1, 0xf1 // crash
    RET


    And changing mem_freebsd.c so that it actually gets called. That worked.


    John.

    --




    ================================================
    This e.mail is private and confidential between Multiplay (UK) Ltd. and the person or entity to whom it is addressed. In the event of misdirection, the recipient is prohibited from using, copying, printing or otherwise disseminating it or any information contained in it.

    In the event of misdirection, illegible or incomplete transmission please telephone +44 845 868 1337
    or return the E.mail to postmaster@multiplay.co.uk.

    --

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupgolang-nuts @
categoriesgo
postedNov 19, '12 at 9:27p
activeNov 20, '12 at 9:56p
posts19
users4
websitegolang.org

People

Translate

site design / logo © 2021 Grokbase