FAQ
I am having trouble with understanding how garbage collector works.
High-throughput data processing application grows and grows memory, three
people have checked to ensure it is not holding active references to
memory, but it grows unlimited, monotinically increasing, to 10+ GB over
time. If work is ceased to application, it will release memory to OS, but
takes a long time and many GC cycles! GC runs, releases partial memory,
runs again, releases more... literally half hour passes before memory usage
decreases significantly, while 100% idle.

This is true in both 1.0.3 and 1.1 Go, on Linux/64-bit. Application is
using libpcap with https://code.google.com/p/golibpcap/ Application is
actually quite simple: captures packets, makes counters of statistics about
packets.

Questions;

    1. Why GC doesn't release all memory? When it runs, it finds a lot of
    memory unused, but seems to not release it.
    2. Why GC operates so slowly? Later it will run again and release same
    memory it found was free-able but didn't free earlier. Why 30 minutes must
    pass?
    3. Does external C library impact GC in negative way?
    4. Is there documentation on how GC works, or best practice to avoid
    bloating?

My estimate is that C application could work with few dozens of MB of
memory, maybe up to couple hundred MB, for this use case.

If needed I, or my colleagues, will fill in more details. But I thought I
should start with general problems and see if there is something common I
miss. How Google builds large high-throughput server systems with Go and
doesn't find these problems?



--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Search Discussions

  • Dave Cheney at Jun 2, 2013 at 12:20 am

    On Sun, Jun 2, 2013 at 10:00 AM, Boris wrote:
    I am having trouble with understanding how garbage collector works.
    High-throughput data processing application grows and grows memory, three
    people have checked to ensure it is not holding active references to memory,
    but it grows unlimited, monotinically increasing, to 10+ GB over time. If
    work is ceased to application, it will release memory to OS, but takes a
    long time and many GC cycles! GC runs, releases partial memory, runs again,
    releases more... literally half hour passes before memory usage decreases
    significantly, while 100% idle.

    This is true in both 1.0.3 and 1.1 Go, on Linux/64-bit. Application is using
    libpcap with https://code.google.com/p/golibpcap/ Application is actually
    quite simple: captures packets, makes counters of statistics about packets.

    Questions;

    Why GC doesn't release all memory? When it runs, it finds a lot of memory
    unused, but seems to not release it.
    The scavenger can only release pages that are completely empty as the
    garbage collector is non compacting.
    Why GC operates so slowly? Later it will run again and release same memory
    it found was free-able but didn't free earlier. Why 30 minutes must pass?
    The scavenger shouldn't release every page it finds immediately, what
    if they are need straight away again? The operation of the scavenger
    is to return memory that is truly unused (ie, your application may
    allocate a lot during startup, but during steady state requires less
    heap space).

    The scavenger also runs infrequently for this reason.
    Does external C library impact GC in negative way?
    It doesn't sound like your application would work with golibpcap
    removed, but as you say that memory is eventually returned to the
    operating system, it doesn't sound at this point like you are leaking
    memory in your C bindings.
    Is there documentation on how GC works, or best practice to avoid bloating?
    Use the source Luke. The only tunables that I know of are GOGC and
    GOGCTRACE, the latter not being a tunable, only a debugging flag.
    My estimate is that C application could work with few dozens of MB of
    memory, maybe up to couple hundred MB, for this use case.

    If needed I, or my colleagues, will fill in more details. But I thought I
    should start with general problems and see if there is something common I
    miss. How Google builds large high-throughput server systems with Go and
    doesn't find these problems?
    I've had several reports from folks who can trigger pathological
    allocation patterns in their code, but in general, your experiences
    appear to be the exception, not the norm. In all cases, it is crucial
    to provide the source to a small program which demonstrates the issue.

    Cheers

    Dave
    --
    You received this message because you are subscribed to the Google Groups
    "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an
    email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.
    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.
  • Rémy Oudompheng at Jun 2, 2013 at 4:24 am
    Le 2 juin 2013 02:01, "Boris" <boris.solovyov@gmail.com> a écrit :
    I am having trouble with understanding how garbage collector works.
    High-throughput data processing application grows and grows memory, three
    people have checked to ensure it is not holding active references to
    memory, but it grows unlimited, monotinically increasing, to 10+ GB over
    time. If work is ceased to application, it will release memory to OS, but
    takes a long time and many GC cycles! GC runs, releases partial memory,
    runs again, releases more... literally half hour passes before memory usage
    decreases significantly, while 100% idle.
    This is true in both 1.0.3 and 1.1 Go, on Linux/64-bit. Application is
    using libpcap with https://code.google.com/p/golibpcap/ Application is
    actually quite simple: captures packets, makes counters of statistics about
    packets.
    Questions;
    Why GC doesn't release all memory? When it runs, it finds a lot of memory
    unused, but seems to not release it.
    Why GC operates so slowly? Later it will run again and release same
    memory it found was free-able but didn't free earlier. Why 30 minutes must
    pass?
    Does external C library impact GC in negative way?
    Is there documentation on how GC works, or best practice to avoid bloating?
    My estimate is that C application could work with few dozens of MB of
    memory, maybe up to couple hundred MB, for this use case.
    If needed I, or my colleagues, will fill in more details. But I thought I
    should start with general problems and see if there is something common I
    miss. How Google builds large high-throughput server systems with Go and
    doesn't find these problems?

    Read GC verbose output by running with GOGCTRACE=1. If it shows your
    program using 4GB of memory (regardless of OS memory usage which might be
    higher), and you think it should be 100MB, then your program should be
    fixed first.

    The best practice is to profile memory allocations and eliminate them.

    The memory usage of a Go memory should not be above 3x its peak memory
    usage (in the sense of actually used memory) over the last hour.

    From your estimate, the program should not go above 1GB of memory usage
    with the current GC logic. If it goes up to 10GB you have another problem.

    Rémy.

    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.
  • Dmitry Vyukov at Jun 2, 2013 at 8:20 am

    On Sun, Jun 2, 2013 at 4:00 AM, Boris wrote:
    I am having trouble with understanding how garbage collector works.
    High-throughput data processing application grows and grows memory, three
    people have checked to ensure it is not holding active references to memory,
    but it grows unlimited, monotinically increasing, to 10+ GB over time. If
    work is ceased to application, it will release memory to OS, but takes a
    long time and many GC cycles! GC runs, releases partial memory, runs again,
    releases more... literally half hour passes before memory usage decreases
    significantly, while 100% idle.

    This is true in both 1.0.3 and 1.1 Go, on Linux/64-bit. Application is using
    libpcap with https://code.google.com/p/golibpcap/ Application is actually
    quite simple: captures packets, makes counters of statistics about packets.

    Questions;

    Why GC doesn't release all memory? When it runs, it finds a lot of memory
    unused, but seems to not release it.
    Why GC operates so slowly? Later it will run again and release same memory
    it found was free-able but didn't free earlier. Why 30 minutes must pass?
    Does external C library impact GC in negative way?
    Is there documentation on how GC works, or best practice to avoid bloating?

    My estimate is that C application could work with few dozens of MB of
    memory, maybe up to couple hundred MB, for this use case.

    If needed I, or my colleagues, will fill in more details. But I thought I
    should start with general problems and see if there is something common I
    miss. How Google builds large high-throughput server systems with Go and
    doesn't find these problems?

    How do you measure memory consumption?
    What do you mean by "release memory to OS"?
    What do you mean by "GC releases memory"?


    Do you use finalizers?
    Do you use files/network connections that are not explicitly closed?

    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupgolang-nuts @
categoriesgo
postedJun 2, '13 at 12:00a
activeJun 2, '13 at 8:20a
posts4
users4
websitegolang.org

People

Translate

site design / logo © 2021 Grokbase