FAQ
Hi,

I have a program that loads a large amount of data into memory when it
starts (up to ~14GB depending on input) and this data remains in memory for
the duration of the program. I'm wondering if this will have any effect on
the working of the garbage collector.

Thanks,

David

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Search Discussions

  • Matthew Holt at May 11, 2014 at 6:49 pm
    Depends on your implementation, but probably not. (Though I don't
    understand the internals of Go's garbage collector.)

    Just curious, is your data 14 GB in memory or 14 GB on disk?

    Related to garbage collection, you will best optimize your program if you
    consider the number of allocations you make as much as the size of your
    allocs.*

    So if the data is loaded into memory and stays there and is not changed
    often, I doubt you'll feel any adverse GC effects.

    (*Gophercon 2014)

    On Sunday, May 11, 2014 12:33:34 PM UTC-6, J. David Lee wrote:

    Hi,

    I have a program that loads a large amount of data into memory when it
    starts (up to ~14GB depending on input) and this data remains in memory for
    the duration of the program. I'm wondering if this will have any effect on
    the working of the garbage collector.

    Thanks,

    David
    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
  • J. David Lee at May 11, 2014 at 7:02 pm
    Hi Matthew,

    Thanks for the response. This is 14 GB in memory, less on disk due to
    compression. My worry is that when memory usage is high a small percentage
    of garbage could become a large amount of memory. I don't know the
    internals of the garbage collector either, but I've noticed that doing
    manual garbage collection while loading this data (after I know that a
    reasonable amount of garbage has been generated) can reduce my program's
    memory footprint considerably. In a recent run the difference was 5 vs 6 GB
    in memory.

    Once data is loaded I'm careful to minimize the number of allocations as
    the program runs.

    Thanks,

    David
    On Sunday, May 11, 2014 1:49:24 PM UTC-5, Matthew Holt wrote:

    Depends on your implementation, but probably not. (Though I don't
    understand the internals of Go's garbage collector.)

    Just curious, is your data 14 GB in memory or 14 GB on disk?

    Related to garbage collection, you will best optimize your program if you
    consider the number of allocations you make as much as the size of your
    allocs.*

    So if the data is loaded into memory and stays there and is not changed
    often, I doubt you'll feel any adverse GC effects.

    (*Gophercon 2014)

    On Sunday, May 11, 2014 12:33:34 PM UTC-6, J. David Lee wrote:

    Hi,

    I have a program that loads a large amount of data into memory when it
    starts (up to ~14GB depending on input) and this data remains in memory for
    the duration of the program. I'm wondering if this will have any effect on
    the working of the garbage collector.

    Thanks,

    David
    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
  • Egon at May 11, 2014 at 7:16 pm
    You should be fine, but use as few pointers as possible. And, as with any
    optimization problems, first measure, then measure... then measure again...
    and then change. First find what the actual bottlenecks are and then try to
    fix it.

    Dmitry recently posted this:
    https://software.intel.com/en-us/blogs/2014/05/10/debugging-performance-issues-in-go-programs
    .

    + egon
    On Sunday, May 11, 2014 9:33:34 PM UTC+3, J. David Lee wrote:

    Hi,

    I have a program that loads a large amount of data into memory when it
    starts (up to ~14GB depending on input) and this data remains in memory for
    the duration of the program. I'm wondering if this will have any effect on
    the working of the garbage collector.

    Thanks,

    David
    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
  • Hamish Ogilvy at May 12, 2014 at 12:35 am
    It will depend on number of objects. You can measure the GC pause directly though. Look at the runtime package, gctrace. If it's an issue you can then dig deeper.

    New GC improvements in 1.3 coming, so maybe try the new beta also.

    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
  • J. David Lee at May 12, 2014 at 4:51 am
    Thanks. gctrace looks like exactly what I need.
    On Sunday, May 11, 2014 7:35:45 PM UTC-5, Hamish Ogilvy wrote:

    It will depend on number of objects. You can measure the GC pause directly
    though. Look at the runtime package, gctrace. If it's an issue you can then
    dig deeper.

    New GC improvements in 1.3 coming, so maybe try the new beta also.
    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupgolang-nuts @
categoriesgo
postedMay 11, '14 at 6:33p
activeMay 12, '14 at 4:51a
posts6
users4
websitegolang.org

People

Translate

site design / logo © 2021 Grokbase