FAQ
maybe i am just approaching the problem the right way so in case you have
some alternative suggestions, I am listening. Here is my problem:

I have this code that loads a lot of data on startup (around 2 Gb
in-memory). I was looking to optimize it and profiled the CPU usage. The
result clearly showed that the program was generating too much garbage
(only GC related functions showed up in the top10 output).

To track that down, i tried to run a memory profile. That did not work as
expected because of the amount of memory loaded on startup, the memory
profile only showed the code paths related with this initial data loading
and nothing else.

So i thought that if there was a way for me to say something like "Data was
loaded. Clear memory stats now" i would be able to see what is actually
happening.

Does this make any sense at all? If now, any alternatives?

Thanks in advance.

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Search Discussions

  • Péter Szilágyi at Oct 28, 2013 at 12:00 pm
    Hi,

       I don't know how to reset the profiler, but just a thought: why don't you
    start the profiler only after loading the data [1]?

    Cheers,
       Peter

    [1] http://golang.org/pkg/runtime/#SetBlockProfileRate

    On Mon, Oct 28, 2013 at 12:37 PM, Bruno Albuquerque wrote:

    maybe i am just approaching the problem the right way so in case you have
    some alternative suggestions, I am listening. Here is my problem:

    I have this code that loads a lot of data on startup (around 2 Gb
    in-memory). I was looking to optimize it and profiled the CPU usage. The
    result clearly showed that the program was generating too much garbage
    (only GC related functions showed up in the top10 output).

    To track that down, i tried to run a memory profile. That did not work as
    expected because of the amount of memory loaded on startup, the memory
    profile only showed the code paths related with this initial data loading
    and nothing else.

    So i thought that if there was a way for me to say something like "Data
    was loaded. Clear memory stats now" i would be able to see what is actually
    happening.

    Does this make any sense at all? If now, any alternatives?

    Thanks in advance.

    --
    You received this message because you are subscribed to the Google Groups
    "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an
    email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.
    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.
  • Bruno Albuquerque at Oct 28, 2013 at 3:10 pm
    Thanks for the suggestion. This did not seem to wok as expected (at least
    when profiling using the test facilities). I will try to manually profile
    it and see if it helps.

    -Bruno


    On Mon, Oct 28, 2013 at 10:00 AM, Péter Szilágyi wrote:

    Hi,

    I don't know how to reset the profiler, but just a thought: why don't
    you start the profiler only after loading the data [1]?

    Cheers,
    Peter

    [1] http://golang.org/pkg/runtime/#SetBlockProfileRate

    On Mon, Oct 28, 2013 at 12:37 PM, Bruno Albuquerque wrote:

    maybe i am just approaching the problem the right way so in case you have
    some alternative suggestions, I am listening. Here is my problem:

    I have this code that loads a lot of data on startup (around 2 Gb
    in-memory). I was looking to optimize it and profiled the CPU usage. The
    result clearly showed that the program was generating too much garbage
    (only GC related functions showed up in the top10 output).

    To track that down, i tried to run a memory profile. That did not work as
    expected because of the amount of memory loaded on startup, the memory
    profile only showed the code paths related with this initial data loading
    and nothing else.

    So i thought that if there was a way for me to say something like "Data
    was loaded. Clear memory stats now" i would be able to see what is actually
    happening.

    Does this make any sense at all? If now, any alternatives?

    Thanks in advance.

    --
    You received this message because you are subscribed to the Google Groups
    "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an
    email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.
    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.
  • Bruno Albuquerque at Oct 28, 2013 at 3:24 pm
    In fact, I just noticed that SetBlockProfileRate is not related to memory
    profiling at all.

    In any case, it seems the -benchmem option for tests can at least give me a
    rough idea of how big the issue is. And it seems to be big:

    BenchmarkCode 20 110851351 ns/op 16353639 B/op 339567 allocs/op

    if i am reading it right, a single iteration of the code I am benchmarking
    resulted in 15 Mb of data being allocated (in 339567 diferent alloc
    operations). Is this interpretation correct?

    -Bruno


    On Mon, Oct 28, 2013 at 1:10 PM, Bruno Albuquerque wrote:

    Thanks for the suggestion. This did not seem to wok as expected (at least
    when profiling using the test facilities). I will try to manually profile
    it and see if it helps.

    -Bruno


    On Mon, Oct 28, 2013 at 10:00 AM, Péter Szilágyi wrote:

    Hi,

    I don't know how to reset the profiler, but just a thought: why don't
    you start the profiler only after loading the data [1]?

    Cheers,
    Peter

    [1] http://golang.org/pkg/runtime/#SetBlockProfileRate

    On Mon, Oct 28, 2013 at 12:37 PM, Bruno Albuquerque wrote:

    maybe i am just approaching the problem the right way so in case you
    have some alternative suggestions, I am listening. Here is my problem:

    I have this code that loads a lot of data on startup (around 2 Gb
    in-memory). I was looking to optimize it and profiled the CPU usage. The
    result clearly showed that the program was generating too much garbage
    (only GC related functions showed up in the top10 output).

    To track that down, i tried to run a memory profile. That did not work
    as expected because of the amount of memory loaded on startup, the memory
    profile only showed the code paths related with this initial data loading
    and nothing else.

    So i thought that if there was a way for me to say something like "Data
    was loaded. Clear memory stats now" i would be able to see what is actually
    happening.

    Does this make any sense at all? If now, any alternatives?

    Thanks in advance.

    --
    You received this message because you are subscribed to the Google
    Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send
    an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.
    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.
  • Aaron Blohowiak at Oct 29, 2013 at 12:11 am
    You should do the GOGC=off -memprofilerate=1 trick for a *single* run of
    your benchmark and then use the resulting memory profile which will show
    all allocations performed ever (because the heap is never GC'd)

    (For this stuff, I especially like "weblist ." in pprof)
    On Monday, October 28, 2013 8:24:34 AM UTC-7, Bruno Albuquerque wrote:

    In fact, I just noticed that SetBlockProfileRate is not related to memory
    profiling at all.

    In any case, it seems the -benchmem option for tests can at least give me
    a rough idea of how big the issue is. And it seems to be big:

    BenchmarkCode 20 110851351 ns/op 16353639 B/op 339567 allocs/op

    if i am reading it right, a single iteration of the code I am benchmarking
    resulted in 15 Mb of data being allocated (in 339567 diferent alloc
    operations). Is this interpretation correct?

    -Bruno



    On Mon, Oct 28, 2013 at 1:10 PM, Bruno Albuquerque <b...@bug-br.org.br<javascript:>
    wrote:
    Thanks for the suggestion. This did not seem to wok as expected (at least
    when profiling using the test facilities). I will try to manually profile
    it and see if it helps.

    -Bruno



    On Mon, Oct 28, 2013 at 10:00 AM, Péter Szilágyi <pet...@gmail.com<javascript:>
    wrote:
    Hi,

    I don't know how to reset the profiler, but just a thought: why don't
    you start the profiler only after loading the data [1]?

    Cheers,
    Peter

    [1] http://golang.org/pkg/runtime/#SetBlockProfileRate


    On Mon, Oct 28, 2013 at 12:37 PM, Bruno Albuquerque <b...@bug-br.org.br<javascript:>
    wrote:
    maybe i am just approaching the problem the right way so in case you
    have some alternative suggestions, I am listening. Here is my problem:

    I have this code that loads a lot of data on startup (around 2 Gb
    in-memory). I was looking to optimize it and profiled the CPU usage. The
    result clearly showed that the program was generating too much garbage
    (only GC related functions showed up in the top10 output).

    To track that down, i tried to run a memory profile. That did not work
    as expected because of the amount of memory loaded on startup, the memory
    profile only showed the code paths related with this initial data loading
    and nothing else.

    So i thought that if there was a way for me to say something like "Data
    was loaded. Clear memory stats now" i would be able to see what is actually
    happening.

    Does this make any sense at all? If now, any alternatives?

    Thanks in advance.

    --
    You received this message because you are subscribed to the Google
    Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send
    an email to golang-nuts...@googlegroups.com <javascript:>.
    For more options, visit https://groups.google.com/groups/opt_out.
    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.
  • Matthew Zimmerman at Oct 29, 2013 at 12:41 am
    I've never had a problem whittling down go tool pprof to only look at
    what I wanted it to look at. Just - (ignore) out whatever memory
    loading function you have that is loading the 2GB of memory (assuming
    that's not where you're using memory you don't know about).

    "top [--cum] [-ignore1] [-ignore2]"

    On Mon, Oct 28, 2013 at 8:11 PM, Aaron Blohowiak
    wrote:
    You should do the GOGC=off -memprofilerate=1 trick for a *single* run of
    your benchmark and then use the resulting memory profile which will show all
    allocations performed ever (because the heap is never GC'd)

    (For this stuff, I especially like "weblist ." in pprof)
    On Monday, October 28, 2013 8:24:34 AM UTC-7, Bruno Albuquerque wrote:

    In fact, I just noticed that SetBlockProfileRate is not related to memory
    profiling at all.

    In any case, it seems the -benchmem option for tests can at least give me
    a rough idea of how big the issue is. And it seems to be big:

    BenchmarkCode 20 110851351 ns/op 16353639 B/op 339567 allocs/op

    if i am reading it right, a single iteration of the code I am benchmarking
    resulted in 15 Mb of data being allocated (in 339567 diferent alloc
    operations). Is this interpretation correct?

    -Bruno



    On Mon, Oct 28, 2013 at 1:10 PM, Bruno Albuquerque <b...@bug-br.org.br>
    wrote:
    Thanks for the suggestion. This did not seem to wok as expected (at least
    when profiling using the test facilities). I will try to manually profile it
    and see if it helps.

    -Bruno



    On Mon, Oct 28, 2013 at 10:00 AM, Péter Szilágyi <pet...@gmail.com>
    wrote:
    Hi,

    I don't know how to reset the profiler, but just a thought: why don't
    you start the profiler only after loading the data [1]?

    Cheers,
    Peter

    [1] http://golang.org/pkg/runtime/#SetBlockProfileRate


    On Mon, Oct 28, 2013 at 12:37 PM, Bruno Albuquerque <b...@bug-br.org.br>
    wrote:
    maybe i am just approaching the problem the right way so in case you
    have some alternative suggestions, I am listening. Here is my problem:

    I have this code that loads a lot of data on startup (around 2 Gb
    in-memory). I was looking to optimize it and profiled the CPU usage. The
    result clearly showed that the program was generating too much garbage (only
    GC related functions showed up in the top10 output).

    To track that down, i tried to run a memory profile. That did not work
    as expected because of the amount of memory loaded on startup, the memory
    profile only showed the code paths related with this initial data loading
    and nothing else.

    So i thought that if there was a way for me to say something like "Data
    was loaded. Clear memory stats now" i would be able to see what is actually
    happening.

    Does this make any sense at all? If now, any alternatives?

    Thanks in advance.

    --
    You received this message because you are subscribed to the Google
    Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send
    an email to golang-nuts...@googlegroups.com.

    For more options, visit https://groups.google.com/groups/opt_out.
    --
    You received this message because you are subscribed to the Google Groups
    "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an
    email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.
    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.
  • Rémy Oudompheng at Oct 29, 2013 at 6:49 am
    Le 29 oct. 2013 01:11, "Aaron Blohowiak" <aaron.blohowiak@gmail.com> a
    écrit :
    You should do the GOGC=off -memprofilerate=1 trick for a *single* run of
    your benchmark and then use the resulting memory profile which will show
    all allocations performed ever (because the heap is never GC'd)
    >

    The memory profile already counts all allocations, there is no need to
    disable GC.

    Rémy.

    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.
  • Bruno Albuquerque at Oct 29, 2013 at 11:53 am
    In the end, what helped me was setting runtime.MemProfileRate to 0 at
    startup and set it to its default value before the section I wanted to
    track (found that out thanks to Péter Szilágyi). This seems to give me what
    I want.


    Thanks for all the help.




    On Tue, Oct 29, 2013 at 4:49 AM, Rémy Oudompheng
    wrote:
    Le 29 oct. 2013 01:11, "Aaron Blohowiak" <aaron.blohowiak@gmail.com> a
    écrit :
    You should do the GOGC=off -memprofilerate=1 trick for a *single* run of
    your benchmark and then use the resulting memory profile which will show
    all allocations performed ever (because the heap is never GC'd)
    The memory profile already counts all allocations, there is no need to
    disable GC.

    Rémy.
    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupgolang-nuts @
categoriesgo
postedOct 28, '13 at 10:38a
activeOct 29, '13 at 11:53a
posts8
users5
websitegolang.org

People

Translate

site design / logo © 2021 Grokbase