FAQ
I've recently re-written part of an old Java utility in Go. It's now a far
simpler design and more performant (in both memory and CPU, which is a nice
surprise, actually), but there's a really slow memory leak. Maybe 10-20
megs per hour, according to the garbage collector. Not terrible, but any
kind of leak just drives me nuts. I just started dumping heap stats every
hour, and I get something like this:

top5
Total: 233.1 MB
     89.1 38.2% 38.2% 89.1 38.2% cnew
     77.0 33.0% 71.2% 77.0 33.0% compress/flate.NewReader
     10.8 4.6% 75.9% 12.2 5.3% net/http.(*Transport).getIdleConnCh
      9.5 4.1% 79.9% 13.5 5.8% importer/twitter.func·005
      9.0 3.9% 83.8% 63.0 27.0% net/http.(*Transport).dialConn

Over time that top line starts getting larger and larger while the ones
below stay about the same. So I think that's where the issue is, and yet
that line is the least helpful line possible. From what I can tell it comes
from this in malloc.goc:

runtime·mallocgc(typ->size*n, flag, 1, 1);

So... I've gone a long way to find out that my leak is caused by memory
being allocated. I'm at a bit of a dead end here. Any help would be greatly
appreciated. Thanks!

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Search Discussions

  • Dave Cheney at Jun 26, 2013 at 1:13 am
    Can you show some code ?

    As an experiment, and I have no evidence to suggest this works, try
    adding this line


    runtime.MemProfileRate = 0

    to the top of your main.main
    On Wed, Jun 26, 2013 at 8:44 AM, Phil Kulak wrote:
    I've recently re-written part of an old Java utility in Go. It's now a far
    simpler design and more performant (in both memory and CPU, which is a nice
    surprise, actually), but there's a really slow memory leak. Maybe 10-20 megs
    per hour, according to the garbage collector. Not terrible, but any kind of
    leak just drives me nuts. I just started dumping heap stats every hour, and
    I get something like this:

    top5
    Total: 233.1 MB
    89.1 38.2% 38.2% 89.1 38.2% cnew
    77.0 33.0% 71.2% 77.0 33.0% compress/flate.NewReader
    10.8 4.6% 75.9% 12.2 5.3% net/http.(*Transport).getIdleConnCh
    9.5 4.1% 79.9% 13.5 5.8% importer/twitter.func·005
    9.0 3.9% 83.8% 63.0 27.0% net/http.(*Transport).dialConn

    Over time that top line starts getting larger and larger while the ones
    below stay about the same. So I think that's where the issue is, and yet
    that line is the least helpful line possible. From what I can tell it comes
    from this in malloc.goc:

    runtime·mallocgc(typ->size*n, flag, 1, 1);

    So... I've gone a long way to find out that my leak is caused by memory
    being allocated. I'm at a bit of a dead end here. Any help would be greatly
    appreciated. Thanks!

    --
    You received this message because you are subscribed to the Google Groups
    "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an
    email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.
    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.
  • Dave Cheney at Jun 26, 2013 at 10:24 am
    Ignore my suggestion about MemProfileRate, it is a red herring.
    On Wed, Jun 26, 2013 at 11:13 AM, Dave Cheney wrote:
    Can you show some code ?

    As an experiment, and I have no evidence to suggest this works, try
    adding this line


    runtime.MemProfileRate = 0

    to the top of your main.main
    On Wed, Jun 26, 2013 at 8:44 AM, Phil Kulak wrote:
    I've recently re-written part of an old Java utility in Go. It's now a far
    simpler design and more performant (in both memory and CPU, which is a nice
    surprise, actually), but there's a really slow memory leak. Maybe 10-20 megs
    per hour, according to the garbage collector. Not terrible, but any kind of
    leak just drives me nuts. I just started dumping heap stats every hour, and
    I get something like this:

    top5
    Total: 233.1 MB
    89.1 38.2% 38.2% 89.1 38.2% cnew
    77.0 33.0% 71.2% 77.0 33.0% compress/flate.NewReader
    10.8 4.6% 75.9% 12.2 5.3% net/http.(*Transport).getIdleConnCh
    9.5 4.1% 79.9% 13.5 5.8% importer/twitter.func·005
    9.0 3.9% 83.8% 63.0 27.0% net/http.(*Transport).dialConn

    Over time that top line starts getting larger and larger while the ones
    below stay about the same. So I think that's where the issue is, and yet
    that line is the least helpful line possible. From what I can tell it comes
    from this in malloc.goc:

    runtime·mallocgc(typ->size*n, flag, 1, 1);

    So... I've gone a long way to find out that my leak is caused by memory
    being allocated. I'm at a bit of a dead end here. Any help would be greatly
    appreciated. Thanks!

    --
    You received this message because you are subscribed to the Google Groups
    "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an
    email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.
    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.
  • Dmitry Vyukov at Jun 26, 2013 at 10:32 am

    On Wed, Jun 26, 2013 at 2:44 AM, Phil Kulak wrote:
    I've recently re-written part of an old Java utility in Go. It's now a far
    simpler design and more performant (in both memory and CPU, which is a nice
    surprise, actually), but there's a really slow memory leak. Maybe 10-20 megs
    per hour, according to the garbage collector. Not terrible, but any kind of
    leak just drives me nuts. I just started dumping heap stats every hour, and
    I get something like this:

    top5
    Total: 233.1 MB
    89.1 38.2% 38.2% 89.1 38.2% cnew
    77.0 33.0% 71.2% 77.0 33.0% compress/flate.NewReader
    10.8 4.6% 75.9% 12.2 5.3% net/http.(*Transport).getIdleConnCh
    9.5 4.1% 79.9% 13.5 5.8% importer/twitter.func·005
    9.0 3.9% 83.8% 63.0 27.0% net/http.(*Transport).dialConn

    Over time that top line starts getting larger and larger while the ones
    below stay about the same. So I think that's where the issue is, and yet
    that line is the least helpful line possible. From what I can tell it comes
    from this in malloc.goc:

    runtime·mallocgc(typ->size*n, flag, 1, 1);

    So... I've gone a long way to find out that my leak is caused by memory
    being allocated. I'm at a bit of a dead end here. Any help would be greatly
    appreciated. Thanks!

    What version of Go are you using?

    The first thing to check is whether you properly close all resources
    (as connections and responses) and drop/zeroize pointers that you
    don't need anymore.

    If that does not help, you may try:
    https://codereview.appspot.com/10446044/
    it needs to be applied to tip, you will need to build your own Go
    toolchain and it works only on amd64.
    When you send signal 6 to the process, it will dump full heap contents
    to stderr (it can be very large). The dump includes GC roots and all
    alive memory blocks, for each block: address, size, type, allocation
    stack and pointers to other blocks.

    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.
  • Phil Kulak at Jun 26, 2013 at 5:49 pm
    I'm using "go version go1.1.1 linux/amd64"

    The app uses a lot of http, but I'm certain I'm closing all responses. I've
    even turned off all connection pooling, but that didn't seem to do anything.

    Those patches look intense. I may have to resort to that. For now I think
    I'll just keep nibbling around the edges. Maybe start disabling certain
    paths and see if that solves the issue. I was hoping that "cnew" line had
    some special meaning to someone, but I guess it really is a dead end.
    Thanks for the help!
    On Wednesday, June 26, 2013 3:32:12 AM UTC-7, Dmitry Vyukov wrote:
    On Wed, Jun 26, 2013 at 2:44 AM, Phil Kulak wrote:
    I've recently re-written part of an old Java utility in Go. It's now a far
    simpler design and more performant (in both memory and CPU, which is a nice
    surprise, actually), but there's a really slow memory leak. Maybe 10-20 megs
    per hour, according to the garbage collector. Not terrible, but any kind of
    leak just drives me nuts. I just started dumping heap stats every hour, and
    I get something like this:

    top5
    Total: 233.1 MB
    89.1 38.2% 38.2% 89.1 38.2% cnew
    77.0 33.0% 71.2% 77.0 33.0% compress/flate.NewReader
    10.8 4.6% 75.9% 12.2 5.3%
    net/http.(*Transport).getIdleConnCh
    9.5 4.1% 79.9% 13.5 5.8% importer/twitter.func·005
    9.0 3.9% 83.8% 63.0 27.0% net/http.(*Transport).dialConn

    Over time that top line starts getting larger and larger while the ones
    below stay about the same. So I think that's where the issue is, and yet
    that line is the least helpful line possible. From what I can tell it comes
    from this in malloc.goc:

    runtime·mallocgc(typ->size*n, flag, 1, 1);

    So... I've gone a long way to find out that my leak is caused by memory
    being allocated. I'm at a bit of a dead end here. Any help would be greatly
    appreciated. Thanks!

    What version of Go are you using?

    The first thing to check is whether you properly close all resources
    (as connections and responses) and drop/zeroize pointers that you
    don't need anymore.

    If that does not help, you may try:
    https://codereview.appspot.com/10446044/
    it needs to be applied to tip, you will need to build your own Go
    toolchain and it works only on amd64.
    When you send signal 6 to the process, it will dump full heap contents
    to stderr (it can be very large). The dump includes GC roots and all
    alive memory blocks, for each block: address, size, type, allocation
    stack and pointers to other blocks.
    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.
  • Dmitry Vyukov at Jun 26, 2013 at 6:01 pm

    On Wed, Jun 26, 2013 at 9:49 PM, Phil Kulak wrote:
    I'm using "go version go1.1.1 linux/amd64"

    The app uses a lot of http, but I'm certain I'm closing all responses. I've
    even turned off all connection pooling, but that didn't seem to do anything.

    Those patches look intense. I may have to resort to that. For now I think
    I'll just keep nibbling around the edges. Maybe start disabling certain
    paths and see if that solves the issue. I was hoping that "cnew" line had
    some special meaning to someone, but I guess it really is a dead end.
    cnew means that the memory is most likely in slices or strings. But
    it's not very useful here.
    Thanks
    for the help!
    On Wednesday, June 26, 2013 3:32:12 AM UTC-7, Dmitry Vyukov wrote:
    On Wed, Jun 26, 2013 at 2:44 AM, Phil Kulak wrote:
    I've recently re-written part of an old Java utility in Go. It's now a
    far
    simpler design and more performant (in both memory and CPU, which is a
    nice
    surprise, actually), but there's a really slow memory leak. Maybe 10-20
    megs
    per hour, according to the garbage collector. Not terrible, but any kind
    of
    leak just drives me nuts. I just started dumping heap stats every hour,
    and
    I get something like this:

    top5
    Total: 233.1 MB
    89.1 38.2% 38.2% 89.1 38.2% cnew
    77.0 33.0% 71.2% 77.0 33.0% compress/flate.NewReader
    10.8 4.6% 75.9% 12.2 5.3%
    net/http.(*Transport).getIdleConnCh
    9.5 4.1% 79.9% 13.5 5.8% importer/twitter.func·005
    9.0 3.9% 83.8% 63.0 27.0% net/http.(*Transport).dialConn

    Over time that top line starts getting larger and larger while the ones
    below stay about the same. So I think that's where the issue is, and yet
    that line is the least helpful line possible. From what I can tell it
    comes
    from this in malloc.goc:

    runtime·mallocgc(typ->size*n, flag, 1, 1);

    So... I've gone a long way to find out that my leak is caused by memory
    being allocated. I'm at a bit of a dead end here. Any help would be
    greatly
    appreciated. Thanks!

    What version of Go are you using?

    The first thing to check is whether you properly close all resources
    (as connections and responses) and drop/zeroize pointers that you
    don't need anymore.

    If that does not help, you may try:
    https://codereview.appspot.com/10446044/
    it needs to be applied to tip, you will need to build your own Go
    toolchain and it works only on amd64.
    When you send signal 6 to the process, it will dump full heap contents
    to stderr (it can be very large). The dump includes GC roots and all
    alive memory blocks, for each block: address, size, type, allocation
    stack and pointers to other blocks.
    --
    You received this message because you are subscribed to the Google Groups
    "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an
    email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.
    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.
  • Phil Kulak at Jun 26, 2013 at 6:51 pm

    cnew means that the memory is most likely in slices or strings. But
    it's not very useful here.
    Ah, that makes sense.

    For what it's worth, I think I found the issue.

    If you look at
    http.Transport: http://golang.org/src/pkg/net/http/transport.go?m=text

    getIdleConnCh() is called for every round trip (called by getConn, which is called by RoundTrip), regardless of weather keepAlives are on. In my case, with an app that's connecting to hundreds of thousands of different hosts, that idleConnCh map just keeps filling up and filling up forever.


    To verify I'm going to build my own, very simple and specialized RoundTripper, with pretty much no state stored between requests, and see if that takes care of the issue.

    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.
  • Dmitry Vyukov at Jun 26, 2013 at 7:01 pm
    Please file the issue at http://golang.org/issue/new
    On Wed, Jun 26, 2013 at 10:51 PM, Phil Kulak wrote:
    cnew means that the memory is most likely in slices or strings. But
    it's not very useful here.
    Ah, that makes sense.

    For what it's worth, I think I found the issue.

    If you look at http.Transport:
    http://golang.org/src/pkg/net/http/transport.go?m=text

    getIdleConnCh() is called for every round trip (called by getConn, which is
    called by RoundTrip), regardless of weather keepAlives are on. In my case,
    with an app that's connecting to hundreds of thousands of different hosts,
    that idleConnCh map just keeps filling up and filling up forever.


    To verify I'm going to build my own, very simple and specialized
    RoundTripper, with pretty much no state stored between requests, and see if
    that takes care of the issue.

    --
    You received this message because you are subscribed to the Google Groups
    "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an
    email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.
    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.
  • Phil Kulak at Jun 26, 2013 at 8:31 pm
    Done and done. Thanks again for the help!

    https://code.google.com/p/go/issues/detail?id=5794
    On Wed, Jun 26, 2013 at 12:01 PM, Dmitry Vyukov wrote:

    Please file the issue at http://golang.org/issue/new
    On Wed, Jun 26, 2013 at 10:51 PM, Phil Kulak wrote:
    cnew means that the memory is most likely in slices or strings. But
    it's not very useful here.
    Ah, that makes sense.

    For what it's worth, I think I found the issue.

    If you look at http.Transport:
    http://golang.org/src/pkg/net/http/transport.go?m=text

    getIdleConnCh() is called for every round trip (called by getConn, which is
    called by RoundTrip), regardless of weather keepAlives are on. In my case,
    with an app that's connecting to hundreds of thousands of different hosts,
    that idleConnCh map just keeps filling up and filling up forever.


    To verify I'm going to build my own, very simple and specialized
    RoundTripper, with pretty much no state stored between requests, and see if
    that takes care of the issue.

    --
    You received this message because you are subscribed to the Google Groups
    "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an
    email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.
    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupgolang-nuts @
categoriesgo
postedJun 26, '13 at 12:23a
activeJun 26, '13 at 8:31p
posts9
users3
websitegolang.org

People

Translate

site design / logo © 2022 Grokbase