FAQ
I'm debugging a program built with Go 1.2, that does lots of operations in
S3 using goamz and MySQL using go-sql-driver.

However, lately it started using too much memory and just growing until it
is caught by the OOM killer. I'm not sure when it started, because I made
some changes to this program to fix some bugs and after working properly I
was cough into this memory issue.

I enabled the pprof http server and got some stats during a test run.

(pprof) top20
Total: 2235.0 MB
   1010.0 45.2% 45.2% 1010.0 45.2% crypto/tls.(*block).reserve
    243.5 10.9% 56.1% 463.5 20.7% crypto/x509.parseCertificate
    189.5 8.5% 64.6% 189.5 8.5% bytes.makeSlice
    170.5 7.6% 72.2% 390.5 17.5% crypto/tls.(*Conn).readHandshake
    147.0 6.6% 78.8% 147.0 6.6% reflect.unsafe_NewArray
    130.0 5.8% 84.6% 130.0 5.8% bufio.NewReaderSize
     80.5 3.6% 88.2% 80.5 3.6% crypto/rc4.NewCipher
     72.5 3.2% 91.4% 72.5 3.2%
crypto/x509/pkix.(*Name).FillFromRDNSequence
     46.0 2.1% 93.5% 46.0 2.1% math/big.nat.make
     27.5 1.2% 94.7% 1283.0 57.4% net/http.(*Transport).dialConn
     25.0 1.1% 95.8% 25.0 1.1% encoding/asn1.parsePrintableString
     18.5 0.8% 96.7% 18.5 0.8% runtime.convT2E
     17.0 0.8% 97.4% 17.0 0.8% crypto/sha1.New
     13.0 0.6% 98.0% 30.0 1.3% crypto/hmac.New
      4.0 0.2% 98.2% 4.0 0.2%
crypto/x509.(*Certificate).buildChains
      4.0 0.2% 98.4% 50.0 2.2% encoding/asn1.parseBigInt
      3.5 0.2% 98.5% 8.0 0.4% net.socket
      3.5 0.2% 98.7% 485.0 21.7% net/http.(*persistConn).readLoop
      3.5 0.2% 98.8% 3.5 0.2% newdefer
      2.5 0.1% 98.9% 2.5 0.1% concatstring

Is there any known memory leaks in the crypto package? Could be a bug in
the goamz package?

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Search Discussions

  • Dave Cheney at Feb 4, 2014 at 8:56 pm
    Maybe, but most of the time memory leaks come from lost timers and
    goroutines. Try hitting your process with SIGQUIT and seeing how many
    goroutines are outstanding.


    On Wed, Feb 5, 2014 at 1:00 AM, Herbert Fischer
    wrote:
    I'm debugging a program built with Go 1.2, that does lots of operations in
    S3 using goamz and MySQL using go-sql-driver.

    However, lately it started using too much memory and just growing until it
    is caught by the OOM killer. I'm not sure when it started, because I made
    some changes to this program to fix some bugs and after working properly I
    was cough into this memory issue.

    I enabled the pprof http server and got some stats during a test run.

    (pprof) top20
    Total: 2235.0 MB
    1010.0 45.2% 45.2% 1010.0 45.2% crypto/tls.(*block).reserve
    243.5 10.9% 56.1% 463.5 20.7% crypto/x509.parseCertificate
    189.5 8.5% 64.6% 189.5 8.5% bytes.makeSlice
    170.5 7.6% 72.2% 390.5 17.5% crypto/tls.(*Conn).readHandshake
    147.0 6.6% 78.8% 147.0 6.6% reflect.unsafe_NewArray
    130.0 5.8% 84.6% 130.0 5.8% bufio.NewReaderSize
    80.5 3.6% 88.2% 80.5 3.6% crypto/rc4.NewCipher
    72.5 3.2% 91.4% 72.5 3.2%
    crypto/x509/pkix.(*Name).FillFromRDNSequence
    46.0 2.1% 93.5% 46.0 2.1% math/big.nat.make
    27.5 1.2% 94.7% 1283.0 57.4% net/http.(*Transport).dialConn
    25.0 1.1% 95.8% 25.0 1.1% encoding/asn1.parsePrintableString
    18.5 0.8% 96.7% 18.5 0.8% runtime.convT2E
    17.0 0.8% 97.4% 17.0 0.8% crypto/sha1.New
    13.0 0.6% 98.0% 30.0 1.3% crypto/hmac.New
    4.0 0.2% 98.2% 4.0 0.2%
    crypto/x509.(*Certificate).buildChains
    4.0 0.2% 98.4% 50.0 2.2% encoding/asn1.parseBigInt
    3.5 0.2% 98.5% 8.0 0.4% net.socket
    3.5 0.2% 98.7% 485.0 21.7% net/http.(*persistConn).readLoop
    3.5 0.2% 98.8% 3.5 0.2% newdefer
    2.5 0.1% 98.9% 2.5 0.1% concatstring

    Is there any known memory leaks in the crypto package? Could be a bug in
    the goamz package?

    --
    You received this message because you are subscribed to the Google Groups
    "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an
    email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.
    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.
  • Herbert Fischer at Feb 5, 2014 at 1:48 am
    Thanks!

    This program uses 60 goroutines as workers and one producer loop, sending
    "messages" that into channels for all workers, that could send another
    "messages" to another workers. When the main producer loop finishes, the
    program keeps running the workers until their channels are depleted/closed.

    However I don't know yet if some package is creating timers or goroutines
    as you suggested. I'll check tomorrow.

    The curious thing is that I was able to reduce memory usage by 5x, by
    forcing HTTP connections with goamz instead of HTTPS. Since the program
    runs from the AWS network, this should be no big deal, and the fix works
    for now.


    On Tue, Feb 4, 2014 at 6:56 PM, Dave Cheney wrote:

    Maybe, but most of the time memory leaks come from lost timers and
    goroutines. Try hitting your process with SIGQUIT and seeing how many
    goroutines are outstanding.


    On Wed, Feb 5, 2014 at 1:00 AM, Herbert Fischer <herbert.fischer@gmail.com
    wrote:
    I'm debugging a program built with Go 1.2, that does lots of operations
    in S3 using goamz and MySQL using go-sql-driver.

    However, lately it started using too much memory and just growing until
    it is caught by the OOM killer. I'm not sure when it started, because I
    made some changes to this program to fix some bugs and after working
    properly I was cough into this memory issue.

    I enabled the pprof http server and got some stats during a test run.

    (pprof) top20
    Total: 2235.0 MB
    1010.0 45.2% 45.2% 1010.0 45.2% crypto/tls.(*block).reserve
    243.5 10.9% 56.1% 463.5 20.7% crypto/x509.parseCertificate
    189.5 8.5% 64.6% 189.5 8.5% bytes.makeSlice
    170.5 7.6% 72.2% 390.5 17.5% crypto/tls.(*Conn).readHandshake
    147.0 6.6% 78.8% 147.0 6.6% reflect.unsafe_NewArray
    130.0 5.8% 84.6% 130.0 5.8% bufio.NewReaderSize
    80.5 3.6% 88.2% 80.5 3.6% crypto/rc4.NewCipher
    72.5 3.2% 91.4% 72.5 3.2%
    crypto/x509/pkix.(*Name).FillFromRDNSequence
    46.0 2.1% 93.5% 46.0 2.1% math/big.nat.make
    27.5 1.2% 94.7% 1283.0 57.4% net/http.(*Transport).dialConn
    25.0 1.1% 95.8% 25.0 1.1% encoding/asn1.parsePrintableString
    18.5 0.8% 96.7% 18.5 0.8% runtime.convT2E
    17.0 0.8% 97.4% 17.0 0.8% crypto/sha1.New
    13.0 0.6% 98.0% 30.0 1.3% crypto/hmac.New
    4.0 0.2% 98.2% 4.0 0.2%
    crypto/x509.(*Certificate).buildChains
    4.0 0.2% 98.4% 50.0 2.2% encoding/asn1.parseBigInt
    3.5 0.2% 98.5% 8.0 0.4% net.socket
    3.5 0.2% 98.7% 485.0 21.7% net/http.(*persistConn).readLoop
    3.5 0.2% 98.8% 3.5 0.2% newdefer
    2.5 0.1% 98.9% 2.5 0.1% concatstring

    Is there any known memory leaks in the crypto package? Could be a bug in
    the goamz package?

    --
    You received this message because you are subscribed to the Google Groups
    "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an
    email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.
    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupgolang-nuts @
categoriesgo
postedFeb 4, '14 at 2:01p
activeFeb 5, '14 at 1:48a
posts3
users2
websitegolang.org

2 users in discussion

Herbert Fischer: 2 posts Dave Cheney: 1 post

People

Translate

site design / logo © 2021 Grokbase