FAQ
Hi, all. I have a simple RPC handler that replies with a large blob of
data. It seems that the RPC handling code or gob encoding code is leaking
memory.

If I have a client that opens a connection, issues the RPC and then
disconnects, then opens another connection and repeats indefinitely, the
behaviour that I observe is that the server will sometimes increase the
"Sys" memory (amount of memory obtained from the OS). The amount
of "Alloc" memory stays around the same. My server regularly calls
runtime.GC() (such as before I dump memory stats).

If I have a client that opens a connection, keeps it open and issues RPCs
over and over, then the server "Alloc" and "Sys" memory will grow
after each RPC. If I eventually drop the connection, the "Alloc" memory
drops back down to the level before I started issuing RPCs. The
"Sys" memory does not go down.

This is all I have in my RPC handler:

func (t *Daemon) Poll(generation uint64, reply *FileSystem) error {
     fs := GetFileSystem()
     if fs != nil && generation != fs.GenerationCount() {
         *reply = *fs
     }
     return nil
}

It looks like either the RPC handling system or the gob encoder is holding
onto a reference to the data for each RPC call.
I'm using go1.4.2.linux-amd64.


--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Search Discussions

  • Richard Gooch at Jul 15, 2015 at 1:52 pm

    On Tuesday, 14 July 2015 06:52:47 UTC-7, Richard Gooch wrote:
    Hi, all. I have a simple RPC handler that replies with a large blob of
    data. It seems that the RPC handling code or gob encoding code is leaking
    memory.

    If I have a client that opens a connection, issues the RPC and then
    disconnects, then opens another connection and repeats indefinitely, the
    behaviour that I observe is that the server will sometimes increase the
    "Sys" memory (amount of memory obtained from the OS). The amount
    of "Alloc" memory stays around the same. My server regularly calls
    runtime.GC() (such as before I dump memory stats).

    If I have a client that opens a connection, keeps it open and issues RPCs
    over and over, then the server "Alloc" and "Sys" memory will grow
    after each RPC. If I eventually drop the connection, the "Alloc" memory
    drops back down to the level before I started issuing RPCs. The
    "Sys" memory does not go down.

    This is all I have in my RPC handler:

    func (t *Daemon) Poll(generation uint64, reply *FileSystem) error {
    fs := GetFileSystem()
    if fs != nil && generation != fs.GenerationCount() {
    *reply = *fs
    }
    return nil
    }

    It looks like either the RPC handling system or the gob encoder is holding
    onto a reference to the data for each RPC call.
    I'm using go1.4.2.linux-amd64.
    I also noticed that (for the case where the client keeps the connection
    open), each time I call the Poll() RPC, the number of
    goroutines goes up by 1. So, it seems that the RPC handling code is leaking
    goroutines.

    Is this expected behaviour? A bug? Am I doing it wrong? I prefer to keep
    the connection open, so as to amortize the
    connection setup time.

    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
  • Nigel Tao at Jul 16, 2015 at 1:22 am

    On Wed, Jul 15, 2015 at 11:52 PM, Richard Gooch wrote:
    I also noticed that (for the case where the client keeps the connection
    open), each time I call the Poll() RPC, the number of
    goroutines goes up by 1. So, it seems that the RPC handling code is leaking
    goroutines.

    Is this expected behaviour? A bug? Am I doing it wrong? I prefer to keep the
    connection open, so as to amortize the
    connection setup time.
    Leaking goroutines isn't expected behavior.

    By "RPC", do you mean the "net/rpc" package in the standard library?

    When you say the number of goroutines goes up by 1, how are you
    counting goroutines?

    If you can boil down a minimal repro case, I'm happy to take a look at
    it. Otherwise, I'd do some printf debugging around service.call in
    net/rpc/server.go to see why the goroutines the server spawns aren't
    exiting.

    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
  • Richard Gooch at Aug 23, 2015 at 5:51 pm

    On Wednesday, 15 July 2015 18:22:25 UTC-7, Nigel Tao wrote:
    On Wed, Jul 15, 2015 at 11:52 PM, Richard Gooch
    <rg+go...@safe-mbox.com <javascript:>> wrote:
    I also noticed that (for the case where the client keeps the connection
    open), each time I call the Poll() RPC, the number of
    goroutines goes up by 1. So, it seems that the RPC handling code is leaking
    goroutines.

    Is this expected behaviour? A bug? Am I doing it wrong? I prefer to keep the
    connection open, so as to amortize the
    connection setup time.
    Leaking goroutines isn't expected behavior.

    By "RPC", do you mean the "net/rpc" package in the standard library?

    When you say the number of goroutines goes up by 1, how are you
    counting goroutines?

    If you can boil down a minimal repro case, I'm happy to take a look at
    it. Otherwise, I'd do some printf debugging around service.call in
    net/rpc/server.go to see why the goroutines the server spawns aren't
    exiting.
      Apologies for the slow response. I got busy with other parts of the system.

    The first thing to point out is that I found a goroutine leak in my code,
    so I fixed that. I've re-run my measurements (with Go 1.5) and it's now
    pretty clear that the RPC code is hanging onto a copy of the data or an
    appropriately sized buffer. It looks most likely that it's holding onto a
    copy of the GOB encoded data. Whether the client issues one or many RPCs,
    the allocated memory in the server jumps by the encoded GOB size. Once the
    client closes the connection, the allocated memory returns to normal.

    Is that known or expected behaviour?

    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
  • Nigel Tao at Aug 23, 2015 at 11:54 pm

    On Mon, Aug 24, 2015 at 3:51 AM, Richard Gooch wrote:
    Is that known or expected behaviour?
    I doubt that that's expected behavior, but once again, a minimal repro
    case would be most helpful in debugging this.

    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
  • Richard Gooch at Aug 24, 2015 at 12:37 am

    On Sunday, 23 August 2015 16:54:12 UTC-7, Nigel Tao wrote:
    On Mon, Aug 24, 2015 at 3:51 AM, Richard Gooch <rg+go...@safe-mbox.com
    <javascript:>> wrote:
    Is that known or expected behaviour?
    I doubt that that's expected behavior, but once again, a minimal repro
    case would be most helpful in debugging this.
    OK, please see attached files. Pass a filename to the rpc-server programme
    which will contain the data that the server will send back to the client.
    Observe how the server allocated memory jumps up once the client has called
    the RPC, and that memory stays allocated until the client closes the
    connection, at which point memory allocation goes back down to around what
    it was before the RPC call.

    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
  • Nigel Tao at Aug 24, 2015 at 2:02 am
    OK, I'm looking at it. Just one quick, unrelated note:

    sleepDuration, _ = time.ParseDuration("10s")
    time.Sleep(sleepDuration)

    can be the simpler

    time.Sleep(10 * time.Second)

    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
  • Richard Gooch at Aug 24, 2015 at 3:55 am

    On Sunday, 23 August 2015 19:03:02 UTC-7, Nigel Tao wrote:
    OK, I'm looking at it. Just one quick, unrelated note:

    sleepDuration, _ = time.ParseDuration("10s")
    time.Sleep(sleepDuration)

    can be the simpler

    time.Sleep(10 * time.Second)
    Ah, yes, good point. Thanks.

    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
  • Richard Gooch at Sep 12, 2015 at 3:17 pm

    On Sunday, 23 August 2015 19:03:02 UTC-7, Nigel Tao wrote:
    OK, I'm looking at it.
    Any updates on this RPC memory leak?

    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
  • Nigel Tao at Sep 14, 2015 at 12:25 am
    Ah, sorry. I poked at it for a bit, but didn't find anything worth reporting.

    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupgolang-nuts @
categoriesgo
postedJul 14, '15 at 1:52p
activeSep 14, '15 at 12:25a
posts10
users2
websitegolang.org

2 users in discussion

Richard Gooch: 6 posts Nigel Tao: 4 posts

People

Translate

site design / logo © 2022 Grokbase