FAQ
Hi, all

I have tested sync.Pool with the following code

package main

import (
  "runtime"
  "sync"
)

var count = 0

var pool = sync.Pool{
  New: func() interface{} {
   count++
   buffer := make([]byte, 1024)
   return buffer
  },
}

func onFinalizer(data interface{}) {
  buffer := data.(*interface{})
  pool.Put(*buffer)
}

func test() {
  data := pool.Get()
  runtime.SetFinalizer(&data, onFinalizer)
}

func main() {
  for {
   for i := 0; i < 10000; i++ {
    test()
    runtime.Gosched()
   }
   println(count)
  }
}

I expect that New function will be called limited times, but the result is
that the program will run until out-of-memory.

Did I use it improperly or something else happened?

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Search Discussions

  • Ian Lance Taylor at Dec 15, 2015 at 5:45 am

    On Mon, Dec 14, 2015 at 8:16 PM, Hoping White wrote:
    I have tested sync.Pool with the following code

    package main

    import (
    "runtime"
    "sync"
    )

    var count = 0

    var pool = sync.Pool{
    New: func() interface{} {
    count++
    buffer := make([]byte, 1024)
    return buffer
    },
    }

    func onFinalizer(data interface{}) {
    buffer := data.(*interface{})
    pool.Put(*buffer)
    }

    func test() {
    data := pool.Get()
    runtime.SetFinalizer(&data, onFinalizer)
    }

    func main() {
    for {
    for i := 0; i < 10000; i++ {
    test()
    runtime.Gosched()
    }
    println(count)
    }
    }

    I expect that New function will be called limited times, but the result is
    that the program will run until out-of-memory.

    Did I use it improperly or something else happened?
    The Go garbage collector is more or less designed to let your program
    get up to its working set. When the GC runs when you have N bytes
    allocated, it sets its next run to occur when you have N*2 bytes
    allocated. In your case, the GC can essentially never free anything.
    Each time it tries, the finalizer brings the value back to life by
    storing it in the pool. So the GC keeps increasing the points at
    which it will run, until your program runs out of memory.

    This is not a good way to use sync.Pool, not only because it doesn't
    work, but because the whole point of sync.Pool is to take work away
    from the GC. When you use a finalizer to add a value back to the
    pool, you are missing the point of the pool. The GC is doing the work
    it requires to find that the value can be freed and then run the
    finalizer. The expensive part of GC is not the allocation; it's
    finding the free memory. Using a finalizer to add a value back to
    sync.Pool forces the GC to do the expensive work in order to save on
    the cheap work. It doesn't make sense.

    Ian

    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
  • Pablo Rozas-Larraondo at Dec 16, 2015 at 12:52 am
    I understand that the GC can never frees any byte slice, as they remain
    reachable inside the pool. But what's still not clear to me is why the Pool
    doesn't return these byte slices when it receives a Get() call.

    As the memory grows the GC is called placing these objects in the Pool, why
    they are never returned and instead the Pool creates new objects?

    Pablo



    On Tuesday, December 15, 2015 at 4:45:58 PM UTC+11, Ian Lance Taylor wrote:

    On Mon, Dec 14, 2015 at 8:16 PM, Hoping White <baiha...@gmail.com
    <javascript:>> wrote:
    I have tested sync.Pool with the following code

    package main

    import (
    "runtime"
    "sync"
    )

    var count = 0

    var pool = sync.Pool{
    New: func() interface{} {
    count++
    buffer := make([]byte, 1024)
    return buffer
    },
    }

    func onFinalizer(data interface{}) {
    buffer := data.(*interface{})
    pool.Put(*buffer)
    }

    func test() {
    data := pool.Get()
    runtime.SetFinalizer(&data, onFinalizer)
    }

    func main() {
    for {
    for i := 0; i < 10000; i++ {
    test()
    runtime.Gosched()
    }
    println(count)
    }
    }

    I expect that New function will be called limited times, but the result is
    that the program will run until out-of-memory.

    Did I use it improperly or something else happened?
    The Go garbage collector is more or less designed to let your program
    get up to its working set. When the GC runs when you have N bytes
    allocated, it sets its next run to occur when you have N*2 bytes
    allocated. In your case, the GC can essentially never free anything.
    Each time it tries, the finalizer brings the value back to life by
    storing it in the pool. So the GC keeps increasing the points at
    which it will run, until your program runs out of memory.

    This is not a good way to use sync.Pool, not only because it doesn't
    work, but because the whole point of sync.Pool is to take work away
    from the GC. When you use a finalizer to add a value back to the
    pool, you are missing the point of the pool. The GC is doing the work
    it requires to find that the value can be freed and then run the
    finalizer. The expensive part of GC is not the allocation; it's
    finding the free memory. Using a finalizer to add a value back to
    sync.Pool forces the GC to do the expensive work in order to save on
    the cheap work. It doesn't make sense.

    Ian
    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
  • Jesse McNelis at Dec 16, 2015 at 1:04 am

    On Wed, Dec 16, 2015 at 11:51 AM, Pablo Rozas-Larraondo wrote:
    I understand that the GC can never frees any byte slice, as they remain
    reachable inside the pool. But what's still not clear to me is why the Pool
    doesn't return these byte slices when it receives a Get() call.

    As the memory grows the GC is called placing these objects in the Pool, why
    they are never returned and instead the Pool creates new objects?
    New objects are created because the pool is empty when the Get() call is made.
    The pool is empty because the finalizers haven't run yet because the
    GC hasn't run yet.

    At some point the GC is run, the objects get put back in the the pool
    and are available for Get() again,
    but eventually that pool runs out again and allocations start again
    until the heap reaches the size of the next GC trigger.

    Each loop through the heap gets bigger and the next GC trigger size
    gets larger until you eventually run out of memory.

    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
  • Pablo Rozas Larraondo at Dec 16, 2015 at 1:40 am
    Very well explained, I understand it now. Thank you!
    On Wed, Dec 16, 2015 at 12:04 PM, Jesse McNelis wrote:

    On Wed, Dec 16, 2015 at 11:51 AM, Pablo Rozas-Larraondo
    wrote:
    I understand that the GC can never frees any byte slice, as they remain
    reachable inside the pool. But what's still not clear to me is why the Pool
    doesn't return these byte slices when it receives a Get() call.

    As the memory grows the GC is called placing these objects in the Pool, why
    they are never returned and instead the Pool creates new objects?
    New objects are created because the pool is empty when the Get() call is
    made.
    The pool is empty because the finalizers haven't run yet because the
    GC hasn't run yet.

    At some point the GC is run, the objects get put back in the the pool
    and are available for Get() again,
    but eventually that pool runs out again and allocations start again
    until the heap reaches the size of the next GC trigger.

    Each loop through the heap gets bigger and the next GC trigger size
    gets larger until you eventually run out of memory.
    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupgolang-nuts @
categoriesgo
postedDec 15, '15 at 4:16a
activeDec 16, '15 at 1:40a
posts5
users4
websitegolang.org

People

Translate

site design / logo © 2021 Grokbase