FAQ
I am trying to use lru_cache

https://code.google.com/p/vitess/source/browse/go/cache/lru_cache.go

data store in cache need to satisfy the interface

// Values that go into LRUCache need to satisfy this interface.
type Value interface {
Size() int
}
How can I easily satisfy this interface for my custom type.
I tried with unsafe, which prints 24 (most likely the pointer size) for
both.
But is it same as Size expect by the above interface.

Besides, I may not be able to use unsafe package in my environment (app
engine).
Thank you for the help.

====
package main

import "unsafe"

type A struct {
a int
name string
}

func main() {
a := A{a: 1, name: "a"}
a2 := A{a: 2222222, name: "Hello World"}
println(unsafe.Sizeof(a))
println(unsafe.Sizeof(a2))
}

--

Search Discussions

  • Bryanturley at Nov 15, 2012 at 3:32 pm

    I tried with unsafe, which prints 24 (most likely the pointer size) for
    both.
    But is it same as Size expect by the above interface.
    24 is the struct size, pointer size is either 4 (32bit platform) or 8
    (64bit platform).

    http://golang.org/pkg/reflect/#Type that will tell you the size of your
    stuff without using unsafe (at least directly...).

    --
  • Bsr at Nov 15, 2012 at 3:49 pm
    Thank you very much.
    Do you think reflect.Type will consider the value it points to (how many
    bytes the strings are ..) etc.

    Meanwhile, I saw an implementation here.
    http://golang.org/src/pkg/encoding/binary/binary.go?s=6460:6488#L246

    Please advise.
    On Thursday, November 15, 2012 10:25:19 AM UTC-5, bryanturley wrote:

    I tried with unsafe, which prints 24 (most likely the pointer size) for
    both.
    But is it same as Size expect by the above interface.
    24 is the struct size, pointer size is either 4 (32bit platform) or 8
    (64bit platform).

    http://golang.org/pkg/reflect/#Type that will tell you the size of your
    stuff without using unsafe (at least directly...).
    --
  • Roger peppe at Nov 15, 2012 at 3:58 pm
    the Size method need not be exactly accurate - it's only used for lru_cache
    to estimate the size of the items in the cache. in fact, the size will
    be greater
    than sizeof reports, because each item added to the cache makes at
    least a couple of heap allocations.

    if you particularly care about the space used by the cache, you
    might be best measuring it empirically - create a cache with a sufficiently
    large limit, add some number of items and see how much memory has
    been allocated. then divide that by the number of items you added
    and use that value for Size.

    this assumes you're using constant-size items, of course, which you're
    not, so you'd need to make some allowance for the size of the string,
    which you may or may not wish to consider as part of the cache space
    allocation.

    On 15 November 2012 15:49, bsr wrote:
    Thank you very much.
    Do you think reflect.Type will consider the value it points to (how many
    bytes the strings are ..) etc.

    Meanwhile, I saw an implementation here.
    http://golang.org/src/pkg/encoding/binary/binary.go?s=6460:6488#L246

    Please advise.

    On Thursday, November 15, 2012 10:25:19 AM UTC-5, bryanturley wrote:

    I tried with unsafe, which prints 24 (most likely the pointer size) for
    both.
    But is it same as Size expect by the above interface.

    24 is the struct size, pointer size is either 4 (32bit platform) or 8
    (64bit platform).

    http://golang.org/pkg/reflect/#Type that will tell you the size of your
    stuff without using unsafe (at least directly...).
    --
    --
  • Bsr at Nov 15, 2012 at 4:10 pm
    Thanks rog. I only initialize the cache once, and overhead is minimal. I
    may try the method from encoding/binary and see how it works.
    thanks
    On Thursday, November 15, 2012 10:58:22 AM UTC-5, rog wrote:

    the Size method need not be exactly accurate - it's only used for
    lru_cache
    to estimate the size of the items in the cache. in fact, the size will
    be greater
    than sizeof reports, because each item added to the cache makes at
    least a couple of heap allocations.

    if you particularly care about the space used by the cache, you
    might be best measuring it empirically - create a cache with a
    sufficiently
    large limit, add some number of items and see how much memory has
    been allocated. then divide that by the number of items you added
    and use that value for Size.

    this assumes you're using constant-size items, of course, which you're
    not, so you'd need to make some allowance for the size of the string,
    which you may or may not wish to consider as part of the cache space
    allocation.

    On 15 November 2012 15:49, bsr <bsr...@gmail.com <javascript:>> wrote:
    Thank you very much.
    Do you think reflect.Type will consider the value it points to (how many
    bytes the strings are ..) etc.

    Meanwhile, I saw an implementation here.
    http://golang.org/src/pkg/encoding/binary/binary.go?s=6460:6488#L246

    Please advise.

    On Thursday, November 15, 2012 10:25:19 AM UTC-5, bryanturley wrote:

    I tried with unsafe, which prints 24 (most likely the pointer size)
    for
    both.
    But is it same as Size expect by the above interface.

    24 is the struct size, pointer size is either 4 (32bit platform) or 8
    (64bit platform).

    http://golang.org/pkg/reflect/#Type that will tell you the size of
    your
    stuff without using unsafe (at least directly...).
    --
    --
  • Sugu Sougoumarane at Nov 15, 2012 at 7:17 pm
    Pretty much everything rog said is on the money. A few more caveats:
    - lru_cache is meant for small caches, typically a few thousand items.
    - For small caches, it's easier to think of the number of items as limit
    rather than size. For example, if you instantiate lru_cache with size 5000,
    and if each object reported a size of 1, then your cache would have a max
    of 5000 items in it. This is how we've typically used it.
    - Any type can be trivially customized to be in lru_cache.
    - If you grep vitess for lru_cache, you'll see a few examples of how we've
    used it. For example,
    here: http://code.google.com/p/vitess/source/browse/go/vt/tabletserver/consolidator.go#84
    On Thursday, November 15, 2012 7:58:22 AM UTC-8, rog wrote:

    the Size method need not be exactly accurate - it's only used for
    lru_cache
    to estimate the size of the items in the cache. in fact, the size will
    be greater
    than sizeof reports, because each item added to the cache makes at
    least a couple of heap allocations.

    if you particularly care about the space used by the cache, you
    might be best measuring it empirically - create a cache with a
    sufficiently
    large limit, add some number of items and see how much memory has
    been allocated. then divide that by the number of items you added
    and use that value for Size.

    this assumes you're using constant-size items, of course, which you're
    not, so you'd need to make some allowance for the size of the string,
    which you may or may not wish to consider as part of the cache space
    allocation.

    On 15 November 2012 15:49, bsr <bsr...@gmail.com <javascript:>> wrote:
    Thank you very much.
    Do you think reflect.Type will consider the value it points to (how many
    bytes the strings are ..) etc.

    Meanwhile, I saw an implementation here.
    http://golang.org/src/pkg/encoding/binary/binary.go?s=6460:6488#L246

    Please advise.

    On Thursday, November 15, 2012 10:25:19 AM UTC-5, bryanturley wrote:

    I tried with unsafe, which prints 24 (most likely the pointer size)
    for
    both.
    But is it same as Size expect by the above interface.

    24 is the struct size, pointer size is either 4 (32bit platform) or 8
    (64bit platform).

    http://golang.org/pkg/reflect/#Type that will tell you the size of
    your
    stuff without using unsafe (at least directly...).
    --
    --
  • Bsr at Nov 15, 2012 at 8:03 pm
    Thanks Sogu. It works great. I would follow your recommendations.
    On Thursday, November 15, 2012 2:10:30 PM UTC-5, Sugu Sougoumarane wrote:

    Pretty much everything rog said is on the money. A few more caveats:
    - lru_cache is meant for small caches, typically a few thousand items.
    - For small caches, it's easier to think of the number of items as limit
    rather than size. For example, if you instantiate lru_cache with size 5000,
    and if each object reported a size of 1, then your cache would have a max
    of 5000 items in it. This is how we've typically used it.
    - Any type can be trivially customized to be in lru_cache.
    - If you grep vitess for lru_cache, you'll see a few examples of how we've
    used it. For example, here:
    http://code.google.com/p/vitess/source/browse/go/vt/tabletserver/consolidator.go#84
    On Thursday, November 15, 2012 7:58:22 AM UTC-8, rog wrote:

    the Size method need not be exactly accurate - it's only used for
    lru_cache
    to estimate the size of the items in the cache. in fact, the size will
    be greater
    than sizeof reports, because each item added to the cache makes at
    least a couple of heap allocations.

    if you particularly care about the space used by the cache, you
    might be best measuring it empirically - create a cache with a
    sufficiently
    large limit, add some number of items and see how much memory has
    been allocated. then divide that by the number of items you added
    and use that value for Size.

    this assumes you're using constant-size items, of course, which you're
    not, so you'd need to make some allowance for the size of the string,
    which you may or may not wish to consider as part of the cache space
    allocation.

    On 15 November 2012 15:49, bsr wrote:
    Thank you very much.
    Do you think reflect.Type will consider the value it points to (how many
    bytes the strings are ..) etc.

    Meanwhile, I saw an implementation here.
    http://golang.org/src/pkg/encoding/binary/binary.go?s=6460:6488#L246

    Please advise.

    On Thursday, November 15, 2012 10:25:19 AM UTC-5, bryanturley wrote:

    I tried with unsafe, which prints 24 (most likely the pointer size)
    for
    both.
    But is it same as Size expect by the above interface.

    24 is the struct size, pointer size is either 4 (32bit platform) or 8
    (64bit platform).

    http://golang.org/pkg/reflect/#Type that will tell you the size of
    your
    stuff without using unsafe (at least directly...).
    --
    --
  • Bryanturley at Nov 15, 2012 at 4:26 pm

    On Thursday, November 15, 2012 9:49:07 AM UTC-6, bsr wrote:
    Thank you very much.
    Do you think reflect.Type will consider the value it points to (how many
    bytes the strings are ..) etc.
    I would be surprised if it did since the string is probably just a length
    and a pointer and the string data is it's own allocation generally.
    There may be another reflect function that does follow pointers or it would
    be (somewhat) easy to write one that does using reflect.
    I am going to agree with rog on "the Size method need not be exactly
    accurate - it's only used for lru_cache
    to estimate the size of the items in the cache."

    Since you are not forced to free all your allocations it is harder to know
    just how much memory you are actually using in a given go program while it
    is running.
    As long as you aren't constantly allocating giant buffers this shouldn't
    really be a problem though.
    Don't get me wrong, go having garbage collection is great.

    --

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupgolang-nuts @
categoriesgo
postedNov 15, '12 at 1:09p
activeNov 15, '12 at 8:03p
posts8
users4
websitegolang.org

People

Translate

site design / logo © 2021 Grokbase