FAQ
In https://golang.org/ref/mem , it talked about how to avoid double-checked
locking to initialize one variable, by using once.Do.

But how to efficiently and thread-safely initialize a multi-threaded map
entries?

For example: there is a highly concurrent map from a string to an object
pointer. Each map entry can only be created on the first map lookup.

map[string]*SomeObject m
string x
if _, ok := m[x]; !ok {
   m[x] = &SomeObject{}
}

If you know any clean way to do this, please let me know.

Chris

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Search Discussions

  • Ian Lance Taylor at Dec 5, 2014 at 7:55 pm

    On Fri, Dec 5, 2014 at 11:36 AM, ChrisLu wrote:
    In https://golang.org/ref/mem , it talked about how to avoid double-checked
    locking to initialize one variable, by using once.Do.

    But how to efficiently and thread-safely initialize a multi-threaded map
    entries?

    For example: there is a highly concurrent map from a string to an object
    pointer. Each map entry can only be created on the first map lookup.

    map[string]*SomeObject m
    string x
    if _, ok := m[x]; !ok {
    m[x] = &SomeObject{}
    }

    If you know any clean way to do this, please let me know.
    The Go builtin map type can't be used as a highly concurrent writable
    map. If that is what you need, you will need to build your own data
    structure with appropriate locking. (It is nontrivial to do this
    safely and efficiently.)

    The closest you can come with the builtin map type is to control all
    map accesses with a sync.RWMutex, which permits concurrent reads and
    serializes writes.

    Ian

    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
  • Egon at Dec 5, 2014 at 8:05 pm

    On Friday, 5 December 2014 21:36:31 UTC+2, ChrisLu wrote:
    In https://golang.org/ref/mem , it talked about how to avoid
    double-checked locking to initialize one variable, by using once.Do.

    But how to efficiently and thread-safely initialize a multi-threaded map
    entries?
    This highly depends on the context where you are working in
    <https://code.google.com/p/go-wiki/wiki/HowToAsk>.

    Here are a few different ways of implementing a concurrent map:
    http://play.golang.org/p/IQ9IktS4Hq

    + Egon

    For example: there is a highly concurrent map from a string to an object
    pointer. Each map entry can only be created on the first map lookup.

    map[string]*SomeObject m
    string x
    if _, ok := m[x]; !ok {
    m[x] = &SomeObject{}
    }

    If you know any clean way to do this, please let me know.

    Chris
    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
  • Adam at Dec 6, 2014 at 1:32 am

    On Friday, December 5, 2014 3:05:17 PM UTC-5, egon wrote:

    On Friday, 5 December 2014 21:36:31 UTC+2, ChrisLu wrote:

    In https://golang.org/ref/mem , it talked about how to avoid
    double-checked locking to initialize one variable, by using once.Do.

    But how to efficiently and thread-safely initialize a multi-threaded map
    entries?
    This highly depends on the context where you are working in
    <https://code.google.com/p/go-wiki/wiki/HowToAsk>.

    Here are a few different ways of implementing a concurrent map:
    http://play.golang.org/p/IQ9IktS4Hq
    could you write this to handle strings.

    + Egon

    For example: there is a highly concurrent map from a string to an object
    pointer. Each map entry can only be created on the first map lookup.

    map[string]*SomeObject m
    string x
    if _, ok := m[x]; !ok {
    m[x] = &SomeObject{}
    }

    If you know any clean way to do this, please let me know.

    Chris
    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
  • Egon at Dec 6, 2014 at 10:20 am

    On Saturday, 6 December 2014 03:32:06 UTC+2, adam wrote:

    On Friday, December 5, 2014 3:05:17 PM UTC-5, egon wrote:


    On Friday, 5 December 2014 21:36:31 UTC+2, ChrisLu wrote:

    In https://golang.org/ref/mem , it talked about how to avoid
    double-checked locking to initialize one variable, by using once.Do.

    But how to efficiently and thread-safely initialize a multi-threaded map
    entries?
    This highly depends on the context where you are working in
    <https://code.google.com/p/go-wiki/wiki/HowToAsk>.

    Here are a few different ways of implementing a concurrent map:
    http://play.golang.org/p/IQ9IktS4Hq
    could you write this to handle strings.
    Yes, but it's simple enough that me doing has no benefit for me... :) And
    has more benefit if you try to write it yourself.

    Anyways the simplest way to start rewriting this for strings is to replace
    all the "map[int]int" with "map[string]int", adjust the methods to use a
    string as a key... and then incrementally start fixing the compilation
    errors.

    The harder one might be the striped version... there you must
    change k%stripes with some sort of hash(k)^stripes, where the k is now a
    string. Something like JS or SDBM hash
    <http://www.cad.polito.it/~squillero/dida/Algoritmi_e_Programmazione_Avanzata/HashFunctions.c>
    should be good enough... of course depending on the strings the ideal hash
    will vary.

    + Egon

    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
  • Adam at Dec 6, 2014 at 4:14 pm
    now could you write it using a custom *MyType
    On Saturday, December 6, 2014 5:19:15 AM UTC-5, egon wrote:
    On Saturday, 6 December 2014 03:32:06 UTC+2, adam wrote:


    On Friday, December 5, 2014 3:05:17 PM UTC-5, egon wrote:


    On Friday, 5 December 2014 21:36:31 UTC+2, ChrisLu wrote:

    In https://golang.org/ref/mem , it talked about how to avoid
    double-checked locking to initialize one variable, by using once.Do.

    But how to efficiently and thread-safely initialize a multi-threaded
    map entries?
    This highly depends on the context where you are working in
    <https://code.google.com/p/go-wiki/wiki/HowToAsk>.

    Here are a few different ways of implementing a concurrent map:
    http://play.golang.org/p/IQ9IktS4Hq
    could you write this to handle strings.
    Yes, but it's simple enough that me doing has no benefit for me... :) And
    has more benefit if you try to write it yourself.

    Anyways the simplest way to start rewriting this for strings is to replace
    all the "map[int]int" with "map[string]int", adjust the methods to use a
    string as a key... and then incrementally start fixing the compilation
    errors.

    The harder one might be the striped version... there you must
    change k%stripes with some sort of hash(k)^stripes, where the k is now a
    string. Something like JS or SDBM hash
    <http://www.cad.polito.it/~squillero/dida/Algoritmi_e_Programmazione_Avanzata/HashFunctions.c>
    should be good enough... of course depending on the strings the ideal hash
    will vary.

    + Egon
    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
  • Adam at Dec 6, 2014 at 4:17 pm

    On Saturday, December 6, 2014 11:14:39 AM UTC-5, adam wrote:
    now could you write it using a custom *MyType
    On Saturday, December 6, 2014 5:19:15 AM UTC-5, egon wrote:
    On Saturday, 6 December 2014 03:32:06 UTC+2, adam wrote:


    On Friday, December 5, 2014 3:05:17 PM UTC-5, egon wrote:


    On Friday, 5 December 2014 21:36:31 UTC+2, ChrisLu wrote:

    In https://golang.org/ref/mem , it talked about how to avoid
    double-checked locking to initialize one variable, by using once.Do.

    But how to efficiently and thread-safely initialize a multi-threaded
    map entries?
    This highly depends on the context where you are working in
    <https://code.google.com/p/go-wiki/wiki/HowToAsk>.

    Here are a few different ways of implementing a concurrent map:
    http://play.golang.org/p/IQ9IktS4Hq
    could you write this to handle strings.
    Yes, but it's simple enough that me doing has no benefit for me... :) And
    has more benefit if you try to write it yourself.
    there is no benefit to anyone having to right the same data type over and
    over and over and over again. Get where I am going with this? Type
    Parameterization......

    Anyways the simplest way to start rewriting this for strings is to
    replace all the "map[int]int" with "map[string]int", adjust the methods to
    use a string as a key... and then incrementally start fixing the
    compilation errors.

    The harder one might be the striped version... there you must
    change k%stripes with some sort of hash(k)^stripes, where the k is now a
    string. Something like JS or SDBM hash
    <http://www.cad.polito.it/~squillero/dida/Algoritmi_e_Programmazione_Avanzata/HashFunctions.c>
    should be good enough... of course depending on the strings the ideal hash
    will vary.

    + Egon
    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
  • Adam at Dec 6, 2014 at 4:21 pm

    On Saturday, December 6, 2014 5:19:15 AM UTC-5, egon wrote:
    On Saturday, 6 December 2014 03:32:06 UTC+2, adam wrote:


    On Friday, December 5, 2014 3:05:17 PM UTC-5, egon wrote:


    On Friday, 5 December 2014 21:36:31 UTC+2, ChrisLu wrote:

    In https://golang.org/ref/mem , it talked about how to avoid
    double-checked locking to initialize one variable, by using once.Do.

    But how to efficiently and thread-safely initialize a multi-threaded
    map entries?
    This highly depends on the context where you are working in
    <https://code.google.com/p/go-wiki/wiki/HowToAsk>.

    Here are a few different ways of implementing a concurrent map:
    http://play.golang.org/p/IQ9IktS4Hq
    could you write this to handle strings.
    Yes, but it's simple enough that me doing has no benefit for me... :) And
    has more benefit if you try to write it yourself.
    there is no benefit to anyone having to write the same data type over and
    over and over and over again. Get where I am going with this? Type
    Parameterization......

    Anyways the simplest way to start rewriting this for strings is to replace
    all the "map[int]int" with "map[string]int", adjust the methods to use a
    string as a key... and then incrementally start fixing the compilation
    errors.

    The harder one might be the striped version... there you must
    change k%stripes with some sort of hash(k)^stripes, where the k is now a
    string. Something like JS or SDBM hash
    <http://www.cad.polito.it/~squillero/dida/Algoritmi_e_Programmazione_Avanzata/HashFunctions.c>
    should be good enough... of course depending on the strings the ideal hash
    will vary.

    + Egon
    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
  • Atdiar at Dec 6, 2014 at 4:28 pm

    On Saturday, December 6, 2014 4:21:56 PM UTC, adam wrote:

    there is no benefit to anyone having to write the same data type over and
    over and over and over again. Get where I am going with this? Type
    Parameterization......

    Anyways the simplest way to start rewriting this for strings is to
    replace all the "map[int]int" with "map[string]int", adjust the methods to
    use a string as a key... and then incrementally start fixing the
    compilation errors.

    The harder one might be the striped version... there you must
    change k%stripes with some sort of hash(k)^stripes, where the k is now a
    string. Something like JS or SDBM hash
    <http://www.cad.polito.it/~squillero/dida/Algoritmi_e_Programmazione_Avanzata/HashFunctions.c>
    should be good enough... of course depending on the strings the ideal hash
    will vary.

    + Egon
    No. What you need is code generation if you don't want to do it manually.
    Alternatively use an extra indirection by using an interface{}.
    Type parametrization is a poor way of doing things if you intend to write
    programs for multicore machines.

    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
  • Akwillis at Dec 6, 2014 at 4:50 pm

    On Saturday, December 6, 2014 11:28:28 AM UTC-5, atd...@gmail.com wrote:

    On Saturday, December 6, 2014 4:21:56 PM UTC, adam wrote:


    there is no benefit to anyone having to write the same data type over and
    over and over and over again. Get where I am going with this? Type
    Parameterization......

    Anyways the simplest way to start rewriting this for strings is to
    replace all the "map[int]int" with "map[string]int", adjust the methods to
    use a string as a key... and then incrementally start fixing the
    compilation errors.

    The harder one might be the striped version... there you must
    change k%stripes with some sort of hash(k)^stripes, where the k is now a
    string. Something like JS or SDBM hash
    <http://www.cad.polito.it/~squillero/dida/Algoritmi_e_Programmazione_Avanzata/HashFunctions.c>
    should be good enough... of course depending on the strings the ideal hash
    will vary.

    + Egon
    No. What you need is code generation if you don't want to do it manually.
    Alternatively use an extra indirection by using an interface{}.
    Type parametrization is a poor way of doing things if you intend to write
    programs for multicore machines.
    Explain the difference between a builtin preprocessor that handles type
    parameterization and go generate. they both would handle to code the same
    way- as a preprocessor. And please explain how using the two approaches
    differ at compile time, making one a poor choice and the other spot on ?


    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
  • Akwillis at Dec 6, 2014 at 4:58 pm

    On Saturday, December 6, 2014 11:28:28 AM UTC-5, atd...@gmail.com wrote:

    On Saturday, December 6, 2014 4:21:56 PM UTC, adam wrote:


    there is no benefit to anyone having to write the same data type over and
    over and over and over again. Get where I am going with this? Type
    Parameterization......

    Anyways the simplest way to start rewriting this for strings is to
    replace all the "map[int]int" with "map[string]int", adjust the methods to
    use a string as a key... and then incrementally start fixing the
    compilation errors.

    The harder one might be the striped version... there you must
    change k%stripes with some sort of hash(k)^stripes, where the k is now a
    string. Something like JS or SDBM hash
    <http://www.cad.polito.it/~squillero/dida/Algoritmi_e_Programmazione_Avanzata/HashFunctions.c>
    should be good enough... of course depending on the strings the ideal hash
    will vary.

    + Egon
    No. What you need is code generation if you don't want to do it manually.
    Alternatively use an extra indirection by using an interface{}.
    Type parametrization is a poor way of doing things if you intend to write
    programs for multicore machines.
      using the hacky tricks with interfaces doubles the size of the data type
    in memory, with large data sets in the multi gigabyte ranges this become
    unrealistic.

    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
  • Atdiar at Dec 6, 2014 at 5:15 pm
    Not true. The interface points to a copy of the value. The original can be
    garbage collected.
    Besides, it's not a hack. That's close to Generics in C#.. You have the
    choice to employ that method, or not. Your call.
    Anyway, I think there is another thread on generics, we can discuss there
    if you want.
    On Saturday, December 6, 2014 4:58:28 PM UTC, akwillis wrote:


    On Saturday, December 6, 2014 11:28:28 AM UTC-5, atd...@gmail.com wrote:


    On Saturday, December 6, 2014 4:21:56 PM UTC, adam wrote:


    there is no benefit to anyone having to write the same data type over
    and over and over and over again. Get where I am going with this? Type
    Parameterization......

    Anyways the simplest way to start rewriting this for strings is to
    replace all the "map[int]int" with "map[string]int", adjust the methods to
    use a string as a key... and then incrementally start fixing the
    compilation errors.

    The harder one might be the striped version... there you must
    change k%stripes with some sort of hash(k)^stripes, where the k is now a
    string. Something like JS or SDBM hash
    <http://www.cad.polito.it/~squillero/dida/Algoritmi_e_Programmazione_Avanzata/HashFunctions.c>
    should be good enough... of course depending on the strings the ideal hash
    will vary.

    + Egon
    No. What you need is code generation if you don't want to do it manually.
    Alternatively use an extra indirection by using an interface{}.
    Type parametrization is a poor way of doing things if you intend to write
    programs for multicore machines.
    using the hacky tricks with interfaces doubles the size of the data type
    in memory, with large data sets in the multi gigabyte ranges this become
    unrealistic.
    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
  • Akwillis at Dec 6, 2014 at 5:33 pm
    Ok, please explain why one uses double the resources of the other. Maybe
    I'm missing something.


    package main

    import "fmt"
    import "runtime"

    func main() {
    n := 100000000
    slice := make([]int, 0, n)
    for i := 1; i <= n; i++ {
    slice = append(slice, i)
    }
    var b runtime.MemStats
    runtime.ReadMemStats(&b)
    fmt.Println(b.Alloc/1000000)
    fmt.Println(b.TotalAlloc/1000000)
    fmt.Println(b.Sys/1000000)
    fmt.Println(b.Lookups)
    fmt.Println(b.Mallocs)
    fmt.Println(b.Frees)
    }


    package main

    import "fmt"
    import "runtime"

    func main() {
    n := 100000000
    slice := make([]interface{}, 0, n)
    for i := 1; i <= n ; i++ {
    slice = append(slice, i)
    }
    var b runtime.MemStats
    runtime.ReadMemStats(&b)
    fmt.Println(b.Alloc/1000000)
    fmt.Println(b.TotalAlloc/1000000)
    fmt.Println(b.Sys/1000000)
    fmt.Println(b.Lookups)
    fmt.Println(b.Mallocs)
    fmt.Println(b.Frees)
    }
    On Saturday, December 6, 2014 12:15:04 PM UTC-5, atd...@gmail.com wrote:

    Not true. The interface points to a copy of the value. The original can be
    garbage collected.
    Besides, it's not a hack. That's close to Generics in C#.. You have the
    choice to employ that method, or not. Your call.
    Anyway, I think there is another thread on generics, we can discuss there
    if you want.
    On Saturday, December 6, 2014 4:58:28 PM UTC, akwillis wrote:


    On Saturday, December 6, 2014 11:28:28 AM UTC-5, atd...@gmail.com wrote:


    On Saturday, December 6, 2014 4:21:56 PM UTC, adam wrote:


    there is no benefit to anyone having to write the same data type over
    and over and over and over again. Get where I am going with this? Type
    Parameterization......

    Anyways the simplest way to start rewriting this for strings is to
    replace all the "map[int]int" with "map[string]int", adjust the methods to
    use a string as a key... and then incrementally start fixing the
    compilation errors.

    The harder one might be the striped version... there you must
    change k%stripes with some sort of hash(k)^stripes, where the k is now a
    string. Something like JS or SDBM hash
    <http://www.cad.polito.it/~squillero/dida/Algoritmi_e_Programmazione_Avanzata/HashFunctions.c>
    should be good enough... of course depending on the strings the ideal hash
    will vary.

    + Egon
    No. What you need is code generation if you don't want to do it
    manually. Alternatively use an extra indirection by using an interface{}.
    Type parametrization is a poor way of doing things if you intend to
    write programs for multicore machines.
    using the hacky tricks with interfaces doubles the size of the data type
    in memory, with large data sets in the multi gigabyte ranges this become
    unrealistic.
    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
  • Atdiar at Dec 6, 2014 at 6:02 pm
    I've told you it can be garbage collected. Never told you you would not
    increase the number of allocations. To be fair, that's a cost that is unto
    you if you want the genericity. Anyway, no need to pollute this thread
    further.
    On Saturday, December 6, 2014 5:33:07 PM UTC, akwillis wrote:

    Ok, please explain why one uses double the resources of the other. Maybe
    I'm missing something.


    package main

    import "fmt"
    import "runtime"

    func main() {
    n := 100000000
    slice := make([]int, 0, n)
    for i := 1; i <= n; i++ {
    slice = append(slice, i)
    }
    var b runtime.MemStats
    runtime.ReadMemStats(&b)
    fmt.Println(b.Alloc/1000000)
    fmt.Println(b.TotalAlloc/1000000)
    fmt.Println(b.Sys/1000000)
    fmt.Println(b.Lookups)
    fmt.Println(b.Mallocs)
    fmt.Println(b.Frees)
    }


    package main

    import "fmt"
    import "runtime"

    func main() {
    n := 100000000
    slice := make([]interface{}, 0, n)
    for i := 1; i <= n ; i++ {
    slice = append(slice, i)
    }
    var b runtime.MemStats
    runtime.ReadMemStats(&b)
    fmt.Println(b.Alloc/1000000)
    fmt.Println(b.TotalAlloc/1000000)
    fmt.Println(b.Sys/1000000)
    fmt.Println(b.Lookups)
    fmt.Println(b.Mallocs)
    fmt.Println(b.Frees)
    }
    On Saturday, December 6, 2014 12:15:04 PM UTC-5, atd...@gmail.com wrote:

    Not true. The interface points to a copy of the value. The original can
    be garbage collected.
    Besides, it's not a hack. That's close to Generics in C#.. You have the
    choice to employ that method, or not. Your call.
    Anyway, I think there is another thread on generics, we can discuss
    there if you want.
    On Saturday, December 6, 2014 4:58:28 PM UTC, akwillis wrote:


    On Saturday, December 6, 2014 11:28:28 AM UTC-5, atd...@gmail.com wrote:


    On Saturday, December 6, 2014 4:21:56 PM UTC, adam wrote:


    there is no benefit to anyone having to write the same data type over
    and over and over and over again. Get where I am going with this? Type
    Parameterization......

    Anyways the simplest way to start rewriting this for strings is to
    replace all the "map[int]int" with "map[string]int", adjust the methods to
    use a string as a key... and then incrementally start fixing the
    compilation errors.

    The harder one might be the striped version... there you must
    change k%stripes with some sort of hash(k)^stripes, where the k is now a
    string. Something like JS or SDBM hash
    <http://www.cad.polito.it/~squillero/dida/Algoritmi_e_Programmazione_Avanzata/HashFunctions.c>
    should be good enough... of course depending on the strings the ideal hash
    will vary.

    + Egon
    No. What you need is code generation if you don't want to do it
    manually. Alternatively use an extra indirection by using an interface{}.
    Type parametrization is a poor way of doing things if you intend to
    write programs for multicore machines.
    using the hacky tricks with interfaces doubles the size of the data
    type in memory, with large data sets in the multi gigabyte ranges this
    become unrealistic.
    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
  • Akwillis at Dec 6, 2014 at 6:26 pm
    Honestly, you are making no sense.
    On Saturday, December 6, 2014 1:02:51 PM UTC-5, atd...@gmail.com wrote:

    I've told you it can be garbage collected. Never told you you would not
    increase the number of allocations. To be fair, that's a cost that is unto
    you if you want the genericity. Anyway, no need to pollute this thread
    further.
    On Saturday, December 6, 2014 5:33:07 PM UTC, akwillis wrote:

    Ok, please explain why one uses double the resources of the other. Maybe
    I'm missing something.


    package main

    import "fmt"
    import "runtime"

    func main() {
    n := 100000000
    slice := make([]int, 0, n)
    for i := 1; i <= n; i++ {
    slice = append(slice, i)
    }
    var b runtime.MemStats
    runtime.ReadMemStats(&b)
    fmt.Println(b.Alloc/1000000)
    fmt.Println(b.TotalAlloc/1000000)
    fmt.Println(b.Sys/1000000)
    fmt.Println(b.Lookups)
    fmt.Println(b.Mallocs)
    fmt.Println(b.Frees)
    }


    package main

    import "fmt"
    import "runtime"

    func main() {
    n := 100000000
    slice := make([]interface{}, 0, n)
    for i := 1; i <= n ; i++ {
    slice = append(slice, i)
    }
    var b runtime.MemStats
    runtime.ReadMemStats(&b)
    fmt.Println(b.Alloc/1000000)
    fmt.Println(b.TotalAlloc/1000000)
    fmt.Println(b.Sys/1000000)
    fmt.Println(b.Lookups)
    fmt.Println(b.Mallocs)
    fmt.Println(b.Frees)
    }
    On Saturday, December 6, 2014 12:15:04 PM UTC-5, atd...@gmail.com wrote:

    Not true. The interface points to a copy of the value. The original can
    be garbage collected.
    Besides, it's not a hack. That's close to Generics in C#.. You have the
    choice to employ that method, or not. Your call.
    Anyway, I think there is another thread on generics, we can discuss
    there if you want.
    On Saturday, December 6, 2014 4:58:28 PM UTC, akwillis wrote:



    On Saturday, December 6, 2014 11:28:28 AM UTC-5, atd...@gmail.com
    wrote:

    On Saturday, December 6, 2014 4:21:56 PM UTC, adam wrote:


    there is no benefit to anyone having to write the same data type over
    and over and over and over again. Get where I am going with this? Type
    Parameterization......

    Anyways the simplest way to start rewriting this for strings is to
    replace all the "map[int]int" with "map[string]int", adjust the methods to
    use a string as a key... and then incrementally start fixing the
    compilation errors.

    The harder one might be the striped version... there you must
    change k%stripes with some sort of hash(k)^stripes, where the k is now a
    string. Something like JS or SDBM hash
    <http://www.cad.polito.it/~squillero/dida/Algoritmi_e_Programmazione_Avanzata/HashFunctions.c>
    should be good enough... of course depending on the strings the ideal hash
    will vary.

    + Egon
    No. What you need is code generation if you don't want to do it
    manually. Alternatively use an extra indirection by using an interface{}.
    Type parametrization is a poor way of doing things if you intend to
    write programs for multicore machines.
    using the hacky tricks with interfaces doubles the size of the data
    type in memory, with large data sets in the multi gigabyte ranges this
    become unrealistic.
    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
  • Atdiar at Dec 6, 2014 at 6:47 pm
    Maybe I just don't understand your intentions.
    What's your issue ? That an interface is 2 words and when it points to
    something such as an uint64, it takes twice more space, relative to what
    you are storing ?

    Then obviously don't use interfaces. I don't think you need me to tell you
    that.
    Use a datastructure that is customized for your type.
    If you have several, just copy the structure and replace the types.

    A tool is sufficient if you need to automatize the process, no need to have
    type parameterization cooked into the language, making it heavy for
    nothing.
    That was a good way of doing things back in the day because the conditions
    were not the same.

    On Saturday, December 6, 2014 6:26:39 PM UTC, akwillis wrote:

    Honestly, you are making no sense.
    On Saturday, December 6, 2014 1:02:51 PM UTC-5, atd...@gmail.com wrote:

    I've told you it can be garbage collected. Never told you you would not
    increase the number of allocations. To be fair, that's a cost that is unto
    you if you want the genericity. Anyway, no need to pollute this thread
    further.
    On Saturday, December 6, 2014 5:33:07 PM UTC, akwillis wrote:

    Ok, please explain why one uses double the resources of the other. Maybe
    I'm missing something.


    package main

    import "fmt"
    import "runtime"

    func main() {
    n := 100000000
    slice := make([]int, 0, n)
    for i := 1; i <= n; i++ {
    slice = append(slice, i)
    }
    var b runtime.MemStats
    runtime.ReadMemStats(&b)
    fmt.Println(b.Alloc/1000000)
    fmt.Println(b.TotalAlloc/1000000)
    fmt.Println(b.Sys/1000000)
    fmt.Println(b.Lookups)
    fmt.Println(b.Mallocs)
    fmt.Println(b.Frees)
    }


    package main

    import "fmt"
    import "runtime"

    func main() {
    n := 100000000
    slice := make([]interface{}, 0, n)
    for i := 1; i <= n ; i++ {
    slice = append(slice, i)
    }
    var b runtime.MemStats
    runtime.ReadMemStats(&b)
    fmt.Println(b.Alloc/1000000)
    fmt.Println(b.TotalAlloc/1000000)
    fmt.Println(b.Sys/1000000)
    fmt.Println(b.Lookups)
    fmt.Println(b.Mallocs)
    fmt.Println(b.Frees)
    }
    On Saturday, December 6, 2014 12:15:04 PM UTC-5, atd...@gmail.com wrote:

    Not true. The interface points to a copy of the value. The original can
    be garbage collected.
    Besides, it's not a hack. That's close to Generics in C#.. You have
    the choice to employ that method, or not. Your call.
    Anyway, I think there is another thread on generics, we can discuss
    there if you want.
    On Saturday, December 6, 2014 4:58:28 PM UTC, akwillis wrote:



    On Saturday, December 6, 2014 11:28:28 AM UTC-5, atd...@gmail.com
    wrote:

    On Saturday, December 6, 2014 4:21:56 PM UTC, adam wrote:


    there is no benefit to anyone having to write the same data type
    over and over and over and over again. Get where I am going with this? Type
    Parameterization......

    Anyways the simplest way to start rewriting this for strings is to
    replace all the "map[int]int" with "map[string]int", adjust the methods to
    use a string as a key... and then incrementally start fixing the
    compilation errors.

    The harder one might be the striped version... there you must
    change k%stripes with some sort of hash(k)^stripes, where the k is now a
    string. Something like JS or SDBM hash
    <http://www.cad.polito.it/~squillero/dida/Algoritmi_e_Programmazione_Avanzata/HashFunctions.c>
    should be good enough... of course depending on the strings the ideal hash
    will vary.

    + Egon
    No. What you need is code generation if you don't want to do it
    manually. Alternatively use an extra indirection by using an interface{}.
    Type parametrization is a poor way of doing things if you intend to
    write programs for multicore machines.
    using the hacky tricks with interfaces doubles the size of the data
    type in memory, with large data sets in the multi gigabyte ranges this
    become unrealistic.
    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
  • Egon at Dec 7, 2014 at 3:12 pm

    On Saturday, 6 December 2014 18:21:56 UTC+2, adam wrote:
    On Saturday, December 6, 2014 5:19:15 AM UTC-5, egon wrote:
    On Saturday, 6 December 2014 03:32:06 UTC+2, adam wrote:


    On Friday, December 5, 2014 3:05:17 PM UTC-5, egon wrote:


    On Friday, 5 December 2014 21:36:31 UTC+2, ChrisLu wrote:
    In https://golang.org/ref/mem , it talked about how to avoid double-checked locking to initialize one variable, by using once.Do.


    But how to efficiently and thread-safely initialize a multi-threaded map entries?


    This highly depends on the context where you are working in.


    Here are a few different ways of implementing a concurrent map:
    http://play.golang.org/p/IQ9IktS4Hq


    could you write this to handle strings.


    Yes, but it's simple enough that me doing has no benefit for me... :) And has more benefit if you try to write it yourself.

    there is no benefit to anyone having to write the same data type over and over and over and over again. Get where I am going with this? Type Parameterization......
    There are multiple parameters to these structures... the number of stripes, whether to use RWMutex or just Mutex, which hash function exactly to use... how long are the strings.... Do you need faster add or write. Do you need merge or some other operation?

    Making all these variants makes no sense even with type parametrization.

    I would rather see a GlobalParameters or SharedPositions, i.e. specific structures that represent an concrete problem... not abstractions upon abstractions upon abstractions.





    Anyways the simplest way to start rewriting this for strings is to replace all the "map[int]int" with "map[string]int", adjust the methods to use a string as a key... and then incrementally start fixing the compilation errors.


    The harder one might be the striped version... there you must change k%stripes with some sort of hash(k)^stripes, where the k is now a string. Something like JS or SDBM hash should be good enough... of course depending on the strings the ideal hash will vary.


    + Egon
    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
  • Nick Craig-Wood at Dec 5, 2014 at 8:15 pm

    On 05/12/14 19:36, ChrisLu wrote:
    In https://golang.org/ref/mem , it talked about how to avoid
    double-checked locking to initialize one variable, by using once.Do.

    But how to efficiently and thread-safely initialize a multi-threaded map
    entries?

    For example: there is a highly concurrent map from a string to an object
    pointer. Each map entry can only be created on the first map lookup.

    map[string]*SomeObject m
    string x
    if _, ok := m[x]; !ok {
    m[x] = &SomeObject{}
    }

    If you know any clean way to do this, please let me know.
    Don't use any locking, use a channel where you send updates or read
    requests to a go routine which manages the map.

    This is the technique I used here

    https://github.com/ncw/go-nflog-acctd/blob/master/main.go#L183

    Or as it says on the tin: Don't communicate by sharing memory; share
    memory by communicating.

    --
    Nick Craig-Wood <nick@craig-wood.com> -- http://www.craig-wood.com/nick

    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
  • Brian Bulkowski at Dec 5, 2014 at 11:40 pm
    I'm really interested in the efficiency differences of these two
    approaches.

    They both work, and they both can/will be hidden behind an interface, so
    it's just an implementation question.

    I did a quick micro-benchmark on the mutex system, and .... I won't say
    anything until I look deeper at my code.

    I did find that RW mutexes are only slightly slower than regular mutexes. (
    straight mutexes maybe 20% faster ). Thus only use RW mutexes if you really
    think you're going to get a fair amount of contention. If it's a mostly
    uncontended mutex, then of course use a straight mutex.

    This is a standard C programmer thing, default to Mutex instead of RWMutex
    because (especially in Linux's mutexes) RW locks tend to be a lot slower -
    touch more state. In linux I think an RW is 2x to 3x slower in the
    uncontended case.... of course Go doesn't use Linux's mutexes because
    you're really communicating with the Go scheduler not the underlying thread
    scheduler.

    -brian
    On Friday, December 5, 2014 12:16:10 PM UTC-8, Nick Craig-Wood wrote:
    On 05/12/14 19:36, ChrisLu wrote:
    In https://golang.org/ref/mem , it talked about how to avoid
    double-checked locking to initialize one variable, by using once.Do.

    But how to efficiently and thread-safely initialize a multi-threaded map
    entries?

    For example: there is a highly concurrent map from a string to an object
    pointer. Each map entry can only be created on the first map lookup.

    map[string]*SomeObject m
    string x
    if _, ok := m[x]; !ok {
    m[x] = &SomeObject{}
    }

    If you know any clean way to do this, please let me know.
    Don't use any locking, use a channel where you send updates or read
    requests to a go routine which manages the map.

    This is the technique I used here

    https://github.com/ncw/go-nflog-acctd/blob/master/main.go#L183

    Or as it says on the tin: Don't communicate by sharing memory; share
    memory by communicating.

    --
    Nick Craig-Wood <ni...@craig-wood.com <javascript:>> --
    http://www.craig-wood.com/nick
    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
  • Dmitry Vyukov at Dec 6, 2014 at 11:14 am

    On Sat, Dec 6, 2014 at 2:40 AM, Brian Bulkowski wrote:
    I'm really interested in the efficiency differences of these two approaches.

    They both work, and they both can/will be hidden behind an interface, so
    it's just an implementation question.

    I did a quick micro-benchmark on the mutex system, and .... I won't say
    anything until I look deeper at my code.

    I did find that RW mutexes are only slightly slower than regular mutexes. (
    straight mutexes maybe 20% faster ). Thus only use RW mutexes if you really
    think you're going to get a fair amount of contention. If it's a mostly
    uncontended mutex, then of course use a straight mutex.

    This is a standard C programmer thing, default to Mutex instead of RWMutex
    because (especially in Linux's mutexes) RW locks tend to be a lot slower -
    touch more state. In linux I think an RW is 2x to 3x slower in the
    uncontended case.... of course Go doesn't use Linux's mutexes because you're
    really communicating with the Go scheduler not the underlying thread
    scheduler.

    You probably measured Mutex.Lock/Unlock vs RWMutex.Lock/Unlock. If you
    measure Mutex.Lock/Unlock vs RWMutex.RLock/RUnlock, then I would
    expect RWMutex to be faster in both uncontended and contended cases.
    So in Go the rule is: if you have write-mostly workload default to
    Mutex, if you have read-mostly workload default to RWMutex.

    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupgolang-nuts @
categoriesgo
postedDec 5, '14 at 7:36p
activeDec 7, '14 at 3:12p
posts20
users9
websitegolang.org

People

Translate

site design / logo © 2022 Grokbase