[go-nuts] idiomatic thread-safe way to initialize map entries

The Go builtin map type can't be used as a highly concurrent writable
map. If that is what you need, you will need to build your own data
structure with appropriate locking. (It is nontrivial to do this
safely and efficiently.)

The closest you can come with the builtin map type is to control all
map accesses with a sync.RWMutex, which permits concurrent reads and
serializes writes.

Ian

This highly depends on the context where you are working in
<https://code.google.com/p/go-wiki/wiki/HowToAsk>.

Here are a few different ways of implementing a concurrent map:
http://play.golang.org/p/IQ9IktS4Hq

+ Egon

could you write this to handle strings.

Yes, but it's simple enough that me doing has no benefit for me... :) And
has more benefit if you try to write it yourself.

Anyways the simplest way to start rewriting this for strings is to replace
all the "map[int]int" with "map[string]int", adjust the methods to use a
string as a key... and then incrementally start fixing the compilation
errors.

The harder one might be the striped version... there you must
change k%stripes with some sort of hash(k)^stripes, where the k is now a
string. Something like JS or SDBM hash
<http://www.cad.polito.it/~squillero/dida/Algoritmi_e_Programmazione_Avanzata/HashFunctions.c>
should be good enough... of course depending on the strings the ideal hash
will vary.

+ Egon

now could you write it using a custom *MyType

there is no benefit to anyone having to right the same data type over and
over and over and over again. Get where I am going with this? Type
Parameterization......

there is no benefit to anyone having to write the same data type over and
over and over and over again. Get where I am going with this? Type
Parameterization......

No. What you need is code generation if you don't want to do it manually.
Alternatively use an extra indirection by using an interface{}.
Type parametrization is a poor way of doing things if you intend to write
programs for multicore machines.

Explain the difference between a builtin preprocessor that handles type
parameterization and go generate. they both would handle to code the same
way- as a preprocessor. And please explain how using the two approaches
differ at compile time, making one a poor choice and the other spot on ?

  using the hacky tricks with interfaces doubles the size of the data type
in memory, with large data sets in the multi gigabyte ranges this become
unrealistic.

Not true. The interface points to a copy of the value. The original can be
garbage collected.
Besides, it's not a hack. That's close to Generics in C#.. You have the
choice to employ that method, or not. Your call.
Anyway, I think there is another thread on generics, we can discuss there
if you want.

Ok, please explain why one uses double the resources of the other. Maybe
I'm missing something.


package main

import "fmt"
import "runtime"

func main() {
n := 100000000
slice := make([]int, 0, n)
for i := 1; i <= n; i++ {
slice = append(slice, i)
}
var b runtime.MemStats
runtime.ReadMemStats(&b)
fmt.Println(b.Alloc/1000000)
fmt.Println(b.TotalAlloc/1000000)
fmt.Println(b.Sys/1000000)
fmt.Println(b.Lookups)
fmt.Println(b.Mallocs)
fmt.Println(b.Frees)
}


package main

import "fmt"
import "runtime"

func main() {
n := 100000000
slice := make([]interface{}, 0, n)
for i := 1; i <= n ; i++ {
slice = append(slice, i)
}
var b runtime.MemStats
runtime.ReadMemStats(&b)
fmt.Println(b.Alloc/1000000)
fmt.Println(b.TotalAlloc/1000000)
fmt.Println(b.Sys/1000000)
fmt.Println(b.Lookups)
fmt.Println(b.Mallocs)
fmt.Println(b.Frees)
}

I've told you it can be garbage collected. Never told you you would not
increase the number of allocations. To be fair, that's a cost that is unto
you if you want the genericity. Anyway, no need to pollute this thread
further.

Honestly, you are making no sense.

Maybe I just don't understand your intentions.
What's your issue ? That an interface is 2 words and when it points to
something such as an uint64, it takes twice more space, relative to what
you are storing ?

Then obviously don't use interfaces. I don't think you need me to tell you
that.
Use a datastructure that is customized for your type.
If you have several, just copy the structure and replace the types.

A tool is sufficient if you need to automatize the process, no need to have
type parameterization cooked into the language, making it heavy for
nothing.
That was a good way of doing things back in the day because the conditions
were not the same.

There are multiple parameters to these structures... the number of stripes, whether to use RWMutex or just Mutex, which hash function exactly to use... how long are the strings.... Do you need faster add or write. Do you need merge or some other operation?

Making all these variants makes no sense even with type parametrization.

I would rather see a GlobalParameters or SharedPositions, i.e. specific structures that represent an concrete problem... not abstractions upon abstractions upon abstractions.

Don't use any locking, use a channel where you send updates or read
requests to a go routine which manages the map.

This is the technique I used here

https://github.com/ncw/go-nflog-acctd/blob/master/main.go#L183

Or as it says on the tin: Don't communicate by sharing memory; share
memory by communicating.

I'm really interested in the efficiency differences of these two
approaches.

They both work, and they both can/will be hidden behind an interface, so
it's just an implementation question.

I did a quick micro-benchmark on the mutex system, and .... I won't say
anything until I look deeper at my code.

I did find that RW mutexes are only slightly slower than regular mutexes. (
straight mutexes maybe 20% faster ). Thus only use RW mutexes if you really
think you're going to get a fair amount of contention. If it's a mostly
uncontended mutex, then of course use a straight mutex.

This is a standard C programmer thing, default to Mutex instead of RWMutex
because (especially in Linux's mutexes) RW locks tend to be a lot slower -
touch more state. In linux I think an RW is 2x to 3x slower in the
uncontended case.... of course Go doesn't use Linux's mutexes because
you're really communicating with the Go scheduler not the underlying thread
scheduler.

-brian

You probably measured Mutex.Lock/Unlock vs RWMutex.Lock/Unlock. If you
measure Mutex.Lock/Unlock vs RWMutex.RLock/RUnlock, then I would
expect RWMutex to be faster in both uncontended and contended cases.
So in Go the rule is: if you have write-mostly workload default to
Mutex, if you have read-mostly workload default to RWMutex.