FAQ
It is advised to use read write locks to read and write from maps from
concurrent routines http://blog.golang.org/go-maps-in-action#Concurrency

Now if I create a map
aMap := make(map[int][]string)

Then create some X routines. Many routines write to the maps however each
routine ALWAYS adds new element to the map and would NEVER modify existing.
After that they will send keys over a channel.

There is one single routine that reads keys on this channel and accesses
map for the values corresponding to passed keys and once done delete the
key value entry from the map.

If none of the routines modify a value that other routines do, I still need
a lock? Also the delete is when we are sure no one would read the key's
value or modify it.


  map[int][]string<-------------------reads &
remove<---------------------------
                                                          ^
X
routines---------------------------------------------------------------[keys]---------------------->
ch-----------------> reader routine

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Search Discussions

  • Caleb Spare at Apr 29, 2014 at 6:46 pm
    If you have a read/write concurrent with a write, you need some kind
    of lock. No exceptions.
    On Tue, Apr 29, 2014 at 11:38 AM, Abhijit Kadam wrote:
    It is advised to use read write locks to read and write from maps from
    concurrent routines http://blog.golang.org/go-maps-in-action#Concurrency

    Now if I create a map
    aMap := make(map[int][]string)

    Then create some X routines. Many routines write to the maps however each
    routine ALWAYS adds new element to the map and would NEVER modify existing.
    After that they will send keys over a channel.

    There is one single routine that reads keys on this channel and accesses map
    for the values corresponding to passed keys and once done delete the key
    value entry from the map.

    If none of the routines modify a value that other routines do, I still need
    a lock? Also the delete is when we are sure no one would read the key's
    value or modify it.


    map[int][]string<-------------------reads &
    remove<---------------------------
    ^
    X
    routines---------------------------------------------------------------[keys]---------------------->
    ch-----------------> reader routine

    --
    You received this message because you are subscribed to the Google Groups
    "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an
    email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
  • Abhijit Kadam at Apr 29, 2014 at 7:14 pm

    On Wednesday, April 30, 2014 12:16:10 AM UTC+5:30, Caleb Spare wrote:
    If you have a read/write concurrent with a write, you need some kind
    of lock. No exceptions.
    What if I use slice instead?

    aSlice := make([][][]string, 10, 10)

    Many writer routines would always append data to a unique index in slice
    (indexes assigned to them before they start) and then pass the index to
    another routine. Other routine which is single reads the data. It will not
    delete it. It will just make the aSlice[passedindex] empty.

    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
  • Rui Ueyama at Apr 29, 2014 at 8:35 pm

    On Tue, Apr 29, 2014 at 12:14 PM, Abhijit Kadam wrote:
    On Wednesday, April 30, 2014 12:16:10 AM UTC+5:30, Caleb Spare wrote:

    If you have a read/write concurrent with a write, you need some kind
    of lock. No exceptions.
    What if I use slice instead?

    aSlice := make([][][]string, 10, 10)

    Many writer routines would always append data to a unique index in slice
    (indexes assigned to them before they start) and then pass the index to
    another routine. Other routine which is single reads the data. It will not
    delete it. It will just make the aSlice[passedindex] empty.
    That seems safe as long as you make a producer goroutine to send an index
    to a consumer goroutine via a channel, so that a write to the slice happens
    before a read by the consumer goroutine. http://golang.org/ref/mem

    That being said, I'd like to ask you why you want to share memory that way.
    It seems to me that there's no need to let the producer goroutines to store
    their results to the shared map or slice. Instead, they can create
    (non-shared) slices and send these slices to the consumer via a channel.
    And the consumer can write them to a map or a slice if it wants to do that.

    --
    You received this message because you are subscribed to the Google Groups
    "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an
    email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
  • Abhijit Kadam at Apr 30, 2014 at 2:22 am
    That is first thing I will do, pass everything by value. I am thinking of
    passing index to save memory, you see the actual data is [][]string string.

    Another reason is to maintain sequence of data to be written by consumer to
    file. This is also I am thinking of how to do.

    Also consumer is writing data to CSV file. I will apply ParallelFor with
    NUMCPU producer routines on original data. Then producers would pass
    processed data to a consumer that would write it to file. Do I need to use
    buffered channels?

    The processing is on 40 MB array. At some cases although it is on large
    array, a single processing is not that much and there is no map reduce kind
    of process where rows are reduced depending on conditions. At such cases I
    will have single routine do all the work as anyways the consumer writer
    routine would be slower as it is doing I/O. Is this correct?

    At other places where there is data transformation I will apply concurrency.

    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
  • Rui Ueyama at Apr 30, 2014 at 3:01 am

    On Tue, Apr 29, 2014 at 7:22 PM, Abhijit Kadam wrote:

    That is first thing I will do, pass everything by value. I am thinking of
    passing index to save memory, you see the actual data is [][]string string.
    Passing a slice by value is very cheap because slice is a reference to an
    array. Slice copy is a constant time operation, just like pointer copy. No
    need to worry about the cost.
    Another reason is to maintain sequence of data to be written by consumer
    to file. This is also I am thinking of how to do.

    Also consumer is writing data to CSV file. I will apply ParallelFor with
    NUMCPU producer routines on original data. Then producers would pass
    processed data to a consumer that would write it to file. Do I need to use
    buffered channels?

    The processing is on 40 MB array. At some cases although it is on large
    array, a single processing is not that much and there is no map reduce kind
    of process where rows are reduced depending on conditions. At such cases I
    will have single routine do all the work as anyways the consumer writer
    routine would be slower as it is doing I/O. Is this correct?

    At other places where there is data transformation I will apply
    concurrency.
    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
  • Abhijit Kadam at Apr 30, 2014 at 3:18 am
    So passing pointers over channel is safe? And it will ensure the data is
    always there for the consumer and not garbage collected ?

    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
  • Rui Ueyama at Apr 30, 2014 at 3:21 am

    On Tue, Apr 29, 2014 at 8:18 PM, Abhijit Kadam wrote:

    So passing pointers over channel is safe? And it will ensure the data is
    always there for the consumer and not garbage collected ?
    Yes.

    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
  • Abhijit Kadam at Apr 30, 2014 at 5:09 am
    It even works if you pass a stack variable and the function from which it
    is passed has returned. So the are blocking the stack or may be not
    allocating the variable on a stack.

    http://play.golang.org/p/L-EN0bDulb

    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
  • Carlos Castillo at Apr 30, 2014 at 11:42 am
    Before the compiler performed escape analysis, any value for which the
    pointer was taken is changed to be allocated on the heap, and is managed by
    the garbage collector. Under this system, it's impossible to actually point
    to a value on the stack.

    Once the compiler performed escape analysis it could determine several
    cases where it was safe for a value to stay on the stack, like when you
    pass a pointer to a value as a parameter. However, passing a pointer
    through a channel always causes the reference to "escape", so the value it
    points to will be allocated on the heap.

    Under both systems it's safe to pass a pointer to a "stack" variable
    through a channel, because no matter which logic you took to get there, the
    compiler will not put it on the stack in the first place.
    On Tuesday, April 29, 2014 10:09:43 PM UTC-7, Abhijit Kadam wrote:


    It even works if you pass a stack variable and the function from which it
    is passed has returned. So the are blocking the stack or may be not
    allocating the variable on a stack.

    http://play.golang.org/p/L-EN0bDulb
    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupgolang-nuts @
categoriesgo
postedApr 29, '14 at 6:39p
activeApr 30, '14 at 11:42a
posts10
users4
websitegolang.org

People

Translate

site design / logo © 2022 Grokbase