FAQ

Search Discussions

  • Anonymous at Sep 7, 2012 at 8:46 pm
  • Egon at Sep 8, 2012 at 6:09 am
    The args are shared. And indirect sharing doesn't bother me since the
    oldThings won't be modified in any way.

    And I found the minimal test case: http://play.golang.org/p/nycttdv3r7 .
    I'm really surprised to see that it doesn't work. I couldn't find any
    explicit documentation regarding this behavior.
    On Friday, September 7, 2012 11:46:25 PM UTC+3, Jan Mercl wrote:

    On Fri, Sep 7, 2012 at 10:36 PM, egon <egon...@gmail.com <javascript:>>
    wrote:
    The doStep function is called concurrently ...
    IMO fine as long as the args to doStep are not shared by the
    concurrent calls in any way (there is also indirect sharing possible
    because of the backing []Thing storage).

    -j
  • Jesse McNelis at Sep 8, 2012 at 6:31 am

    On Sat, Sep 8, 2012 at 4:09 PM, egon wrote:
    The args are shared. And indirect sharing doesn't bother me since the
    oldThings won't be modified in any way.
    If you carefully read the definition for append() you'll see why your
    code doesn't work as you expect.
    http://golang.org/ref/spec#Appending_and_copying_slices
    "If the capacity of s is not large enough to fit the additional
    values, append allocates a new, sufficiently large slice that fits
    both the existing slice elements and the additional values. Thus, the
    returned slice may refer to a different underlying array."

    And I found the minimal test case: http://play.golang.org/p/nycttdv3r7 .
    I'm really surprised to see that it doesn't work. I couldn't find any
    explicit documentation regarding this behavior.
    In this example code you're sharing a slice among 4 goroutines without
    synchronisation so the result you'll get is undefined.
    You can read the memory model for why, http://golang.org/ref/mem




    --
    =====================
    http://jessta.id.au
  • DisposaBoy at Sep 8, 2012 at 6:38 am

    On Saturday, September 8, 2012 7:09:16 AM UTC+1, egon wrote:
    The args are shared. And indirect sharing doesn't bother me since the
    oldThings won't be modified in any way.

    And I found the minimal test case: http://play.golang.org/p/nycttdv3r7 .
    I'm really surprised to see that it doesn't work. I couldn't find any
    explicit documentation regarding this behavior.
    Perhaps it would help if you explained the algorithm you're trying to
    implement and what the expected output it. Reason: you mention that
    oldThings isn't modified but in version the array behind oldThings is
    almost certainly being modified concurrently while in version 2, this
    doesn't happen because you're working on a copy of the data.
  • Jared Anderson at Sep 8, 2012 at 7:14 am
    From http://golang.org/ref/spec#Slice_types :
    A slice, once initialized, is always associated with an underlying array
    that holds its elements. A slice therefore shares storage with its array
    and with other slices of the same array; by contrast, distinct arrays
    always represent distinct storage.

    The array underlying a slice may extend past the end of the slice. The
    capacity is a measure of that extent: it is the sum of the length of the
    slice and the length of the array beyond the slice; a slice of length up to
    that capacity can be created by `slicing' a new one from the original slice
    (§Slices). The capacity of a slice a can be discovered using the built-in
    function cap(a).

    From http://golang.org/ref/spec#Appending_and_copying_slices :
    If the capacity of s is not large enough to fit the additional values,
    append allocates a new, sufficiently large slice that fits both the
    existing slice elements and the additional values. Thus, the returned slice
    may refer to a different underlying array.

    In your test case some times append is writing to the same underlying array
    and other times it is creating a new underlying array for each append. I'm
    guessing you always want to create a new underlying array as you never said
    what was wrong.
    On Fri, Sep 7, 2012 at 10:09 PM, egon wrote:

    The args are shared. And indirect sharing doesn't bother me since the
    oldThings won't be modified in any way.

    And I found the minimal test case: http://play.golang.org/p/nycttdv3r7 .
    I'm really surprised to see that it doesn't work. I couldn't find any
    explicit documentation regarding this behavior.
  • Egon at Sep 8, 2012 at 10:15 pm

    On Saturday, September 8, 2012 10:14:56 AM UTC+3, Jared Anderson wrote:
    In your test case some times append is writing to the same underlying
    array and other times it is creating a new underlying array for each
    append. I'm guessing you always want to create a new underlying array as
    you never said what was wrong.
    I'm fine with the structural sharing - as many functional languages do. I
    was just very surprised to see that it wasn't thread-safe.


    >
  • Jared Anderson at Sep 9, 2012 at 8:13 am
    You can get the same results with no goroutines.
    http://play.golang.org/p/OPacGWV8NY
    On Sat, Sep 8, 2012 at 2:15 PM, egon wrote:
    On Saturday, September 8, 2012 10:14:56 AM UTC+3, Jared Anderson wrote:

    In your test case some times append is writing to the same underlying
    array and other times it is creating a new underlying array for each
    append. I'm guessing you always want to create a new underlying array as
    you never said what was wrong.
    I'm fine with the structural sharing - as many functional languages do. I
    was just very surprised to see that it wasn't thread-safe.
  • Kyle Lemons at Sep 8, 2012 at 6:37 am

    On Fri, Sep 7, 2012 at 1:36 PM, egon wrote:

    I ran into a problem with append, in it's simplified form it is such
    (I thought I would ask group before running off to issue tracker)

    Here are two versions of a function that behave differently:

    // version 1
    func doStep(oldThings []Thing, thing Thing) []Thing {
    newThings := append(oldThings, thing)
    return newThings
    }

    // version 2
    func doStep(oldThings []Thing, thing Thing) []Thing {
    newThings = make([]Thing, len(oldThings), len(oldThings) + 1)
    copy(newThings, oldThings)
    newThings = append(newThings, thing)
    return newThings
    }

    The doStep function is called concurrently and some of the newThings are
    collected and shown. "oldThings" isn't modified but it is accessed from
    multiple processes.
    Version 2 does what it is supposed to do, with version 1 something goes
    wrong.

    Unfortunately I haven't found a minimal test case yet.
    The actual program is a lot longer and can be found at
    github.com/egonelbre/spexs/ just run "./kroko.sh" to test it. The
    offending lines are in "src/spexs/query.go"@26-29<http://github.com/egonelbre/spexs/blob/master/src/spexs/query.go>.
    You should be able to see that these two give different final results.

    Any ideas?
    Append will often allocate more than the space required for a single
    element. Consider the following:
    http://play.golang.org/p/_P8rxcT5fo

    Notice that the backing memory doesn't move after every append, so you
    could end up with more than one process sharing the same backing memory if
    you try to concurrently append to the same slice.
  • ⚛ at Sep 8, 2012 at 7:23 am
    Simplified version of your code:

    type RegToken struct {}
    type Query struct {
    Pat []RegToken
    }

    func NewQuery(parent *Query, token RegToken) *Query {
    q := &Query{}
    q.Pat = append(parent.Pat, token)
    return q
    }

    Consider the following:

    1. len(parent.Pat) is 10, cap(parent.Pat) is 20
    2. q1:=NewQuery(parent,x) does q1.Pat[10]=x, append does not allocate
    memory
    3. q2:=NewQuery(parent,y) does q2.Pat[10]=y, append does not allocate
    memory

    At this point (&q1.Pat[0] == &q2.Pat[0]), that is q1.Pat and q2.Pat are
    sharing the same memory. This is because the capacity of 20 elements was
    sufficient to store an element at index 10 without requiring any memory
    allocation.

    In step 2 and 3, the append:

    q.Pat = append(parent.Pat, token)

    is informally equivalent to:

    q.Pat.len = 11
    q.Pat[10] = token

    Because there is no additional memory allocation here, we have written two
    different values (x and y) to the same memory location at address
    &q.Pat[10].

    The problem you are encountering is *not* caused by multiple goroutines
    running concurrently.
    On Friday, September 7, 2012 10:36:58 PM UTC+2, egon wrote:

    I ran into a problem with append, in it's simplified form it is such
    (I thought I would ask group before running off to issue tracker)

    Here are two versions of a function that behave differently:

    // version 1
    func doStep(oldThings []Thing, thing Thing) []Thing {
    newThings := append(oldThings, thing)
    return newThings
    }

    // version 2
    func doStep(oldThings []Thing, thing Thing) []Thing {
    newThings = make([]Thing, len(oldThings), len(oldThings) + 1)
    copy(newThings, oldThings)
    newThings = append(newThings, thing)
    return newThings
    }

    The doStep function is called concurrently and some of the newThings are
    collected and shown. "oldThings" isn't modified but it is accessed from
    multiple processes.
    Version 2 does what it is supposed to do, with version 1 something goes
    wrong.

    Unfortunately I haven't found a minimal test case yet.
    The actual program is a lot longer and can be found at
    github.com/egonelbre/spexs/ just run "./kroko.sh" to test it. The
    offending lines are in "src/spexs/query.go"@26-29<http://github.com/egonelbre/spexs/blob/master/src/spexs/query.go>.
    You should be able to see that these two give different final results.

    Any ideas?

    + egon

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupgolang-nuts @
categoriesgo
postedSep 7, '12 at 8:37p
activeSep 9, '12 at 8:13a
posts10
users7
websitegolang.org

People

Translate

site design / logo © 2022 Grokbase