FAQ
Recently Dyukov kindly submitted 85e7bee; it updates the GC to ignore maps
which don't contain pointers.

At about the same time I started have weird corruption issues with maps. I
finally tracked them down to a fairly simple case in my code. I can't
replicate it yet in a sandbox or highly simplified format, but I wanted to
get this out soon, so that people know they should probably revert 85e7 if
they're having weird troubles. And of course, hopefully someone smarter
than me will see the problem immediately.

In brief, I create maps as follows:

fixedmap[keystruct][5]valstruct
varmap[keystruct][]valstruct

And proceed to dump between 1 and 50 million entries in them, using a
method like:

a := [5]struct{}
a[0] = somedata
...

fixedmap[key] = a


After each forced GC, or call to runtime.GC(), at least the fixed map is
corrupted. It may return data from another entry, or just what looks like
random bytes.

I spent this morning using git bisect and it it turned up this particular
commit.

I also checked out tip and reverted only this commit; that seems to work
fine.

I am running with two small patches. Tip with these patches works fine, so
I believe it's something in Dyukov's original patch.

encoding/gob/decoder.go:15
-const tooBig = 1 << 30
+const tooBig = 1 << 34

and
--- a/src/runtime/malloc2.go
+++ b/src/runtime/malloc2.go
@@ -136,7 +136,7 @@ const (
         // See http://golang.org/issue/5402 and
http://golang.org/issue/5236.
         // On other 64-bit platforms, we limit the arena to 128GB, or 37
bits.
         // On 32-bit, we don't bother limiting anything, so we use the full
32-bit address.
- _MHeapMap_TotalBits = (_64bit*goos_windows)*35 +
(_64bit*(1-goos_windows))*37 + (1-_64bit)*32
+ _MHeapMap_TotalBits = (_64bit*goos_windows)*35 +
(_64bit*(1-goos_windows))*38 + (1-_64bit)*32

@@ -145,7 +145,7 @@ const (
         // 2, 3, and 4 are all plausible maximums depending
         // on the hardware details of the machine. The garbage
         // collector scales well to 32 cpus.
- _MaxGcproc = 32
+ _MaxGcproc = 64

--
You received this message because you are subscribed to the Google Groups "golang-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-dev+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Search Discussions

  • Russ Cox at Feb 12, 2015 at 4:20 am
    +dvyukov, khr


    On Wed, Feb 11, 2015 at 5:50 PM, wrote:

    Recently Dyukov kindly submitted 85e7bee; it updates the GC to ignore maps
    which don't contain pointers.

    At about the same time I started have weird corruption issues with maps. I
    finally tracked them down to a fairly simple case in my code. I can't
    replicate it yet in a sandbox or highly simplified format, but I wanted to
    get this out soon, so that people know they should probably revert 85e7 if
    they're having weird troubles. And of course, hopefully someone smarter
    than me will see the problem immediately.

    In brief, I create maps as follows:

    fixedmap[keystruct][5]valstruct
    varmap[keystruct][]valstruct

    And proceed to dump between 1 and 50 million entries in them, using a
    method like:

    a := [5]struct{}
    a[0] = somedata
    ...

    fixedmap[key] = a


    After each forced GC, or call to runtime.GC(), at least the fixed map is
    corrupted. It may return data from another entry, or just what looks like
    random bytes.

    I spent this morning using git bisect and it it turned up this particular
    commit.

    I also checked out tip and reverted only this commit; that seems to work
    fine.

    I am running with two small patches. Tip with these patches works fine, so
    I believe it's something in Dyukov's original patch.

    encoding/gob/decoder.go:15
    -const tooBig = 1 << 30
    +const tooBig = 1 << 34

    and
    --- a/src/runtime/malloc2.go
    +++ b/src/runtime/malloc2.go
    @@ -136,7 +136,7 @@ const (
    // See http://golang.org/issue/5402 and
    http://golang.org/issue/5236.
    // On other 64-bit platforms, we limit the arena to 128GB, or 37
    bits.
    // On 32-bit, we don't bother limiting anything, so we use the
    full 32-bit address.
    - _MHeapMap_TotalBits = (_64bit*goos_windows)*35 +
    (_64bit*(1-goos_windows))*37 + (1-_64bit)*32
    + _MHeapMap_TotalBits = (_64bit*goos_windows)*35 +
    (_64bit*(1-goos_windows))*38 + (1-_64bit)*32

    @@ -145,7 +145,7 @@ const (
    // 2, 3, and 4 are all plausible maximums depending
    // on the hardware details of the machine. The garbage
    // collector scales well to 32 cpus.
    - _MaxGcproc = 32
    + _MaxGcproc = 64

    --
    You received this message because you are subscribed to the Google Groups
    "golang-dev" group.
    To unsubscribe from this group and stop receiving emails from it, send an
    email to golang-dev+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
    --
    You received this message because you are subscribed to the Google Groups "golang-dev" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-dev+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
  • Dmitry Vyukov at Feb 12, 2015 at 8:02 am

    On Thu, Feb 12, 2015 at 1:50 AM, wrote:
    Recently Dyukov kindly submitted 85e7bee; it updates the GC to ignore maps
    which don't contain pointers.

    At about the same time I started have weird corruption issues with maps. I
    finally tracked them down to a fairly simple case in my code. I can't
    replicate it yet in a sandbox or highly simplified format, but I wanted to
    get this out soon, so that people know they should probably revert 85e7 if
    they're having weird troubles. And of course, hopefully someone smarter than
    me will see the problem immediately.

    In brief, I create maps as follows:

    fixedmap[keystruct][5]valstruct
    varmap[keystruct][]valstruct

    And proceed to dump between 1 and 50 million entries in them, using a method
    like:

    a := [5]struct{}
    a[0] = somedata
    ...

    fixedmap[key] = a


    After each forced GC, or call to runtime.GC(), at least the fixed map is
    corrupted. It may return data from another entry, or just what looks like
    random bytes.

    I spent this morning using git bisect and it it turned up this particular
    commit.

    I also checked out tip and reverted only this commit; that seems to work
    fine.

    I am running with two small patches. Tip with these patches works fine, so I
    believe it's something in Dyukov's original patch.

    encoding/gob/decoder.go:15
    -const tooBig = 1 << 30
    +const tooBig = 1 << 34

    and
    --- a/src/runtime/malloc2.go
    +++ b/src/runtime/malloc2.go
    @@ -136,7 +136,7 @@ const (
    // See http://golang.org/issue/5402 and
    http://golang.org/issue/5236.
    // On other 64-bit platforms, we limit the arena to 128GB, or 37
    bits.
    // On 32-bit, we don't bother limiting anything, so we use the full
    32-bit address.
    - _MHeapMap_TotalBits = (_64bit*goos_windows)*35 +
    (_64bit*(1-goos_windows))*37 + (1-_64bit)*32
    + _MHeapMap_TotalBits = (_64bit*goos_windows)*35 +
    (_64bit*(1-goos_windows))*38 + (1-_64bit)*32

    @@ -145,7 +145,7 @@ const (
    // 2, 3, and 4 are all plausible maximums depending
    // on the hardware details of the machine. The garbage
    // collector scales well to 32 cpus.
    - _MaxGcproc = 32
    + _MaxGcproc = 64

    Can you please provide the exact reproducer program?
    I am running the following test and it does not fail:

    package main

    import (
    "runtime"
    "fmt"
    "os"
    )

    func main() {
    m := make(map[int][2]int)
    for i := 0; i < 1e6; i++ {
    m[i] = [2]int{i, i+1}
    if (i%111) == 0 {
    runtime.GC()
    if len(m) != i+1 {
    fmt.Printf("bad len: %v/%v\n", len(m), i+1)
    os.Exit(1)
    }
    for j := 0; j <= i; j++ {
    v := m[j]
    if v[0] != j || v[1] != j+1 {
    fmt.Printf("bad value %v: %v/%v\n", j, v[0], v[1])
    os.Exit(1)
    }
    }
    }
    }
    }

    --
    You received this message because you are subscribed to the Google Groups "golang-dev" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-dev+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
  • Dmitry Vyukov at Feb 12, 2015 at 9:10 am
    Is there any chance you have a data race on maps? I can imagine that
    failure modes change after this change. Try to run the program with -race
    flag.

    On Thu, Feb 12, 2015 at 11:01 AM, Dmitry Vyukov wrote:
    On Thu, Feb 12, 2015 at 1:50 AM, wrote:
    Recently Dyukov kindly submitted 85e7bee; it updates the GC to ignore maps
    which don't contain pointers.

    At about the same time I started have weird corruption issues with maps. I
    finally tracked them down to a fairly simple case in my code. I can't
    replicate it yet in a sandbox or highly simplified format, but I wanted to
    get this out soon, so that people know they should probably revert 85e7 if
    they're having weird troubles. And of course, hopefully someone smarter than
    me will see the problem immediately.

    In brief, I create maps as follows:

    fixedmap[keystruct][5]valstruct
    varmap[keystruct][]valstruct

    And proceed to dump between 1 and 50 million entries in them, using a method
    like:

    a := [5]struct{}
    a[0] = somedata
    ...

    fixedmap[key] = a


    After each forced GC, or call to runtime.GC(), at least the fixed map is
    corrupted. It may return data from another entry, or just what looks like
    random bytes.

    I spent this morning using git bisect and it it turned up this particular
    commit.

    I also checked out tip and reverted only this commit; that seems to work
    fine.

    I am running with two small patches. Tip with these patches works fine, so I
    believe it's something in Dyukov's original patch.

    encoding/gob/decoder.go:15
    -const tooBig = 1 << 30
    +const tooBig = 1 << 34

    and
    --- a/src/runtime/malloc2.go
    +++ b/src/runtime/malloc2.go
    @@ -136,7 +136,7 @@ const (
    // See http://golang.org/issue/5402 and
    http://golang.org/issue/5236.
    // On other 64-bit platforms, we limit the arena to 128GB, or 37
    bits.
    // On 32-bit, we don't bother limiting anything, so we use the full
    32-bit address.
    - _MHeapMap_TotalBits = (_64bit*goos_windows)*35 +
    (_64bit*(1-goos_windows))*37 + (1-_64bit)*32
    + _MHeapMap_TotalBits = (_64bit*goos_windows)*35 +
    (_64bit*(1-goos_windows))*38 + (1-_64bit)*32

    @@ -145,7 +145,7 @@ const (
    // 2, 3, and 4 are all plausible maximums depending
    // on the hardware details of the machine. The garbage
    // collector scales well to 32 cpus.
    - _MaxGcproc = 32
    + _MaxGcproc = 64

    Can you please provide the exact reproducer program?
    I am running the following test and it does not fail:

    package main

    import (
    "runtime"
    "fmt"
    "os"
    )

    func main() {
    m := make(map[int][2]int)
    for i := 0; i < 1e6; i++ {
    m[i] = [2]int{i, i+1}
    if (i%111) == 0 {
    runtime.GC()
    if len(m) != i+1 {
    fmt.Printf("bad len: %v/%v\n", len(m), i+1)
    os.Exit(1)
    }
    for j := 0; j <= i; j++ {
    v := m[j]
    if v[0] != j || v[1] != j+1 {
    fmt.Printf("bad value %v: %v/%v\n", j, v[0], v[1])
    os.Exit(1)
    }
    }
    }
    }
    }
    --
    You received this message because you are subscribed to the Google Groups "golang-dev" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-dev+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
  • Peter Vessenes at Feb 12, 2015 at 9:54 pm
    Inre: data races, I don't believe so. When I thought it was a problem with
    my code, I ran a number of trials with the race detector on. Further
    evidence against a race: the loop that sees the data corruption is running
    single threaded when GC triggers. I didn't a/b test with the race detector
    on and git bisect though, but I will assess a known good and bad commit and
    triple check for you.

    Inre: Trial code; I'm working on it; I couldn't get it to trigger with
    simple maps either. I think it has to do with using structs for key and/or
    value.

    More in a bit.
    On Thursday, February 12, 2015 at 1:10:25 AM UTC-8, Dmitry Vyukov wrote:

    Is there any chance you have a data race on maps? I can imagine that
    failure modes change after this change. Try to run the program with -race
    flag.


    On Thu, Feb 12, 2015 at 11:01 AM, Dmitry Vyukov <dvy...@google.com
    <javascript:>> wrote:
    On Thu, Feb 12, 2015 at 1:50 AM, <pe...@coinlab.com <javascript:>>
    wrote:
    Recently Dyukov kindly submitted 85e7bee; it updates the GC to ignore maps
    which don't contain pointers.

    At about the same time I started have weird corruption issues with maps. I
    finally tracked them down to a fairly simple case in my code. I can't
    replicate it yet in a sandbox or highly simplified format, but I wanted to
    get this out soon, so that people know they should probably revert 85e7 if
    they're having weird troubles. And of course, hopefully someone smarter than
    me will see the problem immediately.

    In brief, I create maps as follows:

    fixedmap[keystruct][5]valstruct
    varmap[keystruct][]valstruct

    And proceed to dump between 1 and 50 million entries in them, using a method
    like:

    a := [5]struct{}
    a[0] = somedata
    ...

    fixedmap[key] = a


    After each forced GC, or call to runtime.GC(), at least the fixed map is
    corrupted. It may return data from another entry, or just what looks like
    random bytes.

    I spent this morning using git bisect and it it turned up this
    particular
    commit.

    I also checked out tip and reverted only this commit; that seems to work
    fine.

    I am running with two small patches. Tip with these patches works fine, so I
    believe it's something in Dyukov's original patch.

    encoding/gob/decoder.go:15
    -const tooBig = 1 << 30
    +const tooBig = 1 << 34

    and
    --- a/src/runtime/malloc2.go
    +++ b/src/runtime/malloc2.go
    @@ -136,7 +136,7 @@ const (
    // See http://golang.org/issue/5402 and
    http://golang.org/issue/5236.
    // On other 64-bit platforms, we limit the arena to 128GB, or 37
    bits.
    // On 32-bit, we don't bother limiting anything, so we use the full
    32-bit address.
    - _MHeapMap_TotalBits = (_64bit*goos_windows)*35 +
    (_64bit*(1-goos_windows))*37 + (1-_64bit)*32
    + _MHeapMap_TotalBits = (_64bit*goos_windows)*35 +
    (_64bit*(1-goos_windows))*38 + (1-_64bit)*32

    @@ -145,7 +145,7 @@ const (
    // 2, 3, and 4 are all plausible maximums depending
    // on the hardware details of the machine. The garbage
    // collector scales well to 32 cpus.
    - _MaxGcproc = 32
    + _MaxGcproc = 64

    Can you please provide the exact reproducer program?
    I am running the following test and it does not fail:

    package main

    import (
    "runtime"
    "fmt"
    "os"
    )

    func main() {
    m := make(map[int][2]int)
    for i := 0; i < 1e6; i++ {
    m[i] = [2]int{i, i+1}
    if (i%111) == 0 {
    runtime.GC()
    if len(m) != i+1 {
    fmt.Printf("bad len: %v/%v\n", len(m), i+1)
    os.Exit(1)
    }
    for j := 0; j <= i; j++ {
    v := m[j]
    if v[0] != j || v[1] != j+1 {
    fmt.Printf("bad value %v: %v/%v\n", j, v[0], v[1])
    os.Exit(1)
    }
    }
    }
    }
    }
    --
    You received this message because you are subscribed to the Google Groups "golang-dev" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-dev+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
  • Robert Griesemer at Feb 12, 2015 at 10:26 pm
    FYI, a recent new failure in tools/go/gccgoimporter may (or may not) be
    related to this. I just disabled a test to get the build going again (
    https://go-review.googlesource.com/#/c/4750/ ).

    The failure showed up on the dashboard at the same time.

    - gri
    On Thu, Feb 12, 2015 at 1:10 AM, 'Dmitry Vyukov' via golang-dev wrote:

    Is there any chance you have a data race on maps? I can imagine that
    failure modes change after this change. Try to run the program with -race
    flag.

    On Thu, Feb 12, 2015 at 11:01 AM, Dmitry Vyukov wrote:
    On Thu, Feb 12, 2015 at 1:50 AM, wrote:
    Recently Dyukov kindly submitted 85e7bee; it updates the GC to ignore maps
    which don't contain pointers.

    At about the same time I started have weird corruption issues with maps. I
    finally tracked them down to a fairly simple case in my code. I can't
    replicate it yet in a sandbox or highly simplified format, but I wanted to
    get this out soon, so that people know they should probably revert 85e7 if
    they're having weird troubles. And of course, hopefully someone smarter than
    me will see the problem immediately.

    In brief, I create maps as follows:

    fixedmap[keystruct][5]valstruct
    varmap[keystruct][]valstruct

    And proceed to dump between 1 and 50 million entries in them, using a method
    like:

    a := [5]struct{}
    a[0] = somedata
    ...

    fixedmap[key] = a


    After each forced GC, or call to runtime.GC(), at least the fixed map is
    corrupted. It may return data from another entry, or just what looks like
    random bytes.

    I spent this morning using git bisect and it it turned up this
    particular
    commit.

    I also checked out tip and reverted only this commit; that seems to work
    fine.

    I am running with two small patches. Tip with these patches works fine, so I
    believe it's something in Dyukov's original patch.

    encoding/gob/decoder.go:15
    -const tooBig = 1 << 30
    +const tooBig = 1 << 34

    and
    --- a/src/runtime/malloc2.go
    +++ b/src/runtime/malloc2.go
    @@ -136,7 +136,7 @@ const (
    // See http://golang.org/issue/5402 and
    http://golang.org/issue/5236.
    // On other 64-bit platforms, we limit the arena to 128GB, or 37
    bits.
    // On 32-bit, we don't bother limiting anything, so we use the full
    32-bit address.
    - _MHeapMap_TotalBits = (_64bit*goos_windows)*35 +
    (_64bit*(1-goos_windows))*37 + (1-_64bit)*32
    + _MHeapMap_TotalBits = (_64bit*goos_windows)*35 +
    (_64bit*(1-goos_windows))*38 + (1-_64bit)*32

    @@ -145,7 +145,7 @@ const (
    // 2, 3, and 4 are all plausible maximums depending
    // on the hardware details of the machine. The garbage
    // collector scales well to 32 cpus.
    - _MaxGcproc = 32
    + _MaxGcproc = 64

    Can you please provide the exact reproducer program?
    I am running the following test and it does not fail:

    package main

    import (
    "runtime"
    "fmt"
    "os"
    )

    func main() {
    m := make(map[int][2]int)
    for i := 0; i < 1e6; i++ {
    m[i] = [2]int{i, i+1}
    if (i%111) == 0 {
    runtime.GC()
    if len(m) != i+1 {
    fmt.Printf("bad len: %v/%v\n", len(m), i+1)
    os.Exit(1)
    }
    for j := 0; j <= i; j++ {
    v := m[j]
    if v[0] != j || v[1] != j+1 {
    fmt.Printf("bad value %v: %v/%v\n", j, v[0], v[1])
    os.Exit(1)
    }
    }
    }
    }
    }
    --
    You received this message because you are subscribed to the Google Groups
    "golang-dev" group.
    To unsubscribe from this group and stop receiving emails from it, send an
    email to golang-dev+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
    --
    You received this message because you are subscribed to the Google Groups "golang-dev" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-dev+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
  • Peter Vessenes at Feb 12, 2015 at 10:28 pm
    Dmitri: If it's a race, it's not detected by the race detector.

    I'm still working on replicating the behaviour outside my code.
    On Thursday, February 12, 2015 at 2:26:09 PM UTC-8, gri wrote:

    FYI, a recent new failure in tools/go/gccgoimporter may (or may not) be
    related to this. I just disabled a test to get the build going again (
    https://go-review.googlesource.com/#/c/4750/ ).

    The failure showed up on the dashboard at the same time.

    - gri

    On Thu, Feb 12, 2015 at 1:10 AM, 'Dmitry Vyukov' via golang-dev <
    golan...@googlegroups.com <javascript:>> wrote:
    Is there any chance you have a data race on maps? I can imagine that
    failure modes change after this change. Try to run the program with -race
    flag.


    On Thu, Feb 12, 2015 at 11:01 AM, Dmitry Vyukov <dvy...@google.com
    <javascript:>> wrote:
    On Thu, Feb 12, 2015 at 1:50 AM, <pe...@coinlab.com <javascript:>>
    wrote:
    Recently Dyukov kindly submitted 85e7bee; it updates the GC to ignore maps
    which don't contain pointers.

    At about the same time I started have weird corruption issues with maps. I
    finally tracked them down to a fairly simple case in my code. I can't
    replicate it yet in a sandbox or highly simplified format, but I wanted to
    get this out soon, so that people know they should probably revert 85e7 if
    they're having weird troubles. And of course, hopefully someone
    smarter than
    me will see the problem immediately.

    In brief, I create maps as follows:

    fixedmap[keystruct][5]valstruct
    varmap[keystruct][]valstruct

    And proceed to dump between 1 and 50 million entries in them, using a method
    like:

    a := [5]struct{}
    a[0] = somedata
    ...

    fixedmap[key] = a


    After each forced GC, or call to runtime.GC(), at least the fixed map is
    corrupted. It may return data from another entry, or just what looks like
    random bytes.

    I spent this morning using git bisect and it it turned up this
    particular
    commit.

    I also checked out tip and reverted only this commit; that seems to work
    fine.

    I am running with two small patches. Tip with these patches works
    fine, so I
    believe it's something in Dyukov's original patch.

    encoding/gob/decoder.go:15
    -const tooBig = 1 << 30
    +const tooBig = 1 << 34

    and
    --- a/src/runtime/malloc2.go
    +++ b/src/runtime/malloc2.go
    @@ -136,7 +136,7 @@ const (
    // See http://golang.org/issue/5402 and
    http://golang.org/issue/5236.
    // On other 64-bit platforms, we limit the arena to 128GB, or 37
    bits.
    // On 32-bit, we don't bother limiting anything, so we use the full
    32-bit address.
    - _MHeapMap_TotalBits = (_64bit*goos_windows)*35 +
    (_64bit*(1-goos_windows))*37 + (1-_64bit)*32
    + _MHeapMap_TotalBits = (_64bit*goos_windows)*35 +
    (_64bit*(1-goos_windows))*38 + (1-_64bit)*32

    @@ -145,7 +145,7 @@ const (
    // 2, 3, and 4 are all plausible maximums depending
    // on the hardware details of the machine. The garbage
    // collector scales well to 32 cpus.
    - _MaxGcproc = 32
    + _MaxGcproc = 64

    Can you please provide the exact reproducer program?
    I am running the following test and it does not fail:

    package main

    import (
    "runtime"
    "fmt"
    "os"
    )

    func main() {
    m := make(map[int][2]int)
    for i := 0; i < 1e6; i++ {
    m[i] = [2]int{i, i+1}
    if (i%111) == 0 {
    runtime.GC()
    if len(m) != i+1 {
    fmt.Printf("bad len: %v/%v\n", len(m), i+1)
    os.Exit(1)
    }
    for j := 0; j <= i; j++ {
    v := m[j]
    if v[0] != j || v[1] != j+1 {
    fmt.Printf("bad value %v: %v/%v\n", j, v[0], v[1])
    os.Exit(1)
    }
    }
    }
    }
    }
    --
    You received this message because you are subscribed to the Google Groups
    "golang-dev" group.
    To unsubscribe from this group and stop receiving emails from it, send an
    email to golang-dev+...@googlegroups.com <javascript:>.
    For more options, visit https://groups.google.com/d/optout.
    --
    You received this message because you are subscribed to the Google Groups "golang-dev" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-dev+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
  • Dmitry Vyukov at Feb 13, 2015 at 7:02 am

    On Fri, Feb 13, 2015 at 1:28 AM, Peter Vessenes wrote:

    Dmitri: If it's a race, it's not detected by the race detector.

    I'm still working on replicating the behaviour outside my code.

    Please file an issue at http://golang.org/issue/new when you replicate the
    behavior.


    On Thursday, February 12, 2015 at 2:26:09 PM UTC-8, gri wrote:

    FYI, a recent new failure in tools/go/gccgoimporter may (or may not) be
    related to this. I just disabled a test to get the build going again (
    https://go-review.googlesource.com/#/c/4750/ ).

    The failure showed up on the dashboard at the same time.

    - gri

    On Thu, Feb 12, 2015 at 1:10 AM, 'Dmitry Vyukov' via golang-dev <
    golan...@googlegroups.com> wrote:
    Is there any chance you have a data race on maps? I can imagine that
    failure modes change after this change. Try to run the program with -race
    flag.


    On Thu, Feb 12, 2015 at 11:01 AM, Dmitry Vyukov <dvy...@google.com>
    wrote:
    On Thu, Feb 12, 2015 at 1:50 AM, wrote:
    Recently Dyukov kindly submitted 85e7bee; it updates the GC to ignore maps
    which don't contain pointers.

    At about the same time I started have weird corruption issues with maps. I
    finally tracked them down to a fairly simple case in my code. I can't
    replicate it yet in a sandbox or highly simplified format, but I wanted to
    get this out soon, so that people know they should probably revert 85e7 if
    they're having weird troubles. And of course, hopefully someone
    smarter than
    me will see the problem immediately.

    In brief, I create maps as follows:

    fixedmap[keystruct][5]valstruct
    varmap[keystruct][]valstruct

    And proceed to dump between 1 and 50 million entries in them, using a method
    like:

    a := [5]struct{}
    a[0] = somedata
    ...

    fixedmap[key] = a


    After each forced GC, or call to runtime.GC(), at least the fixed map is
    corrupted. It may return data from another entry, or just what looks like
    random bytes.

    I spent this morning using git bisect and it it turned up this
    particular
    commit.

    I also checked out tip and reverted only this commit; that seems to work
    fine.

    I am running with two small patches. Tip with these patches works
    fine, so I
    believe it's something in Dyukov's original patch.

    encoding/gob/decoder.go:15
    -const tooBig = 1 << 30
    +const tooBig = 1 << 34

    and
    --- a/src/runtime/malloc2.go
    +++ b/src/runtime/malloc2.go
    @@ -136,7 +136,7 @@ const (
    // See http://golang.org/issue/5402 and
    http://golang.org/issue/5236.
    // On other 64-bit platforms, we limit the arena to 128GB, or 37
    bits.
    // On 32-bit, we don't bother limiting anything, so we use the full
    32-bit address.
    - _MHeapMap_TotalBits = (_64bit*goos_windows)*35 +
    (_64bit*(1-goos_windows))*37 + (1-_64bit)*32
    + _MHeapMap_TotalBits = (_64bit*goos_windows)*35 +
    (_64bit*(1-goos_windows))*38 + (1-_64bit)*32

    @@ -145,7 +145,7 @@ const (
    // 2, 3, and 4 are all plausible maximums depending
    // on the hardware details of the machine. The garbage
    // collector scales well to 32 cpus.
    - _MaxGcproc = 32
    + _MaxGcproc = 64

    Can you please provide the exact reproducer program?
    I am running the following test and it does not fail:

    package main

    import (
    "runtime"
    "fmt"
    "os"
    )

    func main() {
    m := make(map[int][2]int)
    for i := 0; i < 1e6; i++ {
    m[i] = [2]int{i, i+1}
    if (i%111) == 0 {
    runtime.GC()
    if len(m) != i+1 {
    fmt.Printf("bad len: %v/%v\n", len(m), i+1)
    os.Exit(1)
    }
    for j := 0; j <= i; j++ {
    v := m[j]
    if v[0] != j || v[1] != j+1 {
    fmt.Printf("bad value %v: %v/%v\n", j, v[0], v[1])
    os.Exit(1)
    }
    }
    }
    }
    }
    --
    You received this message because you are subscribed to the Google
    Groups "golang-dev" group.
    To unsubscribe from this group and stop receiving emails from it, send
    an email to golang-dev+...@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
    --
    You received this message because you are subscribed to the Google Groups "golang-dev" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-dev+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
  • Russ Cox at Feb 13, 2015 at 10:06 pm
    I think the CL that has been pinpointed is too complex and should be rolled
    back. We can fix the pointer scan problem by moving overflow back to the
    bucket header, and then we can fix the pointer walking problem that the
    overflow move fixed by adding a dummy word to the header. I don't want to
    have to look at code like in CL 85e7bee and wonder whether it's correct.
    It's not obviously incorrect but it's not obviously correct either. Why
    bother with the complexity?

    Russ

    --
    You received this message because you are subscribed to the Google Groups "golang-dev" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-dev+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
  • Dmitry Vyukov at Feb 14, 2015 at 10:44 am

    On Sat, Feb 14, 2015 at 1:06 AM, Russ Cox wrote:
    I think the CL that has been pinpointed is too complex and should be rolled
    back. We can fix the pointer scan problem by moving overflow back to the
    bucket header,
    I don't understand how moving overflow pointer to the beginning of
    bucket will fix the scanning of large maps problems.

    and then we can fix the pointer walking problem that the
    overflow move fixed by adding a dummy word to the header. I don't want to
    have to look at code like in CL 85e7bee and wonder whether it's correct.
    It's not obviously incorrect but it's not obviously correct either. Why
    bother with the complexity?

    Russ
    --
    You received this message because you are subscribed to the Google Groups "golang-dev" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-dev+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
  • Dmitry Vyukov at Feb 14, 2015 at 3:17 pm
    I made a stupid mistake. I guess your value is larger than 128 bytes.
    Mailed a fix:
    https://go-review.googlesource.com/#/c/4901/

    Robert, the fix also fixes the gccgoimporter test.


    On Fri, Feb 13, 2015 at 1:28 AM, Peter Vessenes wrote:
    Dmitri: If it's a race, it's not detected by the race detector.

    I'm still working on replicating the behaviour outside my code.
    On Thursday, February 12, 2015 at 2:26:09 PM UTC-8, gri wrote:

    FYI, a recent new failure in tools/go/gccgoimporter may (or may not) be
    related to this. I just disabled a test to get the build going again (
    https://go-review.googlesource.com/#/c/4750/ ).

    The failure showed up on the dashboard at the same time.

    - gri

    On Thu, Feb 12, 2015 at 1:10 AM, 'Dmitry Vyukov' via golang-dev
    wrote:
    Is there any chance you have a data race on maps? I can imagine that
    failure modes change after this change. Try to run the program with -race
    flag.


    On Thu, Feb 12, 2015 at 11:01 AM, Dmitry Vyukov <dvy...@google.com>
    wrote:
    On Thu, Feb 12, 2015 at 1:50 AM, wrote:
    Recently Dyukov kindly submitted 85e7bee; it updates the GC to ignore
    maps
    which don't contain pointers.

    At about the same time I started have weird corruption issues with
    maps. I
    finally tracked them down to a fairly simple case in my code. I can't
    replicate it yet in a sandbox or highly simplified format, but I
    wanted to
    get this out soon, so that people know they should probably revert
    85e7 if
    they're having weird troubles. And of course, hopefully someone
    smarter than
    me will see the problem immediately.

    In brief, I create maps as follows:

    fixedmap[keystruct][5]valstruct
    varmap[keystruct][]valstruct

    And proceed to dump between 1 and 50 million entries in them, using a
    method
    like:

    a := [5]struct{}
    a[0] = somedata
    ...

    fixedmap[key] = a


    After each forced GC, or call to runtime.GC(), at least the fixed map
    is
    corrupted. It may return data from another entry, or just what looks
    like
    random bytes.

    I spent this morning using git bisect and it it turned up this
    particular
    commit.

    I also checked out tip and reverted only this commit; that seems to
    work
    fine.

    I am running with two small patches. Tip with these patches works
    fine, so I
    believe it's something in Dyukov's original patch.

    encoding/gob/decoder.go:15
    -const tooBig = 1 << 30
    +const tooBig = 1 << 34

    and
    --- a/src/runtime/malloc2.go
    +++ b/src/runtime/malloc2.go
    @@ -136,7 +136,7 @@ const (
    // See http://golang.org/issue/5402 and
    http://golang.org/issue/5236.
    // On other 64-bit platforms, we limit the arena to 128GB, or
    37
    bits.
    // On 32-bit, we don't bother limiting anything, so we use the
    full
    32-bit address.
    - _MHeapMap_TotalBits = (_64bit*goos_windows)*35 +
    (_64bit*(1-goos_windows))*37 + (1-_64bit)*32
    + _MHeapMap_TotalBits = (_64bit*goos_windows)*35 +
    (_64bit*(1-goos_windows))*38 + (1-_64bit)*32

    @@ -145,7 +145,7 @@ const (
    // 2, 3, and 4 are all plausible maximums depending
    // on the hardware details of the machine. The garbage
    // collector scales well to 32 cpus.
    - _MaxGcproc = 32
    + _MaxGcproc = 64

    Can you please provide the exact reproducer program?
    I am running the following test and it does not fail:

    package main

    import (
    "runtime"
    "fmt"
    "os"
    )

    func main() {
    m := make(map[int][2]int)
    for i := 0; i < 1e6; i++ {
    m[i] = [2]int{i, i+1}
    if (i%111) == 0 {
    runtime.GC()
    if len(m) != i+1 {
    fmt.Printf("bad len: %v/%v\n", len(m), i+1)
    os.Exit(1)
    }
    for j := 0; j <= i; j++ {
    v := m[j]
    if v[0] != j || v[1] != j+1 {
    fmt.Printf("bad value %v: %v/%v\n", j, v[0], v[1])
    os.Exit(1)
    }
    }
    }
    }
    }
    --
    You received this message because you are subscribed to the Google Groups
    "golang-dev" group.
    To unsubscribe from this group and stop receiving emails from it, send an
    email to golang-dev+...@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
    --
    You received this message because you are subscribed to the Google Groups "golang-dev" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-dev+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
  • Peter Vessenes at Feb 19, 2015 at 7:09 am
    Dmitry,

    Thanks very much for hunting this down. My value is indeed larger than 128
    bytes, but my test code's value was a bit less, hence the difficulty
    replicating.

    As I read the code, do I understand the comments to mean that the 128 byte
    value could be modified to 255 with minimal changes to the runtime? If I
    want to store something larger and get your GC patch benefits, do you have
    recommendations?

    I'm feeling my way around this problem, and trying to figure out how to
    minimize load times, latency and GC pauses for my application. I have
    slightly less than a billion 45 byte records, aggregated up in 50 million
    or so variable length clusters.

    My first draft was map[string][]record; this sucks hard in go for the
    number of records I'm talking about.

    My second draft was map[[64]byte][]record; still has GC problems.

    About 5% of clusters have more than 5 records, so my third step was

    map[[64]byte][5]record for the 95% case and
    map[64]byte][]record for the 5% case.

    Unfortunately, 5% in this case is still slow; 2 to 3 second GC pause times
    typically. And note 5*45 > 128. :)

    Try four is sketched out as: create a giant []byte with all records in it
    ordered by cluster, and store offsets by cluster in a map. This has a bunch
    of nice pros, including fast loading with mmap, but it means that I have to
    write my own memory management routines for the []byte structure unless I
    want to recreate it whenever my data changes.

    At any rate, am I missing an obvious angle on this? Opinions / Advice
    welcome.

    Peter




    On Saturday, February 14, 2015 at 7:17:22 AM UTC-8, Dmitry Vyukov wrote:

    I made a stupid mistake. I guess your value is larger than 128 bytes.
    Mailed a fix:
    https://go-review.googlesource.com/#/c/4901/

    Robert, the fix also fixes the gccgoimporter test.



    On Fri, Feb 13, 2015 at 1:28 AM, Peter Vessenes <pe...@coinlab.com
    <javascript:>> wrote:
    Dmitri: If it's a race, it's not detected by the race detector.

    I'm still working on replicating the behaviour outside my code.
    On Thursday, February 12, 2015 at 2:26:09 PM UTC-8, gri wrote:

    FYI, a recent new failure in tools/go/gccgoimporter may (or may not) be
    related to this. I just disabled a test to get the build going again (
    https://go-review.googlesource.com/#/c/4750/ ).

    The failure showed up on the dashboard at the same time.

    - gri

    On Thu, Feb 12, 2015 at 1:10 AM, 'Dmitry Vyukov' via golang-dev
    wrote:
    Is there any chance you have a data race on maps? I can imagine that
    failure modes change after this change. Try to run the program with
    -race
    flag.


    On Thu, Feb 12, 2015 at 11:01 AM, Dmitry Vyukov <dvy...@google.com>
    wrote:
    On Thu, Feb 12, 2015 at 1:50 AM, wrote:
    Recently Dyukov kindly submitted 85e7bee; it updates the GC to
    ignore
    maps
    which don't contain pointers.

    At about the same time I started have weird corruption issues with
    maps. I
    finally tracked them down to a fairly simple case in my code. I
    can't
    replicate it yet in a sandbox or highly simplified format, but I
    wanted to
    get this out soon, so that people know they should probably revert
    85e7 if
    they're having weird troubles. And of course, hopefully someone
    smarter than
    me will see the problem immediately.

    In brief, I create maps as follows:

    fixedmap[keystruct][5]valstruct
    varmap[keystruct][]valstruct

    And proceed to dump between 1 and 50 million entries in them, using
    a
    method
    like:

    a := [5]struct{}
    a[0] = somedata
    ...

    fixedmap[key] = a


    After each forced GC, or call to runtime.GC(), at least the fixed
    map
    is
    corrupted. It may return data from another entry, or just what
    looks
    like
    random bytes.

    I spent this morning using git bisect and it it turned up this
    particular
    commit.

    I also checked out tip and reverted only this commit; that seems to
    work
    fine.

    I am running with two small patches. Tip with these patches works
    fine, so I
    believe it's something in Dyukov's original patch.

    encoding/gob/decoder.go:15
    -const tooBig = 1 << 30
    +const tooBig = 1 << 34

    and
    --- a/src/runtime/malloc2.go
    +++ b/src/runtime/malloc2.go
    @@ -136,7 +136,7 @@ const (
    // See http://golang.org/issue/5402 and
    http://golang.org/issue/5236.
    // On other 64-bit platforms, we limit the arena to 128GB,
    or
    37
    bits.
    // On 32-bit, we don't bother limiting anything, so we use
    the
    full
    32-bit address.
    - _MHeapMap_TotalBits = (_64bit*goos_windows)*35 +
    (_64bit*(1-goos_windows))*37 + (1-_64bit)*32
    + _MHeapMap_TotalBits = (_64bit*goos_windows)*35 +
    (_64bit*(1-goos_windows))*38 + (1-_64bit)*32

    @@ -145,7 +145,7 @@ const (
    // 2, 3, and 4 are all plausible maximums depending
    // on the hardware details of the machine. The garbage
    // collector scales well to 32 cpus.
    - _MaxGcproc = 32
    + _MaxGcproc = 64

    Can you please provide the exact reproducer program?
    I am running the following test and it does not fail:

    package main

    import (
    "runtime"
    "fmt"
    "os"
    )

    func main() {
    m := make(map[int][2]int)
    for i := 0; i < 1e6; i++ {
    m[i] = [2]int{i, i+1}
    if (i%111) == 0 {
    runtime.GC()
    if len(m) != i+1 {
    fmt.Printf("bad len: %v/%v\n", len(m), i+1)
    os.Exit(1)
    }
    for j := 0; j <= i; j++ {
    v := m[j]
    if v[0] != j || v[1] != j+1 {
    fmt.Printf("bad value %v: %v/%v\n", j, v[0], v[1])
    os.Exit(1)
    }
    }
    }
    }
    }
    --
    You received this message because you are subscribed to the Google
    Groups
    "golang-dev" group.
    To unsubscribe from this group and stop receiving emails from it, send
    an
    email to golang-dev+...@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
    --
    You received this message because you are subscribed to the Google Groups "golang-dev" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-dev+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
  • Dmitry Vyukov at Feb 19, 2015 at 8:20 am

    On Thu, Feb 19, 2015 at 10:09 AM, Peter Vessenes wrote:
    Dmitry,

    Thanks very much for hunting this down. My value is indeed larger than 128
    bytes, but my test code's value was a bit less, hence the difficulty
    replicating.

    As I read the code, do I understand the comments to mean that the 128 byte
    value could be modified to 255 with minimal changes to the runtime? If I
    want to store something larger and get your GC patch benefits, do you have
    recommendations?

    I'm feeling my way around this problem, and trying to figure out how to
    minimize load times, latency and GC pauses for my application. I have
    slightly less than a billion 45 byte records, aggregated up in 50 million or
    so variable length clusters.

    My first draft was map[string][]record; this sucks hard in go for the number
    of records I'm talking about.

    My second draft was map[[64]byte][]record; still has GC problems.

    About 5% of clusters have more than 5 records, so my third step was

    map[[64]byte][5]record for the 95% case and
    map[64]byte][]record for the 5% case.

    Unfortunately, 5% in this case is still slow; 2 to 3 second GC pause times
    typically. And note 5*45 > 128. :)

    Try four is sketched out as: create a giant []byte with all records in it
    ordered by cluster, and store offsets by cluster in a map.
    I would load all data into giant []record and then build a
    map[[64]byte]struct{startIdx uint32; endIdx uint32}.
    This has a bunch
    of nice pros, including fast loading with mmap, but it means that I have to
    write my own memory management routines for the []byte structure unless I
    want to recreate it whenever my data changes.
    I don't understand the requirement to write own memory management.


    At any rate, am I missing an obvious angle on this? Opinions / Advice
    welcome.

    Peter




    On Saturday, February 14, 2015 at 7:17:22 AM UTC-8, Dmitry Vyukov wrote:

    I made a stupid mistake. I guess your value is larger than 128 bytes.
    Mailed a fix:
    https://go-review.googlesource.com/#/c/4901/

    Robert, the fix also fixes the gccgoimporter test.


    On Fri, Feb 13, 2015 at 1:28 AM, Peter Vessenes wrote:
    Dmitri: If it's a race, it's not detected by the race detector.

    I'm still working on replicating the behaviour outside my code.
    On Thursday, February 12, 2015 at 2:26:09 PM UTC-8, gri wrote:

    FYI, a recent new failure in tools/go/gccgoimporter may (or may not) be
    related to this. I just disabled a test to get the build going again (
    https://go-review.googlesource.com/#/c/4750/ ).

    The failure showed up on the dashboard at the same time.

    - gri

    On Thu, Feb 12, 2015 at 1:10 AM, 'Dmitry Vyukov' via golang-dev
    wrote:
    Is there any chance you have a data race on maps? I can imagine that
    failure modes change after this change. Try to run the program with
    -race
    flag.


    On Thu, Feb 12, 2015 at 11:01 AM, Dmitry Vyukov <dvy...@google.com>
    wrote:
    On Thu, Feb 12, 2015 at 1:50 AM, wrote:
    Recently Dyukov kindly submitted 85e7bee; it updates the GC to
    ignore
    maps
    which don't contain pointers.

    At about the same time I started have weird corruption issues with
    maps. I
    finally tracked them down to a fairly simple case in my code. I
    can't
    replicate it yet in a sandbox or highly simplified format, but I
    wanted to
    get this out soon, so that people know they should probably revert
    85e7 if
    they're having weird troubles. And of course, hopefully someone
    smarter than
    me will see the problem immediately.

    In brief, I create maps as follows:

    fixedmap[keystruct][5]valstruct
    varmap[keystruct][]valstruct

    And proceed to dump between 1 and 50 million entries in them, using
    a
    method
    like:

    a := [5]struct{}
    a[0] = somedata
    ...

    fixedmap[key] = a


    After each forced GC, or call to runtime.GC(), at least the fixed
    map
    is
    corrupted. It may return data from another entry, or just what
    looks
    like
    random bytes.

    I spent this morning using git bisect and it it turned up this
    particular
    commit.

    I also checked out tip and reverted only this commit; that seems to
    work
    fine.

    I am running with two small patches. Tip with these patches works
    fine, so I
    believe it's something in Dyukov's original patch.

    encoding/gob/decoder.go:15
    -const tooBig = 1 << 30
    +const tooBig = 1 << 34

    and
    --- a/src/runtime/malloc2.go
    +++ b/src/runtime/malloc2.go
    @@ -136,7 +136,7 @@ const (
    // See http://golang.org/issue/5402 and
    http://golang.org/issue/5236.
    // On other 64-bit platforms, we limit the arena to 128GB,
    or
    37
    bits.
    // On 32-bit, we don't bother limiting anything, so we use
    the
    full
    32-bit address.
    - _MHeapMap_TotalBits = (_64bit*goos_windows)*35 +
    (_64bit*(1-goos_windows))*37 + (1-_64bit)*32
    + _MHeapMap_TotalBits = (_64bit*goos_windows)*35 +
    (_64bit*(1-goos_windows))*38 + (1-_64bit)*32

    @@ -145,7 +145,7 @@ const (
    // 2, 3, and 4 are all plausible maximums depending
    // on the hardware details of the machine. The garbage
    // collector scales well to 32 cpus.
    - _MaxGcproc = 32
    + _MaxGcproc = 64

    Can you please provide the exact reproducer program?
    I am running the following test and it does not fail:

    package main

    import (
    "runtime"
    "fmt"
    "os"
    )

    func main() {
    m := make(map[int][2]int)
    for i := 0; i < 1e6; i++ {
    m[i] = [2]int{i, i+1}
    if (i%111) == 0 {
    runtime.GC()
    if len(m) != i+1 {
    fmt.Printf("bad len: %v/%v\n", len(m), i+1)
    os.Exit(1)
    }
    for j := 0; j <= i; j++ {
    v := m[j]
    if v[0] != j || v[1] != j+1 {
    fmt.Printf("bad value %v: %v/%v\n", j, v[0], v[1])
    os.Exit(1)
    }
    }
    }
    }
    }
    --
    You received this message because you are subscribed to the Google
    Groups
    "golang-dev" group.
    To unsubscribe from this group and stop receiving emails from it, send
    an
    email to golang-dev+...@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
    --
    You received this message because you are subscribed to the Google Groups
    "golang-dev" group.
    To unsubscribe from this group and stop receiving emails from it, send an
    email to golang-dev+unsubscribe@googlegroups.com.

    For more options, visit https://groups.google.com/d/optout.
    --
    You received this message because you are subscribed to the Google Groups "golang-dev" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-dev+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
  • Peter Vessenes at Feb 19, 2015 at 3:24 pm
    Thanks for the thoughts. As new data comes in, the []record will need some
    sort of garbage collection; collections will change their indexes and
    therefore leave 'dead' or free spots in the []record. That said, I think it
    generally gets me what I want, and I can always recompact by copying to a
    new array.

    Thanks,

    Peter
    On Thursday, February 19, 2015 at 12:20:16 AM UTC-8, Dmitry Vyukov wrote:

    On Thu, Feb 19, 2015 at 10:09 AM, Peter Vessenes <pe...@coinlab.com
    <javascript:>> wrote:
    Dmitry,

    Thanks very much for hunting this down. My value is indeed larger than 128
    bytes, but my test code's value was a bit less, hence the difficulty
    replicating.

    As I read the code, do I understand the comments to mean that the 128 byte
    value could be modified to 255 with minimal changes to the runtime? If I
    want to store something larger and get your GC patch benefits, do you have
    recommendations?

    I'm feeling my way around this problem, and trying to figure out how to
    minimize load times, latency and GC pauses for my application. I have
    slightly less than a billion 45 byte records, aggregated up in 50
    million or
    so variable length clusters.

    My first draft was map[string][]record; this sucks hard in go for the number
    of records I'm talking about.

    My second draft was map[[64]byte][]record; still has GC problems.

    About 5% of clusters have more than 5 records, so my third step was

    map[[64]byte][5]record for the 95% case and
    map[64]byte][]record for the 5% case.

    Unfortunately, 5% in this case is still slow; 2 to 3 second GC pause times
    typically. And note 5*45 > 128. :)

    Try four is sketched out as: create a giant []byte with all records in it
    ordered by cluster, and store offsets by cluster in a map.
    I would load all data into giant []record and then build a
    map[[64]byte]struct{startIdx uint32; endIdx uint32}.
    This has a bunch
    of nice pros, including fast loading with mmap, but it means that I have to
    write my own memory management routines for the []byte structure unless I
    want to recreate it whenever my data changes.
    I don't understand the requirement to write own memory management.


    At any rate, am I missing an obvious angle on this? Opinions / Advice
    welcome.

    Peter




    On Saturday, February 14, 2015 at 7:17:22 AM UTC-8, Dmitry Vyukov wrote:

    I made a stupid mistake. I guess your value is larger than 128 bytes.
    Mailed a fix:
    https://go-review.googlesource.com/#/c/4901/

    Robert, the fix also fixes the gccgoimporter test.


    On Fri, Feb 13, 2015 at 1:28 AM, Peter Vessenes wrote:
    Dmitri: If it's a race, it's not detected by the race detector.

    I'm still working on replicating the behaviour outside my code.
    On Thursday, February 12, 2015 at 2:26:09 PM UTC-8, gri wrote:

    FYI, a recent new failure in tools/go/gccgoimporter may (or may not)
    be
    related to this. I just disabled a test to get the build going again
    (
    https://go-review.googlesource.com/#/c/4750/ ).

    The failure showed up on the dashboard at the same time.

    - gri

    On Thu, Feb 12, 2015 at 1:10 AM, 'Dmitry Vyukov' via golang-dev
    wrote:
    Is there any chance you have a data race on maps? I can imagine
    that
    failure modes change after this change. Try to run the program with
    -race
    flag.


    On Thu, Feb 12, 2015 at 11:01 AM, Dmitry Vyukov <dvy...@google.com>
    wrote:
    On Thu, Feb 12, 2015 at 1:50 AM, wrote:
    Recently Dyukov kindly submitted 85e7bee; it updates the GC to
    ignore
    maps
    which don't contain pointers.

    At about the same time I started have weird corruption issues
    with
    maps. I
    finally tracked them down to a fairly simple case in my code. I
    can't
    replicate it yet in a sandbox or highly simplified format, but I
    wanted to
    get this out soon, so that people know they should probably
    revert
    85e7 if
    they're having weird troubles. And of course, hopefully someone
    smarter than
    me will see the problem immediately.

    In brief, I create maps as follows:

    fixedmap[keystruct][5]valstruct
    varmap[keystruct][]valstruct

    And proceed to dump between 1 and 50 million entries in them,
    using
    a
    method
    like:

    a := [5]struct{}
    a[0] = somedata
    ...

    fixedmap[key] = a


    After each forced GC, or call to runtime.GC(), at least the
    fixed
    map
    is
    corrupted. It may return data from another entry, or just what
    looks
    like
    random bytes.

    I spent this morning using git bisect and it it turned up this
    particular
    commit.

    I also checked out tip and reverted only this commit; that seems
    to
    work
    fine.

    I am running with two small patches. Tip with these patches
    works
    fine, so I
    believe it's something in Dyukov's original patch.

    encoding/gob/decoder.go:15
    -const tooBig = 1 << 30
    +const tooBig = 1 << 34

    and
    --- a/src/runtime/malloc2.go
    +++ b/src/runtime/malloc2.go
    @@ -136,7 +136,7 @@ const (
    // See http://golang.org/issue/5402 and
    http://golang.org/issue/5236.
    // On other 64-bit platforms, we limit the arena to
    128GB,
    or
    37
    bits.
    // On 32-bit, we don't bother limiting anything, so we
    use
    the
    full
    32-bit address.
    - _MHeapMap_TotalBits = (_64bit*goos_windows)*35 +
    (_64bit*(1-goos_windows))*37 + (1-_64bit)*32
    + _MHeapMap_TotalBits = (_64bit*goos_windows)*35 +
    (_64bit*(1-goos_windows))*38 + (1-_64bit)*32

    @@ -145,7 +145,7 @@ const (
    // 2, 3, and 4 are all plausible maximums depending
    // on the hardware details of the machine. The garbage
    // collector scales well to 32 cpus.
    - _MaxGcproc = 32
    + _MaxGcproc = 64

    Can you please provide the exact reproducer program?
    I am running the following test and it does not fail:

    package main

    import (
    "runtime"
    "fmt"
    "os"
    )

    func main() {
    m := make(map[int][2]int)
    for i := 0; i < 1e6; i++ {
    m[i] = [2]int{i, i+1}
    if (i%111) == 0 {
    runtime.GC()
    if len(m) != i+1 {
    fmt.Printf("bad len: %v/%v\n", len(m), i+1)
    os.Exit(1)
    }
    for j := 0; j <= i; j++ {
    v := m[j]
    if v[0] != j || v[1] != j+1 {
    fmt.Printf("bad value %v: %v/%v\n", j, v[0], v[1])
    os.Exit(1)
    }
    }
    }
    }
    }
    --
    You received this message because you are subscribed to the Google
    Groups
    "golang-dev" group.
    To unsubscribe from this group and stop receiving emails from it,
    send
    an
    email to golang-dev+...@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
    --
    You received this message because you are subscribed to the Google Groups
    "golang-dev" group.
    To unsubscribe from this group and stop receiving emails from it, send an
    email to golang-dev+...@googlegroups.com <javascript:>.

    For more options, visit https://groups.google.com/d/optout.
    --
    You received this message because you are subscribed to the Google Groups "golang-dev" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-dev+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
  • Dmitry Vyukov at Feb 19, 2015 at 3:32 pm
    The slice does not shrink significantly over time, then you can just
    keep a freelist of unused slots w/o actually removing them and
    shrinking slice. Then indices don't change.

    On Thu, Feb 19, 2015 at 6:24 PM, Peter Vessenes wrote:
    Thanks for the thoughts. As new data comes in, the []record will need some
    sort of garbage collection; collections will change their indexes and
    therefore leave 'dead' or free spots in the []record. That said, I think it
    generally gets me what I want, and I can always recompact by copying to a
    new array.

    Thanks,

    Peter
    On Thursday, February 19, 2015 at 12:20:16 AM UTC-8, Dmitry Vyukov wrote:

    On Thu, Feb 19, 2015 at 10:09 AM, Peter Vessenes <pe...@coinlab.com>
    wrote:
    Dmitry,

    Thanks very much for hunting this down. My value is indeed larger than
    128
    bytes, but my test code's value was a bit less, hence the difficulty
    replicating.

    As I read the code, do I understand the comments to mean that the 128
    byte
    value could be modified to 255 with minimal changes to the runtime? If I
    want to store something larger and get your GC patch benefits, do you
    have
    recommendations?

    I'm feeling my way around this problem, and trying to figure out how to
    minimize load times, latency and GC pauses for my application. I have
    slightly less than a billion 45 byte records, aggregated up in 50
    million or
    so variable length clusters.

    My first draft was map[string][]record; this sucks hard in go for the
    number
    of records I'm talking about.

    My second draft was map[[64]byte][]record; still has GC problems.

    About 5% of clusters have more than 5 records, so my third step was

    map[[64]byte][5]record for the 95% case and
    map[64]byte][]record for the 5% case.

    Unfortunately, 5% in this case is still slow; 2 to 3 second GC pause
    times
    typically. And note 5*45 > 128. :)

    Try four is sketched out as: create a giant []byte with all records in
    it
    ordered by cluster, and store offsets by cluster in a map.
    I would load all data into giant []record and then build a
    map[[64]byte]struct{startIdx uint32; endIdx uint32}.
    This has a bunch
    of nice pros, including fast loading with mmap, but it means that I have
    to
    write my own memory management routines for the []byte structure unless
    I
    want to recreate it whenever my data changes.
    I don't understand the requirement to write own memory management.


    At any rate, am I missing an obvious angle on this? Opinions / Advice
    welcome.

    Peter




    On Saturday, February 14, 2015 at 7:17:22 AM UTC-8, Dmitry Vyukov wrote:

    I made a stupid mistake. I guess your value is larger than 128 bytes.
    Mailed a fix:
    https://go-review.googlesource.com/#/c/4901/

    Robert, the fix also fixes the gccgoimporter test.



    On Fri, Feb 13, 2015 at 1:28 AM, Peter Vessenes <pe...@coinlab.com>
    wrote:
    Dmitri: If it's a race, it's not detected by the race detector.

    I'm still working on replicating the behaviour outside my code.
    On Thursday, February 12, 2015 at 2:26:09 PM UTC-8, gri wrote:

    FYI, a recent new failure in tools/go/gccgoimporter may (or may not)
    be
    related to this. I just disabled a test to get the build going again
    (
    https://go-review.googlesource.com/#/c/4750/ ).

    The failure showed up on the dashboard at the same time.

    - gri

    On Thu, Feb 12, 2015 at 1:10 AM, 'Dmitry Vyukov' via golang-dev
    wrote:
    Is there any chance you have a data race on maps? I can imagine
    that
    failure modes change after this change. Try to run the program with
    -race
    flag.


    On Thu, Feb 12, 2015 at 11:01 AM, Dmitry Vyukov <dvy...@google.com>
    wrote:
    On Thu, Feb 12, 2015 at 1:50 AM, wrote:
    Recently Dyukov kindly submitted 85e7bee; it updates the GC to
    ignore
    maps
    which don't contain pointers.

    At about the same time I started have weird corruption issues
    with
    maps. I
    finally tracked them down to a fairly simple case in my code. I
    can't
    replicate it yet in a sandbox or highly simplified format, but I
    wanted to
    get this out soon, so that people know they should probably
    revert
    85e7 if
    they're having weird troubles. And of course, hopefully someone
    smarter than
    me will see the problem immediately.

    In brief, I create maps as follows:

    fixedmap[keystruct][5]valstruct
    varmap[keystruct][]valstruct

    And proceed to dump between 1 and 50 million entries in them,
    using
    a
    method
    like:

    a := [5]struct{}
    a[0] = somedata
    ...

    fixedmap[key] = a


    After each forced GC, or call to runtime.GC(), at least the
    fixed
    map
    is
    corrupted. It may return data from another entry, or just what
    looks
    like
    random bytes.

    I spent this morning using git bisect and it it turned up this
    particular
    commit.

    I also checked out tip and reverted only this commit; that seems
    to
    work
    fine.

    I am running with two small patches. Tip with these patches
    works
    fine, so I
    believe it's something in Dyukov's original patch.

    encoding/gob/decoder.go:15
    -const tooBig = 1 << 30
    +const tooBig = 1 << 34

    and
    --- a/src/runtime/malloc2.go
    +++ b/src/runtime/malloc2.go
    @@ -136,7 +136,7 @@ const (
    // See http://golang.org/issue/5402 and
    http://golang.org/issue/5236.
    // On other 64-bit platforms, we limit the arena to
    128GB,
    or
    37
    bits.
    // On 32-bit, we don't bother limiting anything, so we
    use
    the
    full
    32-bit address.
    - _MHeapMap_TotalBits = (_64bit*goos_windows)*35 +
    (_64bit*(1-goos_windows))*37 + (1-_64bit)*32
    + _MHeapMap_TotalBits = (_64bit*goos_windows)*35 +
    (_64bit*(1-goos_windows))*38 + (1-_64bit)*32

    @@ -145,7 +145,7 @@ const (
    // 2, 3, and 4 are all plausible maximums depending
    // on the hardware details of the machine. The garbage
    // collector scales well to 32 cpus.
    - _MaxGcproc = 32
    + _MaxGcproc = 64

    Can you please provide the exact reproducer program?
    I am running the following test and it does not fail:

    package main

    import (
    "runtime"
    "fmt"
    "os"
    )

    func main() {
    m := make(map[int][2]int)
    for i := 0; i < 1e6; i++ {
    m[i] = [2]int{i, i+1}
    if (i%111) == 0 {
    runtime.GC()
    if len(m) != i+1 {
    fmt.Printf("bad len: %v/%v\n", len(m), i+1)
    os.Exit(1)
    }
    for j := 0; j <= i; j++ {
    v := m[j]
    if v[0] != j || v[1] != j+1 {
    fmt.Printf("bad value %v: %v/%v\n", j, v[0], v[1])
    os.Exit(1)
    }
    }
    }
    }
    }
    --
    You received this message because you are subscribed to the Google
    Groups
    "golang-dev" group.
    To unsubscribe from this group and stop receiving emails from it,
    send
    an
    email to golang-dev+...@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
    --
    You received this message because you are subscribed to the Google
    Groups
    "golang-dev" group.
    To unsubscribe from this group and stop receiving emails from it, send
    an
    email to golang-dev+...@googlegroups.com.

    For more options, visit https://groups.google.com/d/optout.
    --
    You received this message because you are subscribed to the Google Groups
    "golang-dev" group.
    To unsubscribe from this group and stop receiving emails from it, send an
    email to golang-dev+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
    --
    You received this message because you are subscribed to the Google Groups "golang-dev" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-dev+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
  • Dmitry Vyukov at Feb 19, 2015 at 3:33 pm
    On second thought, if you need to keep some elements adjacent in the
    slice, then it won't works, because holes can be not large enough to
    fit new clusters.

    --
    You received this message because you are subscribed to the Google Groups "golang-dev" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-dev+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
  • Davies Liu at Feb 20, 2015 at 2:10 am
    How about something like this:

    var pools []*[65536]record
    var free []uint32 // index to pools, (highIdx, lowIdx)

    var small_clusters map[[64]byte][8]uint32
    var big_clusters map[[64]byte][]uint32

    There is no pointer in `pools` and `small_clusters`, few pointers in
    `free` and `big_clusters`

    On Thu, Feb 19, 2015 at 7:33 AM, 'Dmitry Vyukov' via golang-dev
    wrote:
    On second thought, if you need to keep some elements adjacent in the
    slice, then it won't works, because holes can be not large enough to
    fit new clusters.

    --
    You received this message because you are subscribed to the Google Groups "golang-dev" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-dev+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.


    --
      - Davies

    --
    You received this message because you are subscribed to the Google Groups "golang-dev" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-dev+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
  • Peter Vessenes at Feb 20, 2015 at 6:47 am
    @dmitry; right, that's the problem. But, you could copy over to a new array
    and re-compact it provided there's enough memory. I think on balance this
    is the right solution given the constraints. Getting to use mmap for the
    large record array is a real win also; it's very slow (and requires a patch
    to the runtime) to parse gobs that large. I don't know how to mmap a map in
    go, so moving the bulk of the data to an array is nice.

    @davies, it's late enough that I can't tell if your suggestion is
    equivalent to one of mine in GC terms. I think it is, in that my
    big_clusters will be large enough to trigger > 2 second GC pauses; I'm
    shooting for more like 100ms or so max. It's that pesky []uint32; I don't
    think I can afford to have any pointers in any of my maps with current GC
    dynamics and the size of my data set. I'll get back to you all with some
    benchmark data once I've got a reasonable final solution sorted.



    On Thursday, February 19, 2015 at 6:10:53 PM UTC-8, Davies wrote:

    How about something like this:

    var pools []*[65536]record
    var free []uint32 // index to pools, (highIdx, lowIdx)

    var small_clusters map[[64]byte][8]uint32
    var big_clusters map[[64]byte][]uint32

    There is no pointer in `pools` and `small_clusters`, few pointers in
    `free` and `big_clusters`

    On Thu, Feb 19, 2015 at 7:33 AM, 'Dmitry Vyukov' via golang-dev
    <golan...@googlegroups.com <javascript:>> wrote:
    On second thought, if you need to keep some elements adjacent in the
    slice, then it won't works, because holes can be not large enough to
    fit new clusters.

    --
    You received this message because you are subscribed to the Google
    Groups "golang-dev" group.
    To unsubscribe from this group and stop receiving emails from it, send
    an email to golang-dev+...@googlegroups.com <javascript:>.
    For more options, visit https://groups.google.com/d/optout.


    --
    - Davies
    --
    You received this message because you are subscribed to the Google Groups "golang-dev" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-dev+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupgolang-dev @
categoriesgo
postedFeb 11, '15 at 11:27p
activeFeb 20, '15 at 6:47a
posts18
users5
websitegolang.org

People

Translate

site design / logo © 2021 Grokbase