FAQ
Thanks for the thoughts. As new data comes in, the []record will need some
sort of garbage collection; collections will change their indexes and
therefore leave 'dead' or free spots in the []record. That said, I think it
generally gets me what I want, and I can always recompact by copying to a
new array.

Thanks,

Peter
On Thursday, February 19, 2015 at 12:20:16 AM UTC-8, Dmitry Vyukov wrote:

On Thu, Feb 19, 2015 at 10:09 AM, Peter Vessenes <pe...@coinlab.com
<javascript:>> wrote:
Dmitry,

Thanks very much for hunting this down. My value is indeed larger than 128
bytes, but my test code's value was a bit less, hence the difficulty
replicating.

As I read the code, do I understand the comments to mean that the 128 byte
value could be modified to 255 with minimal changes to the runtime? If I
want to store something larger and get your GC patch benefits, do you have
recommendations?

I'm feeling my way around this problem, and trying to figure out how to
minimize load times, latency and GC pauses for my application. I have
slightly less than a billion 45 byte records, aggregated up in 50
million or
so variable length clusters.

My first draft was map[string][]record; this sucks hard in go for the number
of records I'm talking about.

My second draft was map[[64]byte][]record; still has GC problems.

About 5% of clusters have more than 5 records, so my third step was

map[[64]byte][5]record for the 95% case and
map[64]byte][]record for the 5% case.

Unfortunately, 5% in this case is still slow; 2 to 3 second GC pause times
typically. And note 5*45 > 128. :)

Try four is sketched out as: create a giant []byte with all records in it
ordered by cluster, and store offsets by cluster in a map.
I would load all data into giant []record and then build a
map[[64]byte]struct{startIdx uint32; endIdx uint32}.
This has a bunch
of nice pros, including fast loading with mmap, but it means that I have to
write my own memory management routines for the []byte structure unless I
want to recreate it whenever my data changes.
I don't understand the requirement to write own memory management.


At any rate, am I missing an obvious angle on this? Opinions / Advice
welcome.

Peter




On Saturday, February 14, 2015 at 7:17:22 AM UTC-8, Dmitry Vyukov wrote:

I made a stupid mistake. I guess your value is larger than 128 bytes.
Mailed a fix:
https://go-review.googlesource.com/#/c/4901/

Robert, the fix also fixes the gccgoimporter test.


On Fri, Feb 13, 2015 at 1:28 AM, Peter Vessenes wrote:
Dmitri: If it's a race, it's not detected by the race detector.

I'm still working on replicating the behaviour outside my code.
On Thursday, February 12, 2015 at 2:26:09 PM UTC-8, gri wrote:

FYI, a recent new failure in tools/go/gccgoimporter may (or may not)
be
related to this. I just disabled a test to get the build going again
(
https://go-review.googlesource.com/#/c/4750/ ).

The failure showed up on the dashboard at the same time.

- gri

On Thu, Feb 12, 2015 at 1:10 AM, 'Dmitry Vyukov' via golang-dev
wrote:
Is there any chance you have a data race on maps? I can imagine
that
failure modes change after this change. Try to run the program with
-race
flag.


On Thu, Feb 12, 2015 at 11:01 AM, Dmitry Vyukov <dvy...@google.com>
wrote:
On Thu, Feb 12, 2015 at 1:50 AM, wrote:
Recently Dyukov kindly submitted 85e7bee; it updates the GC to
ignore
maps
which don't contain pointers.

At about the same time I started have weird corruption issues
with
maps. I
finally tracked them down to a fairly simple case in my code. I
can't
replicate it yet in a sandbox or highly simplified format, but I
wanted to
get this out soon, so that people know they should probably
revert
85e7 if
they're having weird troubles. And of course, hopefully someone
smarter than
me will see the problem immediately.

In brief, I create maps as follows:

fixedmap[keystruct][5]valstruct
varmap[keystruct][]valstruct

And proceed to dump between 1 and 50 million entries in them,
using
a
method
like:

a := [5]struct{}
a[0] = somedata
...

fixedmap[key] = a


After each forced GC, or call to runtime.GC(), at least the
fixed
map
is
corrupted. It may return data from another entry, or just what
looks
like
random bytes.

I spent this morning using git bisect and it it turned up this
particular
commit.

I also checked out tip and reverted only this commit; that seems
to
work
fine.

I am running with two small patches. Tip with these patches
works
fine, so I
believe it's something in Dyukov's original patch.

encoding/gob/decoder.go:15
-const tooBig = 1 << 30
+const tooBig = 1 << 34

and
--- a/src/runtime/malloc2.go
+++ b/src/runtime/malloc2.go
@@ -136,7 +136,7 @@ const (
// See http://golang.org/issue/5402 and
http://golang.org/issue/5236.
// On other 64-bit platforms, we limit the arena to
128GB,
or
37
bits.
// On 32-bit, we don't bother limiting anything, so we
use
the
full
32-bit address.
- _MHeapMap_TotalBits = (_64bit*goos_windows)*35 +
(_64bit*(1-goos_windows))*37 + (1-_64bit)*32
+ _MHeapMap_TotalBits = (_64bit*goos_windows)*35 +
(_64bit*(1-goos_windows))*38 + (1-_64bit)*32

@@ -145,7 +145,7 @@ const (
// 2, 3, and 4 are all plausible maximums depending
// on the hardware details of the machine. The garbage
// collector scales well to 32 cpus.
- _MaxGcproc = 32
+ _MaxGcproc = 64

Can you please provide the exact reproducer program?
I am running the following test and it does not fail:

package main

import (
"runtime"
"fmt"
"os"
)

func main() {
m := make(map[int][2]int)
for i := 0; i < 1e6; i++ {
m[i] = [2]int{i, i+1}
if (i%111) == 0 {
runtime.GC()
if len(m) != i+1 {
fmt.Printf("bad len: %v/%v\n", len(m), i+1)
os.Exit(1)
}
for j := 0; j <= i; j++ {
v := m[j]
if v[0] != j || v[1] != j+1 {
fmt.Printf("bad value %v: %v/%v\n", j, v[0], v[1])
os.Exit(1)
}
}
}
}
}
--
You received this message because you are subscribed to the Google
Groups
"golang-dev" group.
To unsubscribe from this group and stop receiving emails from it,
send
an
email to golang-dev+...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups
"golang-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to golang-dev+...@googlegroups.com <javascript:>.

For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups "golang-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-dev+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Search Discussions

Discussion Posts

Previous

Follow ups

Related Discussions

Discussion Navigation
viewthread | post
posts ‹ prev | 14 of 18 | next ›
Discussion Overview
groupgolang-dev @
categoriesgo
postedFeb 11, '15 at 11:27p
activeFeb 20, '15 at 6:47a
posts18
users5
websitegolang.org

People

Translate

site design / logo © 2021 Grokbase