FAQ

On Tuesday, 26 April 2016 03:44:10 UTC+3, Travis Keep wrote:
We are building a server that uses large amounts of memory.

To help with memory usage, we pre-allocate a set number of pages of fixed
length when the server starts. As the server runs, pages can be recycled
for different data structures as needed. Right now, our server has two main
data structures, lists of struct A and lists of struct B. We want to
control the size of the pages with a command line flag.

Right now, each page looks something like this, but the code below does
not work.

struct A {
...
}

struct B {
...
}

type alist []A
type blist []B

type page struct {
pageMetaDataType
raw []byte // allocated with make([]byte, pageSize)
alist // covers same region of memory as raw
blist // covers same region of memory as raw
}

I then do:
page.raw = make([]byte, 1024)

After allocating raw, I initialize alist and blist like so.

sizeInBytes := len(page.raw)
rawPtr := (*reflect.SliceHeader)(unsafe.Pointer(&page.raw)).Data
aHeader := (*reflect.SliceHeader)(unsafe.Pointer(&page.alist))
bHeader := (*reflect.SliceHeader)(unsafe.Pointer(&page.blist))
aheader.Data = rawPtr
aHeader.Len = 0
aHeader.Cap = sizeInBytes / int(reflect.TypeOf(alist).Elem.Size()) //
Divide page size by size of one A
bheader.Data = rawPtr
bHeader.Len = 0
bHeader.Cap = sizeInBytes / int(reflect.TypeOf(blist).Elem.Size()) //
Divide page size by size of one B


Unfortunately, this scheme does not work for me. I get strange behavior
such as values getting randomly overwritten in my data structures. I have
traced this bug down to the commit where I do nothing but introduce using
pages for multiple purposes by introducing unsafe. So I am pretty sure I am
doing something wrong in the code above.


Questions:

1. reflect.SliceHeader has a uintptr for the data pointer. If I initialize
a slice with SliceHeader, will the GC know about it?
AFAIK, it should.

2. What is the best way to accomplish what I am trying to do? Is it best to
allocate my pages in C and use cgo? I remember reading that the Go garbage
collector will leave C pointers alone.
It depends what exactly are you trying to do. i.e. what business problem is
your program solving. It's hard to reverse engineer any better solutions
from the description you have given.

Yes you can use C pointers, but the approach you showed should work as well.

3. What am I doing wrong?
I suspect the bug is in the code that you didn't show. Do you run your code
with race detector?

How do I do it correctly?
>

It depends how the whole workflow looks.

*A better design:
https://github.com/egonelbre/exp/blob/master/zerostream/op/buffer.go#L32*
*However, if all possible use sync.Pool or techniques pointed out in Go
source.*

+ Egon

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Search Discussions

Discussion Posts

Previous

Follow ups

Related Discussions

Discussion Navigation
viewthread | post
posts ‹ prev | 4 of 14 | next ›
Discussion Overview
groupgolang-nuts @
categoriesgo
postedApr 26, '16 at 12:44a
activeMay 27, '16 at 10:57a
posts14
users7
websitegolang.org

People

Translate

site design / logo © 2021 Grokbase