On Tue, Dec 18, 2012 at 9:49 PM, bryanturley wrote: On Tuesday, December 18, 2012 4:02:59 PM UTC-6, Kyle Lemons wrote: On Tue, Dec 18, 2012 at 4:46 PM, bryanturley wrote:
On Tuesday, December 18, 2012 3:11:29 PM UTC-6, Kyle Lemons wrote:
Could you do this with mmap MAP_FIXED? It would be inherently "unsafe"
of course. I suppose the msync call would take forever, though, so you'd
need to flip the pointers to and copy data back from a MAP_PRIVATE if you
needed to keep serving during that time.
from mmap manpage
Don't interpret addr as a hint: place the mapping at
exactly that address.
Not sure that is what you really want.
You could however use a normal mmap'ed file as a backing store for a
It could be that I haven't used mmap in a long time, but it seems that
you could persist pointers to data that's in an mmap region FIXED at a
given address. This would allow you to write a reasonably straightforward
allocator around the mmap region, and all you would have to write to
another file would be the key/pointer pairs. To reconstitute the data, you
load the mmap up at the same address and populate the map with the same
pointers and voila. (Yes, I realize that this is "unsafe".)
Ah, I see I did that once actually a good bit ago.
Real problem with that would be go has no way to guarantee the size/shape
of a struct.
It's probably not written down anywhere, but I would be surprised if gc
would ever generate two different layouts for the same struct. This is, of
course, a more-than-legitimate worry if you might have to change the
definition of the struct after creating 10gb of data; gob, for instance,
has well-defined rules for how this works.
gcc (and other c compilers) have extra optional attribs to force structs
to be as compact as possible.
I do think there is an offsetof() in reflect though so you could panic if
the running struct definition is different from the struct definition your
loading from disk.
The unsafe package has a sizeof call that could be a first-line-of-defense
if you stored the struct sizes in the key/pointer map file for comparison
with the mmap. Reflecting on the fields and checking alignment would be
even more rigorous.
Also I don't know what the GC does with the contents of an mmap region?
does it scan them for pointers outside the region? obviously it can't reap
memory from inside the region.
You'd have to map the region outside Go's virtual arena, so the GC wouldn't
touch it. You'd have to explicitly allocate and free anything in the
region yourself in terms of your own allocator.
MAP_FIXED sounds to iffy to me today though, probably just less
adventurous than I used to be ;)
It sounds like a fun weekend project, but yeah I'm not sure if the
inflexibility is worth the potential performance gain.