Grokbase › Groups › Go › golang-nuts › December 2012
FAQ
We've got an in memory data store comprised of several large maps (up to
~50 million "unit32 > pointer") and various linking data structs which
totals up to ~10GB. This data also has a fairly high read and write
rate. We'd like to be able to backup this whole structure to disk
periodically, but haven't managed to find an easy way to do this so far.

We've tried gob encoding to disk, but the encoding fails in the GB range
and this also has a huge memory overhead for the file sizes it does work
for. JSON encoding does not support maps, so it doesn't help. Previously
this program was written in C, we would fork the memory and then simply
write the data in the fork to disk. In C, most of our lookups were based on
offsets, so this was pretty simple, but with Go we're using slices, so we
don't have as much control on the memory location and need to preserve
structure. I'm looking at gommap now to see if we can do something similar,
but i can't help but think others have dealt with this issue, so i thought
i would post on here for any suggestions?

And thanks to all for the useful discussions on here, they've been very
helpful.

--

Search Discussions

  • Kevin Gillette at Dec 18, 2012 at 5:15 am ⇧
    If there are no value-to-value (or key-to-key) pointers in the map data,
    what about gob-encoding each key-value pair in a loop (that way gob
    wouldn't be handling much data at once)? The tricky part would be the high
    write throughput -- if there's no element-interdependency, then it's no big
    deal. Otherwise, you may need a different structure besides a map (or a
    structure that wraps generational maps, moving old map key/value pairs into
    the newest generation on reads).
    On Monday, December 17, 2012 5:32:54 PM UTC-7, Hamish Ogilvy wrote:

    We've got an in memory data store comprised of several large maps (up to
    ~50 million "unit32 > pointer") and various linking data structs which
    totals up to ~10GB. This data also has a fairly high read and write
    rate. We'd like to be able to backup this whole structure to disk
    periodically, but haven't managed to find an easy way to do this so far.

    We've tried gob encoding to disk, but the encoding fails in the GB range
    and this also has a huge memory overhead for the file sizes it does work
    for. JSON encoding does not support maps, so it doesn't help. Previously
    this program was written in C, we would fork the memory and then simply
    write the data in the fork to disk. In C, most of our lookups were based on
    offsets, so this was pretty simple, but with Go we're using slices, so we
    don't have as much control on the memory location and need to preserve
    structure. I'm looking at gommap now to see if we can do something similar,
    but i can't help but think others have dealt with this issue, so i thought
    i would post on here for any suggestions?

    And thanks to all for the useful discussions on here, they've been very
    helpful.
    --
  • Hamish Ogilvy at Dec 18, 2012 at 10:46 pm ⇧
    Thanks Kevin, it's a bit more complicated that that unfortunately. We have
    ~5 maps, which lead to other structs, some of the structs have pointers to
    each other, which makes a straight loop pretty much impossible.

    On Tuesday, 18 December 2012 16:15:14 UTC+11, Kevin Gillette wrote:

    If there are no value-to-value (or key-to-key) pointers in the map data,
    what about gob-encoding each key-value pair in a loop (that way gob
    wouldn't be handling much data at once)? The tricky part would be the high
    write throughput -- if there's no element-interdependency, then it's no big
    deal. Otherwise, you may need a different structure besides a map (or a
    structure that wraps generational maps, moving old map key/value pairs into
    the newest generation on reads).
    On Monday, December 17, 2012 5:32:54 PM UTC-7, Hamish Ogilvy wrote:

    We've got an in memory data store comprised of several large maps (up to
    ~50 million "unit32 > pointer") and various linking data structs which
    totals up to ~10GB. This data also has a fairly high read and write
    rate. We'd like to be able to backup this whole structure to disk
    periodically, but haven't managed to find an easy way to do this so far.

    We've tried gob encoding to disk, but the encoding fails in the GB range
    and this also has a huge memory overhead for the file sizes it does work
    for. JSON encoding does not support maps, so it doesn't help. Previously
    this program was written in C, we would fork the memory and then simply
    write the data in the fork to disk. In C, most of our lookups were based on
    offsets, so this was pretty simple, but with Go we're using slices, so we
    don't have as much control on the memory location and need to preserve
    structure. I'm looking at gommap now to see if we can do something similar,
    but i can't help but think others have dealt with this issue, so i thought
    i would post on here for any suggestions?

    And thanks to all for the useful discussions on here, they've been very
    helpful.
    --
  • André Moraes at Dec 19, 2012 at 10:28 am ⇧

    On Tue, Dec 18, 2012 at 8:46 PM, Hamish Ogilvy wrote:
    Thanks Kevin, it's a bit more complicated that that unfortunately. We have
    ~5 maps, which lead to other structs, some of the structs have pointers to
    each other, which makes a straight loop pretty much impossible.
    How do you want to store all those pointers into disk? You can't keep
    the address of them.

    The next time you restart your application the addresses will be
    completely different.

    --
    André Moraes
    http://amoraes.info

    --
  • Clive Saha at Dec 19, 2012 at 12:23 pm ⇧
    I think writing out 10 gb in 1 call is a bad idea - that's a lot of data
    and it will take time during which your structure will have to be frozen.
    Nate's suggestion of on-disk structures backed by an SSD is a good one.
    Even if you don't do that, you should investigate writing out the data
    incrementally as changes happen- with high write rates a 10gb write to disk
    (even ssd) with flush etc. will kill your performance. The MMAP fixed
    solution with custom allocator and custom map/struct implementations is the
    other way to go, but again you're gong to run into the whole problem of
    freezing the memory for long enough to write out all 10 gb.

    This sounds like a fun problem - i'm curious as to how you're handling the
    versioning/freezing problem now in C. Can you keep us updated on what you
    do to solve it and some more details like How high the read/write qps are
    and how much data changes per write etc ? If you give more details and need
    a hand, i'm not doing anything over the long christmas weekend ...

    clive

    --
  • Bryanturley at Dec 18, 2012 at 6:51 pm ⇧

    On Monday, December 17, 2012 6:32:54 PM UTC-6, Hamish Ogilvy wrote:
    We've got an in memory data store comprised of several large maps (up to
    ~50 million "unit32 > pointer") and various linking data structs which
    totals up to ~10GB. This data also has a fairly high read and write
    rate. We'd like to be able to backup this whole structure to disk
    periodically, but haven't managed to find an easy way to do this so far.

    We've tried gob encoding to disk, but the encoding fails in the GB range
    and this also has a huge memory overhead for the file sizes it does work
    for. JSON encoding does not support maps, so it doesn't help. Previously
    this program was written in C, we would fork the memory and then simply
    write the data in the fork to disk.
    You could write clone functions to simulate the fork method (slower since
    no cow memory pages) and have a goroutine write the clones out to disk.

    I would probably write my own map implementation that stores to both memory
    and disk, and do something similar for the linking structures.
    (Assuming a linux server) Tweak some of the kernel settings to have more
    write-behind allowed time/space and get some fast disks.
    Perhaps using a COW block device (lvm2) or filesystem (btrfs? others
    maybe?). Then you would just need sync periods to switch to new snapshots
    in case of a crash.


    --
  • Kyle Lemons at Dec 18, 2012 at 9:11 pm ⇧
    Could you do this with mmap MAP_FIXED? It would be inherently "unsafe" of
    course. I suppose the msync call would take forever, though, so you'd need
    to flip the pointers to and copy data back from a MAP_PRIVATE if you needed
    to keep serving during that time.

    --
  • Bryanturley at Dec 18, 2012 at 9:46 pm ⇧

    On Tuesday, December 18, 2012 3:11:29 PM UTC-6, Kyle Lemons wrote:
    Could you do this with mmap MAP_FIXED? It would be inherently "unsafe" of
    course. I suppose the msync call would take forever, though, so you'd need
    to flip the pointers to and copy data back from a MAP_PRIVATE if you needed
    to keep serving during that time.
    from mmap manpage
    MAP_FIXED
    Don't interpret addr as a hint: place the mapping at exactly
    that address.

    Not sure that is what you really want.

    You could however use a normal mmap'ed file as a backing store for a custom
    map.

    --
  • Kyle Lemons at Dec 18, 2012 at 10:03 pm ⇧

    On Tue, Dec 18, 2012 at 4:46 PM, bryanturley wrote:
    On Tuesday, December 18, 2012 3:11:29 PM UTC-6, Kyle Lemons wrote:

    Could you do this with mmap MAP_FIXED? It would be inherently "unsafe"
    of course. I suppose the msync call would take forever, though, so you'd
    need to flip the pointers to and copy data back from a MAP_PRIVATE if you
    needed to keep serving during that time.
    from mmap manpage
    MAP_FIXED
    Don't interpret addr as a hint: place the mapping at exactly
    that address.

    Not sure that is what you really want.

    You could however use a normal mmap'ed file as a backing store for a
    custom map.
    It could be that I haven't used mmap in a long time, but it seems that you
    could persist pointers to data that's in an mmap region FIXED at a given
    address. This would allow you to write a reasonably straightforward
    allocator around the mmap region, and all you would have to write to
    another file would be the key/pointer pairs. To reconstitute the data, you
    load the mmap up at the same address and populate the map with the same
    pointers and voila. (Yes, I realize that this is "unsafe".)

    --
  • Bryanturley at Dec 19, 2012 at 2:49 am ⇧

    On Tuesday, December 18, 2012 4:02:59 PM UTC-6, Kyle Lemons wrote:
    On Tue, Dec 18, 2012 at 4:46 PM, bryanturley <[email protected]<javascript:>
    wrote:
    On Tuesday, December 18, 2012 3:11:29 PM UTC-6, Kyle Lemons wrote:

    Could you do this with mmap MAP_FIXED? It would be inherently "unsafe"
    of course. I suppose the msync call would take forever, though, so you'd
    need to flip the pointers to and copy data back from a MAP_PRIVATE if you
    needed to keep serving during that time.
    from mmap manpage
    MAP_FIXED
    Don't interpret addr as a hint: place the mapping at
    exactly that address.

    Not sure that is what you really want.

    You could however use a normal mmap'ed file as a backing store for a
    custom map.
    It could be that I haven't used mmap in a long time, but it seems that you
    could persist pointers to data that's in an mmap region FIXED at a given
    address. This would allow you to write a reasonably straightforward
    allocator around the mmap region, and all you would have to write to
    another file would be the key/pointer pairs. To reconstitute the data, you
    load the mmap up at the same address and populate the map with the same
    pointers and voila. (Yes, I realize that this is "unsafe".)
    Ah, I see I did that once actually a good bit ago.
    Real problem with that would be go has no way to guarantee the size/shape
    of a struct.
    gcc (and other c compilers) have extra optional attribs to force structs to
    be as compact as possible.

    I do think there is an offsetof() in reflect though so you could panic if
    the running struct definition is different from the struct definition your
    loading from disk.
    Also I don't know what the GC does with the contents of an mmap region?
    does it scan them for pointers outside the region? obviously it can't reap
    memory from inside the region.

    MAP_FIXED sounds to iffy to me today though, probably just less adventurous
    than I used to be ;)

    --
  • Kyle Lemons at Dec 19, 2012 at 3:58 pm ⇧

    On Tue, Dec 18, 2012 at 9:49 PM, bryanturley wrote:
    On Tuesday, December 18, 2012 4:02:59 PM UTC-6, Kyle Lemons wrote:
    On Tue, Dec 18, 2012 at 4:46 PM, bryanturley wrote:
    On Tuesday, December 18, 2012 3:11:29 PM UTC-6, Kyle Lemons wrote:

    Could you do this with mmap MAP_FIXED? It would be inherently "unsafe"
    of course. I suppose the msync call would take forever, though, so you'd
    need to flip the pointers to and copy data back from a MAP_PRIVATE if you
    needed to keep serving during that time.
    from mmap manpage
    MAP_FIXED
    Don't interpret addr as a hint: place the mapping at
    exactly that address.

    Not sure that is what you really want.

    You could however use a normal mmap'ed file as a backing store for a
    custom map.
    It could be that I haven't used mmap in a long time, but it seems that
    you could persist pointers to data that's in an mmap region FIXED at a
    given address. This would allow you to write a reasonably straightforward
    allocator around the mmap region, and all you would have to write to
    another file would be the key/pointer pairs. To reconstitute the data, you
    load the mmap up at the same address and populate the map with the same
    pointers and voila. (Yes, I realize that this is "unsafe".)
    Ah, I see I did that once actually a good bit ago.
    Real problem with that would be go has no way to guarantee the size/shape
    of a struct.
    It's probably not written down anywhere, but I would be surprised if gc
    would ever generate two different layouts for the same struct. This is, of
    course, a more-than-legitimate worry if you might have to change the
    definition of the struct after creating 10gb of data; gob, for instance,
    has well-defined rules for how this works.

    gcc (and other c compilers) have extra optional attribs to force structs
    to be as compact as possible.

    I do think there is an offsetof() in reflect though so you could panic if
    the running struct definition is different from the struct definition your
    loading from disk.
    The unsafe package has a sizeof call that could be a first-line-of-defense
    if you stored the struct sizes in the key/pointer map file for comparison
    with the mmap. Reflecting on the fields and checking alignment would be
    even more rigorous.

    Also I don't know what the GC does with the contents of an mmap region?
    does it scan them for pointers outside the region? obviously it can't reap
    memory from inside the region.
    You'd have to map the region outside Go's virtual arena, so the GC wouldn't
    touch it. You'd have to explicitly allocate and free anything in the
    region yourself in terms of your own allocator.

    MAP_FIXED sounds to iffy to me today though, probably just less
    adventurous than I used to be ;)
    It sounds like a fun weekend project, but yeah I'm not sure if the
    inflexibility is worth the potential performance gain.

    --
  • Hamish Ogilvy at Dec 20, 2012 at 12:50 am ⇧
    Nate - Looking at testing some SSD's. Any suggestion on providers?
    Softlayer has a decent deal on right
    now: http://www.softlayer.com/info/special-ssd-bw

    Andre - Agree, the addresses will be different. That was one of the
    attractions to gob originally though, as it recreates the structure with
    new addresses for you.

    Clive - You are spot on, the memory forking time becomes annoying at 10GB.
    In C we just live with it, the data store is unavailable for a short time
    while the forking is happening. It's a useful strategy to a point, as this
    also enables us to handle read queries while writing. Unfortunately though
    the memory divergence and fork time are frustrating (especially under high
    write load). Go was attractive immediately as we can read/write multicore
    without complex code or the forking issues. We just need to think about how
    to back the data up and we're home. This thread has been useful and has
    confirmed our approaches are right, we just need to put in the time! We
    aren't actively working on the C code right now, but feel free to email me
    if you'd like to know more!


    On Thursday, 20 December 2012 02:58:33 UTC+11, Kyle Lemons wrote:

    On Tue, Dec 18, 2012 at 9:49 PM, bryanturley <[email protected]<javascript:>
    wrote:
    On Tuesday, December 18, 2012 4:02:59 PM UTC-6, Kyle Lemons wrote:
    On Tue, Dec 18, 2012 at 4:46 PM, bryanturley wrote:
    On Tuesday, December 18, 2012 3:11:29 PM UTC-6, Kyle Lemons wrote:

    Could you do this with mmap MAP_FIXED? It would be inherently
    "unsafe" of course. I suppose the msync call would take forever, though,
    so you'd need to flip the pointers to and copy data back from a MAP_PRIVATE
    if you needed to keep serving during that time.
    from mmap manpage
    MAP_FIXED
    Don't interpret addr as a hint: place the mapping at
    exactly that address.

    Not sure that is what you really want.

    You could however use a normal mmap'ed file as a backing store for a
    custom map.
    It could be that I haven't used mmap in a long time, but it seems that
    you could persist pointers to data that's in an mmap region FIXED at a
    given address. This would allow you to write a reasonably straightforward
    allocator around the mmap region, and all you would have to write to
    another file would be the key/pointer pairs. To reconstitute the data, you
    load the mmap up at the same address and populate the map with the same
    pointers and voila. (Yes, I realize that this is "unsafe".)
    Ah, I see I did that once actually a good bit ago.
    Real problem with that would be go has no way to guarantee the size/shape
    of a struct.
    It's probably not written down anywhere, but I would be surprised if gc
    would ever generate two different layouts for the same struct. This is, of
    course, a more-than-legitimate worry if you might have to change the
    definition of the struct after creating 10gb of data; gob, for instance,
    has well-defined rules for how this works.

    gcc (and other c compilers) have extra optional attribs to force structs
    to be as compact as possible.

    I do think there is an offsetof() in reflect though so you could panic if
    the running struct definition is different from the struct definition your
    loading from disk.
    The unsafe package has a sizeof call that could be a first-line-of-defense
    if you stored the struct sizes in the key/pointer map file for comparison
    with the mmap. Reflecting on the fields and checking alignment would be
    even more rigorous.

    Also I don't know what the GC does with the contents of an mmap region?
    does it scan them for pointers outside the region? obviously it can't reap
    memory from inside the region.
    You'd have to map the region outside Go's virtual arena, so the GC
    wouldn't touch it. You'd have to explicitly allocate and free anything in
    the region yourself in terms of your own allocator.

    MAP_FIXED sounds to iffy to me today though, probably just less
    adventurous than I used to be ;)
    It sounds like a fun weekend project, but yeah I'm not sure if the
    inflexibility is worth the potential performance gain.
    --
  • Nate Finch at Dec 20, 2012 at 2:54 am ⇧
    Newegg.com is generally one of the cheapest places to buy computer hardware, and they're very reliable. I don't know soft layer.
    As for RAM SSDs, I think it's overkill, and I'm not convinced they'd be trustworthy enough with critical data (and all data is critical).

    --
  • Itmitica at Dec 20, 2012 at 5:30 am ⇧
    Flash-based SSDs have a great predisposition for failure due to wearing.
    That makes them not trustworthy enough.

    RAM-based SSDs don't share this wear problem, and they usually have
    batteries to prevent loss of data in case of accidental power off.

    My choice, reliability-wise, would always be RAM-based SSDs. Economically,
    I have to settle for Flash-based SSDs.

    --
  • Bryanturley at Dec 20, 2012 at 4:44 pm ⇧

    On Wednesday, December 19, 2012 11:30:16 PM UTC-6, [email protected] wrote:
    Flash-based SSDs have a great predisposition for failure due to wearing.
    That makes them not trustworthy enough.

    RAM-based SSDs don't share this wear problem, and they usually have
    batteries to prevent loss of data in case of accidental power off.

    My choice, reliability-wise, would always be RAM-based SSDs. Economically,
    I have to settle for Flash-based SSDs.
    flash is a _Random _Access non-volatile _Memory so I am assuming you meant
    some of the volatile DRAM storage devices.
    volatile long-term storage is scary.

    I think any drive that does not have seek times dictated by a spinning
    platter/moving arm are going to a far better solution, just write it out as
    it goes and let the kernels storage stuff handle when it actually hits the
    disk.
    It does a great job optimizing writes when it can, and you can sync
    periodically if you feel you need to.
    Perhaps soft raid 5 or 10 a few of them together if you need more speed.

    Also after reading yesterday btrfs has a atomic clone operation that could
    be used for making backups of the datafiles in the background.
    I have never used it though so you will have to read up on it, You will
    also have to write some code using syscall to use it since that is not a
    common fs operation.
    Shouldn't be to hard though.

    --
  • Bryanturley at Dec 20, 2012 at 4:51 pm ⇧
    Have you thought about using an already written database?
    local or server?


    --
  • Hamish Ogilvy at Dec 20, 2012 at 11:05 pm ⇧
    Thanks Bryan, brtfs looks interesting, will keep an eye on it. Sounds like
    it isn't stable enough yet though.

    As for using an already written database, no can do. We're doing something
    a bit different, it doesn't quite fit with standard DB's, so we wrote our
    own. The system is a similarity engine that accepts small or very large
    queries (like multiple full documents) and self-learns concepts and desired
    context based on user feedback (dynamic query boosting, like Solr, but
    automated and for much larger queries). It supports geo, metrics and lots
    more. Can be used for document search like patents, legal, etc. Also for
    apps like Prismatic, context aware filtering, etc...


    On 21 December 2012 03:51, bryanturley wrote:

    Have you thought about using an already written database?
    local or server?


    --

    --
  • Bryanturley at Dec 20, 2012 at 4:27 pm ⇧

    On Wednesday, December 19, 2012 9:58:33 AM UTC-6, Kyle Lemons wrote:
    On Tue, Dec 18, 2012 at 9:49 PM, bryanturley <[email protected]<javascript:>
    wrote:
    On Tuesday, December 18, 2012 4:02:59 PM UTC-6, Kyle Lemons wrote:
    On Tue, Dec 18, 2012 at 4:46 PM, bryanturley wrote:
    On Tuesday, December 18, 2012 3:11:29 PM UTC-6, Kyle Lemons wrote:

    Could you do this with mmap MAP_FIXED? It would be inherently
    "unsafe" of course. I suppose the msync call would take forever, though,
    so you'd need to flip the pointers to and copy data back from a MAP_PRIVATE
    if you needed to keep serving during that time.
    from mmap manpage
    MAP_FIXED
    Don't interpret addr as a hint: place the mapping at
    exactly that address.

    Not sure that is what you really want.

    You could however use a normal mmap'ed file as a backing store for a
    custom map.
    It could be that I haven't used mmap in a long time, but it seems that
    you could persist pointers to data that's in an mmap region FIXED at a
    given address. This would allow you to write a reasonably straightforward
    allocator around the mmap region, and all you would have to write to
    another file would be the key/pointer pairs. To reconstitute the data, you
    load the mmap up at the same address and populate the map with the same
    pointers and voila. (Yes, I realize that this is "unsafe".)
    Ah, I see I did that once actually a good bit ago.
    Real problem with that would be go has no way to guarantee the size/shape
    of a struct.
    It's probably not written down anywhere, but I would be surprised if gc
    would ever generate two different layouts for the same struct. This is, of
    course, a more-than-legitimate worry if you might have to change the
    definition of the struct after creating 10gb of data; gob, for instance,
    has well-defined rules for how this works.
    Well think of just recently the slices indices are 64bit now, if you had
    slices directly in structs the struct sized just changed.
    And if you moved from x86 to arm the structs will probably change
    shape/size again.

    gcc (and other c compilers) have extra optional attribs to force structs
    to be as compact as possible.

    I do think there is an offsetof() in reflect though so you could panic if
    the running struct definition is different from the struct definition your
    loading from disk.
    The unsafe package has a sizeof call that could be a first-line-of-defense
    if you stored the struct sizes in the key/pointer map file for comparison
    with the mmap. Reflecting on the fields and checking alignment would be
    even more rigorous.
    Yeah sizeof does the size and offsetof does the shape, the compilers do
    some padding at least on the end of structs that I have noticed.

    --
  • Hamish Ogilvy at Dec 18, 2012 at 11:04 pm ⇧
    Thanks Brian and Kyle.

    Brian - Interesting comments re: the kernel changes also. I'm close to
    thinking our own structure that writes out to disk is the way forward also.
    We need to think it through a bit more first though, as the read/write rate
    can get pretty high, so we would need to think very carefully about how the
    disk writes were done, as it could become the new bottleneck pretty quickly
    if not done right.

    Kyle - I think the memory to file mapping can be done, but intuition says
    it's risky. We're still investigating gommap, we will see.

    --
  • Bryanturley at Dec 19, 2012 at 2:51 am ⇧

    On Tuesday, December 18, 2012 5:03:57 PM UTC-6, Hamish Ogilvy wrote:
    Thanks Brian and Kyle.

    Brian - Interesting comments re: the kernel changes also.
    To be clear brYan was talking about changing io stuff in /sys and /proc
    while the system was running not changing the actual kernel source.

    ;)

    --
  • Nate Finch at Dec 19, 2012 at 8:25 am ⇧
    Get an SSD and do all read/writes off disk. You'll be network bound before
    you're disk bound. If you have 10 gigs of data, it's time to stop storing
    it all in memory.
    On Monday, December 17, 2012 7:32:54 PM UTC-5, Hamish Ogilvy wrote:

    We've got an in memory data store comprised of several large maps (up to
    ~50 million "unit32 > pointer") and various linking data structs which
    totals up to ~10GB. This data also has a fairly high read and write
    rate. We'd like to be able to backup this whole structure to disk
    periodically, but haven't managed to find an easy way to do this so far.

    We've tried gob encoding to disk, but the encoding fails in the GB range
    and this also has a huge memory overhead for the file sizes it does work
    for. JSON encoding does not support maps, so it doesn't help. Previously
    this program was written in C, we would fork the memory and then simply
    write the data in the fork to disk. In C, most of our lookups were based on
    offsets, so this was pretty simple, but with Go we're using slices, so we
    don't have as much control on the memory location and need to preserve
    structure. I'm looking at gommap now to see if we can do something similar,
    but i can't help but think others have dealt with this issue, so i thought
    i would post on here for any suggestions?

    And thanks to all for the useful discussions on here, they've been very
    helpful.
    --
  • Itmitica at Dec 19, 2012 at 12:49 pm ⇧
    I assume you mean Flash-based SSDs. RAM-based SSDs seems like what they
    need, if they can afford it. That's how they can store it all in memory.
    On Wednesday, December 19, 2012 10:24:57 AM UTC+2, Nate Finch wrote:

    Get an SSD and do all read/writes off disk. You'll be network bound before
    you're disk bound. If you have 10 gigs of data, it's time to stop storing
    it all in memory.
    --

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupgolang-nuts @
categoriesgo
postedDec 18, '12 at 3:38a
activeDec 20, '12 at 11:05p
posts22
users8
websitegolang.org

8 users in discussion

Bryanturley: 7 posts Hamish Ogilvy: 5 posts Kyle Lemons: 3 posts Nate Finch: 2 posts Itmitica: 2 posts Kevin Gillette: 1 post André Moraes: 1 post Clive Saha: 1 post

Content

People

Support

Translate

site design / logo © 2023 Grokbase