FAQ
Posting this here because I think it's too vague a question for the IRC
channel.

I'm building a toy key-value store. The key-value store is all in-memory
with a background goroutine that picks up the changes and persist them to
disk. When a call to `Put` is made, the affected value is flagged as Dirty
and eventually persisted by the background goroutine.

The API of my store takes <string, []byte> pairs. My concern is that right
now, a client can give me a slice of byte, then modify it without me
picking up the change. It also modifies the value for all other clients (I
want my store to be concurrent-safe).

So the following test fails:

http://play.golang.org/p/Ru-Uk4weew

You can see it in context here [1]:

https://github.com/aybabtme/dskvs/blob/proto/dskvs_test.go#L296

The opposite is also true, I can currently modify the inner bytes of the
store by modifying the slice it returns me (see [2]).

I'm wondering which approach I should take:

    1. Let the clients shoot themselves in the foot if they modify the value
    they receive
    2. Give the clients copies for every request?
    3. Ask the clients to provide their own buffer on which I write. This
    would give freedom to chose between speed/unsafe (1) or slow/safe (2), at
    the cost of an uglier API.
    4. ... something else?


[1]: related question is that I'm getting lost as to how I should organize
my test code in an idiomatic way. I find that dskvs_test.go is getting
bloated. I'm also not quite sure of the approach I should take to properly
exercise the correct concurrent behavior of my library during tests; I have
one such test that is a naive attempt at it [3].

[2]: https://github.com/aybabtme/dskvs/blob/proto/dskvs_test.go#L331

[3]: https://github.com/aybabtme/dskvs/blob/proto/dskvs_test.go#L617

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Search Discussions

  • Antoine Grondin at Jul 2, 2013 at 4:04 am
    I'll add, the cost of doing (2) for read/written data is an increase by 50%
    in response times, from 10s to 15s for 100M read/write queries.
    On Monday, July 1, 2013 11:56:11 PM UTC-4, Antoine Grondin wrote:

    Posting this here because I think it's too vague a question for the IRC
    channel.

    I'm building a toy key-value store. The key-value store is all in-memory
    with a background goroutine that picks up the changes and persist them to
    disk. When a call to `Put` is made, the affected value is flagged as Dirty
    and eventually persisted by the background goroutine.

    The API of my store takes <string, []byte> pairs. My concern is that
    right now, a client can give me a slice of byte, then modify it without me
    picking up the change. It also modifies the value for all other clients (I
    want my store to be concurrent-safe).

    So the following test fails:

    http://play.golang.org/p/Ru-Uk4weew

    You can see it in context here [1]:

    https://github.com/aybabtme/dskvs/blob/proto/dskvs_test.go#L296

    The opposite is also true, I can currently modify the inner bytes of the
    store by modifying the slice it returns me (see [2]).

    I'm wondering which approach I should take:

    1. Let the clients shoot themselves in the foot if they modify the
    value they receive
    2. Give the clients copies for every request?
    3. Ask the clients to provide their own buffer on which I write. This
    would give freedom to chose between speed/unsafe (1) or slow/safe (2), at
    the cost of an uglier API.
    4. ... something else?


    [1]: related question is that I'm getting lost as to how I should organize
    my test code in an idiomatic way. I find that dskvs_test.go is getting
    bloated. I'm also not quite sure of the approach I should take to properly
    exercise the correct concurrent behavior of my library during tests; I have
    one such test that is a naive attempt at it [3].

    [2]: https://github.com/aybabtme/dskvs/blob/proto/dskvs_test.go#L331

    [3]: https://github.com/aybabtme/dskvs/blob/proto/dskvs_test.go#L617
    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.
  • Nigel Tao at Jul 2, 2013 at 6:24 am

    On Tue, Jul 2, 2013 at 1:56 PM, Antoine Grondin wrote:
    Let the clients shoot themselves in the foot if they modify the value they
    receive
    Give the clients copies for every request?
    Ask the clients to provide their own buffer on which I write. This would
    give freedom to chose between speed/unsafe (1) or slow/safe (2), at the cost
    of an uglier API.
    ... something else?
    FWIW, in my unfinished port of LevelDB to Go, for the in-memory
    storage, I copy bytes when putting things into the database, and
    return live references when reading out of the database. See the "The
    caller should not modify the contents of the returned slice" comment
    at https://code.google.com/p/leveldb-go/source/browse/leveldb/db/db.go

    There was a design discussion a couple of months ago on adding
    "read-only slices" to the language, which would make it less likely
    for users to shoot themselves in the foot even if you don't copy. The
    discussion is at
    https://groups.google.com/forum/#!topic/golang-dev/Y7j4B2r_eDw and
    there might have been other spin-off discussions on golang-dev around
    that time.

    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.
  • Egon at Jul 2, 2013 at 8:38 am

    On Tuesday, July 2, 2013 6:56:11 AM UTC+3, Antoine Grondin wrote:
    Posting this here because I think it's too vague a question for the IRC
    channel.

    I'm building a toy key-value store. The key-value store is all in-memory
    with a background goroutine that picks up the changes and persist them to
    disk. When a call to `Put` is made, the affected value is flagged as Dirty
    and eventually persisted by the background goroutine.

    The API of my store takes <string, []byte> pairs. My concern is that
    right now, a client can give me a slice of byte, then modify it without me
    picking up the change. It also modifies the value for all other clients (I
    want my store to be concurrent-safe).

    So the following test fails:

    http://play.golang.org/p/Ru-Uk4weew

    You can see it in context here [1]:

    https://github.com/aybabtme/dskvs/blob/proto/dskvs_test.go#L296

    The opposite is also true, I can currently modify the inner bytes of the
    store by modifying the slice it returns me (see [2]).

    I'm wondering which approach I should take:

    1. Let the clients shoot themselves in the foot if they modify the
    value they receive
    2. Give the clients copies for every request?
    3. Ask the clients to provide their own buffer on which I write. This
    would give freedom to chose between speed/unsafe (1) or slow/safe (2), at
    the cost of an uglier API.
    4. ... something else?

    4. Make the Put/Get blocking to the outside world, that way the user
    "knows" the data is being used by the API. If the user needs async then
    they can always use "go db.Put(data)" and be aware it's being used
    concurrently.


    [1]: related question is that I'm getting lost as to how I should organize
    my test code in an idiomatic way. I find that dskvs_test.go is getting
    bloated. I'm also not quite sure of the approach I should take to properly
    exercise the correct concurrent behavior of my library during tests; I have
    one such test that is a naive attempt at it [3].

    [2]: https://github.com/aybabtme/dskvs/blob/proto/dskvs_test.go#L331

    [3]: https://github.com/aybabtme/dskvs/blob/proto/dskvs_test.go#L617
    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.
  • Antoine Grondin at Jul 4, 2013 at 8:20 am
    Thanks Nigel, I remember reading about this proposal. Also the way
    you deal with it in LevelDB is interesting.

    @egon, the data is locked for writes by a RWMutex when the store
    happens to persists them, or when I copy the bytes in a safe area (my
    current fix to this issue). Aside from that, most calls to the store
    are not blocking.

    Cheers,

    --
    Antoine Grondin

    Live long and prosper

    On Tue, Jul 2, 2013 at 4:38 AM, egon wrote:

    On Tuesday, July 2, 2013 6:56:11 AM UTC+3, Antoine Grondin wrote:

    Posting this here because I think it's too vague a question for the IRC
    channel.

    I'm building a toy key-value store. The key-value store is all in-memory
    with a background goroutine that picks up the changes and persist them to
    disk. When a call to `Put` is made, the affected value is flagged as Dirty
    and eventually persisted by the background goroutine.

    The API of my store takes <string, []byte> pairs. My concern is that
    right now, a client can give me a slice of byte, then modify it without me
    picking up the change. It also modifies the value for all other clients (I
    want my store to be concurrent-safe).

    So the following test fails:

    http://play.golang.org/p/Ru-Uk4weew

    You can see it in context here [1]:

    https://github.com/aybabtme/dskvs/blob/proto/dskvs_test.go#L296

    The opposite is also true, I can currently modify the inner bytes of the
    store by modifying the slice it returns me (see [2]).

    I'm wondering which approach I should take:

    Let the clients shoot themselves in the foot if they modify the value they
    receive
    Give the clients copies for every request?
    Ask the clients to provide their own buffer on which I write. This would
    give freedom to chose between speed/unsafe (1) or slow/safe (2), at the cost
    of an uglier API.
    ... something else?
    4. Make the Put/Get blocking to the outside world, that way the user "knows"
    the data is being used by the API. If the user needs async then they can
    always use "go db.Put(data)" and be aware it's being used concurrently.

    [1]: related question is that I'm getting lost as to how I should organize
    my test code in an idiomatic way. I find that dskvs_test.go is getting
    bloated. I'm also not quite sure of the approach I should take to properly
    exercise the correct concurrent behavior of my library during tests; I have
    one such test that is a naive attempt at it [3].

    [2]: https://github.com/aybabtme/dskvs/blob/proto/dskvs_test.go#L331

    [3]: https://github.com/aybabtme/dskvs/blob/proto/dskvs_test.go#L617
    --
    You received this message because you are subscribed to a topic in the
    Google Groups "golang-nuts" group.
    To unsubscribe from this topic, visit
    https://groups.google.com/d/topic/golang-nuts/N6NvENtUGao/unsubscribe.
    To unsubscribe from this group and all its topics, send an email to
    golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.
    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.
  • Tomwilde at Jul 4, 2013 at 1:22 pm

    The key-value store is all in-memory with a background goroutine that
    picks up the changes and persist them to disk.

    I don't see a call to syscall.Mlock or syscall.Mlockall in your source
    code. You are susceptible to swapping/paging.

    Cheers!

    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupgolang-nuts @
categoriesgo
postedJul 2, '13 at 3:56a
activeJul 4, '13 at 1:22p
posts6
users4
websitegolang.org

People

Translate

site design / logo © 2021 Grokbase