FAQ
Very frequently people ask questions of the form

"I have set X, and hash table key Y, and want all key/value pairs from Y
where key is in X"

example:

hset users "1" "{'name': 'bob', 'age': 45}" "2" {'name': 'sally', 'age':
22}" [...]
sadd userlist 1 3 22 112 [...]

# now I want to issue one command to get all of the JSON blobs where

or

"I have set X, and a collection of keys named with convention "Y:A", and
want all key/value pairs where for a given Y, A is in X"

example:

hset user:1 name bob age 45
hset user:2 name sally age 22
[...]

sadd userlist 1 3 22 112

# now I want to issue one command to hgetall all of the k/v pairs from
user:N where N in userlist

and there are various mixtures of the two approaches, all of which suffer
from either or both of the approaches' problems.

The former form (storing JSON- or otherwise-serialized- string data) sees a
lot of use because many programming problems involve nested objects/tables,
and although you lose the cleanliness and random attribute access and space
efficiency of the latter form, trying to emulate nested behavior quickly
becomes quite difficult.

In both cases, there's no immediately obvious answer to the problem that
doesn't involve multiple roundtrips. In declining order of goodness it
sort of goes like this:

solution #1 (using the app to build the second command from results of
first)
----------------

x = r.smembers "userlist"
results = r.hgetall "users" x

# two round trips
# note that results is not a hash but an array of values, so you have to
then match up your values with x
# note also that large values of x may cause issues

solution #2 (using lua)
----------------

hmgetviaset = <<LUA
local keys = redis.call("smembers", KEYS[2])
local ret = {}
for i,v in ipairs(keys) do
ret[i] = {i, redis.call("hmget", KEYS[1], i)}
end
return ret
LUA

r.hset "users", "1", "bobserialized"
r.hset "users", "2", "sallyserialized"
r.sadd "userlist", 1
r.sadd "userlist", 2

puts r.eval(hmgetviaset, ["users", "userlist"])

# one round trip
# requires that you are a member of this mailing list to obtain and/or
understand this script and/or enough knowledge to write it yourself

solution #3 (the naive, obvious solution; using the app to iterate)
-----------------

x = r.smembers userlist
x.map!{|i| r.hmget "users" i}

# possibly many round trips
# didn't include pipeline code because pipelining can be faster but causes
issues for even medium values of x


All of these solutions have goodness and badness in them; the app solution
is fine but is a bit finicky, especially for larger values of x, and
requires you to pass the entire set through the network twice before redis
will start to give you your solution. The lua solution is okay but
obviously for large values of x will impact the server, and performs a
number of unnecessary duplications as well; and additionally is arcane and
esoteric even for an arcane and esoteric database platform, and absent a
lot of currently nonexistent documentation is going to be tough for anyone
to get to without asking questions on this forum. Obviously the client
solution is not, shall we say, web scale.

So in a roundabout way, let me bring up the idea that redis could benefit
by understanding natively what a foreign key is, either by providing some
dynamic means of identifying foreign keys, or by having commands that
natively understand foreign keys.

for example:

HMGET "hashkey" USING "setname"

'USING' being a new keyword that uses all of the values of SETNAME as keys
for the HMGET.

or

HMSGET "hashkey" "setname"

etc., etc.

Thoughts?

F.

--
You received this message because you are subscribed to the Google Groups "Redis DB" group.
To post to this group, send email to redis-db@googlegroups.com.
To unsubscribe from this group, send email to redis-db+unsubscribe@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/redis-db?hl=en.

Search Discussions

  • Josiah Carlson at Nov 19, 2012 at 9:00 pm
    If you know the keys to your hash, you can use SORT with multiple GET
    parameters.

    That is 1 round-trip without scripting.

    - Josiah
    On Mon, Nov 19, 2012 at 11:32 AM, Felix Gallo wrote:
    Very frequently people ask questions of the form

    "I have set X, and hash table key Y, and want all key/value pairs from Y
    where key is in X"

    example:

    hset users "1" "{'name': 'bob', 'age': 45}" "2" {'name': 'sally', 'age':
    22}" [...]
    sadd userlist 1 3 22 112 [...]

    # now I want to issue one command to get all of the JSON blobs where

    or

    "I have set X, and a collection of keys named with convention "Y:A", and
    want all key/value pairs where for a given Y, A is in X"

    example:

    hset user:1 name bob age 45
    hset user:2 name sally age 22
    [...]

    sadd userlist 1 3 22 112

    # now I want to issue one command to hgetall all of the k/v pairs from
    user:N where N in userlist

    and there are various mixtures of the two approaches, all of which suffer
    from either or both of the approaches' problems.

    The former form (storing JSON- or otherwise-serialized- string data) sees a
    lot of use because many programming problems involve nested objects/tables,
    and although you lose the cleanliness and random attribute access and space
    efficiency of the latter form, trying to emulate nested behavior quickly
    becomes quite difficult.

    In both cases, there's no immediately obvious answer to the problem that
    doesn't involve multiple roundtrips. In declining order of goodness it sort
    of goes like this:

    solution #1 (using the app to build the second command from results of
    first)
    ----------------

    x = r.smembers "userlist"
    results = r.hgetall "users" x

    # two round trips
    # note that results is not a hash but an array of values, so you have to
    then match up your values with x
    # note also that large values of x may cause issues

    solution #2 (using lua)
    ----------------

    hmgetviaset = <<LUA
    local keys = redis.call("smembers", KEYS[2])
    local ret = {}
    for i,v in ipairs(keys) do
    ret[i] = {i, redis.call("hmget", KEYS[1], i)}
    end
    return ret
    LUA

    r.hset "users", "1", "bobserialized"
    r.hset "users", "2", "sallyserialized"
    r.sadd "userlist", 1
    r.sadd "userlist", 2

    puts r.eval(hmgetviaset, ["users", "userlist"])

    # one round trip
    # requires that you are a member of this mailing list to obtain and/or
    understand this script and/or enough knowledge to write it yourself

    solution #3 (the naive, obvious solution; using the app to iterate)
    -----------------

    x = r.smembers userlist
    x.map!{|i| r.hmget "users" i}

    # possibly many round trips
    # didn't include pipeline code because pipelining can be faster but causes
    issues for even medium values of x


    All of these solutions have goodness and badness in them; the app solution
    is fine but is a bit finicky, especially for larger values of x, and
    requires you to pass the entire set through the network twice before redis
    will start to give you your solution. The lua solution is okay but
    obviously for large values of x will impact the server, and performs a
    number of unnecessary duplications as well; and additionally is arcane and
    esoteric even for an arcane and esoteric database platform, and absent a lot
    of currently nonexistent documentation is going to be tough for anyone to
    get to without asking questions on this forum. Obviously the client
    solution is not, shall we say, web scale.

    So in a roundabout way, let me bring up the idea that redis could benefit by
    understanding natively what a foreign key is, either by providing some
    dynamic means of identifying foreign keys, or by having commands that
    natively understand foreign keys.

    for example:

    HMGET "hashkey" USING "setname"

    'USING' being a new keyword that uses all of the values of SETNAME as keys
    for the HMGET.

    or

    HMSGET "hashkey" "setname"

    etc., etc.

    Thoughts?

    F.

    --
    You received this message because you are subscribed to the Google Groups
    "Redis DB" group.
    To post to this group, send email to redis-db@googlegroups.com.
    To unsubscribe from this group, send email to
    redis-db+unsubscribe@googlegroups.com.
    For more options, visit this group at
    http://groups.google.com/group/redis-db?hl=en.
    --
    You received this message because you are subscribed to the Google Groups "Redis DB" group.
    To post to this group, send email to redis-db@googlegroups.com.
    To unsubscribe from this group, send email to redis-db+unsubscribe@googlegroups.com.
    For more options, visit this group at http://groups.google.com/group/redis-db?hl=en.
  • Dvir Volk at Nov 19, 2012 at 9:03 pm
    Dammit Josiah, you always beat me to the punch ;-)

    sent from my Sinclair ZX48
    On Nov 19, 2012 11:00 PM, "Josiah Carlson" wrote:

    If you know the keys to your hash, you can use SORT with multiple GET
    parameters.

    That is 1 round-trip without scripting.

    - Josiah
    On Mon, Nov 19, 2012 at 11:32 AM, Felix Gallo wrote:
    Very frequently people ask questions of the form

    "I have set X, and hash table key Y, and want all key/value pairs from Y
    where key is in X"

    example:

    hset users "1" "{'name': 'bob', 'age': 45}" "2" {'name': 'sally', 'age':
    22}" [...]
    sadd userlist 1 3 22 112 [...]

    # now I want to issue one command to get all of the JSON blobs where

    or

    "I have set X, and a collection of keys named with convention "Y:A", and
    want all key/value pairs where for a given Y, A is in X"

    example:

    hset user:1 name bob age 45
    hset user:2 name sally age 22
    [...]

    sadd userlist 1 3 22 112

    # now I want to issue one command to hgetall all of the k/v pairs from
    user:N where N in userlist

    and there are various mixtures of the two approaches, all of which suffer
    from either or both of the approaches' problems.

    The former form (storing JSON- or otherwise-serialized- string data) sees a
    lot of use because many programming problems involve nested
    objects/tables,
    and although you lose the cleanliness and random attribute access and space
    efficiency of the latter form, trying to emulate nested behavior quickly
    becomes quite difficult.

    In both cases, there's no immediately obvious answer to the problem that
    doesn't involve multiple roundtrips. In declining order of goodness it sort
    of goes like this:

    solution #1 (using the app to build the second command from results of
    first)
    ----------------

    x = r.smembers "userlist"
    results = r.hgetall "users" x

    # two round trips
    # note that results is not a hash but an array of values, so you have to
    then match up your values with x
    # note also that large values of x may cause issues

    solution #2 (using lua)
    ----------------

    hmgetviaset = <<LUA
    local keys = redis.call("smembers", KEYS[2])
    local ret = {}
    for i,v in ipairs(keys) do
    ret[i] = {i, redis.call("hmget", KEYS[1], i)}
    end
    return ret
    LUA

    r.hset "users", "1", "bobserialized"
    r.hset "users", "2", "sallyserialized"
    r.sadd "userlist", 1
    r.sadd "userlist", 2

    puts r.eval(hmgetviaset, ["users", "userlist"])

    # one round trip
    # requires that you are a member of this mailing list to obtain and/or
    understand this script and/or enough knowledge to write it yourself

    solution #3 (the naive, obvious solution; using the app to iterate)
    -----------------

    x = r.smembers userlist
    x.map!{|i| r.hmget "users" i}

    # possibly many round trips
    # didn't include pipeline code because pipelining can be faster but causes
    issues for even medium values of x


    All of these solutions have goodness and badness in them; the app solution
    is fine but is a bit finicky, especially for larger values of x, and
    requires you to pass the entire set through the network twice before redis
    will start to give you your solution. The lua solution is okay but
    obviously for large values of x will impact the server, and performs a
    number of unnecessary duplications as well; and additionally is arcane and
    esoteric even for an arcane and esoteric database platform, and absent a lot
    of currently nonexistent documentation is going to be tough for anyone to
    get to without asking questions on this forum. Obviously the client
    solution is not, shall we say, web scale.

    So in a roundabout way, let me bring up the idea that redis could
    benefit by
    understanding natively what a foreign key is, either by providing some
    dynamic means of identifying foreign keys, or by having commands that
    natively understand foreign keys.

    for example:

    HMGET "hashkey" USING "setname"

    'USING' being a new keyword that uses all of the values of SETNAME as keys
    for the HMGET.

    or

    HMSGET "hashkey" "setname"

    etc., etc.

    Thoughts?

    F.

    --
    You received this message because you are subscribed to the Google Groups
    "Redis DB" group.
    To post to this group, send email to redis-db@googlegroups.com.
    To unsubscribe from this group, send email to
    redis-db+unsubscribe@googlegroups.com.
    For more options, visit this group at
    http://groups.google.com/group/redis-db?hl=en.
    --
    You received this message because you are subscribed to the Google Groups
    "Redis DB" group.
    To post to this group, send email to redis-db@googlegroups.com.
    To unsubscribe from this group, send email to
    redis-db+unsubscribe@googlegroups.com.
    For more options, visit this group at
    http://groups.google.com/group/redis-db?hl=en.
    --
    You received this message because you are subscribed to the Google Groups "Redis DB" group.
    To post to this group, send email to redis-db@googlegroups.com.
    To unsubscribe from this group, send email to redis-db+unsubscribe@googlegroups.com.
    For more options, visit this group at http://groups.google.com/group/redis-db?hl=en.
  • Josiah Carlson at Nov 19, 2012 at 9:19 pm
    I gave the world an hour and a half to beat me... We had a fire drill,
    I went to lunch, and we took an office photo. Maybe next time I'll get
    in a game of ping-pong before replying just to give you a shot ;)

    - Josiah
    On Mon, Nov 19, 2012 at 1:03 PM, Dvir Volk wrote:
    Dammit Josiah, you always beat me to the punch ;-)

    sent from my Sinclair ZX48
    On Nov 19, 2012 11:00 PM, "Josiah Carlson" wrote:

    If you know the keys to your hash, you can use SORT with multiple GET
    parameters.

    That is 1 round-trip without scripting.

    - Josiah

    On Mon, Nov 19, 2012 at 11:32 AM, Felix Gallo <felixgallo@gmail.com>
    wrote:
    Very frequently people ask questions of the form

    "I have set X, and hash table key Y, and want all key/value pairs from Y
    where key is in X"

    example:

    hset users "1" "{'name': 'bob', 'age': 45}" "2" {'name': 'sally', 'age':
    22}" [...]
    sadd userlist 1 3 22 112 [...]

    # now I want to issue one command to get all of the JSON blobs where

    or

    "I have set X, and a collection of keys named with convention "Y:A", and
    want all key/value pairs where for a given Y, A is in X"

    example:

    hset user:1 name bob age 45
    hset user:2 name sally age 22
    [...]

    sadd userlist 1 3 22 112

    # now I want to issue one command to hgetall all of the k/v pairs from
    user:N where N in userlist

    and there are various mixtures of the two approaches, all of which
    suffer
    from either or both of the approaches' problems.

    The former form (storing JSON- or otherwise-serialized- string data)
    sees a
    lot of use because many programming problems involve nested
    objects/tables,
    and although you lose the cleanliness and random attribute access and
    space
    efficiency of the latter form, trying to emulate nested behavior quickly
    becomes quite difficult.

    In both cases, there's no immediately obvious answer to the problem that
    doesn't involve multiple roundtrips. In declining order of goodness it
    sort
    of goes like this:

    solution #1 (using the app to build the second command from results of
    first)
    ----------------

    x = r.smembers "userlist"
    results = r.hgetall "users" x

    # two round trips
    # note that results is not a hash but an array of values, so you have to
    then match up your values with x
    # note also that large values of x may cause issues

    solution #2 (using lua)
    ----------------

    hmgetviaset = <<LUA
    local keys = redis.call("smembers", KEYS[2])
    local ret = {}
    for i,v in ipairs(keys) do
    ret[i] = {i, redis.call("hmget", KEYS[1], i)}
    end
    return ret
    LUA

    r.hset "users", "1", "bobserialized"
    r.hset "users", "2", "sallyserialized"
    r.sadd "userlist", 1
    r.sadd "userlist", 2

    puts r.eval(hmgetviaset, ["users", "userlist"])

    # one round trip
    # requires that you are a member of this mailing list to obtain and/or
    understand this script and/or enough knowledge to write it yourself

    solution #3 (the naive, obvious solution; using the app to iterate)
    -----------------

    x = r.smembers userlist
    x.map!{|i| r.hmget "users" i}

    # possibly many round trips
    # didn't include pipeline code because pipelining can be faster but
    causes
    issues for even medium values of x


    All of these solutions have goodness and badness in them; the app
    solution
    is fine but is a bit finicky, especially for larger values of x, and
    requires you to pass the entire set through the network twice before
    redis
    will start to give you your solution. The lua solution is okay but
    obviously for large values of x will impact the server, and performs a
    number of unnecessary duplications as well; and additionally is arcane
    and
    esoteric even for an arcane and esoteric database platform, and absent a
    lot
    of currently nonexistent documentation is going to be tough for anyone
    to
    get to without asking questions on this forum. Obviously the client
    solution is not, shall we say, web scale.

    So in a roundabout way, let me bring up the idea that redis could
    benefit by
    understanding natively what a foreign key is, either by providing some
    dynamic means of identifying foreign keys, or by having commands that
    natively understand foreign keys.

    for example:

    HMGET "hashkey" USING "setname"

    'USING' being a new keyword that uses all of the values of SETNAME as
    keys
    for the HMGET.

    or

    HMSGET "hashkey" "setname"

    etc., etc.

    Thoughts?

    F.

    --
    You received this message because you are subscribed to the Google
    Groups
    "Redis DB" group.
    To post to this group, send email to redis-db@googlegroups.com.
    To unsubscribe from this group, send email to
    redis-db+unsubscribe@googlegroups.com.
    For more options, visit this group at
    http://groups.google.com/group/redis-db?hl=en.
    --
    You received this message because you are subscribed to the Google Groups
    "Redis DB" group.
    To post to this group, send email to redis-db@googlegroups.com.
    To unsubscribe from this group, send email to
    redis-db+unsubscribe@googlegroups.com.
    For more options, visit this group at
    http://groups.google.com/group/redis-db?hl=en.
    --
    You received this message because you are subscribed to the Google Groups
    "Redis DB" group.
    To post to this group, send email to redis-db@googlegroups.com.
    To unsubscribe from this group, send email to
    redis-db+unsubscribe@googlegroups.com.
    For more options, visit this group at
    http://groups.google.com/group/redis-db?hl=en.
    --
    You received this message because you are subscribed to the Google Groups "Redis DB" group.
    To post to this group, send email to redis-db@googlegroups.com.
    To unsubscribe from this group, send email to redis-db+unsubscribe@googlegroups.com.
    For more options, visit this group at http://groups.google.com/group/redis-db?hl=en.
  • Felix Gallo at Nov 19, 2012 at 9:33 pm
    oh god, just when I thought I had discovered all the dark corners. So
    yeah, that's a fourth option: use the SORT command to retrieve lookthrough
    values. My skin is crawling.

    Further, it looks like that only works in one of the use cases: where each
    record has its own top level redis key.

    So

    set user:1 "bob"
    set user:2 "mary"
    sadd userlist 1 2

    sort userlist by 1 get user:*

    produces the proper response, but in the case where you have a hash, and
    you want to look through into a key, the documentation's proposed syntax
    doesn't appear to work:

    hset users 1 "bob"
    hset users 2 "mary"
    sadd userlist 1 2

    sort userlist by 1 get users->*

    gives back no error but nulls for the data.

    anyway -- if anything the point is strengthened -- there are multiple ways
    to do it but all of them have various grotesque problems of speed,
    explosivity, discoverability, or in this case...sanity...

    F.
    On Mon, Nov 19, 2012 at 1:19 PM, Josiah Carlson wrote:

    I gave the world an hour and a half to beat me... We had a fire drill,
    I went to lunch, and we took an office photo. Maybe next time I'll get
    in a game of ping-pong before replying just to give you a shot ;)

    - Josiah
    On Mon, Nov 19, 2012 at 1:03 PM, Dvir Volk wrote:
    Dammit Josiah, you always beat me to the punch ;-)

    sent from my Sinclair ZX48
    On Nov 19, 2012 11:00 PM, "Josiah Carlson" wrote:

    If you know the keys to your hash, you can use SORT with multiple GET
    parameters.

    That is 1 round-trip without scripting.

    - Josiah

    On Mon, Nov 19, 2012 at 11:32 AM, Felix Gallo <felixgallo@gmail.com>
    wrote:
    Very frequently people ask questions of the form

    "I have set X, and hash table key Y, and want all key/value pairs
    from Y
    where key is in X"

    example:

    hset users "1" "{'name': 'bob', 'age': 45}" "2" {'name': 'sally',
    'age':
    22}" [...]
    sadd userlist 1 3 22 112 [...]

    # now I want to issue one command to get all of the JSON blobs where

    or

    "I have set X, and a collection of keys named with convention "Y:A",
    and
    want all key/value pairs where for a given Y, A is in X"

    example:

    hset user:1 name bob age 45
    hset user:2 name sally age 22
    [...]

    sadd userlist 1 3 22 112

    # now I want to issue one command to hgetall all of the k/v pairs from
    user:N where N in userlist

    and there are various mixtures of the two approaches, all of which
    suffer
    from either or both of the approaches' problems.

    The former form (storing JSON- or otherwise-serialized- string data)
    sees a
    lot of use because many programming problems involve nested
    objects/tables,
    and although you lose the cleanliness and random attribute access and
    space
    efficiency of the latter form, trying to emulate nested behavior
    quickly
    becomes quite difficult.

    In both cases, there's no immediately obvious answer to the problem
    that
    doesn't involve multiple roundtrips. In declining order of goodness
    it
    sort
    of goes like this:

    solution #1 (using the app to build the second command from results of
    first)
    ----------------

    x = r.smembers "userlist"
    results = r.hgetall "users" x

    # two round trips
    # note that results is not a hash but an array of values, so you have
    to
    then match up your values with x
    # note also that large values of x may cause issues

    solution #2 (using lua)
    ----------------

    hmgetviaset = <<LUA
    local keys = redis.call("smembers", KEYS[2])
    local ret = {}
    for i,v in ipairs(keys) do
    ret[i] = {i, redis.call("hmget", KEYS[1], i)}
    end
    return ret
    LUA

    r.hset "users", "1", "bobserialized"
    r.hset "users", "2", "sallyserialized"
    r.sadd "userlist", 1
    r.sadd "userlist", 2

    puts r.eval(hmgetviaset, ["users", "userlist"])

    # one round trip
    # requires that you are a member of this mailing list to obtain and/or
    understand this script and/or enough knowledge to write it yourself

    solution #3 (the naive, obvious solution; using the app to iterate)
    -----------------

    x = r.smembers userlist
    x.map!{|i| r.hmget "users" i}

    # possibly many round trips
    # didn't include pipeline code because pipelining can be faster but
    causes
    issues for even medium values of x


    All of these solutions have goodness and badness in them; the app
    solution
    is fine but is a bit finicky, especially for larger values of x, and
    requires you to pass the entire set through the network twice before
    redis
    will start to give you your solution. The lua solution is okay but
    obviously for large values of x will impact the server, and performs a
    number of unnecessary duplications as well; and additionally is arcane
    and
    esoteric even for an arcane and esoteric database platform, and
    absent a
    lot
    of currently nonexistent documentation is going to be tough for anyone
    to
    get to without asking questions on this forum. Obviously the client
    solution is not, shall we say, web scale.

    So in a roundabout way, let me bring up the idea that redis could
    benefit by
    understanding natively what a foreign key is, either by providing some
    dynamic means of identifying foreign keys, or by having commands that
    natively understand foreign keys.

    for example:

    HMGET "hashkey" USING "setname"

    'USING' being a new keyword that uses all of the values of SETNAME as
    keys
    for the HMGET.

    or

    HMSGET "hashkey" "setname"

    etc., etc.

    Thoughts?

    F.

    --
    You received this message because you are subscribed to the Google
    Groups
    "Redis DB" group.
    To post to this group, send email to redis-db@googlegroups.com.
    To unsubscribe from this group, send email to
    redis-db+unsubscribe@googlegroups.com.
    For more options, visit this group at
    http://groups.google.com/group/redis-db?hl=en.
    --
    You received this message because you are subscribed to the Google
    Groups
    "Redis DB" group.
    To post to this group, send email to redis-db@googlegroups.com.
    To unsubscribe from this group, send email to
    redis-db+unsubscribe@googlegroups.com.
    For more options, visit this group at
    http://groups.google.com/group/redis-db?hl=en.
    --
    You received this message because you are subscribed to the Google Groups
    "Redis DB" group.
    To post to this group, send email to redis-db@googlegroups.com.
    To unsubscribe from this group, send email to
    redis-db+unsubscribe@googlegroups.com.
    For more options, visit this group at
    http://groups.google.com/group/redis-db?hl=en.
    --
    You received this message because you are subscribed to the Google Groups
    "Redis DB" group.
    To post to this group, send email to redis-db@googlegroups.com.
    To unsubscribe from this group, send email to
    redis-db+unsubscribe@googlegroups.com.
    For more options, visit this group at
    http://groups.google.com/group/redis-db?hl=en.
    --
    You received this message because you are subscribed to the Google Groups "Redis DB" group.
    To post to this group, send email to redis-db@googlegroups.com.
    To unsubscribe from this group, send email to redis-db+unsubscribe@googlegroups.com.
    For more options, visit this group at http://groups.google.com/group/redis-db?hl=en.
  • Jesús Gabriel y Galán at Nov 19, 2012 at 9:37 pm

    On Mon, Nov 19, 2012 at 10:33 PM, Felix Gallo wrote:
    oh god, just when I thought I had discovered all the dark corners. So yeah,
    that's a fourth option: use the SORT command to retrieve lookthrough values.
    My skin is crawling.

    Further, it looks like that only works in one of the use cases: where each
    record has its own top level redis key.

    So

    set user:1 "bob"
    set user:2 "mary"
    sadd userlist 1 2

    sort userlist by 1 get user:*

    produces the proper response, but in the case where you have a hash, and you
    want to look through into a key, the documentation's proposed syntax doesn't
    appear to work:

    hset users 1 "bob"
    hset users 2 "mary"
    sadd userlist 1 2

    sort userlist by 1 get users->*

    gives back no error but nulls for the data.

    anyway -- if anything the point is strengthened -- there are multiple ways
    to do it but all of them have various grotesque problems of speed,
    explosivity, discoverability, or in this case...sanity...
    Hi,

    Reading this thread with my problem in the other thread in mind, I got
    this working:

    sort "u:1234::test" BY si:*->m GET si:*->s GET si:*->st

    where u:<id>::<id> is the list of keys, si:* are the hashes, and m, s
    and st hash keys. In my case the order is not important, but anyway I
    had to choose a hash key that could be converted to double.
    I'll test the performance for my use case.

    Thanks,

    Jesus.

    --
    You received this message because you are subscribed to the Google Groups "Redis DB" group.
    To post to this group, send email to redis-db@googlegroups.com.
    To unsubscribe from this group, send email to redis-db+unsubscribe@googlegroups.com.
    For more options, visit this group at http://groups.google.com/group/redis-db?hl=en.
  • Dvir Volk at Nov 19, 2012 at 9:46 pm
    I only saw it because i was bored while walking my dog :)

    sent from my Sinclair ZX48
    On Nov 19, 2012 11:19 PM, "Josiah Carlson" wrote:

    I gave the world an hour and a half to beat me... We had a fire drill,
    I went to lunch, and we took an office photo. Maybe next time I'll get
    in a game of ping-pong before replying just to give you a shot ;)

    - Josiah
    On Mon, Nov 19, 2012 at 1:03 PM, Dvir Volk wrote:
    Dammit Josiah, you always beat me to the punch ;-)

    sent from my Sinclair ZX48
    On Nov 19, 2012 11:00 PM, "Josiah Carlson" wrote:

    If you know the keys to your hash, you can use SORT with multiple GET
    parameters.

    That is 1 round-trip without scripting.

    - Josiah

    On Mon, Nov 19, 2012 at 11:32 AM, Felix Gallo <felixgallo@gmail.com>
    wrote:
    Very frequently people ask questions of the form

    "I have set X, and hash table key Y, and want all key/value pairs
    from Y
    where key is in X"

    example:

    hset users "1" "{'name': 'bob', 'age': 45}" "2" {'name': 'sally',
    'age':
    22}" [...]
    sadd userlist 1 3 22 112 [...]

    # now I want to issue one command to get all of the JSON blobs where

    or

    "I have set X, and a collection of keys named with convention "Y:A",
    and
    want all key/value pairs where for a given Y, A is in X"

    example:

    hset user:1 name bob age 45
    hset user:2 name sally age 22
    [...]

    sadd userlist 1 3 22 112

    # now I want to issue one command to hgetall all of the k/v pairs from
    user:N where N in userlist

    and there are various mixtures of the two approaches, all of which
    suffer
    from either or both of the approaches' problems.

    The former form (storing JSON- or otherwise-serialized- string data)
    sees a
    lot of use because many programming problems involve nested
    objects/tables,
    and although you lose the cleanliness and random attribute access and
    space
    efficiency of the latter form, trying to emulate nested behavior
    quickly
    becomes quite difficult.

    In both cases, there's no immediately obvious answer to the problem
    that
    doesn't involve multiple roundtrips. In declining order of goodness
    it
    sort
    of goes like this:

    solution #1 (using the app to build the second command from results of
    first)
    ----------------

    x = r.smembers "userlist"
    results = r.hgetall "users" x

    # two round trips
    # note that results is not a hash but an array of values, so you have
    to
    then match up your values with x
    # note also that large values of x may cause issues

    solution #2 (using lua)
    ----------------

    hmgetviaset = <<LUA
    local keys = redis.call("smembers", KEYS[2])
    local ret = {}
    for i,v in ipairs(keys) do
    ret[i] = {i, redis.call("hmget", KEYS[1], i)}
    end
    return ret
    LUA

    r.hset "users", "1", "bobserialized"
    r.hset "users", "2", "sallyserialized"
    r.sadd "userlist", 1
    r.sadd "userlist", 2

    puts r.eval(hmgetviaset, ["users", "userlist"])

    # one round trip
    # requires that you are a member of this mailing list to obtain and/or
    understand this script and/or enough knowledge to write it yourself

    solution #3 (the naive, obvious solution; using the app to iterate)
    -----------------

    x = r.smembers userlist
    x.map!{|i| r.hmget "users" i}

    # possibly many round trips
    # didn't include pipeline code because pipelining can be faster but
    causes
    issues for even medium values of x


    All of these solutions have goodness and badness in them; the app
    solution
    is fine but is a bit finicky, especially for larger values of x, and
    requires you to pass the entire set through the network twice before
    redis
    will start to give you your solution. The lua solution is okay but
    obviously for large values of x will impact the server, and performs a
    number of unnecessary duplications as well; and additionally is arcane
    and
    esoteric even for an arcane and esoteric database platform, and
    absent a
    lot
    of currently nonexistent documentation is going to be tough for anyone
    to
    get to without asking questions on this forum. Obviously the client
    solution is not, shall we say, web scale.

    So in a roundabout way, let me bring up the idea that redis could
    benefit by
    understanding natively what a foreign key is, either by providing some
    dynamic means of identifying foreign keys, or by having commands that
    natively understand foreign keys.

    for example:

    HMGET "hashkey" USING "setname"

    'USING' being a new keyword that uses all of the values of SETNAME as
    keys
    for the HMGET.

    or

    HMSGET "hashkey" "setname"

    etc., etc.

    Thoughts?

    F.

    --
    You received this message because you are subscribed to the Google
    Groups
    "Redis DB" group.
    To post to this group, send email to redis-db@googlegroups.com.
    To unsubscribe from this group, send email to
    redis-db+unsubscribe@googlegroups.com.
    For more options, visit this group at
    http://groups.google.com/group/redis-db?hl=en.
    --
    You received this message because you are subscribed to the Google
    Groups
    "Redis DB" group.
    To post to this group, send email to redis-db@googlegroups.com.
    To unsubscribe from this group, send email to
    redis-db+unsubscribe@googlegroups.com.
    For more options, visit this group at
    http://groups.google.com/group/redis-db?hl=en.
    --
    You received this message because you are subscribed to the Google Groups
    "Redis DB" group.
    To post to this group, send email to redis-db@googlegroups.com.
    To unsubscribe from this group, send email to
    redis-db+unsubscribe@googlegroups.com.
    For more options, visit this group at
    http://groups.google.com/group/redis-db?hl=en.
    --
    You received this message because you are subscribed to the Google Groups
    "Redis DB" group.
    To post to this group, send email to redis-db@googlegroups.com.
    To unsubscribe from this group, send email to
    redis-db+unsubscribe@googlegroups.com.
    For more options, visit this group at
    http://groups.google.com/group/redis-db?hl=en.
    --
    You received this message because you are subscribed to the Google Groups "Redis DB" group.
    To post to this group, send email to redis-db@googlegroups.com.
    To unsubscribe from this group, send email to redis-db+unsubscribe@googlegroups.com.
    For more options, visit this group at http://groups.google.com/group/redis-db?hl=en.
  • Dvir Volk at Nov 19, 2012 at 9:02 pm
    Hey Felix
    You know the SORT command can already do more or less that?
    But i dont think it's optimal. Lua is the way to go IMHO.

    sent from my Sinclair ZX48
    On Nov 19, 2012 9:33 PM, "Felix Gallo" wrote:

    Very frequently people ask questions of the form

    "I have set X, and hash table key Y, and want all key/value pairs from Y
    where key is in X"

    example:

    hset users "1" "{'name': 'bob', 'age': 45}" "2" {'name': 'sally', 'age':
    22}" [...]
    sadd userlist 1 3 22 112 [...]

    # now I want to issue one command to get all of the JSON blobs where

    or

    "I have set X, and a collection of keys named with convention "Y:A", and
    want all key/value pairs where for a given Y, A is in X"

    example:

    hset user:1 name bob age 45
    hset user:2 name sally age 22
    [...]

    sadd userlist 1 3 22 112

    # now I want to issue one command to hgetall all of the k/v pairs from
    user:N where N in userlist

    and there are various mixtures of the two approaches, all of which suffer
    from either or both of the approaches' problems.

    The former form (storing JSON- or otherwise-serialized- string data) sees
    a lot of use because many programming problems involve nested
    objects/tables, and although you lose the cleanliness and random attribute
    access and space efficiency of the latter form, trying to emulate nested
    behavior quickly becomes quite difficult.

    In both cases, there's no immediately obvious answer to the problem that
    doesn't involve multiple roundtrips. In declining order of goodness it
    sort of goes like this:

    solution #1 (using the app to build the second command from results of
    first)
    ----------------

    x = r.smembers "userlist"
    results = r.hgetall "users" x

    # two round trips
    # note that results is not a hash but an array of values, so you have to
    then match up your values with x
    # note also that large values of x may cause issues

    solution #2 (using lua)
    ----------------

    hmgetviaset = <<LUA
    local keys = redis.call("smembers", KEYS[2])
    local ret = {}
    for i,v in ipairs(keys) do
    ret[i] = {i, redis.call("hmget", KEYS[1], i)}
    end
    return ret
    LUA

    r.hset "users", "1", "bobserialized"
    r.hset "users", "2", "sallyserialized"
    r.sadd "userlist", 1
    r.sadd "userlist", 2

    puts r.eval(hmgetviaset, ["users", "userlist"])

    # one round trip
    # requires that you are a member of this mailing list to obtain and/or
    understand this script and/or enough knowledge to write it yourself

    solution #3 (the naive, obvious solution; using the app to iterate)
    -----------------

    x = r.smembers userlist
    x.map!{|i| r.hmget "users" i}

    # possibly many round trips
    # didn't include pipeline code because pipelining can be faster but causes
    issues for even medium values of x


    All of these solutions have goodness and badness in them; the app solution
    is fine but is a bit finicky, especially for larger values of x, and
    requires you to pass the entire set through the network twice before redis
    will start to give you your solution. The lua solution is okay but
    obviously for large values of x will impact the server, and performs a
    number of unnecessary duplications as well; and additionally is arcane and
    esoteric even for an arcane and esoteric database platform, and absent a
    lot of currently nonexistent documentation is going to be tough for anyone
    to get to without asking questions on this forum. Obviously the client
    solution is not, shall we say, web scale.

    So in a roundabout way, let me bring up the idea that redis could benefit
    by understanding natively what a foreign key is, either by providing some
    dynamic means of identifying foreign keys, or by having commands that
    natively understand foreign keys.

    for example:

    HMGET "hashkey" USING "setname"

    'USING' being a new keyword that uses all of the values of SETNAME as keys
    for the HMGET.

    or

    HMSGET "hashkey" "setname"

    etc., etc.

    Thoughts?

    F.

    --
    You received this message because you are subscribed to the Google Groups
    "Redis DB" group.
    To post to this group, send email to redis-db@googlegroups.com.
    To unsubscribe from this group, send email to
    redis-db+unsubscribe@googlegroups.com.
    For more options, visit this group at
    http://groups.google.com/group/redis-db?hl=en.
    --
    You received this message because you are subscribed to the Google Groups "Redis DB" group.
    To post to this group, send email to redis-db@googlegroups.com.
    To unsubscribe from this group, send email to redis-db+unsubscribe@googlegroups.com.
    For more options, visit this group at http://groups.google.com/group/redis-db?hl=en.

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupredis-db @
categoriesredis
postedNov 19, '12 at 7:33p
activeNov 19, '12 at 9:46p
posts8
users4
websiteredis.io
irc#redis

People

Translate

site design / logo © 2022 Grokbase