FAQ
I'm using redis 2.6.12 sorted lists to store unique id pointers.
Unfortunately, it seems like redis is using a lot of memory to store what
amounts to very small sets of 20 character ids. The lists are added to via
ZADD.

A list of keys:

redis msgds1:6379> keys *
1) "u:BI7aaab:c:7c9fef0de6715658Li2:msgun"
2) "u:3MFncab:c:1da0540de6715e68b81:msg"
3) "u:CYFncab:c:01d4bffce671547akY4:msg"
4) "u:BI7aaab:c:2253e64be6715aacMu3:msgun"
5) "u:BI7aaab:c:ffcbbda9e6715244Yi:msg"
6)
"u:rnNucab:c:f7a511a669eddf85f664ca12844dd7e151709c3cfe7a6f7d35000000:msg"
7) "u:nNFncab:c:4a53dd69e671575cT11:msg"
8) "u:BI7aaab:c:221697fce6715ff0vO2:msg"
9) "u:BI7aaab:c:7376454be6715d60~44:msg"
10) "u:aUFncab:c:f582ec69e6715d50sE4:msg"
11) "u:T5Fncab:c:ed0e8ca9e6715cc2M.:msg"
12) "u:BI7aaab:c:6162c60de67156bfLU5:msgun"
13) "u:FRFncab:c:7a697b1de6715d50Qn:msg"
14) "u:BI7aaab:c:23e4170de6715aacNh4:msgun"
15) "u:IlFncab:c:3aa43e0de6715da2pT2:msg"
16) "u:BI7aaab:c:1b4cb60de6715547Ox3:msg"
17)
"u:ajDkcab:c:697843e027396a5f5d89499cdaa2c77f5168b012bbbe14c14b001c3c:msg"
18) "u:BI7aaab:c:2d0f9c1de671542dyr6:msg"
19) "u:BI7aaab:c:1a1be60de6715b66852:msgun"
20) "u:n7Fncab:c:0151fe69e6715045oQ1:msg"
21) "u:BI7aaab:c:56fca02de6715999k57:msgun"
22) "u:BI7aaab:c:d01b4e0de67158f9Se2:msg"
23) "u:BI7aaab:c:c2761d0de67156b29x:msg"
24)
"u:OJNscab:c:697843e027396a5f5d89499cdaa2c77f5168b012bbbe14c14b001c3c:msg"
25) "u:RCFncab:c:c6381e69e67155475C:msgun"
26) "u:xHFncab:c:6855eda9e6715ceami7:msg"
27) "u:CaGncab:c:91b83b1de6715d50wt7:msg"
28) "u:BI7aaab:c:257fbda9e67156bfCP3:msg"
29) "u:BI7aaab:c:92adafa9e67150b48J6:msgun"
30) "u:ojGncab:c:edf20e69e6715d602S1:msg"
31) "u:BI7aaab:c:70e04d0de671547aGn5:msg"
32) "u:BI7aaab:c:3d1f4d4be67158f9DY:msg"
33) "u:BI7aaab:c:582bc02de6715244Th3:msgun"
Each of those keys contain a list of the pointers:

redis msgds1:6379> zrange u:BI7aaab:c:3d1f4d4be67158f9DY:msg 0 -1
1) "d0705d4be67158f9DY"
2) "feb7d26ce6715aacZ~5"
3) "10dd4b7ce67158f9-n4"
4) "c831e40de671547akY4"
But memory usage seems high:

redis msgds1:6379> info
# Server
redis_version:2.6.12
redis_git_sha1:00000000
redis_git_dirty:0
redis_mode:standalone
os:Linux 2.6.32-358.2.1.el6.x86_64 x86_64
arch_bits:64
multiplexing_api:epoll
gcc_version:4.4.7
process_id:22900
run_id:4178a6495f8f4f1116ccbd80e8ab7cdb3a96d51c
tcp_port:6379
uptime_in_seconds:7272
uptime_in_days:0
hz:10
lru_clock:361421

# Clients
connected_clients:1
client_longest_output_list:0
client_biggest_input_buf:0
blocked_clients:0

# Memory
used_memory:5534512
used_memory_human:5.28M
used_memory_rss:13873152
used_memory_peak:5602616
used_memory_peak_human:5.34M
used_memory_lua:31744
mem_fragmentation_ratio:2.51
mem_allocator:jemalloc-3.2.0

# Persistence
loading:0
rdb_changes_since_last_save:103
rdb_bgsave_in_progress:0
rdb_last_save_time:1366755746
rdb_last_bgsave_status:ok
rdb_last_bgsave_time_sec:-1
rdb_current_bgsave_time_sec:-1
aof_enabled:1
aof_rewrite_in_progress:0
aof_rewrite_scheduled:0
aof_last_rewrite_time_sec:-1
aof_current_rewrite_time_sec:-1
aof_last_bgrewrite_status:ok
aof_current_size:6863
aof_base_size:5376
aof_pending_rewrite:0
aof_buffer_length:0
aof_rewrite_buffer_length:0
aof_pending_bio_fsync:0
aof_delayed_fsync:0

# Stats
total_connections_received:60
total_commands_processed:61
instantaneous_ops_per_sec:0
rejected_connections:0
expired_keys:0
evicted_keys:0
keyspace_hits:26
keyspace_misses:0
pubsub_channels:0
pubsub_patterns:0
latest_fork_usec:0

# Replication
role:master
connected_slaves:0

# CPU
used_cpu_sys:0.82
used_cpu_user:0.81
used_cpu_sys_children:0.00
used_cpu_user_children:0.00

# Keyspace
db0:keys=33,expires=0
I've restarted the server but the keys, even when read from disk, still
take up 5mb. All of these keys are 4-5 entries at most.

Here's my server config:

daemonize yes
databases 1
dir /var/lib/redis
pidfile /var/run/redis.pid
port 6379
timeout 600
maxclients 100000

loglevel notice
logfile /var/log/redis.log
slowlog-log-slower-than 1000000
slowlog-max-len 1024

stop-writes-on-bgsave-error yes
rdbchecksum no

appendonly yes
appendfsync everysec
no-appendfsync-on-rewrite yes

auto-aof-rewrite-percentage 10
auto-aof-rewrite-min-size 1gb

activerehashing yes
Can anyone explain the memory usage here?

--
You received this message because you are subscribed to the Google Groups "Redis DB" group.
To unsubscribe from this group and stop receiving emails from it, send an email to redis-db+unsubscribe@googlegroups.com.
To post to this group, send email to redis-db@googlegroups.com.
Visit this group at http://groups.google.com/group/redis-db?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.

Search Discussions

  • Salvatore Sanfilippo at Apr 24, 2013 at 8:12 am
    Hello, did you checked using ZCARD what is the length of every of
    those sorted set? Thanks.
    On Wed, Apr 24, 2013 at 2:37 AM, News Group wrote:
    I'm using redis 2.6.12 sorted lists to store unique id pointers.
    Unfortunately, it seems like redis is using a lot of memory to store what
    amounts to very small sets of 20 character ids. The lists are added to via
    ZADD.

    A list of keys:
    redis msgds1:6379> keys *
    1) "u:BI7aaab:c:7c9fef0de6715658Li2:msgun"
    2) "u:3MFncab:c:1da0540de6715e68b81:msg"
    3) "u:CYFncab:c:01d4bffce671547akY4:msg"
    4) "u:BI7aaab:c:2253e64be6715aacMu3:msgun"
    5) "u:BI7aaab:c:ffcbbda9e6715244Yi:msg"
    6)
    "u:rnNucab:c:f7a511a669eddf85f664ca12844dd7e151709c3cfe7a6f7d35000000:msg"
    7) "u:nNFncab:c:4a53dd69e671575cT11:msg"
    8) "u:BI7aaab:c:221697fce6715ff0vO2:msg"
    9) "u:BI7aaab:c:7376454be6715d60~44:msg"
    10) "u:aUFncab:c:f582ec69e6715d50sE4:msg"
    11) "u:T5Fncab:c:ed0e8ca9e6715cc2M.:msg"
    12) "u:BI7aaab:c:6162c60de67156bfLU5:msgun"
    13) "u:FRFncab:c:7a697b1de6715d50Qn:msg"
    14) "u:BI7aaab:c:23e4170de6715aacNh4:msgun"
    15) "u:IlFncab:c:3aa43e0de6715da2pT2:msg"
    16) "u:BI7aaab:c:1b4cb60de6715547Ox3:msg"
    17)
    "u:ajDkcab:c:697843e027396a5f5d89499cdaa2c77f5168b012bbbe14c14b001c3c:msg"
    18) "u:BI7aaab:c:2d0f9c1de671542dyr6:msg"
    19) "u:BI7aaab:c:1a1be60de6715b66852:msgun"
    20) "u:n7Fncab:c:0151fe69e6715045oQ1:msg"
    21) "u:BI7aaab:c:56fca02de6715999k57:msgun"
    22) "u:BI7aaab:c:d01b4e0de67158f9Se2:msg"
    23) "u:BI7aaab:c:c2761d0de67156b29x:msg"
    24)
    "u:OJNscab:c:697843e027396a5f5d89499cdaa2c77f5168b012bbbe14c14b001c3c:msg"
    25) "u:RCFncab:c:c6381e69e67155475C:msgun"
    26) "u:xHFncab:c:6855eda9e6715ceami7:msg"
    27) "u:CaGncab:c:91b83b1de6715d50wt7:msg"
    28) "u:BI7aaab:c:257fbda9e67156bfCP3:msg"
    29) "u:BI7aaab:c:92adafa9e67150b48J6:msgun"
    30) "u:ojGncab:c:edf20e69e6715d602S1:msg"
    31) "u:BI7aaab:c:70e04d0de671547aGn5:msg"
    32) "u:BI7aaab:c:3d1f4d4be67158f9DY:msg"
    33) "u:BI7aaab:c:582bc02de6715244Th3:msgun"

    Each of those keys contain a list of the pointers:
    redis msgds1:6379> zrange u:BI7aaab:c:3d1f4d4be67158f9DY:msg 0 -1
    1) "d0705d4be67158f9DY"
    2) "feb7d26ce6715aacZ~5"
    3) "10dd4b7ce67158f9-n4"
    4) "c831e40de671547akY4"

    But memory usage seems high:
    redis msgds1:6379> info
    # Server
    redis_version:2.6.12
    redis_git_sha1:00000000
    redis_git_dirty:0
    redis_mode:standalone
    os:Linux 2.6.32-358.2.1.el6.x86_64 x86_64
    arch_bits:64
    multiplexing_api:epoll
    gcc_version:4.4.7
    process_id:22900
    run_id:4178a6495f8f4f1116ccbd80e8ab7cdb3a96d51c
    tcp_port:6379
    uptime_in_seconds:7272
    uptime_in_days:0
    hz:10
    lru_clock:361421

    # Clients
    connected_clients:1
    client_longest_output_list:0
    client_biggest_input_buf:0
    blocked_clients:0

    # Memory
    used_memory:5534512
    used_memory_human:5.28M
    used_memory_rss:13873152
    used_memory_peak:5602616
    used_memory_peak_human:5.34M
    used_memory_lua:31744
    mem_fragmentation_ratio:2.51
    mem_allocator:jemalloc-3.2.0

    # Persistence
    loading:0
    rdb_changes_since_last_save:103
    rdb_bgsave_in_progress:0
    rdb_last_save_time:1366755746
    rdb_last_bgsave_status:ok
    rdb_last_bgsave_time_sec:-1
    rdb_current_bgsave_time_sec:-1
    aof_enabled:1
    aof_rewrite_in_progress:0
    aof_rewrite_scheduled:0
    aof_last_rewrite_time_sec:-1
    aof_current_rewrite_time_sec:-1
    aof_last_bgrewrite_status:ok
    aof_current_size:6863
    aof_base_size:5376
    aof_pending_rewrite:0
    aof_buffer_length:0
    aof_rewrite_buffer_length:0
    aof_pending_bio_fsync:0
    aof_delayed_fsync:0

    # Stats
    total_connections_received:60
    total_commands_processed:61
    instantaneous_ops_per_sec:0
    rejected_connections:0
    expired_keys:0
    evicted_keys:0
    keyspace_hits:26
    keyspace_misses:0
    pubsub_channels:0
    pubsub_patterns:0
    latest_fork_usec:0

    # Replication
    role:master
    connected_slaves:0

    # CPU
    used_cpu_sys:0.82
    used_cpu_user:0.81
    used_cpu_sys_children:0.00
    used_cpu_user_children:0.00

    # Keyspace
    db0:keys=33,expires=0

    I've restarted the server but the keys, even when read from disk, still take
    up 5mb. All of these keys are 4-5 entries at most.

    Here's my server config:
    daemonize yes
    databases 1
    dir /var/lib/redis
    pidfile /var/run/redis.pid
    port 6379
    timeout 600
    maxclients 100000

    loglevel notice
    logfile /var/log/redis.log
    slowlog-log-slower-than 1000000
    slowlog-max-len 1024

    stop-writes-on-bgsave-error yes
    rdbchecksum no

    appendonly yes
    appendfsync everysec
    no-appendfsync-on-rewrite yes

    auto-aof-rewrite-percentage 10
    auto-aof-rewrite-min-size 1gb

    activerehashing yes

    Can anyone explain the memory usage here?

    --
    You received this message because you are subscribed to the Google Groups
    "Redis DB" group.
    To unsubscribe from this group and stop receiving emails from it, send an
    email to redis-db+unsubscribe@googlegroups.com.
    To post to this group, send email to redis-db@googlegroups.com.
    Visit this group at http://groups.google.com/group/redis-db?hl=en.
    For more options, visit https://groups.google.com/groups/opt_out.


    --
    Salvatore 'antirez' Sanfilippo
    open source developer - VMware
    http://invece.org

    Beauty is more important in computing than anywhere else in technology
    because software is so complicated. Beauty is the ultimate defence
    against complexity.
    — David Gelernter

    --
    You received this message because you are subscribed to the Google Groups "Redis DB" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to redis-db+unsubscribe@googlegroups.com.
    To post to this group, send email to redis-db@googlegroups.com.
    Visit this group at http://groups.google.com/group/redis-db?hl=en.
    For more options, visit https://groups.google.com/groups/opt_out.
  • News Group at Apr 24, 2013 at 12:07 pm

    On Wednesday, 24 April 2013 04:12:30 UTC-4, Salvatore Sanfilippo wrote:
    Hello, did you checked using ZCARD what is the length of every of
    those sorted set? Thanks.
    Most of them are very short - 1 to 5.

    redis msgds1:6379> zcard u:BI7aaab:c:3d1f4d4be67158f9DY:msg
    (integer) 4
    --
    You received this message because you are subscribed to the Google Groups "Redis DB" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to redis-db+unsubscribe@googlegroups.com.
    To post to this group, send email to redis-db@googlegroups.com.
    Visit this group at http://groups.google.com/group/redis-db?hl=en.
    For more options, visit https://groups.google.com/groups/opt_out.
  • Didier Spezia at Apr 24, 2013 at 1:10 pm
    Here is my take:
    it comes from the value of maxclients (set to 100000 in your system).

    Redis preallocates arrays corresponding to the maximum number of expected
    connections.
    Each item is a 28 bytes structure defined as:

    typedef struct aeFileEvent {
    int mask; /* one of AE_(READABLE|WRITABLE) */
    aeFileProc *rfileProc;
    aeFileProc *wfileProc;
    void *clientData;
    } aeFileEvent;

    and ae.c allocates two arrays corresponding to registered and fired events:

    eventLoop->events = zmalloc(sizeof(aeFileEvent)*setsize);
    eventLoop->fired = zmalloc(sizeof(aeFiredEvent)*setsize);

    So 100K * 28 * 2 = 5.6 MB which in line with what you observe.

    Regards,
    Didier.

    --
    You received this message because you are subscribed to the Google Groups "Redis DB" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to redis-db+unsubscribe@googlegroups.com.
    To post to this group, send email to redis-db@googlegroups.com.
    Visit this group at http://groups.google.com/group/redis-db?hl=en.
    For more options, visit https://groups.google.com/groups/opt_out.
  • News Group at Apr 24, 2013 at 1:46 pm

    On Wednesday, 24 April 2013 09:10:28 UTC-4, Didier Spezia wrote:

    Here is my take:
    it comes from the value of maxclients (set to 100000 in your system).
    Interesting - that explains it perfectly. Thanks!

    --
    You received this message because you are subscribed to the Google Groups "Redis DB" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to redis-db+unsubscribe@googlegroups.com.
    To post to this group, send email to redis-db@googlegroups.com.
    Visit this group at http://groups.google.com/group/redis-db?hl=en.
    For more options, visit https://groups.google.com/groups/opt_out.

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupredis-db @
categoriesredis
postedApr 24, '13 at 8:00a
activeApr 24, '13 at 1:46p
posts5
users3
websiteredis.io
irc#redis

People

Translate

site design / logo © 2022 Grokbase