FAQ
Howdy,

we recently updated three redis nodes and since the update are experiencing
some strange connection problems:

In high load times the instances refuse new connections for about 1 minute.
This happens about once an hour. But not every hour and the gap is not
exactly 60 minutes.

We set maxclients to 32k and also the max open files limit in ubuntu is
32k. There are between 6k and 11k clients on average. We have ~25k commands
per second.

There are no entries in the slowlog corresponding with the connection drops.

As it happens on all three nodes at the same time i think we might hit
another limit. Is there anything known?

Thanks,

David

--
You received this message because you are subscribed to the Google Groups "Redis DB" group.
To unsubscribe from this group and stop receiving emails from it, send an email to redis-db+unsubscribe@googlegroups.com.
To post to this group, send email to redis-db@googlegroups.com.
Visit this group at http://groups.google.com/group/redis-db.
For more options, visit https://groups.google.com/groups/opt_out.

Search Discussions

  • Salvatore Sanfilippo at Aug 30, 2013 at 10:09 am
    Hello David,

    I can't think to a specific cause for that... but I would try to
    replicate it against 2.6.16 if possible, as 2.6.10 is a bit old (even
    if I can't find a directly related issue in the history).
    Anyway, I'm interested to help in the debugging. First, the period,
    what happens every hour? Maybe you have RDB persistence configured to
    save hourly?

    Salvatore
    On Fri, Aug 30, 2013 at 12:00 PM, David wrote:
    Howdy,

    we recently updated three redis nodes and since the update are experiencing
    some strange connection problems:

    In high load times the instances refuse new connections for about 1 minute.
    This happens about once an hour. But not every hour and the gap is not
    exactly 60 minutes.

    We set maxclients to 32k and also the max open files limit in ubuntu is 32k.
    There are between 6k and 11k clients on average. We have ~25k commands per
    second.

    There are no entries in the slowlog corresponding with the connection drops.

    As it happens on all three nodes at the same time i think we might hit
    another limit. Is there anything known?

    Thanks,

    David

    --
    You received this message because you are subscribed to the Google Groups
    "Redis DB" group.
    To unsubscribe from this group and stop receiving emails from it, send an
    email to redis-db+unsubscribe@googlegroups.com.
    To post to this group, send email to redis-db@googlegroups.com.
    Visit this group at http://groups.google.com/group/redis-db.
    For more options, visit https://groups.google.com/groups/opt_out.


    --
    Salvatore 'antirez' Sanfilippo
    open source developer - GoPivotal
    http://invece.org

    Beauty is more important in computing than anywhere else in technology
    because software is so complicated. Beauty is the ultimate defence
    against complexity.
            — David Gelernter

    --
    You received this message because you are subscribed to the Google Groups "Redis DB" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to redis-db+unsubscribe@googlegroups.com.
    To post to this group, send email to redis-db@googlegroups.com.
    Visit this group at http://groups.google.com/group/redis-db.
    For more options, visit https://groups.google.com/groups/opt_out.
  • Salvatore Sanfilippo at Aug 30, 2013 at 10:10 am
    Another question, did you upgraded other things together with Redis
    when you switched from 2.2 to 2.6? For instance the Linux kernel or
    something like that.

    On Fri, Aug 30, 2013 at 12:09 PM, Salvatore Sanfilippo
    wrote:
    Hello David,

    I can't think to a specific cause for that... but I would try to
    replicate it against 2.6.16 if possible, as 2.6.10 is a bit old (even
    if I can't find a directly related issue in the history).
    Anyway, I'm interested to help in the debugging. First, the period,
    what happens every hour? Maybe you have RDB persistence configured to
    save hourly?

    Salvatore
    On Fri, Aug 30, 2013 at 12:00 PM, David wrote:
    Howdy,

    we recently updated three redis nodes and since the update are experiencing
    some strange connection problems:

    In high load times the instances refuse new connections for about 1 minute.
    This happens about once an hour. But not every hour and the gap is not
    exactly 60 minutes.

    We set maxclients to 32k and also the max open files limit in ubuntu is 32k.
    There are between 6k and 11k clients on average. We have ~25k commands per
    second.

    There are no entries in the slowlog corresponding with the connection drops.

    As it happens on all three nodes at the same time i think we might hit
    another limit. Is there anything known?

    Thanks,

    David

    --
    You received this message because you are subscribed to the Google Groups
    "Redis DB" group.
    To unsubscribe from this group and stop receiving emails from it, send an
    email to redis-db+unsubscribe@googlegroups.com.
    To post to this group, send email to redis-db@googlegroups.com.
    Visit this group at http://groups.google.com/group/redis-db.
    For more options, visit https://groups.google.com/groups/opt_out.


    --
    Salvatore 'antirez' Sanfilippo
    open source developer - GoPivotal
    http://invece.org

    Beauty is more important in computing than anywhere else in technology
    because software is so complicated. Beauty is the ultimate defence
    against complexity.
    — David Gelernter


    --
    Salvatore 'antirez' Sanfilippo
    open source developer - GoPivotal
    http://invece.org

    Beauty is more important in computing than anywhere else in technology
    because software is so complicated. Beauty is the ultimate defence
    against complexity.
            — David Gelernter

    --
    You received this message because you are subscribed to the Google Groups "Redis DB" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to redis-db+unsubscribe@googlegroups.com.
    To post to this group, send email to redis-db@googlegroups.com.
    Visit this group at http://groups.google.com/group/redis-db.
    For more options, visit https://groups.google.com/groups/opt_out.
  • David at Aug 30, 2013 at 10:25 am
    Hey Salvatore,

    its about once in 60 minutes (sometimes in 16, sometimes in 87 minutes)
    that many clients drop connections and answers of redis are getting really
    slow (like 200ms response time, where 5ms are normal). It happens on all
    three nodes at the same time. For the clients the three are in a consistent
    hashing circle.

    Yes we upgraded all the Ubuntu packages, so also the Linux Kernel.

    Under one of the nodes we have a slave since the update, but the last dump
    for the slave is 3 days ago. We have disabled all save options, so RDB
    should not be the cause.

    Thanks for your help,

    David


    2013/8/30 Salvatore Sanfilippo <antirez@gmail.com>
    Another question, did you upgraded other things together with Redis
    when you switched from 2.2 to 2.6? For instance the Linux kernel or
    something like that.

    On Fri, Aug 30, 2013 at 12:09 PM, Salvatore Sanfilippo
    wrote:
    Hello David,

    I can't think to a specific cause for that... but I would try to
    replicate it against 2.6.16 if possible, as 2.6.10 is a bit old (even
    if I can't find a directly related issue in the history).
    Anyway, I'm interested to help in the debugging. First, the period,
    what happens every hour? Maybe you have RDB persistence configured to
    save hourly?

    Salvatore
    On Fri, Aug 30, 2013 at 12:00 PM, David wrote:
    Howdy,

    we recently updated three redis nodes and since the update are
    experiencing
    some strange connection problems:

    In high load times the instances refuse new connections for about 1
    minute.
    This happens about once an hour. But not every hour and the gap is not
    exactly 60 minutes.

    We set maxclients to 32k and also the max open files limit in ubuntu is
    32k.
    There are between 6k and 11k clients on average. We have ~25k commands
    per
    second.

    There are no entries in the slowlog corresponding with the connection
    drops.
    As it happens on all three nodes at the same time i think we might hit
    another limit. Is there anything known?

    Thanks,

    David

    --
    You received this message because you are subscribed to the Google
    Groups
    "Redis DB" group.
    To unsubscribe from this group and stop receiving emails from it, send
    an
    email to redis-db+unsubscribe@googlegroups.com.
    To post to this group, send email to redis-db@googlegroups.com.
    Visit this group at http://groups.google.com/group/redis-db.
    For more options, visit https://groups.google.com/groups/opt_out.


    --
    Salvatore 'antirez' Sanfilippo
    open source developer - GoPivotal
    http://invece.org

    Beauty is more important in computing than anywhere else in technology
    because software is so complicated. Beauty is the ultimate defence
    against complexity.
    — David Gelernter


    --
    Salvatore 'antirez' Sanfilippo
    open source developer - GoPivotal
    http://invece.org

    Beauty is more important in computing than anywhere else in technology
    because software is so complicated. Beauty is the ultimate defence
    against complexity.
    — David Gelernter

    --
    You received this message because you are subscribed to the Google Groups
    "Redis DB" group.
    To unsubscribe from this group and stop receiving emails from it, send an
    email to redis-db+unsubscribe@googlegroups.com.
    To post to this group, send email to redis-db@googlegroups.com.
    Visit this group at http://groups.google.com/group/redis-db.
    For more options, visit https://groups.google.com/groups/opt_out.
    --
    You received this message because you are subscribed to the Google Groups "Redis DB" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to redis-db+unsubscribe@googlegroups.com.
    To post to this group, send email to redis-db@googlegroups.com.
    Visit this group at http://groups.google.com/group/redis-db.
    For more options, visit https://groups.google.com/groups/opt_out.
  • David at Sep 2, 2013 at 1:10 pm
    Howdy,

    is there a known problem with updated Linux Kernels in Ubuntu?

    Can I provide any more information to identify the cause of connection
    drops?

    Thanks,
    David


    2013/8/30 David <edlerd@gmail.com>
    Hey Salvatore,

    its about once in 60 minutes (sometimes in 16, sometimes in 87 minutes)
    that many clients drop connections and answers of redis are getting really
    slow (like 200ms response time, where 5ms are normal). It happens on all
    three nodes at the same time. For the clients the three are in a consistent
    hashing circle.

    Yes we upgraded all the Ubuntu packages, so also the Linux Kernel.

    Under one of the nodes we have a slave since the update, but the last dump
    for the slave is 3 days ago. We have disabled all save options, so RDB
    should not be the cause.

    Thanks for your help,

    David


    2013/8/30 Salvatore Sanfilippo <antirez@gmail.com>
    Another question, did you upgraded other things together with Redis
    when you switched from 2.2 to 2.6? For instance the Linux kernel or
    something like that.

    On Fri, Aug 30, 2013 at 12:09 PM, Salvatore Sanfilippo
    wrote:
    Hello David,

    I can't think to a specific cause for that... but I would try to
    replicate it against 2.6.16 if possible, as 2.6.10 is a bit old (even
    if I can't find a directly related issue in the history).
    Anyway, I'm interested to help in the debugging. First, the period,
    what happens every hour? Maybe you have RDB persistence configured to
    save hourly?

    Salvatore
    On Fri, Aug 30, 2013 at 12:00 PM, David wrote:
    Howdy,

    we recently updated three redis nodes and since the update are
    experiencing
    some strange connection problems:

    In high load times the instances refuse new connections for about 1
    minute.
    This happens about once an hour. But not every hour and the gap is not
    exactly 60 minutes.

    We set maxclients to 32k and also the max open files limit in ubuntu
    is 32k.
    There are between 6k and 11k clients on average. We have ~25k commands
    per
    second.

    There are no entries in the slowlog corresponding with the connection
    drops.
    As it happens on all three nodes at the same time i think we might hit
    another limit. Is there anything known?

    Thanks,

    David

    --
    You received this message because you are subscribed to the Google
    Groups
    "Redis DB" group.
    To unsubscribe from this group and stop receiving emails from it, send
    an
    email to redis-db+unsubscribe@googlegroups.com.
    To post to this group, send email to redis-db@googlegroups.com.
    Visit this group at http://groups.google.com/group/redis-db.
    For more options, visit https://groups.google.com/groups/opt_out.


    --
    Salvatore 'antirez' Sanfilippo
    open source developer - GoPivotal
    http://invece.org

    Beauty is more important in computing than anywhere else in technology
    because software is so complicated. Beauty is the ultimate defence
    against complexity.
    — David Gelernter


    --
    Salvatore 'antirez' Sanfilippo
    open source developer - GoPivotal
    http://invece.org

    Beauty is more important in computing than anywhere else in technology
    because software is so complicated. Beauty is the ultimate defence
    against complexity.
    — David Gelernter

    --
    You received this message because you are subscribed to the Google Groups
    "Redis DB" group.
    To unsubscribe from this group and stop receiving emails from it, send an
    email to redis-db+unsubscribe@googlegroups.com.
    To post to this group, send email to redis-db@googlegroups.com.
    Visit this group at http://groups.google.com/group/redis-db.
    For more options, visit https://groups.google.com/groups/opt_out.
    --
    You received this message because you are subscribed to the Google Groups "Redis DB" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to redis-db+unsubscribe@googlegroups.com.
    To post to this group, send email to redis-db@googlegroups.com.
    Visit this group at http://groups.google.com/group/redis-db.
    For more options, visit https://groups.google.com/groups/opt_out.

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupredis-db @
categoriesredis
postedAug 30, '13 at 10:00a
activeSep 2, '13 at 1:10p
posts5
users2
websiteredis.io
irc#redis

2 users in discussion

David: 3 posts Salvatore Sanfilippo: 2 posts

People

Translate

site design / logo © 2022 Grokbase