Grokbase › Groups › Go › golang-nuts › October 2013
FAQ
Hello Friendly Gophers,

We have an issue where we are getting this error for a number of different
services:

read tcp [ip]:[port]: i/o timeout

It seems to happen to many of our services independent of where or what it
is connecting to (cassandra, memcache, zookeeper, etc) so I am thinking it
is an OS level issue and not a code specific issue. We are running on
Ubuntu boxes in Amazon, specifically m1.small's if you are familiar. These
are fairly underpowered machines but they provide enough for our purposes
(or, in light of this issue, perhaps not).

Originally we thought it might be a file descriptor issue but we don't have
that many open connections (netstat says ~1000).

What could cause these stray i/o timeouts in the net package? Has anyone
else seen this using the "net/http" package?

Thanks
Jono

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].
For more options, visit https://groups.google.com/groups/opt_out.

Search Discussions

  • James Bardin at Oct 16, 2013 at 6:10 pm ⇧

    On Wednesday, October 16, 2013 1:03:31 PM UTC-4, [email protected] wrote:


    What could cause these stray i/o timeouts in the net package? Has anyone
    else seen this using the "net/http" package?
    Are you filtering anything with iptables on the servers?

    This is usually what would happen if the remote server disappears (i.e.
    network down). You send some data, and wait for a response that will never
    come until the tcp window times out. Short of network partitions, this
    could be from an improperly configured firewall; losing track of connection
    state (you can only track so many connections at once on a very busy
    server, or the conntrack state times out) and silently dropping the rest.
    It's been a while since I've looked at their config , but Redhat (and
    CentOS) based systems used to have problems like this occasionally, as they
    relied on conntrack state for everything, only accepted incoming
    connections flagged as NEW or ESTABLISHED, while silently dropping "bad"
    packets.


    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].
    For more options, visit https://groups.google.com/groups/opt_out.
  • Jonathan at Oct 17, 2013 at 3:17 pm ⇧
    We have been able to reproduce this locally on a Mac where all the services
    are on one machine so it is not an OS specific issue nor is it network
    error. I have set the timeout to be 60s (in gossie) and I am still seeing
    the error so it is way beyond the deadline. Is there a chance that
    connections are being reused in the net package without the timeout
    resetting?

    The libraries we are seeing this in are:
    Gossie (https://github.com/carloscm/gossie)
    Memcache (https://github.com/bradfitz/gomemcache)
    go-zookeeper (https://github.com/samuel/go-zookeeper)

    Any more thoughts...?
    On Wednesday, October 16, 2013 7:10:30 PM UTC+1, James Bardin wrote:


    On Wednesday, October 16, 2013 1:03:31 PM UTC-4, [email protected] wrote:



    What could cause these stray i/o timeouts in the net package? Has anyone
    else seen this using the "net/http" package?
    Are you filtering anything with iptables on the servers?

    This is usually what would happen if the remote server disappears (i.e.
    network down). You send some data, and wait for a response that will never
    come until the tcp window times out. Short of network partitions, this
    could be from an improperly configured firewall; losing track of connection
    state (you can only track so many connections at once on a very busy
    server, or the conntrack state times out) and silently dropping the rest.
    It's been a while since I've looked at their config , but Redhat (and
    CentOS) based systems used to have problems like this occasionally, as they
    relied on conntrack state for everything, only accepted incoming
    connections flagged as NEW or ESTABLISHED, while silently dropping "bad"
    packets.

    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].
    For more options, visit https://groups.google.com/groups/opt_out.
  • James Bardin at Oct 17, 2013 at 3:48 pm ⇧

    On Thursday, October 17, 2013 10:17:14 AM UTC-4, [email protected] wrote:
    We have been able to reproduce this locally on a Mac where all the
    services are on one machine so it is not an OS specific issue nor is it
    network error. I have set the timeout to be 60s (in gossie) and I am still
    seeing the error so it is way beyond the deadline. Is there a chance that
    connections are being reused in the net package without the timeout
    resetting?

    The libraries we are seeing this in are:
    Gossie (https://github.com/carloscm/gossie)
    Memcache (https://github.com/bradfitz/gomemcache)
    go-zookeeper (https://github.com/samuel/go-zookeeper)

    Any more thoughts...?
    Are you only able to reproduce this with gossie?
    I ask because that gossie library has been abandoned, and the thrift
    library it's using may be a dead fork as well.

    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].
    For more options, visit https://groups.google.com/groups/opt_out.
  • Jonathan at Oct 17, 2013 at 3:52 pm ⇧
    No, it happens on memcache and go-zookeeper as well.

    It is unfortunate that gossie is now abandoned. We are not really sure what
    to do about that one. But that is another issue.

    Interestingly, we don't see this in the nsq library and it is similarly
    setting the timeout (using conn.SetDeadline()).
    On Thursday, October 17, 2013 4:48:52 PM UTC+1, James Bardin wrote:



    On Thursday, October 17, 2013 10:17:14 AM UTC-4, [email protected]:
    We have been able to reproduce this locally on a Mac where all the
    services are on one machine so it is not an OS specific issue nor is it
    network error. I have set the timeout to be 60s (in gossie) and I am still
    seeing the error so it is way beyond the deadline. Is there a chance that
    connections are being reused in the net package without the timeout
    resetting?

    The libraries we are seeing this in are:
    Gossie (https://github.com/carloscm/gossie)
    Memcache (https://github.com/bradfitz/gomemcache)
    go-zookeeper (https://github.com/samuel/go-zookeeper)

    Any more thoughts...?
    Are you only able to reproduce this with gossie?
    I ask because that gossie library has been abandoned, and the thrift
    library it's using may be a dead fork as well.
    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].
    For more options, visit https://groups.google.com/groups/opt_out.
  • D at Apr 2, 2015 at 7:17 am ⇧
    Hi,

    Any update on this? I am also getting the same issue with Memcache (
    https://github.com/bradfitz/gomemcache)

    Thanks,
    On Thursday, 17 October 2013 21:22:32 UTC+5:30, [email protected] wrote:

    No, it happens on memcache and go-zookeeper as well.

    It is unfortunate that gossie is now abandoned. We are not really sure
    what to do about that one. But that is another issue.

    Interestingly, we don't see this in the nsq library and it is similarly
    setting the timeout (using conn.SetDeadline()).
    On Thursday, October 17, 2013 4:48:52 PM UTC+1, James Bardin wrote:



    On Thursday, October 17, 2013 10:17:14 AM UTC-4, [email protected]
    wrote:
    We have been able to reproduce this locally on a Mac where all the
    services are on one machine so it is not an OS specific issue nor is it
    network error. I have set the timeout to be 60s (in gossie) and I am still
    seeing the error so it is way beyond the deadline. Is there a chance that
    connections are being reused in the net package without the timeout
    resetting?

    The libraries we are seeing this in are:
    Gossie (https://github.com/carloscm/gossie)
    Memcache (https://github.com/bradfitz/gomemcache)
    go-zookeeper (https://github.com/samuel/go-zookeeper)

    Any more thoughts...?
    Are you only able to reproduce this with gossie?
    I ask because that gossie library has been abandoned, and the thrift
    library it's using may be a dead fork as well.
    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].
    For more options, visit https://groups.google.com/d/optout.
  • Brad Fitzpatrick at Apr 2, 2015 at 8:13 am ⇧
    D,

    Your bug report lacks critical information: Go version, operating system,
    operating system version, etc. And relevant code / repro would be nice too.


    On Thu, Apr 2, 2015 at 9:17 AM, D wrote:

    Hi,

    Any update on this? I am also getting the same issue with Memcache (
    https://github.com/bradfitz/gomemcache)

    Thanks,

    On Thursday, 17 October 2013 21:22:32 UTC+5:30, [email protected]
    wrote:
    No, it happens on memcache and go-zookeeper as well.

    It is unfortunate that gossie is now abandoned. We are not really sure
    what to do about that one. But that is another issue.

    Interestingly, we don't see this in the nsq library and it is similarly
    setting the timeout (using conn.SetDeadline()).
    On Thursday, October 17, 2013 4:48:52 PM UTC+1, James Bardin wrote:



    On Thursday, October 17, 2013 10:17:14 AM UTC-4, [email protected]
    wrote:
    We have been able to reproduce this locally on a Mac where all the
    services are on one machine so it is not an OS specific issue nor is it
    network error. I have set the timeout to be 60s (in gossie) and I am still
    seeing the error so it is way beyond the deadline. Is there a chance that
    connections are being reused in the net package without the timeout
    resetting?

    The libraries we are seeing this in are:
    Gossie (https://github.com/carloscm/gossie)
    Memcache (https://github.com/bradfitz/gomemcache)
    go-zookeeper (https://github.com/samuel/go-zookeeper)

    Any more thoughts...?
    Are you only able to reproduce this with gossie?
    I ask because that gossie library has been abandoned, and the thrift
    library it's using may be a dead fork as well.
    --
    You received this message because you are subscribed to the Google Groups
    "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an
    email to [email protected].
    For more options, visit https://groups.google.com/d/optout.
    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].
    For more options, visit https://groups.google.com/d/optout.
  • Debraj Manna at Apr 2, 2015 at 9:28 am ⇧
    Go Version go1.4.1 linux/amd64
    OS - Linux XXX 3.2.0-4-amd64 #1 SMP Debian 3.2.65-1+deb7u2 x86_64 GNU/Linux

    The error message I am seeing:-
    read tcp 127.0.0.1:11290: i/o timeout

    This problem I am seeing most of the time with GetMulti() call and when
    memcache is under load.

    I can not share the exact code now. I will try to get a toy repro code of
    the problem soon.
    On Thu, Apr 2, 2015 at 1:43 PM, Brad Fitzpatrick wrote:

    D,

    Your bug report lacks critical information: Go version, operating system,
    operating system version, etc. And relevant code / repro would be nice too.


    On Thu, Apr 2, 2015 at 9:17 AM, D wrote:

    Hi,

    Any update on this? I am also getting the same issue with Memcache (
    https://github.com/bradfitz/gomemcache)

    Thanks,

    On Thursday, 17 October 2013 21:22:32 UTC+5:30, [email protected]
    wrote:
    No, it happens on memcache and go-zookeeper as well.

    It is unfortunate that gossie is now abandoned. We are not really sure
    what to do about that one. But that is another issue.

    Interestingly, we don't see this in the nsq library and it is similarly
    setting the timeout (using conn.SetDeadline()).
    On Thursday, October 17, 2013 4:48:52 PM UTC+1, James Bardin wrote:



    On Thursday, October 17, 2013 10:17:14 AM UTC-4, [email protected]
    wrote:
    We have been able to reproduce this locally on a Mac where all the
    services are on one machine so it is not an OS specific issue nor is it
    network error. I have set the timeout to be 60s (in gossie) and I am still
    seeing the error so it is way beyond the deadline. Is there a chance that
    connections are being reused in the net package without the timeout
    resetting?

    The libraries we are seeing this in are:
    Gossie (https://github.com/carloscm/gossie)
    Memcache (https://github.com/bradfitz/gomemcache)
    go-zookeeper (https://github.com/samuel/go-zookeeper)

    Any more thoughts...?
    Are you only able to reproduce this with gossie?
    I ask because that gossie library has been abandoned, and the thrift
    library it's using may be a dead fork as well.
    --
    You received this message because you are subscribed to the Google Groups
    "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an
    email to [email protected].
    For more options, visit https://groups.google.com/d/optout.
    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].
    For more options, visit https://groups.google.com/d/optout.
  • Debraj Manna at Apr 2, 2015 at 3:31 pm ⇧
    Go Version go1.4.1 linux/amd64
    OS - Linux XXX 3.2.0-4-amd64 #1 SMP Debian 3.2.65-1+deb7u2 x86_64 GNU/Linux

    The error message I am seeing:-
    read tcp 127.0.0.1:11290: i/o timeout

    This problem I am seeing most of the time with GetMulti() call and when
    memcache is under load.

    I can not share the exact code now. I will try to get a toy repro code of
    the problem soon.
    On Thursday, April 2, 2015 at 1:43:33 PM UTC+5:30, bradfitz wrote:

    D,

    Your bug report lacks critical information: Go version, operating system,
    operating system version, etc. And relevant code / repro would be nice too.



    On Thu, Apr 2, 2015 at 9:17 AM, D <[email protected] <javascript:>>
    wrote:
    Hi,

    Any update on this? I am also getting the same issue with Memcache (
    https://github.com/bradfitz/gomemcache)

    Thanks,

    On Thursday, 17 October 2013 21:22:32 UTC+5:30, [email protected]
    wrote:
    No, it happens on memcache and go-zookeeper as well.

    It is unfortunate that gossie is now abandoned. We are not really sure
    what to do about that one. But that is another issue.

    Interestingly, we don't see this in the nsq library and it is similarly
    setting the timeout (using conn.SetDeadline()).
    On Thursday, October 17, 2013 4:48:52 PM UTC+1, James Bardin wrote:



    On Thursday, October 17, 2013 10:17:14 AM UTC-4, [email protected]
    wrote:
    We have been able to reproduce this locally on a Mac where all the
    services are on one machine so it is not an OS specific issue nor is it
    network error. I have set the timeout to be 60s (in gossie) and I am still
    seeing the error so it is way beyond the deadline. Is there a chance that
    connections are being reused in the net package without the timeout
    resetting?

    The libraries we are seeing this in are:
    Gossie (https://github.com/carloscm/gossie)
    Memcache (https://github.com/bradfitz/gomemcache)
    go-zookeeper (https://github.com/samuel/go-zookeeper)

    Any more thoughts...?
    Are you only able to reproduce this with gossie?
    I ask because that gossie library has been abandoned, and the thrift
    library it's using may be a dead fork as well.
    --
    You received this message because you are subscribed to the Google Groups
    "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an
    email to [email protected] <javascript:>.
    For more options, visit https://groups.google.com/d/optout.
    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].
    For more options, visit https://groups.google.com/d/optout.

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupgolang-nuts @
categoriesgo
postedOct 16, '13 at 5:03p
activeApr 2, '15 at 3:31p
posts9
users6
websitegolang.org

6 users in discussion

Debraj Manna: 2 posts Jonathan: 2 posts James Bardin: 2 posts Jonomacd: 1 post Debraj Manna: 1 post Brad Fitzpatrick: 1 post

Content

People

Support

Translate

site design / logo © 2023 Grokbase