FAQ
I'm currently trying to break the 64511 concurrent tcp sessions barrier in
my GO application.

I have optimized my kernel for high tcp concurrency as suggested by Richard
Jones on his blog back in 2008 (see metabrew.com<http://www.metabrew.com/article/a-million-user-comet-application-with-mochiweb-part-3>blog below). Unfortunately it doesn't have any effect on the number of
concurrent sessions.

As you have only 64511 unprivileged ports available per IP address I have
added several additional IP addresses to my system and made go loop though
these addresses. When I use a LocalAddr with port 0 go directly starts
complaining with "address already in use" as soon I pass the +/- 64511
connections. So I tied to loop over my ip range in combination with a
specified port for the LocalAddr. Unfortunately this doesn't improve the
number of concurrent connections.

1 Million TCP connections (Linux):
http://www.metabrew.com/article/a-million-user-comet-application-with-mochiweb-part-3

2 Million TCP connections (FreeBSD):
http://blog.whatsapp.com/index.php/2012/01/1-million-is-so-2011/

I'm running a Ubuntu Server with a 3.2 kernel.

My connection handling code:

RETRY:

// get the firs source ip:port address in queue
ipNext = <-iprange

// manage our on port numbers per ip to overcome the 64511 limit
host, port, _ := net.SplitHostPort(ipNext)
nextPort, _ := strconv.Atoi(port)
nextPort++
if nextPort > 65535 {
nextPort = 1024
}
// add this address with the next port number back to the end of the queue
iprange <- host +":"+ strconv.Itoa(nextPort)

d := net.Dialer{Timeout: 1*time.Second, Deadline:
time.Now().Add(2*time.Second)}
d.LocalAddr, err = net.ResolveTCPAddr("tcp4", ipNext)
if err != nil {
         log.Fatal(err)
}
dc, err := d.Dial("tcp", dst)

if err != nil && strings.Contains(err.Error(), "address already in use") {
goto RETRY
}

Any suggestions on how to pass this barrier?

Thanks,

Paul

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Search Discussions

  • Kyle Lemons at Aug 16, 2013 at 6:03 pm

    On Fri, Aug 16, 2013 at 3:45 AM, Paul van Brouwershaven wrote:

    I'm currently trying to break the 64511 concurrent tcp sessions barrier in
    my GO application.

    I have optimized my kernel for high tcp concurrency as suggested by
    Richard Jones on his blog back in 2008 (see metabrew.com<http://www.metabrew.com/article/a-million-user-comet-application-with-mochiweb-part-3>blog below). Unfortunately it doesn't have any effect on the number of
    concurrent sessions.

    As you have only 64511 unprivileged ports available per IP address I have
    added several additional IP addresses to my system and made go loop though
    these addresses. When I use a LocalAddr with port 0 go directly starts
    complaining with "address already in use" as soon I pass the +/- 64511
    connections.
    This is probably something that can be fixed; I'd file an issue.

    So I tied to loop over my ip range in combination with a specified port
    for the LocalAddr. Unfortunately this doesn't improve the number of
    concurrent connections.
    What is the failure mode when you do this? Do you still get "address
    already in use" after 64511? Do you know that your router can handle NAT
    with one (physical) port having more than 65536 source ports? Does the
    tcpdump have any hints?

    1 Million TCP connections (Linux):

    http://www.metabrew.com/article/a-million-user-comet-application-with-mochiweb-part-3

    2 Million TCP connections (FreeBSD):
    http://blog.whatsapp.com/index.php/2012/01/1-million-is-so-2011/

    I'm running a Ubuntu Server with a 3.2 kernel.

    My connection handling code:

    RETRY:

    // get the firs source ip:port address in queue
    ipNext = <-iprange

    // manage our on port numbers per ip to overcome the 64511 limit
    host, port, _ := net.SplitHostPort(ipNext)
    nextPort, _ := strconv.Atoi(port)
    nextPort++
    if nextPort > 65535 {
    nextPort = 1024
    }
    // add this address with the next port number back to the end of the queue
    iprange <- host +":"+ strconv.Itoa(nextPort)

    d := net.Dialer{Timeout: 1*time.Second, Deadline:
    time.Now().Add(2*time.Second)}
    d.LocalAddr, err = net.ResolveTCPAddr("tcp4", ipNext)
    if err != nil {
    log.Fatal(err)
    }
    dc, err := d.Dial("tcp", dst)

    if err != nil && strings.Contains(err.Error(), "address already in use") {
    goto RETRY
    }

    Any suggestions on how to pass this barrier?

    Thanks,

    Paul

    --
    You received this message because you are subscribed to the Google Groups
    "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an
    email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.
    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.
  • Paul van Brouwershaven at Aug 17, 2013 at 7:43 am

    On Friday, 16 August 2013 20:02:50 UTC+2, Kyle Lemons wrote:

    On Fri, Aug 16, 2013 at 3:45 AM, Paul van Brouwershaven <
    pa...@vanbrouwershaven.com <javascript:>> wrote:
    I'm currently trying to break the 64511 concurrent tcp sessions barrier
    in my GO application.

    I have optimized my kernel for high tcp concurrency as suggested by
    Richard Jones on his blog back in 2008 (see metabrew.com<http://www.metabrew.com/article/a-million-user-comet-application-with-mochiweb-part-3>blog below). Unfortunately it doesn't have any effect on the number of
    concurrent sessions.

    As you have only 64511 unprivileged ports available per IP address I have
    added several additional IP addresses to my system and made go loop though
    these addresses. When I use a LocalAddr with port 0 go directly starts
    complaining with "address already in use" as soon I pass the +/- 64511
    connections.
    This is probably something that can be fixed; I'd file an issue.
    I have filed an issue: https://code.google.com/p/go/issues/detail?id=6176

    So I tied to loop over my ip range in combination with a specified port
    for the LocalAddr. Unfortunately this doesn't improve the number of
    concurrent connections.
    What is the failure mode when you do this? Do you still get "address
    already in use" after 64511? Do you know that your router can handle NAT
    with one (physical) port having more than 65536 source ports? Does the
    tcpdump have any hints?
    Basically you can't have more than 65535 ports, so that's the reason I'm
    trying to use 64511 (65535-1024) ports per IP address. This would mean that
    you have 1.0.0.1:65000 and 1.0.0.2:65000 having a connection at the same
    time (as advised by Richard Jones). Unfortunately this gives me the same
    "address already in use" error message as when I specify port 0 (auto).

    I'm counting the number of TCP sessions with:
    netstat -n | awk '/^tcp/ {t[$NF]++}END{for(state in t){print state,
    t[state]} }'

    tcpdump is not giving me any hints, I will send you a small dump off list.

    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.
  • Kyle Lemons at Aug 17, 2013 at 6:33 pm

    On Sat, Aug 17, 2013 at 12:43 AM, Paul van Brouwershaven wrote:

    On Friday, 16 August 2013 20:02:50 UTC+2, Kyle Lemons wrote:

    On Fri, Aug 16, 2013 at 3:45 AM, Paul van Brouwershaven <
    pa...@vanbrouwershaven.com> wrote:
    I'm currently trying to break the 64511 concurrent tcp sessions barrier
    in my GO application.

    I have optimized my kernel for high tcp concurrency as suggested by
    Richard Jones on his blog back in 2008 (see metabrew.com<http://www.metabrew.com/article/a-million-user-comet-application-with-mochiweb-part-3>blog below). Unfortunately it doesn't have any effect on the number of
    concurrent sessions.

    As you have only 64511 unprivileged ports available per IP address I
    have added several additional IP addresses to my system and made go loop
    though these addresses. When I use a LocalAddr with port 0 go directly
    starts complaining with "address already in use" as soon I pass the +/-
    64511 connections.
    This is probably something that can be fixed; I'd file an issue.
    I have filed an issue: https://code.google.com/p/go/issues/detail?id=6176

    So I tied to loop over my ip range in combination with a specified port
    for the LocalAddr. Unfortunately this doesn't improve the number of
    concurrent connections.
    What is the failure mode when you do this? Do you still get "address
    already in use" after 64511? Do you know that your router can handle NAT
    with one (physical) port having more than 65536 source ports? Does the
    tcpdump have any hints?
    Basically you can't have more than 65535 ports, so that's the reason I'm
    trying to use 64511 (65535-1024) ports per IP address. This would mean that
    you have 1.0.0.1:65000 and 1.0.0.2:65000 having a connection at the same
    time (as advised by Richard Jones). Unfortunately this gives me the same
    "address already in use" error message as when I specify port 0 (auto).
    Right, I understand the approach. However, it seems likely that a
    (commodity) router might make the inappropriate assumption that one single
    (physical) port will have only one IP address, and only build in a NAT
    table that has enough rows to accommodate the 65535 source ports per
    (physical) port. If this hypothesis were true, adding more IPs wouldn't
    improve the total number of TCP flows you can maintain, but adding a second
    IP on a second interface connected to a second port on the router would
    double the effective number of flows you could maintain.

    I'm counting the number of TCP sessions with:
    netstat -n | awk '/^tcp/ {t[$NF]++}END{for(state in t){print state,
    t[state]} }'

    tcpdump is not giving me any hints, I will send you a small dump off list.
    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.
  • Paul van Brouwershaven at Aug 18, 2013 at 7:42 am

    On Saturday, 17 August 2013 20:33:36 UTC+2, Kyle Lemons wrote:

    Right, I understand the approach. However, it seems likely that a
    (commodity) router might make the inappropriate assumption that one single
    (physical) port will have only one IP address, and only build in a NAT
    table that has enough rows to accommodate the 65535 source ports per
    (physical) port. If this hypothesis were true, adding more IPs wouldn't
    improve the total number of TCP flows you can maintain, but adding a second
    IP on a second interface connected to a second port on the router would
    double the effective number of flows you could maintain.
    The server is located in a data center behind a first class router, in the
    past we were located in a different data center that indeed couldn't handle
    the traffic. To verify I contacted the network operations team and
    confirmed that the router is not even seeing an increase in memory or CPU
    usage.

    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.
  • Oleku Konko at Aug 17, 2013 at 8:54 am
    This is very Interesting and i also think its an issue that needs to be
    fixed. Better still add IP pool as a default feature net & http package.

    +1 nice links


    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.
  • Benjamin Measures at Aug 17, 2013 at 10:16 pm

    On Friday, 16 August 2013 11:45:40 UTC+1, Paul van Brouwershaven wrote:

    I'm currently trying to break the 64511 concurrent tcp sessions barrier in
    my GO application. [...]
    1 Million TCP connections (Linux):

    http://www.metabrew.com/article/a-million-user-comet-application-with-mochiweb-part-3
    2 Million TCP connections (FreeBSD):
    http://blog.whatsapp.com/index.php/2012/01/1-million-is-so-2011/

    Note those two articles are not about the same thing. Make sure you're not
    confusing the difference between outgoing tcp connections and incoming tcp
    connections.

    If you just want a 1M tcp connections badge, do it with incoming
    connections.

    [...] I have added several additional IP addresses to my system and made go
    loop though these addresses. When I use a LocalAddr with port 0 go directly
    starts complaining with "address already in use" as soon I pass the +/-
    64511 connections. So I tied to loop over my ip range in combination with a
    specified port for the LocalAddr. Unfortunately this doesn't improve the
    number of concurrent connections.
    Does "doesn't improve" mean you get the same error, or something else?

    My connection handling code:
    RETRY:

    // get the firs source ip:port address in queue
    ipNext = <-iprange

    // manage our on port numbers per ip to overcome the 64511 limit
    host, port, _ := net.SplitHostPort(ipNext)
    nextPort, _ := strconv.Atoi(port)
    nextPort++
    if nextPort > 65535 {
    nextPort = 1024
    }
    // add this address with the next port number back to the end of the queue
    iprange <- host +":"+ strconv.Itoa(nextPort)

    d := net.Dialer{Timeout: 1*time.Second, Deadline:
    time.Now().Add(2*time.Second)}
    d.LocalAddr, err = net.ResolveTCPAddr("tcp4", ipNext)
    if err != nil {
    log.Fatal(err)
    }
    dc, err := d.Dial("tcp", dst)

    if err != nil && strings.Contains(err.Error(), "address already in use") {
    goto RETRY
    }
    I don't see where the host (for ipNext, for LocalAddr) gets incremented.
    Based on the above, I can only presume it doesn't and this would explain
    the behaviour you're seeing.

    Personally, I'd go for a simpler approach. For each local IP address launch
    a goroutine that dials out repeatedly (with exponential backoff, if you
    will). I wouldn't bother with trying to manage ephemeral ports either -
    just keep dialing independently on each IP and whatever limit you hit will
    probably be the ephemeral limit anyways.

    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.
  • Paul van Brouwershaven at Aug 18, 2013 at 7:51 am

    On Sunday, 18 August 2013 00:16:31 UTC+2, Benjamin Measures wrote:
    On Friday, 16 August 2013 11:45:40 UTC+1, Paul van Brouwershaven wrote:

    I'm currently trying to break the 64511 concurrent tcp sessions barrier
    in my GO application. [...]
    1 Million TCP connections (Linux):

    http://www.metabrew.com/article/a-million-user-comet-application-with-mochiweb-part-3
    2 Million TCP connections (FreeBSD):
    http://blog.whatsapp.com/index.php/2012/01/1-million-is-so-2011/

    Note those two articles are not about the same thing. Make sure you're not
    confusing the difference between outgoing tcp connections and incoming tcp
    connections.
    That's a really good point!

    If you just want a 1M tcp connections badge, do it with incoming
    connections.
    Unfortunately I need to make outgoing connections.

    [...] I have added several additional IP addresses to my system and made go
    loop though these addresses. When I use a LocalAddr with port 0 go directly
    starts complaining with "address already in use" as soon I pass the +/-
    64511 connections. So I tried to loop over my ip range in combination with
    a specified port for the LocalAddr. Unfortunately this doesn't improve the
    number of concurrent connections.
    Does "doesn't improve" mean you get the same error, or something else?
    I get the same "address already in use" error, I will try to make a small
    test program that I can share as soon I have a few minutes.

    I don't see where the host (for ipNext, for LocalAddr) gets incremented.
    Based on the above, I can only presume it doesn't and this would explain
    the behaviour you're seeing.

    Personally, I'd go for a simpler approach. For each local IP address
    launch a goroutine that dials out repeatedly (with exponential backoff, if
    you will). I wouldn't bother with trying to manage ephemeral ports either -
    just keep dialing independently on each IP and whatever limit you hit will
    probably be the ephemeral limit anyways.
    ipNext is a go channel, "ipNext = <-iprange" read one IP address of the
    channel, "iprange <- host +":"+ strconv.Itoa(nextPort)" adds that same IP
    address back to the channel but with the next port in range.

    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.
  • Dave Cheney at Aug 18, 2013 at 8:05 am
    Unfortunately I need to make outgoing connections.
    Please try using net.DialTCP to control the source address you connect
    from. You can get the list of available addresses from the net package
    using InterfaceAddrs(). You should construct a *net.TCPAddr for the
    source address with one of the IP addresses assigned to your outgoing
    interface and the port set to 0 to allow the operating system to
    choose an ephemeral outgoing port.

    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.
  • Paul van Brouwershaven at Aug 18, 2013 at 8:35 am

    On Sun, Aug 18, 2013 at 10:05 AM, Dave Cheney wrote:

    Please try using net.DialTCP to control the source address you connect
    from. You can get the list of available addresses from the net package
    using InterfaceAddrs(). You should construct a *net.TCPAddr for the
    source address with one of the IP addresses assigned to your outgoing
    interface and the port set to 0 to allow the operating system to
    choose an ephemeral outgoing port.
    I'm using the new net.Dialer which is much better than net.DialTCP as you
    can also specify a timeout and a local address. In the past I used a
    customized net.DialTCP to get to the same functionality.

    d := net.Dialer{Timeout: 1*time.Second, Deadline:
    time.Now().Add(2*time.Second)}
    d.LocalAddr, err = net.ResolveTCPAddr("tcp4", ipNext)

    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.
  • Dave Cheney at Aug 18, 2013 at 8:46 am
    Can you please post executable sample code.

    On Sun, Aug 18, 2013 at 6:35 PM, Paul van Brouwershaven
    wrote:
    On Sun, Aug 18, 2013 at 10:05 AM, Dave Cheney wrote:

    Please try using net.DialTCP to control the source address you connect
    from. You can get the list of available addresses from the net package
    using InterfaceAddrs(). You should construct a *net.TCPAddr for the
    source address with one of the IP addresses assigned to your outgoing
    interface and the port set to 0 to allow the operating system to
    choose an ephemeral outgoing port.

    I'm using the new net.Dialer which is much better than net.DialTCP as you
    can also specify a timeout and a local address. In the past I used a
    customized net.DialTCP to get to the same functionality.

    d := net.Dialer{Timeout: 1*time.Second, Deadline:
    time.Now().Add(2*time.Second)}
    d.LocalAddr, err = net.ResolveTCPAddr("tcp4", ipNext)
    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.
  • Paul van Brouwershaven at Aug 19, 2013 at 11:00 am

    On Sunday, 18 August 2013 10:46:53 UTC+2, Dave Cheney wrote:

    Can you please post executable sample code.

    Here is a demo code that is running 60k concurrent connections with a
    timeout of 2 seconds.

    http://play.golang.org/p/mQZ2nlUQsr

    Please note that you need to increase your file descriptors as below and
    you a multicore system to properly run this code!

    sysctl -w fs.file-max=999999
    ulimit -n `cat /proc/sys/fs/file-max`

    The code will print a status output like below every 5 seconds:

    2013/08/19 10:51:33 GO: 60007 || TCP: ESTABLISHED 4 SYN_SENT 28232 ||
    ERROR: dial tcp 10.2.149.179:80: address already in use

    /tmp/count.sh contains a count on current open tcp sessions:

    #!/bin/sh
    netstat -n | awk '/^tcp/ {t[$NF]++}END{for(state in t){print state,
    t[state]} }'


    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.
  • Dave Cheney at Aug 19, 2013 at 11:35 am
    Please try running your program under the race detector. I can see at least one data race, on lasterror which may be confusing your results.


    On 19/08/2013, at 21:00, Paul van Brouwershaven wrote:
    On Sunday, 18 August 2013 10:46:53 UTC+2, Dave Cheney wrote:
    Can you please post executable sample code.
    Here is a demo code that is running 60k concurrent connections with a timeout of 2 seconds.

    http://play.golang.org/p/mQZ2nlUQsr

    Please note that you need to increase your file descriptors as below and you a multicore system to properly run this code!

    sysctl -w fs.file-max=999999
    ulimit -n `cat /proc/sys/fs/file-max`

    The code will print a status output like below every 5 seconds:

    2013/08/19 10:51:33 GO: 60007 || TCP: ESTABLISHED 4 SYN_SENT 28232 || ERROR: dial tcp 10.2.149.179:80: address already in use

    /tmp/count.sh contains a count on current open tcp sessions:

    #!/bin/sh
    netstat -n | awk '/^tcp/ {t[$NF]++}END{for(state in t){print state, t[state]} }'


    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.
    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.
  • Paul van Brouwershaven at Aug 19, 2013 at 1:06 pm

    On Mon, Aug 19, 2013 at 1:35 PM, Dave Cheney wrote:

    Please try running your program under the race detector. I can see at
    least one data race, on lasterror which may be confusing your results.
    The race detector won't run with this number of routines, and with a low
    number of routines it doesn't report any errors.

    "lasterror" was the only variable accessed across routines, so I simply
    removed it as this is only a program that demonstrates the error.

    http://play.golang.org/p/wXxX_nIOk-

    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.
  • Dave Cheney at Aug 19, 2013 at 1:38 pm
    I don't understand why you need to use so many goroutines, why not just have one per source IP address? You will also need one goroutine per accepting IP address.


    On 19/08/2013, at 23:06, Paul van Brouwershaven wrote:
    On Mon, Aug 19, 2013 at 1:35 PM, Dave Cheney wrote:
    Please try running your program under the race detector. I can see at least one data race, on lasterror which may be confusing your results.
    The race detector won't run with this number of routines, and with a low number of routines it doesn't report any errors.

    "lasterror" was the only variable accessed across routines, so I simply removed it as this is only a program that demonstrates the error.

    http://play.golang.org/p/wXxX_nIOk-
    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.
  • Paul van Brouwershaven at Aug 19, 2013 at 1:51 pm
    That would slowdown the program as I would only have a few concurrent
    requests (as many as source IP addresses I have).

    On Mon, Aug 19, 2013 at 3:38 PM, Dave Cheney wrote:

    I don't understand why you need to use so many goroutines, why not just
    have one per source IP address? You will also need one goroutine per
    accepting IP address.



    On 19/08/2013, at 23:06, Paul van Brouwershaven wrote:
    On Mon, Aug 19, 2013 at 1:35 PM, Dave Cheney wrote:

    Please try running your program under the race detector. I can see at
    least one data race, on lasterror which may be confusing your results.
    The race detector won't run with this number of routines, and with a low
    number of routines it doesn't report any errors.

    "lasterror" was the only variable accessed across routines, so I simply
    removed it as this is only a program that demonstrates the error.

    http://play.golang.org/p/wXxX_nIOk-


    --
    Regards,

    Paul van Brouwershaven

    http://linkedin.com/in/pvanbrouwershaven
    http://facebook.com/p.vanbrouwershaven
    http://twitter.com/vanbroup

    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.
  • Dave Cheney at Aug 20, 2013 at 12:42 am

    That would slowdown the program as I would only have a few concurrent
    requests (as many as source IP addresses I have).
    Well, it's just a test, right ? I think it's more important to get
    your test right, rather than complete fast and show the wrong result.
    Also you will need multiple listeners bound to separate IP address to
    get more than 65k possible sockets.

    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.
  • Paul van Brouwershaven at Aug 20, 2013 at 6:36 am

    On Tue, Aug 20, 2013 at 2:42 AM, Dave Cheney wrote:

    That would slowdown the program as I would only have a few concurrent
    requests (as many as source IP addresses I have).
    Well, it's just a test, right ? I think it's more important to get
    your test right, rather than complete fast and show the wrong result.
    Also you will need multiple listeners bound to separate IP address to
    get more than 65k possible sockets.
    I'm running with a high concurrency to get to the 60k+ connections, running
    with less concurrency would cause the program to make less connections or
    keep the connection open longer than necessary.

    The race detector is not reporting any issues at a low concurrency, but I
    removed the lasterror that could likely cause an issue anyway.

    Is there anything else that you would like me to do?

    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.
  • Dave Cheney at Aug 20, 2013 at 7:04 am

    I'm running with a high concurrency to get to the 60k+ connections, running
    with less concurrency would cause the program to make less connections or
    keep the connection open longer than necessary.
    Why do you need to do that ? If you want to test how many connections
    you can have open in total, then open the connection, stick it in a
    map (so it doesn't get garbage collected), and do it again. When you
    start getting errors, move on to the next source IP.

    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.
  • Paul van Brouwershaven at Aug 20, 2013 at 9:19 am

    On Tue, Aug 20, 2013 at 9:04 AM, Dave Cheney wrote:

    I'm running with a high concurrency to get to the 60k+ connections, running
    with less concurrency would cause the program to make less connections or
    keep the connection open longer than necessary.
    Why do you need to do that ? If you want to test how many connections
    you can have open in total, then open the connection, stick it in a
    map (so it doesn't get garbage collected), and do it again. When you
    start getting errors, move on to the next source IP.
    I updated the program to run with a map, instead of using a goroutine and I
    also changed that I only move to a new ip address as soon I'm receiving an
    error message.

    The problem is that I'm not getting more concurrent tcp sessions than the
    number of routines I use. I tested for race conditions again and did not
    found any issues.

    You can find the updated test program here:
    http://play.golang.org/p/hNxbXLjSbS

    When running with 64000 routines, I get slightly over the 64511 (64542) but
    this keeps under the 65535 limit.

    2013/08/20 08:55:54 sockets: used 683 TCP: inuse 590 orphan 31 tw 2 alloc
    592 mem 771 UDP: inuse 0 mem 0 UDPLITE: inuse 0 RAW: inuse 0 FRAG: inuse 0
    memory 0
    2013/08/20 08:56:17 sockets: used 64677 TCP: inuse 64503 orphan 31 tw 2
    alloc 64586 mem 772 UDP: inuse 0 mem 0 UDPLITE: inuse 0 RAW: inuse 0 FRAG:
    inuse 0 memory 0
    2013/08/20 08:56:37 sockets: used 64680 TCP: inuse 64484 orphan 31 tw 0
    alloc 64589 mem 772 UDP: inuse 0 mem 0 UDPLITE: inuse 0 RAW: inuse 0 FRAG:
    inuse 1 memory 1480
    2013/08/20 08:57:17 sockets: used 64715 TCP: inuse *64521 *orphan 31 tw 0
    alloc 64624 mem 772 UDP: inuse 0 mem 0 UDPLITE: inuse 0 RAW: inuse 0 FRAG:
    inuse 0 memory 0
    2013/08/20 08:57:46 sockets: used 64723 TCP: inuse *64542 *orphan 31 tw 0
    alloc 64632 mem 771 UDP: inuse 0 mem 0 UDPLITE: inuse 0 RAW: inuse 0 FRAG:
    inuse 1 memory 1480
    2013/08/20 08:58:30 sockets: used 64407 TCP: inuse 62746 orphan 31 tw 0
    alloc 64316 mem 772 UDP: inuse 0 mem 0 UDPLITE: inuse 0 RAW: inuse 0 FRAG:
    inuse 0 memory 0

    When I run the test program with 64100 routines I can even get to 64632 but
    soon run into the error "address already in use" error.

    2013/08/20 09:07:33 sockets: used 708 TCP: inuse 614 orphan 30 tw 0 alloc
    616 mem 770 UDP: inuse 0 mem 0 UDPLITE: inuse 0 RAW: inuse 0 FRAG: inuse 0
    memory 0
    2013/08/20 09:08:04 sockets: used 64807 TCP: inuse *64632* orphan 30 tw 4
    alloc 64715 mem 771 UDP: inuse 0 mem 0 UDPLITE: inuse 0 RAW: inuse 0 FRAG:
    inuse 0 memory 0
    2013/08/20 09:08:23 *.*.*.1 -> 10.5.175.151 || dial tcp 10.5.175.151:80:
    address already in use
    2013/08/20 09:08:32 *.*.*.2 -> 10.5.175.134 || dial tcp 10.5.175.134:80:
    address already in use
    2013/08/20 09:08:32 *.*.*.3 -> 10.5.175.157 || dial tcp 10.5.175.157:80:
    address already in use
    2013/08/20 09:08:32 *.*.*.4 -> 10.5.175.161 || dial tcp 10.5.175.161:80:
    address already in use
    2013/08/20 09:08:32 *.*.*.5 -> 10.5.175.149 || dial tcp 10.5.175.149:80:
    address already in use
    2013/08/20 09:08:32 *.*.*.6 -> 10.5.175.166 || dial tcp 10.5.175.166:80:
    address already in use
    2013/08/20 09:08:32 *.*.*.7 -> 10.5.175.146 || dial tcp 10.5.175.146:80:
    address already in use

    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.
  • Kyle Lemons at Aug 20, 2013 at 4:49 pm

    On Tue, Aug 20, 2013 at 2:19 AM, Paul van Brouwershaven wrote:
    On Tue, Aug 20, 2013 at 9:04 AM, Dave Cheney wrote:

    I'm running with a high concurrency to get to the 60k+ connections, running
    with less concurrency would cause the program to make less connections or
    keep the connection open longer than necessary.
    Why do you need to do that ? If you want to test how many connections
    you can have open in total, then open the connection, stick it in a
    map (so it doesn't get garbage collected), and do it again. When you
    start getting errors, move on to the next source IP.
    I updated the program to run with a map, instead of using a goroutine and
    I also changed that I only move to a new ip address as soon I'm receiving
    an error message.

    The problem is that I'm not getting more concurrent tcp sessions than the
    number of routines I use. I tested for race conditions again and did not
    found any issues.

    You can find the updated test program here:
    http://play.golang.org/p/hNxbXLjSbS

    When running with 64000 routines, I get slightly over the 64511 (64542)
    but this keeps under the 65535 limit.

    2013/08/20 08:55:54 sockets: used 683 TCP: inuse 590 orphan 31 tw 2 alloc
    592 mem 771 UDP: inuse 0 mem 0 UDPLITE: inuse 0 RAW: inuse 0 FRAG: inuse 0
    memory 0
    2013/08/20 08:56:17 sockets: used 64677 TCP: inuse 64503 orphan 31 tw 2
    alloc 64586 mem 772 UDP: inuse 0 mem 0 UDPLITE: inuse 0 RAW: inuse 0 FRAG:
    inuse 0 memory 0
    2013/08/20 08:56:37 sockets: used 64680 TCP: inuse 64484 orphan 31 tw 0
    alloc 64589 mem 772 UDP: inuse 0 mem 0 UDPLITE: inuse 0 RAW: inuse 0 FRAG:
    inuse 1 memory 1480
    2013/08/20 08:57:17 sockets: used 64715 TCP: inuse *64521 *orphan 31 tw 0
    alloc 64624 mem 772 UDP: inuse 0 mem 0 UDPLITE: inuse 0 RAW: inuse 0 FRAG:
    inuse 0 memory 0
    2013/08/20 08:57:46 sockets: used 64723 TCP: inuse *64542 *orphan 31 tw 0
    alloc 64632 mem 771 UDP: inuse 0 mem 0 UDPLITE: inuse 0 RAW: inuse 0 FRAG:
    inuse 1 memory 1480
    2013/08/20 08:58:30 sockets: used 64407 TCP: inuse 62746 orphan 31 tw 0
    alloc 64316 mem 772 UDP: inuse 0 mem 0 UDPLITE: inuse 0 RAW: inuse 0 FRAG:
    inuse 0 memory 0

    When I run the test program with 64100 routines I can even get to 64632
    but soon run into the error "address already in use" error.

    2013/08/20 09:07:33 sockets: used 708 TCP: inuse 614 orphan 30 tw 0 alloc
    616 mem 770 UDP: inuse 0 mem 0 UDPLITE: inuse 0 RAW: inuse 0 FRAG: inuse 0
    memory 0
    2013/08/20 09:08:04 sockets: used 64807 TCP: inuse *64632* orphan 30 tw 4
    alloc 64715 mem 771 UDP: inuse 0 mem 0 UDPLITE: inuse 0 RAW: inuse 0 FRAG:
    inuse 0 memory 0
    2013/08/20 09:08:23 *.*.*.1 -> 10.5.175.151 || dial tcp 10.5.175.151:80:
    address already in use
    2013/08/20 09:08:32 *.*.*.2 -> 10.5.175.134 || dial tcp 10.5.175.134:80:
    address already in use
    2013/08/20 09:08:32 *.*.*.3 -> 10.5.175.157 || dial tcp 10.5.175.157:80:
    address already in use
    2013/08/20 09:08:32 *.*.*.4 -> 10.5.175.161 || dial tcp 10.5.175.161:80:
    address already in use
    2013/08/20 09:08:32 *.*.*.5 -> 10.5.175.149 || dial tcp 10.5.175.149:80:
    address already in use
    2013/08/20 09:08:32 *.*.*.6 -> 10.5.175.166 || dial tcp 10.5.175.166:80:
    address already in use
    2013/08/20 09:08:32 *.*.*.7 -> 10.5.175.146 || dial tcp 10.5.175.146:80:
    address already in use
    Are you getting these address already in use by coming up with the ports
    yourself or by letting the OS choose them?
    --
    You received this message because you are subscribed to the Google Groups
    "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an
    email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.
    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.
  • Paul van Brouwershaven at Aug 20, 2013 at 5:22 pm
    Are you getting these address already in use by coming up with the ports
    yourself or by letting the OS choose them?

    I'm letting the OS select the ports now. But I get the same error when I
    make the selection.

    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.
  • Benjamin Measures at Aug 20, 2013 at 11:13 pm

    On Tuesday, 20 August 2013 10:19:44 UTC+1, Paul van Brouwershaven wrote:

    2013/08/20 08:57:46 sockets: used 64723 TCP: inuse *64542 *orphan 31 tw 0
    alloc 64632 mem 771 UDP: inuse 0 mem 0 UDPLITE: inuse 0 RAW: inuse 0 FRAG:
    inuse 1 memory 1480
    2013/08/20 08:58:30 sockets: used 64407 TCP: inuse 62746 orphan 31 tw 0
    alloc 64316 mem 772 UDP: inuse 0 mem 0 UDPLITE: inuse 0 RAW: inuse 0 FRAG:
    inuse 0 memory 0
    The TCP sockets inuse has decreased. This would suggest sockets are being
    closed.

    When I run the test program with 64100 routines I can even get to 64632 but
    soon run into the error "address already in use" error.
    2013/08/20 09:07:33 sockets: used 708 TCP: inuse 614 orphan 30 tw 0 alloc
    616 mem 770 UDP: inuse 0 mem 0 UDPLITE: inuse 0 RAW: inuse 0 FRAG: inuse 0
    memory 0
    2013/08/20 09:08:04 sockets: used 64807 TCP: inuse *64632* orphan 30 tw 4
    alloc 64715 mem 771 UDP: inuse 0 mem 0 UDPLITE: inuse 0 RAW: inuse 0 FRAG:
    inuse 0 memory 0
    2013/08/20 09:08:23 *.*.*.1 -> 10.5.175.151 || dial tcp 10.5.175.151:80:
    address already in use
    Using more goroutines will cause connections to be opened (and closed) at a
    greater rate. That the error doesn't appear until ~20s after peak/high
    inuse, would support the CLOSE_WAIT issue as outlined earlier in this
    thread.

    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.
  • Dave Cheney at Aug 21, 2013 at 5:40 am
    Please try this sample code

    http://play.golang.org/p/y2319158oP

    I get to 114688 connections without any serious tcp tuning.

    On Wed, Aug 21, 2013 at 9:13 AM, Benjamin Measures
    wrote:
    On Tuesday, 20 August 2013 10:19:44 UTC+1, Paul van Brouwershaven wrote:

    2013/08/20 08:57:46 sockets: used 64723 TCP: inuse 64542 orphan 31 tw 0
    alloc 64632 mem 771 UDP: inuse 0 mem 0 UDPLITE: inuse 0 RAW: inuse 0 FRAG:
    inuse 1 memory 1480
    2013/08/20 08:58:30 sockets: used 64407 TCP: inuse 62746 orphan 31 tw 0
    alloc 64316 mem 772 UDP: inuse 0 mem 0 UDPLITE: inuse 0 RAW: inuse 0 FRAG:
    inuse 0 memory 0

    The TCP sockets inuse has decreased. This would suggest sockets are being
    closed.
    When I run the test program with 64100 routines I can even get to 64632
    but soon run into the error "address already in use" error.

    2013/08/20 09:07:33 sockets: used 708 TCP: inuse 614 orphan 30 tw 0 alloc
    616 mem 770 UDP: inuse 0 mem 0 UDPLITE: inuse 0 RAW: inuse 0 FRAG: inuse 0
    memory 0
    2013/08/20 09:08:04 sockets: used 64807 TCP: inuse 64632 orphan 30 tw 4
    alloc 64715 mem 771 UDP: inuse 0 mem 0 UDPLITE: inuse 0 RAW: inuse 0 FRAG:
    inuse 0 memory 0
    2013/08/20 09:08:23 *.*.*.1 -> 10.5.175.151 || dial tcp 10.5.175.151:80:
    address already in use

    Using more goroutines will cause connections to be opened (and closed) at a
    greater rate. That the error doesn't appear until ~20s after peak/high
    inuse, would support the CLOSE_WAIT issue as outlined earlier in this
    thread.
    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.
  • Paul van Brouwershaven at Aug 21, 2013 at 7:34 am
    I tested your sample and could get a 128997 established connections, but
    this is only 63964 outgoing connections. This is about the same result as
    with my test but you now have doubled this by counting incoming and
    outgoing connections.

    2013/08/21 06:54:20 starting listeners
    2013/08/21 06:54:20 listeners started, starting clients
    2013/08/21 06:54:20 clients started, waiting for completion
    2013/08/21 06:54:23 client 127.0.0.106:0: connection *7993*: dial tcp
    127.0.0.206:8000: address already in use
    2013/08/21 06:54:23 client 127.0.0.107:0: connection *7993*: dial tcp
    127.0.0.207:8000: address already in use
    2013/08/21 06:54:24 client 127.0.0.102:0: connection *7993*: dial tcp
    127.0.0.202:8000: address already in use
    2013/08/21 06:54:24 client 127.0.0.103:0: connection *7996*: dial tcp
    127.0.0.203:8000: address already in use
    2013/08/21 06:54:24 client 127.0.0.100:0: connection *7995*: dial tcp
    127.0.0.200:8000: address already in use
    2013/08/21 06:54:24 client 127.0.0.104:0: connection *7997*: dial tcp
    127.0.0.204:8000: address already in use
    2013/08/21 06:54:24 client 127.0.0.101:0: connection *7999*: dial tcp
    127.0.0.201:8000: address already in use
    2013/08/21 06:54:24 client 127.0.0.105:0: connection *7998*: dial tcp
    127.0.0.205:8000: address already in use

      --------- +

      *63964*
    *
    *
    2013/08/21 06:54:24 All clients connected, pausing now to let you
    investigate the process

    2013/08/21 06:54:24 sockets: used 129118 TCP: inuse 129023 orphan 11 tw 2
    alloc 129025 mem 1774 UDP: inuse 0 mem 0 UDPLITE: inuse 0 RAW: inuse 0
    FRAG: inuse 0 memory 0

    ESTABLISHED 128997

    As soon I extend the number of source and target addresses I get the
    following:

    2013/08/21 07:30:19 starting listeners
    2013/08/21 07:30:19 listeners started, starting clients
    2013/08/21 07:30:19 clients started, waiting for completion
    2013/08/21 07:30:23 client 127.0.0.106:0: connection *6395*: dial tcp
    127.0.0.206:8000: address already in use
    2013/08/21 07:30:23 client 127.0.0.101:0: connection *6395*: dial tcp
    127.0.0.201:8000: address already in use
    2013/08/21 07:30:23 client 127.0.0.103:0: connection *6397*: dial tcp
    127.0.0.203:8000: address already in use
    2013/08/21 07:30:23 client 127.0.0.102:0: connection *6395*: dial tcp
    127.0.0.202:8000: address already in use
    2013/08/21 07:30:23 client 127.0.0.105:0: connection *6396*: dial tcp
    127.0.0.205:8000: address already in use
    2013/08/21 07:30:23 client 127.0.0.104:0: connection *6394*: dial tcp
    127.0.0.204:8000: address already in use
    2013/08/21 07:30:23 client 127.0.0.109:0: connection *6391*: dial tcp
    127.0.0.109:8000: address already in use
    2013/08/21 07:30:23 client 127.0.0.100:0: connection *6394*: dial tcp
    127.0.0.200:8000: address already in use
    2013/08/21 07:30:23 client 127.0.0.108:0: connection *6392*: dial tcp
    127.0.0.108:8000: address already in use
    2013/08/21 07:30:23 client 127.0.0.107:0: connection *6391*: dial tcp
    127.0.0.207:8000: address already in use

      --------- +

      *63940*

    2013/08/21 07:30:23 All clients connected, pausing now to let you
    investigate the process

    2013/08/21 07:30:23 sockets: used 129137 TCP: inuse 129036 orphan 11 tw 0
    alloc 129038 mem 1778 UDP: inuse 0 mem 0 UDPLITE: inuse 0 RAW: inuse 0
    FRAG: inuse 0 memory 0



    On Wed, Aug 21, 2013 at 7:40 AM, Dave Cheney wrote:

    Please try this sample code

    http://play.golang.org/p/y2319158oP

    I get to 114688 connections without any serious tcp tuning.

    On Wed, Aug 21, 2013 at 9:13 AM, Benjamin Measures
    wrote:
    On Tuesday, 20 August 2013 10:19:44 UTC+1, Paul van Brouwershaven wrote:

    2013/08/20 08:57:46 sockets: used 64723 TCP: inuse 64542 orphan 31 tw 0
    alloc 64632 mem 771 UDP: inuse 0 mem 0 UDPLITE: inuse 0 RAW: inuse 0
    FRAG:
    inuse 1 memory 1480
    2013/08/20 08:58:30 sockets: used 64407 TCP: inuse 62746 orphan 31 tw 0
    alloc 64316 mem 772 UDP: inuse 0 mem 0 UDPLITE: inuse 0 RAW: inuse 0
    FRAG:
    inuse 0 memory 0

    The TCP sockets inuse has decreased. This would suggest sockets are being
    closed.
    When I run the test program with 64100 routines I can even get to 64632
    but soon run into the error "address already in use" error.

    2013/08/20 09:07:33 sockets: used 708 TCP: inuse 614 orphan 30 tw 0
    alloc
    616 mem 770 UDP: inuse 0 mem 0 UDPLITE: inuse 0 RAW: inuse 0 FRAG:
    inuse 0
    memory 0
    2013/08/20 09:08:04 sockets: used 64807 TCP: inuse 64632 orphan 30 tw 4
    alloc 64715 mem 771 UDP: inuse 0 mem 0 UDPLITE: inuse 0 RAW: inuse 0
    FRAG:
    inuse 0 memory 0
    2013/08/20 09:08:23 *.*.*.1 -> 10.5.175.151 || dial tcp 10.5.175.151:80
    :
    address already in use

    Using more goroutines will cause connections to be opened (and closed) at a
    greater rate. That the error doesn't appear until ~20s after peak/high
    inuse, would support the CLOSE_WAIT issue as outlined earlier in this
    thread.
    --
    You received this message because you are subscribed to a topic in the
    Google Groups "golang-nuts" group.
    To unsubscribe from this topic, visit
    https://groups.google.com/d/topic/golang-nuts/Mi7QkAqP7II/unsubscribe.
    To unsubscribe from this group and all its topics, send an email to
    golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.


    --
    Regards,

    Paul van Brouwershaven

    http://linkedin.com/in/pvanbrouwershaven
    http://facebook.com/p.vanbrouwershaven
    http://twitter.com/vanbroup

    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.
  • Dave Cheney at Aug 21, 2013 at 7:52 am
    Ok, I don't really understand what is required now. I've demonstrated you can hold up 120k sockets from a single process. I think this can easily be extended to a client and a server process, then the rest is tcp tuning. What am I missing ?
    On 21/08/2013, at 17:34, Paul van Brouwershaven wrote:

    I tested your sample and could get a 128997 established connections, but this is only 63964 outgoing connections. This is about the same result as with my test but you now have doubled this by counting incoming and outgoing connections.

    2013/08/21 06:54:20 starting listeners
    2013/08/21 06:54:20 listeners started, starting clients
    2013/08/21 06:54:20 clients started, waiting for completion
    2013/08/21 06:54:23 client 127.0.0.106:0: connection 7993: dial tcp 127.0.0.206:8000: address already in use
    2013/08/21 06:54:23 client 127.0.0.107:0: connection 7993: dial tcp 127.0.0.207:8000: address already in use
    2013/08/21 06:54:24 client 127.0.0.102:0: connection 7993: dial tcp 127.0.0.202:8000: address already in use
    2013/08/21 06:54:24 client 127.0.0.103:0: connection 7996: dial tcp 127.0.0.203:8000: address already in use
    2013/08/21 06:54:24 client 127.0.0.100:0: connection 7995: dial tcp 127.0.0.200:8000: address already in use
    2013/08/21 06:54:24 client 127.0.0.104:0: connection 7997: dial tcp 127.0.0.204:8000: address already in use
    2013/08/21 06:54:24 client 127.0.0.101:0: connection 7999: dial tcp 127.0.0.201:8000: address already in use
    2013/08/21 06:54:24 client 127.0.0.105:0: connection 7998: dial tcp 127.0.0.205:8000: address already in use
    --------- +
    63964

    2013/08/21 06:54:24 All clients connected, pausing now to let you investigate the process

    2013/08/21 06:54:24 sockets: used 129118 TCP: inuse 129023 orphan 11 tw 2 alloc 129025 mem 1774 UDP: inuse 0 mem 0 UDPLITE: inuse 0 RAW: inuse 0 FRAG: inuse 0 memory 0

    ESTABLISHED 128997

    As soon I extend the number of source and target addresses I get the following:

    2013/08/21 07:30:19 starting listeners
    2013/08/21 07:30:19 listeners started, starting clients
    2013/08/21 07:30:19 clients started, waiting for completion
    2013/08/21 07:30:23 client 127.0.0.106:0: connection 6395: dial tcp 127.0.0.206:8000: address already in use
    2013/08/21 07:30:23 client 127.0.0.101:0: connection 6395: dial tcp 127.0.0.201:8000: address already in use
    2013/08/21 07:30:23 client 127.0.0.103:0: connection 6397: dial tcp 127.0.0.203:8000: address already in use
    2013/08/21 07:30:23 client 127.0.0.102:0: connection 6395: dial tcp 127.0.0.202:8000: address already in use
    2013/08/21 07:30:23 client 127.0.0.105:0: connection 6396: dial tcp 127.0.0.205:8000: address already in use
    2013/08/21 07:30:23 client 127.0.0.104:0: connection 6394: dial tcp 127.0.0.204:8000: address already in use
    2013/08/21 07:30:23 client 127.0.0.109:0: connection 6391: dial tcp 127.0.0.109:8000: address already in use
    2013/08/21 07:30:23 client 127.0.0.100:0: connection 6394: dial tcp 127.0.0.200:8000: address already in use
    2013/08/21 07:30:23 client 127.0.0.108:0: connection 6392: dial tcp 127.0.0.108:8000: address already in use
    2013/08/21 07:30:23 client 127.0.0.107:0: connection 6391: dial tcp 127.0.0.207:8000: address already in use
    --------- +
    63940

    2013/08/21 07:30:23 All clients connected, pausing now to let you investigate the process

    2013/08/21 07:30:23 sockets: used 129137 TCP: inuse 129036 orphan 11 tw 0 alloc 129038 mem 1778 UDP: inuse 0 mem 0 UDPLITE: inuse 0 RAW: inuse 0 FRAG: inuse 0 memory 0



    On Wed, Aug 21, 2013 at 7:40 AM, Dave Cheney wrote:
    Please try this sample code

    http://play.golang.org/p/y2319158oP

    I get to 114688 connections without any serious tcp tuning.

    On Wed, Aug 21, 2013 at 9:13 AM, Benjamin Measures
    wrote:
    On Tuesday, 20 August 2013 10:19:44 UTC+1, Paul van Brouwershaven wrote:

    2013/08/20 08:57:46 sockets: used 64723 TCP: inuse 64542 orphan 31 tw 0
    alloc 64632 mem 771 UDP: inuse 0 mem 0 UDPLITE: inuse 0 RAW: inuse 0 FRAG:
    inuse 1 memory 1480
    2013/08/20 08:58:30 sockets: used 64407 TCP: inuse 62746 orphan 31 tw 0
    alloc 64316 mem 772 UDP: inuse 0 mem 0 UDPLITE: inuse 0 RAW: inuse 0 FRAG:
    inuse 0 memory 0

    The TCP sockets inuse has decreased. This would suggest sockets are being
    closed.
    When I run the test program with 64100 routines I can even get to 64632
    but soon run into the error "address already in use" error.

    2013/08/20 09:07:33 sockets: used 708 TCP: inuse 614 orphan 30 tw 0 alloc
    616 mem 770 UDP: inuse 0 mem 0 UDPLITE: inuse 0 RAW: inuse 0 FRAG: inuse 0
    memory 0
    2013/08/20 09:08:04 sockets: used 64807 TCP: inuse 64632 orphan 30 tw 4
    alloc 64715 mem 771 UDP: inuse 0 mem 0 UDPLITE: inuse 0 RAW: inuse 0 FRAG:
    inuse 0 memory 0
    2013/08/20 09:08:23 *.*.*.1 -> 10.5.175.151 || dial tcp 10.5.175.151:80:
    address already in use

    Using more goroutines will cause connections to be opened (and closed) at a
    greater rate. That the error doesn't appear until ~20s after peak/high
    inuse, would support the CLOSE_WAIT issue as outlined earlier in this
    thread.
    --
    You received this message because you are subscribed to a topic in the Google Groups "golang-nuts" group.
    To unsubscribe from this topic, visit https://groups.google.com/d/topic/golang-nuts/Mi7QkAqP7II/unsubscribe.
    To unsubscribe from this group and all its topics, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.


    --
    Regards,

    Paul van Brouwershaven

    http://linkedin.com/in/pvanbrouwershaven
    http://facebook.com/p.vanbrouwershaven
    http://twitter.com/vanbroup
    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.
  • Kyle Lemons at Aug 21, 2013 at 8:06 am
    Interestingly:

    2013/08/21 08:00:46 starting listeners
    2013/08/21 08:00:46 listeners started, starting clients
    2013/08/21 08:00:46 clients started, waiting for completion
    2013/08/21 08:01:52 client 127.0.0.101:0: connection 8189: dial tcp
    127.0.0.201:8000: connection timed out
    2013/08/21 08:01:52 client 127.0.0.100:0: connection 8189: dial tcp
    127.0.0.200:8000: connection timed out
    2013/08/21 08:01:52 client 127.0.0.105:0: connection 8191: dial tcp
    127.0.0.205:8000: connection timed out
    2013/08/21 08:01:52 client 127.0.0.102:0: connection 8189: dial tcp
    127.0.0.202:8000: connection timed out
    2013/08/21 08:01:52 client 127.0.0.107:0: connection 8189: dial tcp
    127.0.0.207:8000: connection timed out
    2013/08/21 08:01:52 client 127.0.0.106:0: connection 8189: dial tcp
    127.0.0.206:8000: connection timed out
    2013/08/21 08:01:52 client 127.0.0.103:0: connection 8188: dial tcp
    127.0.0.203:8000: connection timed out
    2013/08/21 08:01:52 client 127.0.0.104:0: connection 8190: dial tcp
    127.0.0.204:8000: connection timed out
    2013/08/21 08:01:52 All clients connected, pausing now to let you
    investigate the process

    for a total of 65514. No tuning other than what you posted in
    https://code.google.com/p/go/issues/detail?id=6176#c4

    On Wed, Aug 21, 2013 at 12:52 AM, Dave Cheney wrote:

    Ok, I don't really understand what is required now. I've demonstrated you
    can hold up 120k sockets from a single process. I think this can easily be
    extended to a client and a server process, then the rest is tcp tuning.
    What am I missing ?

    On 21/08/2013, at 17:34, Paul van Brouwershaven wrote:

    I tested your sample and could get a 128997 established connections, but
    this is only 63964 outgoing connections. This is about the same result as
    with my test but you now have doubled this by counting incoming and
    outgoing connections.

    2013/08/21 06:54:20 starting listeners
    2013/08/21 06:54:20 listeners started, starting clients
    2013/08/21 06:54:20 clients started, waiting for completion
    2013/08/21 06:54:23 client 127.0.0.106:0: connection *7993*: dial tcp
    127.0.0.206:8000: address already in use
    2013/08/21 06:54:23 client 127.0.0.107:0: connection *7993*: dial tcp
    127.0.0.207:8000: address already in use
    2013/08/21 06:54:24 client 127.0.0.102:0: connection *7993*: dial tcp
    127.0.0.202:8000: address already in use
    2013/08/21 06:54:24 client 127.0.0.103:0: connection *7996*: dial tcp
    127.0.0.203:8000: address already in use
    2013/08/21 06:54:24 client 127.0.0.100:0: connection *7995*: dial tcp
    127.0.0.200:8000: address already in use
    2013/08/21 06:54:24 client 127.0.0.104:0: connection *7997*: dial tcp
    127.0.0.204:8000: address already in use
    2013/08/21 06:54:24 client 127.0.0.101:0: connection *7999*: dial tcp
    127.0.0.201:8000: address already in use
    2013/08/21 06:54:24 client 127.0.0.105:0: connection *7998*: dial tcp
    127.0.0.205:8000: address already in use

    --------- +

    *63964*
    *
    *
    2013/08/21 06:54:24 All clients connected, pausing now to let you
    investigate the process

    2013/08/21 06:54:24 sockets: used 129118 TCP: inuse 129023 orphan 11 tw 2
    alloc 129025 mem 1774 UDP: inuse 0 mem 0 UDPLITE: inuse 0 RAW: inuse 0
    FRAG: inuse 0 memory 0

    ESTABLISHED 128997

    As soon I extend the number of source and target addresses I get the
    following:

    2013/08/21 07:30:19 starting listeners
    2013/08/21 07:30:19 listeners started, starting clients
    2013/08/21 07:30:19 clients started, waiting for completion
    2013/08/21 07:30:23 client 127.0.0.106:0: connection *6395*: dial tcp
    127.0.0.206:8000: address already in use
    2013/08/21 07:30:23 client 127.0.0.101:0: connection *6395*: dial tcp
    127.0.0.201:8000: address already in use
    2013/08/21 07:30:23 client 127.0.0.103:0: connection *6397*: dial tcp
    127.0.0.203:8000: address already in use
    2013/08/21 07:30:23 client 127.0.0.102:0: connection *6395*: dial tcp
    127.0.0.202:8000: address already in use
    2013/08/21 07:30:23 client 127.0.0.105:0: connection *6396*: dial tcp
    127.0.0.205:8000: address already in use
    2013/08/21 07:30:23 client 127.0.0.104:0: connection *6394*: dial tcp
    127.0.0.204:8000: address already in use
    2013/08/21 07:30:23 client 127.0.0.109:0: connection *6391*: dial tcp
    127.0.0.109:8000: address already in use
    2013/08/21 07:30:23 client 127.0.0.100:0: connection *6394*: dial tcp
    127.0.0.200:8000: address already in use
    2013/08/21 07:30:23 client 127.0.0.108:0: connection *6392*: dial tcp
    127.0.0.108:8000: address already in use
    2013/08/21 07:30:23 client 127.0.0.107:0: connection *6391*: dial tcp
    127.0.0.207:8000: address already in use

    --------- +

    *63940*

    2013/08/21 07:30:23 All clients connected, pausing now to let you
    investigate the process

    2013/08/21 07:30:23 sockets: used 129137 TCP: inuse 129036 orphan 11 tw 0
    alloc 129038 mem 1778 UDP: inuse 0 mem 0 UDPLITE: inuse 0 RAW: inuse 0
    FRAG: inuse 0 memory 0



    On Wed, Aug 21, 2013 at 7:40 AM, Dave Cheney wrote:

    Please try this sample code

    http://play.golang.org/p/y2319158oP

    I get to 114688 connections without any serious tcp tuning.

    On Wed, Aug 21, 2013 at 9:13 AM, Benjamin Measures
    wrote:
    On Tuesday, 20 August 2013 10:19:44 UTC+1, Paul van Brouwershaven wrote:

    2013/08/20 08:57:46 sockets: used 64723 TCP: inuse 64542 orphan 31 tw 0
    alloc 64632 mem 771 UDP: inuse 0 mem 0 UDPLITE: inuse 0 RAW: inuse 0
    FRAG:
    inuse 1 memory 1480
    2013/08/20 08:58:30 sockets: used 64407 TCP: inuse 62746 orphan 31 tw 0
    alloc 64316 mem 772 UDP: inuse 0 mem 0 UDPLITE: inuse 0 RAW: inuse 0
    FRAG:
    inuse 0 memory 0

    The TCP sockets inuse has decreased. This would suggest sockets are being
    closed.
    When I run the test program with 64100 routines I can even get to 64632
    but soon run into the error "address already in use" error.

    2013/08/20 09:07:33 sockets: used 708 TCP: inuse 614 orphan 30 tw 0
    alloc
    616 mem 770 UDP: inuse 0 mem 0 UDPLITE: inuse 0 RAW: inuse 0 FRAG:
    inuse 0
    memory 0
    2013/08/20 09:08:04 sockets: used 64807 TCP: inuse 64632 orphan 30 tw 4
    alloc 64715 mem 771 UDP: inuse 0 mem 0 UDPLITE: inuse 0 RAW: inuse 0
    FRAG:
    inuse 0 memory 0
    2013/08/20 09:08:23 *.*.*.1 -> 10.5.175.151 || dial tcp
    10.5.175.151:80:
    address already in use

    Using more goroutines will cause connections to be opened (and closed) at a
    greater rate. That the error doesn't appear until ~20s after peak/high
    inuse, would support the CLOSE_WAIT issue as outlined earlier in this
    thread.
    --
    You received this message because you are subscribed to a topic in the
    Google Groups "golang-nuts" group.
    To unsubscribe from this topic, visit
    https://groups.google.com/d/topic/golang-nuts/Mi7QkAqP7II/unsubscribe.
    To unsubscribe from this group and all its topics, send an email to
    golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.


    --
    Regards,

    Paul van Brouwershaven

    http://linkedin.com/in/pvanbrouwershaven
    http://facebook.com/p.vanbrouwershaven
    http://twitter.com/vanbroup

    --
    You received this message because you are subscribed to the Google Groups
    "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an
    email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.
    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.
  • Paul van Brouwershaven at Aug 21, 2013 at 8:18 am

    On Wed, Aug 21, 2013 at 10:06 AM, Kyle Lemons wrote:

    for a total of 65514. No tuning other than what you posted in
    https://code.google.com/p/go/issues/detail?id=6176#c4
    This is still lower than the 65535, the slight difference why you can't
    reach the 655535 exactly is more depending on other connections running on
    your test machine.

    But it's the intention to pass this 65535, which in this case should be go
    up to 7 (ip numbers) * 65535 (ports) = 458745 outgoing connections.

    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.
  • Kyle Lemons at Aug 21, 2013 at 9:31 am
    Do you have a C program that can do this? I have been playing around and
    still can't exceed 65518 outgoing connections even across 64 processes,
    regardless of how many destination IPs I use.

    I'm more and more convinced that this is a linux issue, not a Go issue.

    On Wed, Aug 21, 2013 at 1:18 AM, Paul van Brouwershaven wrote:
    On Wed, Aug 21, 2013 at 10:06 AM, Kyle Lemons wrote:

    for a total of 65514. No tuning other than what you posted in
    https://code.google.com/p/go/issues/detail?id=6176#c4
    This is still lower than the 65535, the slight difference why you can't
    reach the 655535 exactly is more depending on other connections running on
    your test machine.

    But it's the intention to pass this 65535, which in this case should be
    go up to 7 (ip numbers) * 65535 (ports) = 458745 outgoing connections.

    --
    You received this message because you are subscribed to the Google Groups
    "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an
    email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.
    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.
  • Kyle Lemons at Aug 21, 2013 at 9:41 am
    Aha, here's a fun clue:

    Aug 21 09:32:01 localhost kernel: [18603209.749683] nf_conntrack: table
    full, dropping packet.

    Quick google search leads me
    to /proc/sys/net/ipv4/netfilter/ip_conntrack_max

    which I doubled (to 131072), and now I can get 131056 consistently.

    On Wed, Aug 21, 2013 at 2:30 AM, Kyle Lemons wrote:

    Do you have a C program that can do this? I have been playing around and
    still can't exceed 65518 outgoing connections even across 64 processes,
    regardless of how many destination IPs I use.

    I'm more and more convinced that this is a linux issue, not a Go issue.


    On Wed, Aug 21, 2013 at 1:18 AM, Paul van Brouwershaven <
    paul@vanbrouwershaven.com> wrote:
    On Wed, Aug 21, 2013 at 10:06 AM, Kyle Lemons wrote:

    for a total of 65514. No tuning other than what you posted in
    https://code.google.com/p/go/issues/detail?id=6176#c4
    This is still lower than the 65535, the slight difference why you can't
    reach the 655535 exactly is more depending on other connections running on
    your test machine.

    But it's the intention to pass this 65535, which in this case should be
    go up to 7 (ip numbers) * 65535 (ports) = 458745 outgoing connections.

    --
    You received this message because you are subscribed to the Google Groups
    "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an
    email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.
    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.
  • Paul van Brouwershaven at Aug 21, 2013 at 9:47 am
    If you remove the nf_conntrack module from your kernel (and have some fun
    with your manual firewall) you can go even much higher. The connection
    tracking module is abusing resources of your system.

    On Wed, Aug 21, 2013 at 11:41 AM, Kyle Lemons wrote:

    Aha, here's a fun clue:

    Aug 21 09:32:01 localhost kernel: [18603209.749683] nf_conntrack: table
    full, dropping packet.

    Quick google search leads me
    to /proc/sys/net/ipv4/netfilter/ip_conntrack_max

    which I doubled (to 131072), and now I can get 131056 consistently.

    On Wed, Aug 21, 2013 at 2:30 AM, Kyle Lemons wrote:

    Do you have a C program that can do this? I have been playing around and
    still can't exceed 65518 outgoing connections even across 64 processes,
    regardless of how many destination IPs I use.

    I'm more and more convinced that this is a linux issue, not a Go issue.


    On Wed, Aug 21, 2013 at 1:18 AM, Paul van Brouwershaven <
    paul@vanbrouwershaven.com> wrote:
    On Wed, Aug 21, 2013 at 10:06 AM, Kyle Lemons wrote:

    for a total of 65514. No tuning other than what you posted in
    https://code.google.com/p/go/issues/detail?id=6176#c4
    This is still lower than the 65535, the slight difference why you can't
    reach the 655535 exactly is more depending on other connections running on
    your test machine.

    But it's the intention to pass this 65535, which in this case should be
    go up to 7 (ip numbers) * 65535 (ports) = 458745 outgoing connections.

    --
    You received this message because you are subscribed to the Google
    Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send
    an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.

    --
    Regards,

    Paul van Brouwershaven

    http://linkedin.com/in/pvanbrouwershaven
    http://facebook.com/p.vanbrouwershaven
    http://twitter.com/vanbroup

    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.
  • Paul van Brouwershaven at Aug 21, 2013 at 9:45 am
    It looks like I found the problem, I went back to handling my ports
    manually but now I excluded all ports that are in use as listener on my
    local system, as for example 27017 for mongodb.

    Then I moved the ipNext selection into the retry, as I want to try a new
    port and not the same port again.

    Now I have 151289 connections :-)

    2013/08/21 09:40:24 sockets: used 151289 TCP: inuse 151188 orphan 11 tw 4
    alloc 151190 mem 1775 UDP: inuse 0 mem 0 UDPLITE: inuse 0 RAW: inuse 0
    FRAG: inuse 0 memory 0


    On Wed, Aug 21, 2013 at 11:30 AM, Kyle Lemons wrote:

    Do you have a C program that can do this? I have been playing around and
    still can't exceed 65518 outgoing connections even across 64 processes,
    regardless of how many destination IPs I use.

    I'm more and more convinced that this is a linux issue, not a Go issue.


    On Wed, Aug 21, 2013 at 1:18 AM, Paul van Brouwershaven <
    paul@vanbrouwershaven.com> wrote:
    On Wed, Aug 21, 2013 at 10:06 AM, Kyle Lemons wrote:

    for a total of 65514. No tuning other than what you posted in
    https://code.google.com/p/go/issues/detail?id=6176#c4
    This is still lower than the 65535, the slight difference why you can't
    reach the 655535 exactly is more depending on other connections running on
    your test machine.

    But it's the intention to pass this 65535, which in this case should be
    go up to 7 (ip numbers) * 65535 (ports) = 458745 outgoing connections.

    --
    You received this message because you are subscribed to the Google Groups
    "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an
    email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.

    --
    Regards,

    Paul van Brouwershaven

    http://linkedin.com/in/pvanbrouwershaven
    http://facebook.com/p.vanbrouwershaven
    http://twitter.com/vanbroup

    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.
  • Paul van Brouwershaven at Aug 21, 2013 at 8:14 am
    You demonstrated that you can make 120k TCP sessions,* *63964
    outgoing, and 63964
    incoming.

    This is not 120k outgoing, handling incoming connections is not a big deal
    as they could use the same port.

    On Wed, Aug 21, 2013 at 9:52 AM, Dave Cheney wrote:

    Ok, I don't really understand what is required now. I've demonstrated you
    can hold up 120k sockets from a single process. I think this can easily be
    extended to a client and a server process, then the rest is tcp tuning.
    What am I missing ?

    On 21/08/2013, at 17:34, Paul van Brouwershaven wrote:

    I tested your sample and could get a 128997 established connections, but
    this is only 63964 outgoing connections. This is about the same result as
    with my test but you now have doubled this by counting incoming and
    outgoing connections.

    2013/08/21 06:54:20 starting listeners
    2013/08/21 06:54:20 listeners started, starting clients
    2013/08/21 06:54:20 clients started, waiting for completion
    2013/08/21 06:54:23 client 127.0.0.106:0: connection *7993*: dial tcp
    127.0.0.206:8000: address already in use
    2013/08/21 06:54:23 client 127.0.0.107:0: connection *7993*: dial tcp
    127.0.0.207:8000: address already in use
    2013/08/21 06:54:24 client 127.0.0.102:0: connection *7993*: dial tcp
    127.0.0.202:8000: address already in use
    2013/08/21 06:54:24 client 127.0.0.103:0: connection *7996*: dial tcp
    127.0.0.203:8000: address already in use
    2013/08/21 06:54:24 client 127.0.0.100:0: connection *7995*: dial tcp
    127.0.0.200:8000: address already in use
    2013/08/21 06:54:24 client 127.0.0.104:0: connection *7997*: dial tcp
    127.0.0.204:8000: address already in use
    2013/08/21 06:54:24 client 127.0.0.101:0: connection *7999*: dial tcp
    127.0.0.201:8000: address already in use
    2013/08/21 06:54:24 client 127.0.0.105:0: connection *7998*: dial tcp
    127.0.0.205:8000: address already in use

    --------- +

    *63964*
    *
    *
    2013/08/21 06:54:24 All clients connected, pausing now to let you
    investigate the process

    2013/08/21 06:54:24 sockets: used 129118 TCP: inuse 129023 orphan 11 tw 2
    alloc 129025 mem 1774 UDP: inuse 0 mem 0 UDPLITE: inuse 0 RAW: inuse 0
    FRAG: inuse 0 memory 0

    ESTABLISHED 128997

    As soon I extend the number of source and target addresses I get the
    following:

    2013/08/21 07:30:19 starting listeners
    2013/08/21 07:30:19 listeners started, starting clients
    2013/08/21 07:30:19 clients started, waiting for completion
    2013/08/21 07:30:23 client 127.0.0.106:0: connection *6395*: dial tcp
    127.0.0.206:8000: address already in use
    2013/08/21 07:30:23 client 127.0.0.101:0: connection *6395*: dial tcp
    127.0.0.201:8000: address already in use
    2013/08/21 07:30:23 client 127.0.0.103:0: connection *6397*: dial tcp
    127.0.0.203:8000: address already in use
    2013/08/21 07:30:23 client 127.0.0.102:0: connection *6395*: dial tcp
    127.0.0.202:8000: address already in use
    2013/08/21 07:30:23 client 127.0.0.105:0: connection *6396*: dial tcp
    127.0.0.205:8000: address already in use
    2013/08/21 07:30:23 client 127.0.0.104:0: connection *6394*: dial tcp
    127.0.0.204:8000: address already in use
    2013/08/21 07:30:23 client 127.0.0.109:0: connection *6391*: dial tcp
    127.0.0.109:8000: address already in use
    2013/08/21 07:30:23 client 127.0.0.100:0: connection *6394*: dial tcp
    127.0.0.200:8000: address already in use
    2013/08/21 07:30:23 client 127.0.0.108:0: connection *6392*: dial tcp
    127.0.0.108:8000: address already in use
    2013/08/21 07:30:23 client 127.0.0.107:0: connection *6391*: dial tcp
    127.0.0.207:8000: address already in use

    --------- +

    *63940*

    2013/08/21 07:30:23 All clients connected, pausing now to let you
    investigate the process

    2013/08/21 07:30:23 sockets: used 129137 TCP: inuse 129036 orphan 11 tw 0
    alloc 129038 mem 1778 UDP: inuse 0 mem 0 UDPLITE: inuse 0 RAW: inuse 0
    FRAG: inuse 0 memory 0



    On Wed, Aug 21, 2013 at 7:40 AM, Dave Cheney wrote:

    Please try this sample code

    http://play.golang.org/p/y2319158oP

    I get to 114688 connections without any serious tcp tuning.

    On Wed, Aug 21, 2013 at 9:13 AM, Benjamin Measures
    wrote:
    On Tuesday, 20 August 2013 10:19:44 UTC+1, Paul van Brouwershaven wrote:

    2013/08/20 08:57:46 sockets: used 64723 TCP: inuse 64542 orphan 31 tw 0
    alloc 64632 mem 771 UDP: inuse 0 mem 0 UDPLITE: inuse 0 RAW: inuse 0
    FRAG:
    inuse 1 memory 1480
    2013/08/20 08:58:30 sockets: used 64407 TCP: inuse 62746 orphan 31 tw 0
    alloc 64316 mem 772 UDP: inuse 0 mem 0 UDPLITE: inuse 0 RAW: inuse 0
    FRAG:
    inuse 0 memory 0

    The TCP sockets inuse has decreased. This would suggest sockets are being
    closed.
    When I run the test program with 64100 routines I can even get to 64632
    but soon run into the error "address already in use" error.

    2013/08/20 09:07:33 sockets: used 708 TCP: inuse 614 orphan 30 tw 0
    alloc
    616 mem 770 UDP: inuse 0 mem 0 UDPLITE: inuse 0 RAW: inuse 0 FRAG:
    inuse 0
    memory 0
    2013/08/20 09:08:04 sockets: used 64807 TCP: inuse 64632 orphan 30 tw 4
    alloc 64715 mem 771 UDP: inuse 0 mem 0 UDPLITE: inuse 0 RAW: inuse 0
    FRAG:
    inuse 0 memory 0
    2013/08/20 09:08:23 *.*.*.1 -> 10.5.175.151 || dial tcp
    10.5.175.151:80:
    address already in use

    Using more goroutines will cause connections to be opened (and closed) at a
    greater rate. That the error doesn't appear until ~20s after peak/high
    inuse, would support the CLOSE_WAIT issue as outlined earlier in this
    thread.
    --
    You received this message because you are subscribed to a topic in the
    Google Groups "golang-nuts" group.
    To unsubscribe from this topic, visit
    https://groups.google.com/d/topic/golang-nuts/Mi7QkAqP7II/unsubscribe.
    To unsubscribe from this group and all its topics, send an email to
    golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.


    --
    Regards,

    Paul van Brouwershaven

    http://linkedin.com/in/pvanbrouwershaven
    http://facebook.com/p.vanbrouwershaven
    http://twitter.com/vanbroup

    --
    Regards,

    Paul van Brouwershaven

    http://linkedin.com/in/pvanbrouwershaven
    http://facebook.com/p.vanbrouwershaven
    http://twitter.com/vanbroup

    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.
  • Paul van Brouwershaven at Aug 21, 2013 at 6:09 am

    On Wed, Aug 21, 2013 at 1:13 AM, Benjamin Measures wrote:
    On Tuesday, 20 August 2013 10:19:44 UTC+1, Paul van Brouwershaven wrote:

    2013/08/20 08:57:46 sockets: used 64723 TCP: inuse *64542 *orphan 31 tw
    0 alloc 64632 mem 771 UDP: inuse 0 mem 0 UDPLITE: inuse 0 RAW: inuse 0
    FRAG: inuse 1 memory 1480
    2013/08/20 08:58:30 sockets: used 64407 TCP: inuse 62746 orphan 31 tw 0
    alloc 64316 mem 772 UDP: inuse 0 mem 0 UDPLITE: inuse 0 RAW: inuse 0 FRAG:
    inuse 0 memory 0
    The TCP sockets inuse has decreased. This would suggest sockets are being
    closed.
    The sockets should be closed, in 1 or 2 seconds as requested in Dailer
    Timeout and Deadline.

    When I run the test program with 64100 routines I can even get to 64632
    but soon run into the error "address already in use" error.
    2013/08/20 09:07:33 sockets: used 708 TCP: inuse 614 orphan 30 tw 0 alloc
    616 mem 770 UDP: inuse 0 mem 0 UDPLITE: inuse 0 RAW: inuse 0 FRAG: inuse 0
    memory 0
    2013/08/20 09:08:04 sockets: used 64807 TCP: inuse *64632* orphan 30 tw
    4 alloc 64715 mem 771 UDP: inuse 0 mem 0 UDPLITE: inuse 0 RAW: inuse 0
    FRAG: inuse 0 memory 0
    2013/08/20 09:08:23 *.*.*.1 -> 10.5.175.151 || dial tcp 10.5.175.151:80:
    address already in use
    Using more goroutines will cause connections to be opened (and closed) at
    a greater rate. That the error doesn't appear until ~20s after peak/high
    inuse, would support the CLOSE_WAIT issue as outlined earlier in this
    thread.
    The error doesn't appear immediately with +/- 64000 routines because we are
    still running under the TCP limit that I'm trying to pass. Each IP address
    is limited to 65535 ephemeral ports (
    http://en.wikipedia.org/wiki/Ephemeral_port). Where my ephemeral port range
    is set from 1024 to 65535. To use more ephemeral ports then possible on a
    single server you need to add additional IP addresses because the ports
    should be bounded to a specific IP address.

    While you can tune a kernel to shorten the CLOSE_WAIT time and reusing the
    CLOSE_WAIT ports it should not matter in my case (for now).

    By using two source IP addresses I should be able to make 129022 (2x64511)
    connections, the state of the TCP should have no effect on the total number
    of sessions, it would have effect if you are looking for established
    sessions which I'm not.

    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.
  • Benjamin Measures at Aug 20, 2013 at 12:24 am

    On Monday, 19 August 2013 12:00:28 UTC+1, Paul van Brouwershaven wrote:

    Here is a demo code that is running 60k concurrent connections with a
    timeout of 2 seconds.
    http://play.golang.org/p/mQZ2nlUQsr

    The code will print a status output like below every 5 seconds:
    2013/08/19 10:51:33 GO: 60007 || TCP: ESTABLISHED 4 SYN_SENT 28232 ||
    ERROR: dial tcp 10.2.149.179:80 <http://10.2.149.179/>: address already
    in use
    Your goroutines are closing the connection almost immediately after dialing:

    dconn, err := d.Dial("tcp", ipaddr+":80")
    if err != nil {
       [...]
    }
    defer dconn.Close()


    The closing of the connections would explain why you can never get >60k
    connections ESTABLISHED (or SYN_SENT).

    Furthermore, closing the connections will cause the tcp socket to go into
    TIME_WAIT state (for up to 60s by default on Linux), during which time you
    cannot rebind the socket. Eventually, you'll just run out of ports and this
    will happen far sooner than you'd expect from looking at the ESTABLISHED
    connections alone.


    BTW, it seems you're dialing out to a different ip:port each time -
    technically, connections are identified by the tuple
    proto,laddr,lport,daddr,dport. So, theoretically, you could create multiple
    connections (think >1M) bound to the same laddr,lport. SO_REUSEPORT was not
    long added to Linux though, so it may not be in Go
    yet: http://grokbase.com/t/gg/golang-dev/13373v3nfm/net-so-reuseport-in-linux-3-9

    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.
  • Paul van Brouwershaven at Aug 20, 2013 at 6:29 am

    On Tue, Aug 20, 2013 at 2:24 AM, Benjamin Measures wrote:

    Your goroutines are closing the connection almost immediately after
    dialing:

    dconn, err := d.Dial("tcp", ipaddr+":80")
    if err != nil {
    [...]
    }
    defer dconn.Close()
    It are very short connections, but they don't close immediately. The "defer
    dconn.Close()" closes the connection when the function returns, I'm not
    downloading anything in the example program but the timing is the same.

    The closing of the connections would explain why you can never get >60k
    connections ESTABLISHED (or SYN_SENT).
    Furthermore, closing the connections will cause the tcp socket to go into
    TIME_WAIT state (for up to 60s by default on Linux), during which time you
    cannot rebind the socket. Eventually, you'll just run out of ports and this
    will happen far sooner than you'd expect from looking at the ESTABLISHED
    connections alone.
    Most of the connection are in SYN_SENT status (waiting on connection), I
    don't have issues with TIME_WAIT. I'm running in high concurrency and it's
    no problem to start more than > 60k connections per second, except that I'm
    getting "address already in use" errors as soon I go over the +/- 64511
    limit.

    TIME_WAIT 2917
    CLOSE_WAIT 5
    FIN_WAIT1 2049
    SYN_SENT 54926
    ESTABLISHED 836
    FIN_WAIT2 1063
    CLOSING 13
    LAST_ACK 61

    BTW, it seems you're dialing out to a different ip:port each time -
    technically, connections are identified by the tuple
    proto,laddr,lport,daddr,dport. So, theoretically, you could create multiple
    connections (think >1M) bound to the same laddr,lport. SO_REUSEPORT was not
    long added to Linux though, so it may not be in Go yet:
    http://grokbase.com/t/gg/golang-dev/13373v3nfm/net-so-reuseport-in-linux-3-9
    Sounds interesting but is this not for incoming connections only?

    *"For TCP, so_reuseport allows multiple listener sockets to be bound to the
    same port."*

    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.
  • Kyle Lemons at Aug 20, 2013 at 7:18 am

    On Mon, Aug 19, 2013 at 11:29 PM, Paul van Brouwershaven wrote:

    On Tue, Aug 20, 2013 at 2:24 AM, Benjamin Measures <saint.abroad@gmail.com
    wrote:
    Your goroutines are closing the connection almost immediately after
    dialing:

    dconn, err := d.Dial("tcp", ipaddr+":80")
    if err != nil {
    [...]
    }
    defer dconn.Close()
    It are very short connections, but they don't close immediately. The
    "defer dconn.Close()" closes the connection when the function returns, I'm
    not downloading anything in the example program but the timing is the same.

    The closing of the connections would explain why you can never get >60k
    connections ESTABLISHED (or SYN_SENT).
    Furthermore, closing the connections will cause the tcp socket to go into
    TIME_WAIT state (for up to 60s by default on Linux), during which time you
    cannot rebind the socket. Eventually, you'll just run out of ports and this
    will happen far sooner than you'd expect from looking at the ESTABLISHED
    connections alone.
    Most of the connection are in SYN_SENT status (waiting on connection), I
    don't have issues with TIME_WAIT. I'm running in high concurrency and it's
    no problem to start more than > 60k connections per second, except that I'm
    getting "address already in use" errors as soon I go over the +/- 64511
    limit.

    TIME_WAIT 2917
    CLOSE_WAIT 5
    FIN_WAIT1 2049
    SYN_SENT 54926
    ESTABLISHED 836
    FIN_WAIT2 1063
    CLOSING 13
    LAST_ACK 61
    I don't think you can really say you have 1M TCP connections unless you
    have 1M that say ESTABLISHED there. I'd work on getting to 64511
    ESTABLISHED first, and then move on to multiple IPs. It still looks to me
    like some router somewhere isn't able to actually route enough connections
    for you, thus tons of SYN_SENT -- no SYN/ACK, because the ACK can't be
    routed back to you.

    BTW, it seems you're dialing out to a different ip:port each time -
    technically, connections are identified by the tuple
    proto,laddr,lport,daddr,dport. So, theoretically, you could create multiple
    connections (think >1M) bound to the same laddr,lport. SO_REUSEPORT was not
    long added to Linux though, so it may not be in Go yet:
    http://grokbase.com/t/gg/golang-dev/13373v3nfm/net-so-reuseport-in-linux-3-9
    Sounds interesting but is this not for incoming connections only?

    *"For TCP, so_reuseport allows multiple listener sockets to be bound to
    the same port."*

    --
    You received this message because you are subscribed to the Google Groups
    "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an
    email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.
    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.
  • Dave Cheney at Aug 20, 2013 at 7:26 am

    I don't think you can really say you have 1M TCP connections unless you have
    1M that say ESTABLISHED there. I'd work on getting to 64511 ESTABLISHED
    first, and then move on to multiple IPs. It still looks to me like some
    router somewhere isn't able to actually route enough connections for you,
    thus tons of SYN_SENT -- no SYN/ACK, because the ACK can't be routed back to
    you.
    Also, check dmesg on the server side. If you see things like SYN
    cookies enabled, it means your OS thinks it is under attack and will
    be slowing down accepting connections from that source IP.

    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.
  • Paul van Brouwershaven at Aug 20, 2013 at 8:08 am

    On Tue, Aug 20, 2013 at 9:18 AM, Kyle Lemons wrote:

    I don't think you can really say you have 1M TCP connections unless you
    have 1M that say ESTABLISHED there. I'd work on getting to 64511
    ESTABLISHED first, and then move on to multiple IPs. It still looks to me
    like some router somewhere isn't able to actually route enough connections
    for you, thus tons of SYN_SENT -- no SYN/ACK, because the ACK can't be
    routed back to you.
    That would be an option, although I'm connection to 'random' addresses in
    the private IP space. I know that most of these connection will never get
    established.

    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupgolang-nuts @
categoriesgo
postedAug 16, '13 at 10:45a
activeAug 21, '13 at 9:47a
posts39
users5
websitegolang.org

People

Translate

site design / logo © 2022 Grokbase