FAQ
As it is already raised as issue here
- https://code.google.com/p/go/issues/detail?id=6785, I am trying to get
opinions from folks on how to reduce the contention till issue is fixed -

When there is an attempt to reuse connections at higher rate, there is
contention in GetIdleCon and PutIdleCon -

316 @ 0x415989 0x415a0b 0x429eee 0x42a130 0x49ad56 0x45056e 0x450ab8 0x44ef26 0x43390d 0x4330db 0x43456f 0x433bbf 0x400e4f 0x415c20
# 0x42a130 sync.runtime_Semacquire+0x30 /usr/local/go/src/pkg/runtime/sema.goc:199
# 0x49ad56 sync.(*Mutex).Lock+0xd6 /usr/local/go/src/pkg/sync/mutex.go:66
# 0x45056e net/http.(*Transport).getIdleConn+0x9e /usr/local/go/src/pkg/net/http/transport.go:402
# 0x450ab8 net/http.(*Transport).getConn+0xa8 /usr/local/go/src/pkg/net/http/transport.go:452
# 0x44ef26 net/http.(*Transport).RoundTrip+0x416 /usr/local/go/src/pkg/net/http/transport.go:201
# 0x43390d net/http.send+0x43d /usr/local/go/src/pkg/net/http/client.go:195
# 0x4330db net/http.(*Client).send+0x15b /usr/local/go/src/pkg/net/http/client.go:118
# 0x43456f net/http.(*Client).doFollowingRedirects+0x97f /usr/local/go/src/pkg/net/http/client.go:343
# 0x433bbf net/http.(*Client).Get+0xaf /usr/local/go/src/pkg/net/http/client.go:275
# 0x400e4f main.startClient+0x16f

347 @ 0x415989 0x415a0b 0x429eee 0x42a130 0x49ad56 0x44fd5e 0x455b87 0x454c1b 0x454931 0x483291 0x480f6d 0x400f0b 0x415c20
# 0x42a130 sync.runtime_Semacquire+0x30 /usr/local/go/src/pkg/runtime/sema.goc:199
# 0x49ad56 sync.(*Mutex).Lock+0xd6 /usr/local/go/src/pkg/sync/mutex.go:66
# 0x44fd5e net/http.(*Transport).putIdleConn+0xce /usr/local/go/src/pkg/net/http/transport.go:342
# 0x455b87 net/http.func·023+0x97 /usr/local/go/src/pkg/net/http/transport.go:853
# 0x454c1b net/http.(*bodyEOFSignal).condfn+0xab /usr/local/go/src/pkg/net/http/transport.go:1161
# 0x454931 net/http.(*bodyEOFSignal).Read+0x321 /usr/local/go/src/pkg/net/http/transport.go:1133
# 0x483291 io/ioutil.devNull.ReadFrom+0xb1 /usr/local/go/src/pkg/io/ioutil/ioutil.go:151
# 0x480f6d io.Copy+0x13d /usr/local/go/src/pkg/io/io.go:349
# 0x400f0b main.startClient+0x22b


This results in -

1. New connections being made and teared down very fast and thus exceeding the ports available.

2. System resources are not used optimal. For a simple hello world server, for loop clients using http.Client gives 112k rps with 64% CPU. Network is not bottleneck here. Because even with 1k response and 100ms response time from server I still get 82k rps with this client at high concurrency.

Code : http://play.golang.org/p/MA-a2ZAVNe . N go-routines are created for startClient in main, where n is the concurrency desired by user. I used client (*http.Client) object per go-routine instead of global, as I thought, contention will be across one client object. But is it not the case?


Can someone please suggest alternatives/options to avoid contention in putting and getting connection?


--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Search Discussions

  • Dmitry Vyukov at Sep 12, 2014 at 12:13 am

    On Thu, Sep 11, 2014 at 2:57 PM, Suraj Narkhede wrote:
    As it is already raised as issue here -
    https://code.google.com/p/go/issues/detail?id=6785, I am trying to get
    opinions from folks on how to reduce the contention till issue is fixed -

    When there is an attempt to reuse connections at higher rate, there is
    contention in GetIdleCon and PutIdleCon -

    316 @ 0x415989 0x415a0b 0x429eee 0x42a130 0x49ad56 0x45056e 0x450ab8
    0x44ef26 0x43390d 0x4330db 0x43456f 0x433bbf 0x400e4f 0x415c20
    # 0x42a130 sync.runtime_Semacquire+0x30 /usr/local/go/src/pkg/runtime/sema.goc:199
    # 0x49ad56 sync.(*Mutex).Lock+0xd6 /usr/local/go/src/pkg/sync/mutex.go:66
    # 0x45056e net/http.(*Transport).getIdleConn+0x9e /usr/local/go/src/pkg/net/http/transport.go:402
    # 0x450ab8 net/http.(*Transport).getConn+0xa8 /usr/local/go/src/pkg/net/http/transport.go:452
    # 0x44ef26 net/http.(*Transport).RoundTrip+0x416 /usr/local/go/src/pkg/net/http/transport.go:201
    # 0x43390d net/http.send+0x43d /usr/local/go/src/pkg/net/http/client.go:195
    # 0x4330db net/http.(*Client).send+0x15b /usr/local/go/src/pkg/net/http/client.go:118
    # 0x43456f net/http.(*Client).doFollowingRedirects+0x97f /usr/local/go/src/pkg/net/http/client.go:343
    # 0x433bbf net/http.(*Client).Get+0xaf /usr/local/go/src/pkg/net/http/client.go:275
    # 0x400e4f main.startClient+0x16f

    347 @ 0x415989 0x415a0b 0x429eee 0x42a130 0x49ad56 0x44fd5e 0x455b87
    0x454c1b 0x454931 0x483291 0x480f6d 0x400f0b 0x415c20
    # 0x42a130 sync.runtime_Semacquire+0x30 /usr/local/go/src/pkg/runtime/sema.goc:199
    # 0x49ad56 sync.(*Mutex).Lock+0xd6 /usr/local/go/src/pkg/sync/mutex.go:66
    # 0x44fd5e net/http.(*Transport).putIdleConn+0xce /usr/local/go/src/pkg/net/http/transport.go:342
    # 0x455b87 net/http.func·023+0x97 /usr/local/go/src/pkg/net/http/transport.go:853
    # 0x454c1b net/http.(*bodyEOFSignal).condfn+0xab /usr/local/go/src/pkg/net/http/transport.go:1161
    # 0x454931 net/http.(*bodyEOFSignal).Read+0x321 /usr/local/go/src/pkg/net/http/transport.go:1133
    # 0x483291 io/ioutil.devNull.ReadFrom+0xb1 /usr/local/go/src/pkg/io/ioutil/ioutil.go:151
    # 0x480f6d io.Copy+0x13d /usr/local/go/src/pkg/io/io.go:349
    # 0x400f0b main.startClient+0x22b


    This results in -

    1. New connections being made and teared down very fast and thus exceeding
    the ports available.

    2. System resources are not used optimal. For a simple hello world server,
    for loop clients using http.Client gives 112k rps with 64% CPU. Network is
    not bottleneck here. Because even with 1k response and 100ms response time
    from server I still get 82k rps with this client at high concurrency.

    Code : http://play.golang.org/p/MA-a2ZAVNe . N go-routines are created for
    startClient in main, where n is the concurrency desired by user. I used
    client (*http.Client) object per go-routine instead of global, as I thought,
    contention will be across one client object. But is it not the case?


    Can someone please suggest alternatives/options to avoid contention in
    putting and getting connection?

    You also need to create Transport per goroutine, because by default
    all Client's use the single DefaultTransport.

    Have you tried tuning Transport.MaxIdleConnsPerHost? Try to use a
    single transport but set MaxIdleConnsPerHost to number of goroutines.

    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
  • Suraj Narkhede at Sep 12, 2014 at 12:21 am
    Thanks Dmitry. Just changed the code to do it with transport per go-routine
    and throughput increased by 33% now. :). I am still working on it and aim
    to reach throughput of 300k which I get using other clients. Currently at
    187k.

    MaxIdleConnsPerHost i had set to 10000, but it did not help much due to
    contention.
    Will keep it posted here.


    On Thursday, September 11, 2014 5:13:19 PM UTC-7, Dmitry Vyukov wrote:

    On Thu, Sep 11, 2014 at 2:57 PM, Suraj Narkhede <suraj...@gmail.com
    <javascript:>> wrote:
    As it is already raised as issue here -
    https://code.google.com/p/go/issues/detail?id=6785, I am trying to get
    opinions from folks on how to reduce the contention till issue is fixed -
    When there is an attempt to reuse connections at higher rate, there is
    contention in GetIdleCon and PutIdleCon -

    316 @ 0x415989 0x415a0b 0x429eee 0x42a130 0x49ad56 0x45056e 0x450ab8
    0x44ef26 0x43390d 0x4330db 0x43456f 0x433bbf 0x400e4f 0x415c20
    # 0x42a130 sync.runtime_Semacquire+0x30
    /usr/local/go/src/pkg/runtime/sema.goc:199
    # 0x49ad56 sync.(*Mutex).Lock+0xd6
    /usr/local/go/src/pkg/sync/mutex.go:66
    # 0x45056e net/http.(*Transport).getIdleConn+0x9e
    /usr/local/go/src/pkg/net/http/transport.go:402
    # 0x450ab8 net/http.(*Transport).getConn+0xa8
    /usr/local/go/src/pkg/net/http/transport.go:452
    # 0x44ef26 net/http.(*Transport).RoundTrip+0x416
    /usr/local/go/src/pkg/net/http/transport.go:201
    # 0x43390d net/http.send+0x43d
    /usr/local/go/src/pkg/net/http/client.go:195
    # 0x4330db net/http.(*Client).send+0x15b
    /usr/local/go/src/pkg/net/http/client.go:118
    # 0x43456f net/http.(*Client).doFollowingRedirects+0x97f
    /usr/local/go/src/pkg/net/http/client.go:343
    # 0x433bbf net/http.(*Client).Get+0xaf
    /usr/local/go/src/pkg/net/http/client.go:275
    # 0x400e4f main.startClient+0x16f

    347 @ 0x415989 0x415a0b 0x429eee 0x42a130 0x49ad56 0x44fd5e 0x455b87
    0x454c1b 0x454931 0x483291 0x480f6d 0x400f0b 0x415c20
    # 0x42a130 sync.runtime_Semacquire+0x30
    /usr/local/go/src/pkg/runtime/sema.goc:199
    # 0x49ad56 sync.(*Mutex).Lock+0xd6
    /usr/local/go/src/pkg/sync/mutex.go:66
    # 0x44fd5e net/http.(*Transport).putIdleConn+0xce
    /usr/local/go/src/pkg/net/http/transport.go:342
    # 0x455b87 net/http.func·023+0x97
    /usr/local/go/src/pkg/net/http/transport.go:853
    # 0x454c1b net/http.(*bodyEOFSignal).condfn+0xab
    /usr/local/go/src/pkg/net/http/transport.go:1161
    # 0x454931 net/http.(*bodyEOFSignal).Read+0x321
    /usr/local/go/src/pkg/net/http/transport.go:1133
    # 0x483291 io/ioutil.devNull.ReadFrom+0xb1
    /usr/local/go/src/pkg/io/ioutil/ioutil.go:151
    # 0x480f6d io.Copy+0x13d /usr/local/go/src/pkg/io/io.go:349
    # 0x400f0b main.startClient+0x22b


    This results in -

    1. New connections being made and teared down very fast and thus exceeding
    the ports available.

    2. System resources are not used optimal. For a simple hello world server,
    for loop clients using http.Client gives 112k rps with 64% CPU. Network is
    not bottleneck here. Because even with 1k response and 100ms response time
    from server I still get 82k rps with this client at high concurrency.

    Code : http://play.golang.org/p/MA-a2ZAVNe . N go-routines are created for
    startClient in main, where n is the concurrency desired by user. I used
    client (*http.Client) object per go-routine instead of global, as I thought,
    contention will be across one client object. But is it not the case?


    Can someone please suggest alternatives/options to avoid contention in
    putting and getting connection?

    You also need to create Transport per goroutine, because by default
    all Client's use the single DefaultTransport.

    Have you tried tuning Transport.MaxIdleConnsPerHost? Try to use a
    single transport but set MaxIdleConnsPerHost to number of goroutines.
    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupgolang-nuts @
categoriesgo
postedSep 11, '14 at 9:57p
activeSep 12, '14 at 12:21a
posts3
users2
websitegolang.org

2 users in discussion

Suraj Narkhede: 2 posts Dmitry Vyukov: 1 post

People

Translate

site design / logo © 2022 Grokbase