FAQ
I'm trying to write a go server for managing many (mostly idle) long-lived
connections on TCP. These services are largely routers for an evented
database layers & don't do a whole lot of heavy lifting. After some cursory
research, it looks like my primary limitation it looks like my main
constraint will be the minimum goroutine stack size. At 4KB, 1MM
connections means 4GB RAM for the stack of goroutines which are all blocked
on a single epoll. This seems a bit silly; I could buy servers with more
RAM, but it'd be more useful to improve server density before getting the
upgrade.

I want to try reducing the initial stack size since these threads do very
little work themselves. After some investigation, it seems like this is a
const in the go toolchain. Is this correct? If I fiddle with this constant,
will I destabilize anything else? Would the go community be interested in a
patch that makes the value configurable with a flag to go build?

Thanks!
--Thomas

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Search Discussions

  • Dave Cheney at Jun 12, 2013 at 8:37 pm
    Without knowing your program my feeling is you are optimising the wrong thing. How many pages of kernel memory will 1,000,000 active tcp connections consume ?
    On 13/06/2013, at 4:40, thomas.bouldin@gmail.com wrote:

    I'm trying to write a go server for managing many (mostly idle) long-lived connections on TCP. These services are largely routers for an evented database layers & don't do a whole lot of heavy lifting. After some cursory research, it looks like my primary limitation it looks like my main constraint will be the minimum goroutine stack size. At 4KB, 1MM connections means 4GB RAM for the stack of goroutines which are all blocked on a single epoll. This seems a bit silly; I could buy servers with more RAM, but it'd be more useful to improve server density before getting the upgrade.

    I want to try reducing the initial stack size since these threads do very little work themselves. After some investigation, it seems like this is a const in the go toolchain. Is this correct? If I fiddle with this constant, will I destabilize anything else? Would the go community be interested in a patch that makes the value configurable with a flag to go build?

    Thanks!
    --Thomas
    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.
    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.
  • Minux at Jun 12, 2013 at 8:40 pm

    On Thu, Jun 13, 2013 at 2:40 AM, wrote:

    I'm trying to write a go server for managing many (mostly idle) long-lived
    connections on TCP. These services are largely routers for an evented
    database layers & don't do a whole lot of heavy lifting. After some cursory
    research, it looks like my primary limitation it looks like my main
    constraint will be the minimum goroutine stack size. At 4KB, 1MM
    connections means 4GB RAM for the stack of goroutines which are all blocked
    on a single epoll. This seems a bit silly; I could buy servers with more
    RAM, but it'd be more useful to improve server density before getting the
    upgrade.

    I want to try reducing the initial stack size since these threads do very
    little work themselves. After some investigation, it seems like this is a
    const in the go toolchain. Is this correct? If I fiddle with this constant,
    will I destabilize anything else? Would the go community be
    it's StackMin in pkg/runtime/stack.h.
    you can't reduce it (from 4096) too much, but 2048 could still pass most of
    the all.bash
    tests.
    (all failed tests have something to do with testing malloc counts, and
    unexpected stack
    split happens)

    interested in a patch that makes the value configurable with a flag to go
    build?
    the general tendency seems to avoid excessive knobs, the runtime should try
    to figure
    it out itself.
    also, i think there is some implicit dependency on StackMin is about 4096
    in package
    tests and some /test tests (e.g. closure.go).

    btw, i expect managing 1M live connections on one host is going to be messy
    even if
    you reduced the per goroutine stack.

    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.
  • Dmitry Vyukov at Jun 13, 2013 at 6:56 am
    I think modern OS can consume <10K per tcp connection (if you reduce
    buffers), so 4K for stack still looks like considerable amount of
    memory.
    On Thu, Jun 13, 2013 at 12:40 AM, minux wrote:
    On Thu, Jun 13, 2013 at 2:40 AM, wrote:

    I'm trying to write a go server for managing many (mostly idle) long-lived
    connections on TCP. These services are largely routers for an evented
    database layers & don't do a whole lot of heavy lifting. After some cursory
    research, it looks like my primary limitation it looks like my main
    constraint will be the minimum goroutine stack size. At 4KB, 1MM connections
    means 4GB RAM for the stack of goroutines which are all blocked on a single
    epoll. This seems a bit silly; I could buy servers with more RAM, but it'd
    be more useful to improve server density before getting the upgrade.

    I want to try reducing the initial stack size since these threads do very
    little work themselves. After some investigation, it seems like this is a
    const in the go toolchain. Is this correct? If I fiddle with this constant,
    will I destabilize anything else? Would the go community be
    it's StackMin in pkg/runtime/stack.h.
    you can't reduce it (from 4096) too much, but 2048 could still pass most of
    the all.bash
    tests.
    (all failed tests have something to do with testing malloc counts, and
    unexpected stack
    split happens)
    interested in a patch that makes the value configurable with a flag to go
    build?
    the general tendency seems to avoid excessive knobs, the runtime should try
    to figure
    it out itself.
    also, i think there is some implicit dependency on StackMin is about 4096 in
    package
    tests and some /test tests (e.g. closure.go).

    btw, i expect managing 1M live connections on one host is going to be messy
    even if
    you reduced the per goroutine stack.

    --
    You received this message because you are subscribed to the Google Groups
    "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an
    email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.
    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.
  • Dave Cheney at Jun 13, 2013 at 10:21 am
    Sure, but my point is the cost of an active tcp connection is not 0 bytes of kernel memory, hence the argument that needing 4gb of heap for goroutine stacks is not as unreasonable as first suggested.


    On 13/06/2013, at 16:56, Dmitry Vyukov wrote:

    I think modern OS can consume <10K per tcp connection (if you reduce
    buffers), so 4K for stack still looks like considerable amount of
    memory.
    On Thu, Jun 13, 2013 at 12:40 AM, minux wrote:
    On Thu, Jun 13, 2013 at 2:40 AM, wrote:

    I'm trying to write a go server for managing many (mostly idle) long-lived
    connections on TCP. These services are largely routers for an evented
    database layers & don't do a whole lot of heavy lifting. After some cursory
    research, it looks like my primary limitation it looks like my main
    constraint will be the minimum goroutine stack size. At 4KB, 1MM connections
    means 4GB RAM for the stack of goroutines which are all blocked on a single
    epoll. This seems a bit silly; I could buy servers with more RAM, but it'd
    be more useful to improve server density before getting the upgrade.

    I want to try reducing the initial stack size since these threads do very
    little work themselves. After some investigation, it seems like this is a
    const in the go toolchain. Is this correct? If I fiddle with this constant,
    will I destabilize anything else? Would the go community be
    it's StackMin in pkg/runtime/stack.h.
    you can't reduce it (from 4096) too much, but 2048 could still pass most of
    the all.bash
    tests.
    (all failed tests have something to do with testing malloc counts, and
    unexpected stack
    split happens)
    interested in a patch that makes the value configurable with a flag to go
    build?
    the general tendency seems to avoid excessive knobs, the runtime should try
    to figure
    it out itself.
    also, i think there is some implicit dependency on StackMin is about 4096 in
    package
    tests and some /test tests (e.g. closure.go).

    btw, i expect managing 1M live connections on one host is going to be messy
    even if
    you reduced the per goroutine stack.

    --
    You received this message because you are subscribed to the Google Groups
    "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an
    email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.
    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.
    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.
  • Roger peppe at Jun 13, 2013 at 12:36 pm

    On 13 June 2013 11:21, Dave Cheney wrote:
    Sure, but my point is the cost of an active tcp connection is not 0 bytes of kernel memory, hence the argument that needing 4gb of heap for goroutine stacks is not as unreasonable as first suggested.
    It's not uncommon to use several goroutines per connection (for example: one to
    read from the channel, one to multiplex that data and wait for timeout events;
    or one goroutine for each outstanding concurrent RPC request).

    I think that reducing the minimum amount of memory for a goroutine may
    have significantly beneficial effects on the memory usage of real go programs.
    I wonder if there's a straightforward way of adding some instrumentation to
    check that hypothesis.

    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.
  • Thomas Bouldin at Jun 14, 2013 at 6:32 pm
    My service isn't that far off the previously mentioned chat app: I have
    clients who connect, demux to a thread responsible for fetching offline
    messages from database, and then block on a network event or a
    notifications from another thread which receives events from other servers
    in the same service and routes the event to the appropriate connection. For
    capacity planning I've written a simple version which just accepts sockets,
    reads, & writes to a queue that simulates the database delegate goroutine.

    I get that the TCP buffers will have a non-negligible cost as well. I may
    look at whether those can be shrunk since my payloads are small and clients
    do not have high bandwidth; in the meantime goroutine stack size seems like
    the easiest & safest knob to twist. On an AWS m1.large, which has 7.5GB
    RAM, I was able to increase the number of sustained connections by 25% from
    800K to 1M by cutting the initial stack size from 4096 to 2048.

    Fiddling with (*TCPConn*)SetReadBufferBytes/SetWriteBufferBytes has helped
    me eek out more connections, though I need to do more research to decide
    whether or not that's particularly safe.

    On Thu, Jun 13, 2013 at 5:36 AM, roger peppe wrote:
    On 13 June 2013 11:21, Dave Cheney wrote:
    Sure, but my point is the cost of an active tcp connection is not 0
    bytes of kernel memory, hence the argument that needing 4gb of heap for
    goroutine stacks is not as unreasonable as first suggested.

    It's not uncommon to use several goroutines per connection (for example:
    one to
    read from the channel, one to multiplex that data and wait for timeout
    events;
    or one goroutine for each outstanding concurrent RPC request).

    I think that reducing the minimum amount of memory for a goroutine may
    have significantly beneficial effects on the memory usage of real go
    programs.
    I wonder if there's a straightforward way of adding some instrumentation to
    check that hypothesis.
    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupgolang-nuts @
categoriesgo
postedJun 12, '13 at 7:55p
activeJun 14, '13 at 6:32p
posts7
users5
websitegolang.org

People

Translate

site design / logo © 2021 Grokbase