FAQ
I have a very active Go program that handles network traffic using a custom
protocol and makes extensive use of channel communication and goroutines. I
am experiencing a problem where the program panics like this:

runtime/cgo: pthread_create failed: Resource temporarily unavailable
SIGABRT: abort
PC=0x800b05cbc


goroutine 1 [chan receive]:
net.(*pollServer).WaitRead(0xc2004dfc30, 0xc20032ac00, 0xc2000c0e70, 0x23,
0xc2000c0e01, ...)
/usr/local/go/src/pkg/net/fd_unix.go:241 +0x63
net.(*netFD).accept(0xc20032ac00, 0x4c8b20, 0x0, 0xc2000c0e70, 0x23, ...)
/usr/local/go/src/pkg/net/fd_unix.go:637 +0x194
net.(*TCPListener).AcceptTCP(0xc2003463d0, 0x1077395320, 0x0, 0x0,
0x4172c3, ...)
/usr/local/go/src/pkg/net/tcpsock_posix.go:237 +0x62
net.(*TCPListener).Accept(0xc2003463d0, 0x0, 0x0, 0x0, 0x0, ...)
/usr/local/go/src/pkg/net/tcpsock_posix.go:247 +0x49
crypto/tls.(*listener).Accept(0xc20032e5e0, 0x0, 0x0, 0x0, 0x0, ...)
/usr/local/go/src/pkg/crypto/tls/tls.go:46 +0x51
X.X(0x0, 0x0)
/usr/local/X/X/X.go:170 +0x777
main.main()
/usr/local/X/X/rg.go:331 +0x568

The operating system is FreeBSD amd64, I am on Go tip and have
kern.threads.max_threads_per_proc=8192. Am I simply running out of threads?
When the program panics there are the following numbers of goroutines
running in these states:

1 [finalizer wait]
4 [running]
8,177 [syscall]
8,282 [select]
16,538 [runnable]
21,085 [chan receive]
26,025 [semacquire]

Is the problem that 8,177 routines in the syscall state?

John.


--

Search Discussions

  • Han-Wen Nienhuys at Nov 29, 2012 at 3:59 pm

    On Thu, Nov 29, 2012 at 4:24 PM, John Graham-Cumming wrote:
    I have a very active Go program that handles network traffic using a custom
    protocol and makes extensive use of channel communication and goroutines. I
    am experiencing a problem where the program panics like this:

    runtime/cgo: pthread_create failed: Resource temporarily unavailable
    SIGABRT: abort
    PC=0x800b05cbc


    goroutine 1 [chan receive]:
    net.(*pollServer).WaitRead(0xc2004dfc30, 0xc20032ac00, 0xc2000c0e70, 0x23,
    0xc2000c0e01, ...)
    /usr/local/go/src/pkg/net/fd_unix.go:241 +0x63
    net.(*netFD).accept(0xc20032ac00, 0x4c8b20, 0x0, 0xc2000c0e70, 0x23, ...)
    /usr/local/go/src/pkg/net/fd_unix.go:637 +0x194
    net.(*TCPListener).AcceptTCP(0xc2003463d0, 0x1077395320, 0x0, 0x0, 0x4172c3,
    ...)
    /usr/local/go/src/pkg/net/tcpsock_posix.go:237 +0x62
    net.(*TCPListener).Accept(0xc2003463d0, 0x0, 0x0, 0x0, 0x0, ...)
    /usr/local/go/src/pkg/net/tcpsock_posix.go:247 +0x49
    crypto/tls.(*listener).Accept(0xc20032e5e0, 0x0, 0x0, 0x0, 0x0, ...)
    /usr/local/go/src/pkg/crypto/tls/tls.go:46 +0x51
    X.X(0x0, 0x0)
    /usr/local/X/X/X.go:170 +0x777
    main.main()
    /usr/local/X/X/rg.go:331 +0x568

    The operating system is FreeBSD amd64, I am on Go tip and have
    kern.threads.max_threads_per_proc=8192. Am I simply running out of threads?
    When the program panics there are the following numbers of goroutines
    running in these states:

    1 [finalizer wait]
    4 [running]
    8,177 [syscall]
    8,282 [select]
    16,538 [runnable]
    21,085 [chan receive]
    26,025 [semacquire]

    Is the problem that 8,177 routines in the syscall state?
    I think so. Any outstanding blocking syscall is run in a separate
    thread so the rest of the program can go on with work.


    --
    Han-Wen Nienhuys
    Google Munich
    hanwen@google.com

    --
  • Dave Cheney at Nov 29, 2012 at 7:17 pm
    (reply to all this time)

    Yes, that is likely to be the cause. Inside the guts of
    syscall.Syscall{,6}, just before setting up the arguments for the
    actual syscall, there is a call to runtime.entersyscall.

    runtime.entersyscall notifies the scheduler that the thread that is
    hosting this goroutine is about to block. This generally means that a
    new thread is created (unless there is a spare one parked in the
    scheduler) to replace the thread that is about to block. It is this
    new thread that takes over running goroutines until the original
    thread unblocks.

    The net package attempts to reduce the number of os threads blocked on
    syscalls by using kqueue/epoll/select, but no solution currently
    exists for disk io, which is a common cause of this sort of error.

    Cheers
    On Fri, Nov 30, 2012 at 2:24 AM, John Graham-Cumming wrote:
    I have a very active Go program that handles network traffic using a custom
    protocol and makes extensive use of channel communication and goroutines. I
    am experiencing a problem where the program panics like this:

    runtime/cgo: pthread_create failed: Resource temporarily unavailable
    SIGABRT: abort
    PC=0x800b05cbc


    goroutine 1 [chan receive]:
    net.(*pollServer).WaitRead(0xc2004dfc30, 0xc20032ac00, 0xc2000c0e70, 0x23,
    0xc2000c0e01, ...)
    /usr/local/go/src/pkg/net/fd_unix.go:241 +0x63
    net.(*netFD).accept(0xc20032ac00, 0x4c8b20, 0x0, 0xc2000c0e70, 0x23, ...)
    /usr/local/go/src/pkg/net/fd_unix.go:637 +0x194
    net.(*TCPListener).AcceptTCP(0xc2003463d0, 0x1077395320, 0x0, 0x0, 0x4172c3,
    ...)
    /usr/local/go/src/pkg/net/tcpsock_posix.go:237 +0x62
    net.(*TCPListener).Accept(0xc2003463d0, 0x0, 0x0, 0x0, 0x0, ...)
    /usr/local/go/src/pkg/net/tcpsock_posix.go:247 +0x49
    crypto/tls.(*listener).Accept(0xc20032e5e0, 0x0, 0x0, 0x0, 0x0, ...)
    /usr/local/go/src/pkg/crypto/tls/tls.go:46 +0x51
    X.X(0x0, 0x0)
    /usr/local/X/X/X.go:170 +0x777
    main.main()
    /usr/local/X/X/rg.go:331 +0x568

    The operating system is FreeBSD amd64, I am on Go tip and have
    kern.threads.max_threads_per_proc=8192. Am I simply running out of threads?
    When the program panics there are the following numbers of goroutines
    running in these states:

    1 [finalizer wait]
    4 [running]
    8,177 [syscall]
    8,282 [select]
    16,538 [runnable]
    21,085 [chan receive]
    26,025 [semacquire]

    Is the problem that 8,177 routines in the syscall state?

    John.


    --
    --
  • Russ Cox at Nov 29, 2012 at 8:42 pm
    This is a problem I would like to do better with in Go 1.1. If you
    still have the stack traces from the 8,177 goroutines in system calls,
    I would be interested in the distribution of the first line of their
    stack traces. That will tell us which functions are making 'blocking'
    system calls that we might profitably change to non-blocking (from
    Go's perspective not the OS's).

    Also related: golang.org/issue/4056.

    Thanks.
    Russ

    --
  • Mikio Hara at Nov 30, 2012 at 2:01 am

    On Fri, Nov 30, 2012 at 5:36 AM, Russ Cox wrote:

    Also related: golang.org/issue/4056.
    And golang.org/issue/2490.

    --

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupgolang-nuts @
categoriesgo
postedNov 29, '12 at 3:24p
activeNov 30, '12 at 2:01a
posts5
users5
websitegolang.org

People

Translate

site design / logo © 2021 Grokbase