[go-nuts] Does Go runtime shrinks thread pool?

load testing I noticed that program allocates (MemStats.Alloc) 200 MB of
memory. pprof showed only about 2. I dig further and found that there are
4k of OS threads, most of them are sitting at sys_linux_amd64.s:238[1] (Go
1.0.3).
1.1) runtime ever shrinks thread pool?
When one goroutine blocks in syscall on one of the OS thread and there is
no available OS thread to
host other ready goroutines, the runtime has no choice but to create
another OS thread.

The Go runtime currently doesn't ever shrink its thread pool.

A related issue: https://code.google.com/p/go/issues/detail?id=4056
for now, you will have to limit the number concurrent OS thread blocking
yourself
(mostly cgo calls, because calling into C world is treated the same as
blocking in
syscall).

So, it appears that for certain syscalls, like reading from a socket, the
runtime uses some kind of polling or pushing provided by the OS allowing it
to free the thread occupied by the goroutine blocked in such a call.

What about the other calls? Is it the present issue only related to cgo? If
not, is it documented which calls cause an OS thread to be blocked when the
goroutine running on it blocks?

I understand, that Go doesn't want to expose much info about the
multiplexing mechanism to the user, but if user code may cause thousands of
OS threads to be created, it should be common knowledge which syscalls do
that.

In Erlang, processes are scheduled preemptively and one OS thread can run
thousands of concurrent processes. It also has an issue with calling to C
code that is linked into the VM -- the scheduler can't preempt the process
before it has returned from the C land. This caveat is known and understood
by Erlang users.

Go has cooperative goroutines that might be not so cooperative depending on
the code they are running. So I think it should be well documented which
use cases cause a goroutine to misbehave. Is it already?

No. Any such documentation would be different for different Go
implementations, and would change over time depending on changes to
the scheduler.
No, because that too is implementation dependent. The implementations
will improve over time and make it harder or, I hope, impossible, for
a goroutine to misbehave in that way.

Ian

It does not.

This is related to https://code.google.com/p/go/issues/detail?id=4056 .

Ian

Can you please retest with Go1.1 beta/tip and report whether it's
better/worser/same?

Hi,
Actually, I would prefer recommended thread pool to size to hard limit. So idle threads would be slowly terminated one by one.

Also, 200 MB is not a big problem. 50KB per thread is amazing.

Will do in few more days.

–-–
Alexey "AlekSi" Palazhchenko