[go-nuts] problem with daisy_chain

On Wednesday, April 30, 2014 2:33:13 PM UTC+2, Ian Davis wrote:

On Wed, Apr 30, 2014, at 01:20 PM, Massimiliano Tomassoli wrote:

The following code runs successfully with n = 100000, but not with n =
200000. Why? Out of memory?


Where are you running it? I just ran it with n=200000 on my laptop and it
printed 200001 about 3 seconds later

Windows 7 64, 4 GB ram.
I used
go run daisy_chain.go

--
You received this message because you are subscribed to the Google Groups
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to golang-nuts+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

On Wednesday, April 30, 2014 2:44:20 PM UTC+2, Henrik Johansson wrote:
Add this at the end of your program and gradually increase the number from
something that works.

mem := &runtime.MemStats{}
runtime.ReadMemStats(mem)

fmt.Println(mem.HeapAlloc)

On Wed, Apr 30, 2014 at 10:01 AM, Massimiliano Tomassoli wrote:
With n = 100000 it prints 21521264 and with n = 110000 it crashes. There
must be a bug... I'm using "go1.2.1 windows/386".

--
You received this message because you are subscribed to the Google Groups
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to golang-nuts+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

On Wed, Apr 30, 2014 at 4:06 PM, 'Michael Jones' via golang-nuts wrote:

mtj-macbookpro:mem mtj$ cat mem.go
package main

import (
"fmt"
)

func f(left, right chan int) {
left <- 1 + <-right
}

func main() {
const n = 200000
leftmost := make(chan int)
right := leftmost
left := leftmost
for i := 0; i < n; i++ {
right = make(chan int)
go f(left, right)
left = right
}
go func(c chan int) { c <- 1 }(right)
fmt.Println(<-leftmost)
}
mtj-macbookpro:mem mtj$ export GOMAXPROCS=1
mtj-macbookpro:mem mtj$ time go run mem.go
200001

real 0m1.175s
user 0m0.749s
sys 0m0.393s
mtj-macbookpro:mem mtj$ export GOMAXPROCS=8
mtj-macbookpro:mem mtj$ time go run mem.go
200001

real 0m1.089s
user 0m1.423s
sys 0m0.850s



On Wed, Apr 30, 2014 at 10:01 AM, Massimiliano Tomassoli <
kiuhnm03@gmail.com> wrote:


--
*Michael T. Jones | Chief Technology Advocate | mtj@google.com
<mtj@google.com> | +1 650-335-5765 <%2B1%20650-335-5765>*

--
You received this message because you are subscribed to the Google Groups
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to golang-nuts+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

On Wednesday, April 30, 2014 4:23:00 PM UTC+2, Henrik Johansson wrote:
You have 32 bit windows I guess which means you probably have much less
effectively available if I remember correctly.
I think it is just so simple that you are hitting your systems ceiling.

No, as I said, I have the 64 bit version. I can allocate ~2.5 GB with no
problems.
Go crashes after just 450 MB (with n = 110,000).

--
You received this message because you are subscribed to the Google Groups
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to golang-nuts+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

On Wednesday, April 30, 2014 4:40:17 PM UTC+2, Henrik Johansson wrote:
But didn't your "go version" say 386? I have no idea what it should say on
windows 7 64.
Is it possible to run the 386 version of Go on 64 bit windows?

On Wednesday, April 30, 2014 8:24:21 AM UTC-7, Massimiliano Tomassoli wrote:
You're right, I'm using the 32-bit version of go.
I installed the 64-bit version and now it works, but I still think there's
something wrong with the 32-bit version.
Windows reserves 2 GB to 32-bit processes (
http://msdn.microsoft.com/en-us/library/windows/desktop/aa366912%28v=vs.85%29.aspx
).

On Wednesday, April 30, 2014 5:42:18 PM UTC+2, Carlos Castillo wrote:
In Go 1.2 goroutine stacks are 8k in size to start, so 8k x 200000 =
1.6GB. Also, the way the GC works results in double the amount of virtual
memory being used than it actually needs before the next collection happens
(on average).
If you use go 1.1, or the 1.3 beta, which both have 4k starting stacks,
the address space needed for stacks alone is halved.

Also note that it's address space that determines whether an allocation
will fail/succeed, and has no direct relation to the amount of memory that
is in use by the program (ie: it's working set), or the amount of RAM the
OS has given to it. Also, for an allocation to succeed that much contiguous
address space must exist as well, so memory fragmentation is also an issue.

There is nothing wrong with the 32-bit version of go, you are just hitting
the limits imposed by a 32-bit OS.

On Wednesday, 30 April 2014 18:05:39 UTC+1, Massimiliano Tomassoli wrote:
I still think there is something wrong. In other languages (32 bit) I can
allocate up to 1.5 GB of memory at least and in Go the daisy-chain examples
crashes with just n = 110,000.

On Wednesday, April 30, 2014 10:29:39 PM UTC+2, Benjamin Measures wrote:
Allocating 1.5GB in, say, 64KB chunks is very different to allocating
1.5GB in [...] 187.5MB, 375MB, 750MB (contiguous) chunks. Read Carlos'
reply again and let it sink in.

If that's not enough to convince you, try a 32-bit Java VM with -Xms1536m
(and understand how bad memory fragmentation is in Windows 32-bit).

On Thursday, May 1, 2014 1:13:24 AM UTC+2, James Wendel wrote:
You code creates 200,000 goroutines. Each ones takes up 8192 bytes. 8k *
200000 = 1.6GB of memory.

On Thu, May 1, 2014 at 2:45 AM, Massimiliano Tomassoli wrote:

It crashes with just 110,000 goroutines.

On Thursday, May 1, 2014 5:41:21 PM UTC+2, Jesper Louis Andersen wrote:

On Thu, May 1, 2014 at 2:45 AM, Massimiliano Tomassoli <kiuh...@gmail.com<javascript:>

This should not come as a surprise. Depending on the allocation strategy,
you might end up in a situation where you can get way less memory due to
memory fragmentation. 32bit architectures have small address spaces[0]
which means you can't easily find regions which have enough contiguous free
memory due to fragmentation.

On Friday, May 2, 2014 12:52:01 AM UTC+2, Andy Balholm wrote:
I suspect that you're running into
https://code.google.com/p/go/issues/detail?id=2323

The Go heap is allocated as a contiguous chunk of address space. Some DLLs
allocate chunks of memory at strange places that get in the way, preventing
it from finding a contiguous chunk of almost 2 GB. Apparently this problem
is worst when running 32-bit Go programs on 64-bit Windows.

On Wednesday, April 30, 2014 6:24:29 PM UTC+2, James Wendel wrote:
The 32-bit Windows Go build has garbage collection issues. See:
https://code.google.com/p/go/issues/detail?id=909
If you were to try the 32-bit build of Go 1.3 it may behave a lot better.

I've just read the comments. There are definitely some issues with the
32-bit version of Go and they're not just related to the GC.

On Wednesday, April 30, 2014 2:20:45 PM UTC+2, Massimiliano Tomassoli wrote:

The following code runs successfully with n = 100000, but not with n =
200000. Why? Out of memory?

package main

import (
"fmt"
)

func f(left, right chan int) {
left <- 1 + <-right
}

func main() {
const n = 200000
leftmost := make(chan int)
right := leftmost
left := leftmost
for i := 0; i < n; i++ {
right = make(chan int)
go f(left, right)
left = right
}
go func(c chan int) { c <- 1 }(right)
fmt.Println(<-leftmost)
}