[go-nuts] [ANN] btcd - A full node Bitcoin implementation written in Go

Quite interested about the memory leaks you found in Go. Can you elaborate?

Good job!
I think golang is very suitable for application like this. (Another one I
can think of is probably erlang)
Do you have any performance benchmark compared to other implementations?
I'd like to see the results.

All of the referenced memory leaks have been fixed as of Go tip, but a
basic
summary is:

1) Maps were leaking when GOMAXPROCS > 1
    (https://groups.google.com/forum/#!topic/golang-nuts/TGAe2MzVxO0)
2) When append created a new underlying array, it still considered the old
array
    live. When that old array contained pointers, those pointers in turn
would
    still be considered live. As a result they would never be garbage
collected.
    (https://groups.google.com/forum/#!topic/golang-nuts/CxMqNWZ4RrM)
3) The third issue, as it turns out, was really the same underlying issue
as #2
    just manifesting in a different way. It involved a mix of sub-slices and
    appends in such a way that the underlying array had to grow, and when
that
    happened it led to the issue in #2.

Because downloading and validating the Bitcoin block chain involves
churning an
obscene amount of data, these leaks were huge issues for us. We were
spiking
upwards of 4GB of memory usage. After the fixes, we are seeing ~180MB.
These
were particularly challenging to track down because the memory profiling of
Go
claimed everything was fine and wasn't showing any of the leaked memory was
allocated, but the OS clearly showed it was and as a result all we could
tell
is memory was just being gobbled up at alarming rates with no indication
why.

I am unhappy to report that btcd compiles successfully under NetBSD (6.0
beta 2) and start running quite successfully, but eventually fails,
seemingly during some garbage collection operation, but that is an
uneducated guess on my part. A snippet of the output says:

SIGSEGV: segmentation violation
PC=0x80f86c4

runtime.settype_flush(0x8e80800)
         /archive/project/GoLang/src/pkg/runtime/malloc.goc:596 +0x84
gc(0xbba2ca70)
         /archive/project/GoLang/src/pkg/runtime/mgc0.c:2072 +0x5e
mgc(0x8eac640)
         /archive/project/GoLang/src/pkg/runtime/mgc0.c:2050 +0x2b
runtime.mcall(0x804c7bb)
         /archive/project/GoLang/src/pkg/runtime/asm_386.s:190 +0x3c

goroutine 20 [garbage collection]:
runtime.gc(0x1)
         /archive/project/GoLang/src/pkg/runtime/mgc0.c:2021 +0x183
fp=0xbba2ca7c

I am very pleased that bitcoin handling is now possible in Go and I'm
looking forward to the meme spreading across architectures, so this
particular setback is quite important to me and if I can help resolve it,
I'll do it with pleasure. Unfortunately, I suffer from a shortage of
computing power (I have access to lots of slow, old PCs, but nothing
modern) and very limited understanding of the bitcoin meme, although I have
a friend high in the local (South African) finance industry I would dearly
like to impress.

Lucio.

PS: The full error report and preceding typescript is available. I confess
I do not understand much of it.

That looks like a Go runtime bug, not a btcd bug. The NetBSD port is
very immature.

I don't think the NetBSD port is very immature.
yes, the amd64 builder fails a lot, but i kinda suspect this is the builder
problem,
as the 386 builder (note that Lucio is using the 386 port) is working fine.

I don't want to make it sound worse than it is. Go works reasonably
well on NetBSD and it has been tested on it far more than some other
very prominent software. My personal standards for software maturity
are extremely tight, don't buy too much in my exact wording.

That being said, NetBSD is only surpassed by lack of maturity by the
recent DragonflyBSD port.

Tip. Never learnt to use anything else. How can I help?

Lucio.