Grokbase › Groups › Go › golang-nuts › May 2014
FAQ
Hi,

I have been using go-bindata (https://github.com/jteeuwen/go-bindata) to
convert some assets (templates and image files) to Go source files.

When compiling the resulting source code the go compiler uses an insane
amount of memory. I compile on Win8.1 (64-bit) and Ubuntu 14.04 (64-bit).
On Windows compiling the source spikes at around 1.6 GB memory use - but on
Ubuntu the memory usage goes to about 7 GB.

The source file from Go-bindata is about 7 MB (a bit larger than the
original assets ~700 KB).

Is there a logical reason to the:
- Amount of memory used?
- Different amount of memory used?
- Difference in source code size and compiler memory usage?

I can't really share the source/assets - but if it brings any value I can
spend some time to re-create the scenario with sharable data.

--
Michael Banzon
http://michaelbanzon.com/

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Search Discussions

  • Davmaz at Sep 28, 2015 at 10:30 pm ⇧
    Hello all - I'm trying to build go programs on a Xilinx Zynq development
    board that has 512MB of RAM. There is no other storage. When I have all the
    tools in place (cross compiled and put in the /usr/local/go directories) a
    du -hs / reports 185MB used. I can run go doc fmt and most other go
    commands. BUT if I try to *build* the simplest (fmt.Println("Hello!"))
    programs, it eventually kernel panics.

    I can cross compile and copy over any reasonable program on the (Zynq/ARM)
    target. I'd like to build on the target platform so I can eventually test
    and profile code natively.

    Any suggestions?

    On Sunday, May 18, 2014 at 8:58:14 PM UTC-4, Dave Cheney wrote:

    Does go-bindata produce a large array like

    var asset = []byte{ 0x30, 0x01, 0x44 } ?

    If so, this will cause the compiler to use a lot of memory because of the
    way that every constant literal is stored inside the compiler as a multi
    precision number. The best way to solve this is to instead keep your asset
    as a base64 encoded string literal

    var asset = "somebase64text"

    and feed the asset through a base64 reader before consuming it.

    This generates a much more efficient compiled form.
    On Sunday, 18 May 2014 23:25:26 UTC+10, mbanzon wrote:

    Hi,

    I have been using go-bindata (https://github.com/jteeuwen/go-bindata) to
    convert some assets (templates and image files) to Go source files.

    When compiling the resulting source code the go compiler uses an insane
    amount of memory. I compile on Win8.1 (64-bit) and Ubuntu 14.04 (64-bit).
    On Windows compiling the source spikes at around 1.6 GB memory use - but on
    Ubuntu the memory usage goes to about 7 GB.

    The source file from Go-bindata is about 7 MB (a bit larger than the
    original assets ~700 KB).

    Is there a logical reason to the:
    - Amount of memory used?
    - Different amount of memory used?
    - Difference in source code size and compiler memory usage?

    I can't really share the source/assets - but if it brings any value I can
    spend some time to re-create the scenario with sharable data.

    --
    Michael Banzon
    http://michaelbanzon.com/
    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
  • Dave Cheney at Sep 28, 2015 at 10:56 pm ⇧
    It is possible that your go installation is incorrect. Can you please post the output of go build -x.

    It /tmp on disk, or in ram?

    Dave

    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
  • Dave Mazzoni at Sep 28, 2015 at 11:27 pm ⇧
    I think the installation is correct. But my approach is wrong. I moved all
    the go tools over to the target/Arm system under the assumption it was the
    only way to run go test and go pprof on the target/Arm embedded system.

    The real payoff would be to do all the heavy lifting (compiling, testing,
    profiling, etc) on the host OS/Linux amd64 and transfer those executables
    to the target.

    Let me pose the question this way: is there a way I can cross-compile
    everything I need on the host and then transfer the files to the target *to
    do testing and profiling*? It seems I would need a way to
    build/cross-compile an "instrumented" executable for the target and then
    copy and run it there.

    Make sense?
    On Mon, Sep 28, 2015 at 6:56 PM, Dave Cheney wrote:

    It is possible that your go installation is incorrect. Can you please post
    the output of go build -x.

    It /tmp on disk, or in ram?

    Dave

    --
    You received this message because you are subscribed to a topic in the
    Google Groups "golang-nuts" group.
    To unsubscribe from this topic, visit
    https://groups.google.com/d/topic/golang-nuts/tOp5mx_SJMI/unsubscribe.
    To unsubscribe from this group and all its topics, send an email to
    golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
  • Dave Cheney at Sep 28, 2015 at 11:32 pm ⇧
    You can build test binaries with the -c flag, this should support the usual
    GOOS/GOARCH flags, but not tested.

    The go tool also supports the -execcmd (from memory) flag which effectively
    inserts a shim, the cmd, before running your program, check out the various
    wrappers in the misc directory, this is who we do the darwin/arm and
    android builds.

    Profiling will work as expected with a cross compiled binary, just scp back
    the binary and the profile to your workstation and use pprof as usual.

    Dave
    On Tue, 29 Sep 2015 09:27 Dave Mazzoni wrote:

    I think the installation is correct. But my approach is wrong. I moved all
    the go tools over to the target/Arm system under the assumption it was the
    only way to run go test and go pprof on the target/Arm embedded system.

    The real payoff would be to do all the heavy lifting (compiling, testing,
    profiling, etc) on the host OS/Linux amd64 and transfer those executables
    to the target.

    Let me pose the question this way: is there a way I can cross-compile
    everything I need on the host and then transfer the files to the target *to
    do testing and profiling*? It seems I would need a way to
    build/cross-compile an "instrumented" executable for the target and then
    copy and run it there.

    Make sense?
    On Mon, Sep 28, 2015 at 6:56 PM, Dave Cheney wrote:

    It is possible that your go installation is incorrect. Can you please
    post the output of go build -x.

    It /tmp on disk, or in ram?

    Dave
    --
    You received this message because you are subscribed to a topic in the
    Google Groups "golang-nuts" group.
    To unsubscribe from this topic, visit
    https://groups.google.com/d/topic/golang-nuts/tOp5mx_SJMI/unsubscribe.
    To unsubscribe from this group and all its topics, send an email to
    golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
  • Ian Lance Taylor at Sep 29, 2015 at 12:17 am ⇧

    On Mon, Sep 28, 2015 at 4:32 PM, Dave Cheney wrote:
    The go tool also supports the -execcmd (from memory) flag which effectively
    inserts a shim, the cmd, before running your program, check out the various
    wrappers in the misc directory, this is who we do the darwin/arm and android
    builds.
    Note that it's -exec, not -execcmd. See
    https://golang.org/cmd/go/#hdr-Compile_and_run_Go_program for more
    details.

    Ian

    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
  • Dave Mazzoni at Sep 29, 2015 at 12:20 am ⇧
    Perfect -- it's working. This is a huge win for the embedded community.
    Thanks!
    On Mon, Sep 28, 2015 at 8:17 PM, Ian Lance Taylor wrote:
    On Mon, Sep 28, 2015 at 4:32 PM, Dave Cheney wrote:

    The go tool also supports the -execcmd (from memory) flag which
    effectively
    inserts a shim, the cmd, before running your program, check out the various
    wrappers in the misc directory, this is who we do the darwin/arm and android
    builds.
    Note that it's -exec, not -execcmd. See
    https://golang.org/cmd/go/#hdr-Compile_and_run_Go_program for more
    details.

    Ian
    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
  • Dave Mazzoni at Sep 29, 2015 at 11:59 pm ⇧
    Almost... While I can get go tests to cross-compile correctly using the -c
    option, when I reintroduce C code with CGO, I now get runtime errors on the
    target:
    # ./ad9361_test.go
    ./ad9361_test.go: line 1: package: not found
    ./ad9361_test.go: line 3: syntax error: unexpected newline (expecting ")")
    Am I missing something?
    On Mon, Sep 28, 2015 at 8:19 PM, Dave Mazzoni wrote:

    Perfect -- it's working. This is a huge win for the embedded community.
    Thanks!
    On Mon, Sep 28, 2015 at 8:17 PM, Ian Lance Taylor wrote:
    On Mon, Sep 28, 2015 at 4:32 PM, Dave Cheney wrote:

    The go tool also supports the -execcmd (from memory) flag which
    effectively
    inserts a shim, the cmd, before running your program, check out the various
    wrappers in the misc directory, this is who we do the darwin/arm and android
    builds.
    Note that it's -exec, not -execcmd. See
    https://golang.org/cmd/go/#hdr-Compile_and_run_Go_program for more
    details.

    Ian
    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
  • Dave Cheney at Sep 30, 2015 at 12:01 am ⇧
    Cross compilation of CGO packages is not supported by default as it requires a cross compiling c compiler for your target.

    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupgolang-nuts @
categoriesgo
postedMay 18, '14 at 1:25p
activeSep 30, '15 at 12:01a
posts9
users4
websitegolang.org

4 users in discussion

Dave Mazzoni: 4 posts Dave Cheney: 3 posts Michael Banzon: 1 post Ian Lance Taylor: 1 post

Content

People

Support

Translate

site design / logo © 2022 Grokbase