Grokbase › Groups › Go › golang-nuts › March 2013
FAQ
Hello all

A quick note about a few growing pains we've been hitting as our code base
has increased in size.

I recently started looking at our package hierarchy when one of my tests
started taking about 3 seconds to link, even with everything in page cache
(transparent hugepages are off, so I'm avoiding the problem I described in
issue 3080).

The test depends on 163 packages. Some of them have nothing to do with the
test. We need to clean up our package hierarchy. We could probably get it
down to about 100 packages with careful refactoring. Maybe 120 is more
realistic.

According to perf, the two hot spots in 6l are for loops in zaddr and patch
that together take up about 50%+ of the link time.

Our $GOPATH/bin contains about 80 binaries, totaling 964 MB.

The biggest binary that includes almost all our code is 25 MB. We usually
deploy about 20 of these binaries to a server.

We could get a significant saving in link time and storage (which
translates to a smaller initramfs for our deployment) by using a "busybox
trick" to build all our commands as a single binary that looks at argv[0]
to decide what to run. Having most of our code in a shared library would be
the other option.

Cheers

Albert

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].
For more options, visit https://groups.google.com/groups/opt_out.

Search Discussions

  • Gerard at Mar 20, 2013 at 10:01 am ⇧
    Are you asking a question?

    Op woensdag 20 maart 2013 10:48:42 UTC+1 schreef Albert Strasheim het
    volgende:
    Hello all

    A quick note about a few growing pains we've been hitting as our code base
    has increased in size.

    I recently started looking at our package hierarchy when one of my tests
    started taking about 3 seconds to link, even with everything in page cache
    (transparent hugepages are off, so I'm avoiding the problem I described in
    issue 3080).

    The test depends on 163 packages. Some of them have nothing to do with the
    test. We need to clean up our package hierarchy. We could probably get it
    down to about 100 packages with careful refactoring. Maybe 120 is more
    realistic.

    According to perf, the two hot spots in 6l are for loops in zaddr and
    patch that together take up about 50%+ of the link time.

    Our $GOPATH/bin contains about 80 binaries, totaling 964 MB.

    The biggest binary that includes almost all our code is 25 MB. We usually
    deploy about 20 of these binaries to a server.

    We could get a significant saving in link time and storage (which
    translates to a smaller initramfs for our deployment) by using a "busybox
    trick" to build all our commands as a single binary that looks at argv[0]
    to decide what to run. Having most of our code in a shared library would be
    the other option.

    Cheers

    Albert
    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].
    For more options, visit https://groups.google.com/groups/opt_out.
  • Dave Cheney at Mar 20, 2013 at 10:07 am ⇧
    I think it is feedback from a Go user who is doing some very large
    things with the language. Albert is a great supporter of the language
    and has dedicated significant computational resources to stress
    testing the development branch.
    On Wed, Mar 20, 2013 at 9:01 PM, Gerard wrote:
    Are you asking a question?

    Op woensdag 20 maart 2013 10:48:42 UTC+1 schreef Albert Strasheim het
    volgende:
    Hello all

    A quick note about a few growing pains we've been hitting as our code base
    has increased in size.

    I recently started looking at our package hierarchy when one of my tests
    started taking about 3 seconds to link, even with everything in page cache
    (transparent hugepages are off, so I'm avoiding the problem I described in
    issue 3080).

    The test depends on 163 packages. Some of them have nothing to do with the
    test. We need to clean up our package hierarchy. We could probably get it
    down to about 100 packages with careful refactoring. Maybe 120 is more
    realistic.

    According to perf, the two hot spots in 6l are for loops in zaddr and
    patch that together take up about 50%+ of the link time.

    Our $GOPATH/bin contains about 80 binaries, totaling 964 MB.

    The biggest binary that includes almost all our code is 25 MB. We usually
    deploy about 20 of these binaries to a server.

    We could get a significant saving in link time and storage (which
    translates to a smaller initramfs for our deployment) by using a "busybox
    trick" to build all our commands as a single binary that looks at argv[0] to
    decide what to run. Having most of our code in a shared library would be the
    other option.

    Cheers

    Albert
    --
    You received this message because you are subscribed to the Google Groups
    "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an
    email to [email protected].
    For more options, visit https://groups.google.com/groups/opt_out.
    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].
    For more options, visit https://groups.google.com/groups/opt_out.
  • Ryan Slade at Mar 20, 2013 at 12:42 pm ⇧
    Hi Albert

    Purely out of curiosity, are you able to share any more details about the
    system you are working on? I'm always interested to hear about large
    systems using Go and I'm sure the rest of the community would be too.

    Thanks
    Ryan
    On Wednesday, 20 March 2013 09:48:42 UTC, Albert Strasheim wrote:

    Hello all

    A quick note about a few growing pains we've been hitting as our code base
    has increased in size.

    I recently started looking at our package hierarchy when one of my tests
    started taking about 3 seconds to link, even with everything in page cache
    (transparent hugepages are off, so I'm avoiding the problem I described in
    issue 3080).

    The test depends on 163 packages. Some of them have nothing to do with the
    test. We need to clean up our package hierarchy. We could probably get it
    down to about 100 packages with careful refactoring. Maybe 120 is more
    realistic.

    According to perf, the two hot spots in 6l are for loops in zaddr and
    patch that together take up about 50%+ of the link time.

    Our $GOPATH/bin contains about 80 binaries, totaling 964 MB.

    The biggest binary that includes almost all our code is 25 MB. We usually
    deploy about 20 of these binaries to a server.

    We could get a significant saving in link time and storage (which
    translates to a smaller initramfs for our deployment) by using a "busybox
    trick" to build all our commands as a single binary that looks at argv[0]
    to decide what to run. Having most of our code in a shared library would be
    the other option.

    Cheers

    Albert
    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].
    For more options, visit https://groups.google.com/groups/opt_out.
  • Damian Gryski at Mar 20, 2013 at 1:30 pm ⇧
    Le mercredi 20 mars 2013 13:42:51 UTC+1, Ryan Slade a écrit :
    Purely out of curiosity, are you able to share any more details about the
    system you are working on? I'm always interested to hear about large
    systems using Go and I'm sure the rest of the community would be too.
    And then we can add a link to
    https://code.google.com/p/go-wiki/wiki/GoUsers :)

    Damian

    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].
    For more options, visit https://groups.google.com/groups/opt_out.
  • Taru Karttunen at Mar 20, 2013 at 1:52 pm ⇧

    On 20.03 02:48, Albert Strasheim wrote:
    The biggest binary that includes almost all our code is 25 MB. We usually
    deploy about 20 of these binaries to a server.

    We could get a significant saving in link time and storage (which
    translates to a smaller initramfs for our deployment) by using a "busybox
    trick" to build all our commands as a single binary that looks at argv[0]
    to decide what to run. Having most of our code in a shared library would be
    the other option.
    We use stripping the binary and the argv[0] trick to great effects.

    - Taru Karttunen

    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].
    For more options, visit https://groups.google.com/groups/opt_out.
  • Rob Pike at Mar 20, 2013 at 5:33 pm ⇧
    The linker has long been known to be the bottleneck for large
    binaries. More tuning remains to be done.

    -rob

    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].
    For more options, visit https://groups.google.com/groups/opt_out.

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupgolang-nuts @
categoriesgo
postedMar 20, '13 at 9:48a
activeMar 20, '13 at 5:33p
posts7
users7
websitegolang.org

7 users in discussion

Albert Strasheim: 1 post Damian Gryski: 1 post Taru Karttunen: 1 post Dave Cheney: 1 post Ryan Slade: 1 post Rob Pike: 1 post Gerard: 1 post

Content

People

Support

Translate

site design / logo © 2023 Grokbase