FAQ
Hi

By now the Go packages are compiled into an intermediate representation and
stored in an ad hoc binary format, as far as I can tell.
Is it possible and practical to employ existing binary format like
FlatBuffers for Go's package archives?

My original idea was to implement a marshal/unmarshal library for the .a
files as a building block for writing other tools in the future.
Then I thought, what if the data structures can be stored in the archives
can be formatted with existing serialization library?

What do you think?



--
You received this message because you are subscribed to the Google Groups "golang-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-dev+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Search Discussions

  • Ian Lance Taylor at Nov 23, 2015 at 4:34 am

    On Sun, Nov 22, 2015 at 8:02 PM, murali wrote:
    By now the Go packages are compiled into an intermediate representation and
    stored in an ad hoc binary format, as far as I can tell.
    Is it possible and practical to employ existing binary format like
    FlatBuffers for Go's package archives?

    My original idea was to implement a marshal/unmarshal library for the .a
    files as a building block for writing other tools in the future.
    Then I thought, what if the data structures can be stored in the archives
    can be formatted with existing serialization library?

    What do you think?
    The Go archive format is the same as that recognized by the Unix ar
    tool, although Go uses it slightly differently. The objects stored in
    the Go archive format are, as you say, ad hoc. I wouldn't call it an
    intermediate representation, though; it contains fully compiled
    instructions with relocations, like other object file formats.

    We might change that in the future (https://golang.org/issue/11123).
    If we did, we might use a serialization format. But it seems to me we
    would use one already supported by the Go standard library, like json
    or gob. I don't see a clear reason to use something new.

    Ian

    --
    You received this message because you are subscribed to the Google Groups "golang-dev" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-dev+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
  • Russ Cox at Nov 23, 2015 at 4:38 am

    On Sun, Nov 22, 2015 at 11:02 PM, murali wrote:

    Hi

    By now the Go packages are compiled into an intermediate representation
    and stored in an ad hoc binary format, as far as I can tell.
    Is it possible and practical to employ existing binary format like
    FlatBuffers for Go's package archives?

    My original idea was to implement a marshal/unmarshal library for the .a
    files as a building block for writing other tools in the future.
    Then I thought, what if the data structures can be stored in the archives
    can be formatted with existing serialization library?

    What do you think?
    Please do not spend time on this.

    The overall container - the .a file - is a well-understood and very widely
    used format. It dates back to very early Unix. I hope at some point to
    replace it with a different format, namely the zip file format, which will
    have some nicer properties but again be something well understood. I say
    this because you mentioned the .a file, but I don't think you meant that.

    I think you meant the .o file, which as generated by the Go compiler
    contains: (1) compiled code, (2) relocations for the machine code, (3) the
    package export data, and (4) directives for the linker. (In files generated
    by the assembler, (3) and (4) are empty.)

    (1) The compiled code is actual machine code, as dictated by the
    architecture. This choice of representation is very important for reducing
    work done by the linker (see golang.org/s/go13linker).
    (2) The relocations are slightly ad-hoc but decently well defined. If we
    were going to change to a different serialization, it would probably be to
    adopt ELF's definition, not to invent another.
    (3) The package export data is being redone by Robert and will be very nice
    and clean when he's finished.
    (4) The linker directives are simple strings.

    Automatic serialization like FlatBuffers is off the table for (1). For (2),
    (3), and (4), the fundamental problem is that basically all automatic
    serialization assumes that you've got the exact data that needs to be
    serialized in a contained unit in your program. Not so in any of those. The
    relevant information and directives are part of larger structures. Yes, we
    could define new serialization-ready data structures and write code to walk
    the larger structures, generate the new structures, and serialize them. But
    it's more efficient and more direct just to write the serialization code,
    and then you can make sure the actual serialization has properties you care
    about (like using real machine code in (1), or possibly an existing
    definition like ELF in (2), or correctly handling cyclic data structures in
    (3), or being trivial for people to read in (4)).

    In general I think people overvalue automatic serialization, because they
    overestimate the cost of manual serialization and underestimate the
    complexity of the automated systems.

    Russ

    --
    You received this message because you are subscribed to the Google Groups "golang-dev" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-dev+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
  • Murali at Nov 23, 2015 at 5:55 am
    Understood. Thanks for your (Ian and Russ) elaboration!
    On Monday, November 23, 2015 at 12:38:05 PM UTC+8, rsc wrote:

    On Sun, Nov 22, 2015 at 11:02 PM, murali <mur...@castech.com.tw
    <javascript:>> wrote:
    Hi

    By now the Go packages are compiled into an intermediate representation
    and stored in an ad hoc binary format, as far as I can tell.
    Is it possible and practical to employ existing binary format like
    FlatBuffers for Go's package archives?

    My original idea was to implement a marshal/unmarshal library for the .a
    files as a building block for writing other tools in the future.
    Then I thought, what if the data structures can be stored in the archives
    can be formatted with existing serialization library?

    What do you think?
    Please do not spend time on this.

    The overall container - the .a file - is a well-understood and very widely
    used format. It dates back to very early Unix. I hope at some point to
    replace it with a different format, namely the zip file format, which will
    have some nicer properties but again be something well understood. I say
    this because you mentioned the .a file, but I don't think you meant that.

    I think you meant the .o file, which as generated by the Go compiler
    contains: (1) compiled code, (2) relocations for the machine code, (3) the
    package export data, and (4) directives for the linker. (In files generated
    by the assembler, (3) and (4) are empty.)

    (1) The compiled code is actual machine code, as dictated by the
    architecture. This choice of representation is very important for reducing
    work done by the linker (see golang.org/s/go13linker).
    (2) The relocations are slightly ad-hoc but decently well defined. If we
    were going to change to a different serialization, it would probably be to
    adopt ELF's definition, not to invent another.
    (3) The package export data is being redone by Robert and will be very
    nice and clean when he's finished.
    (4) The linker directives are simple strings.

    Automatic serialization like FlatBuffers is off the table for (1). For
    (2), (3), and (4), the fundamental problem is that basically all automatic
    serialization assumes that you've got the exact data that needs to be
    serialized in a contained unit in your program. Not so in any of those. The
    relevant information and directives are part of larger structures. Yes, we
    could define new serialization-ready data structures and write code to walk
    the larger structures, generate the new structures, and serialize them. But
    it's more efficient and more direct just to write the serialization code,
    and then you can make sure the actual serialization has properties you care
    about (like using real machine code in (1), or possibly an existing
    definition like ELF in (2), or correctly handling cyclic data structures in
    (3), or being trivial for people to read in (4)).

    In general I think people overvalue automatic serialization, because they
    overestimate the cost of manual serialization and underestimate the
    complexity of the automated systems.

    Russ
    --
    You received this message because you are subscribed to the Google Groups "golang-dev" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-dev+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupgolang-dev @
categoriesgo
postedNov 23, '15 at 4:12a
activeNov 23, '15 at 5:55a
posts4
users3
websitegolang.org

People

Translate

site design / logo © 2021 Grokbase