On Sun, Nov 22, 2015 at 11:02 PM, murali wrote:


By now the Go packages are compiled into an intermediate representation
and stored in an ad hoc binary format, as far as I can tell.
Is it possible and practical to employ existing binary format like
FlatBuffers for Go's package archives?

My original idea was to implement a marshal/unmarshal library for the .a
files as a building block for writing other tools in the future.
Then I thought, what if the data structures can be stored in the archives
can be formatted with existing serialization library?

What do you think?
Please do not spend time on this.

The overall container - the .a file - is a well-understood and very widely
used format. It dates back to very early Unix. I hope at some point to
replace it with a different format, namely the zip file format, which will
have some nicer properties but again be something well understood. I say
this because you mentioned the .a file, but I don't think you meant that.

I think you meant the .o file, which as generated by the Go compiler
contains: (1) compiled code, (2) relocations for the machine code, (3) the
package export data, and (4) directives for the linker. (In files generated
by the assembler, (3) and (4) are empty.)

(1) The compiled code is actual machine code, as dictated by the
architecture. This choice of representation is very important for reducing
work done by the linker (see golang.org/s/go13linker).
(2) The relocations are slightly ad-hoc but decently well defined. If we
were going to change to a different serialization, it would probably be to
adopt ELF's definition, not to invent another.
(3) The package export data is being redone by Robert and will be very nice
and clean when he's finished.
(4) The linker directives are simple strings.

Automatic serialization like FlatBuffers is off the table for (1). For (2),
(3), and (4), the fundamental problem is that basically all automatic
serialization assumes that you've got the exact data that needs to be
serialized in a contained unit in your program. Not so in any of those. The
relevant information and directives are part of larger structures. Yes, we
could define new serialization-ready data structures and write code to walk
the larger structures, generate the new structures, and serialize them. But
it's more efficient and more direct just to write the serialization code,
and then you can make sure the actual serialization has properties you care
about (like using real machine code in (1), or possibly an existing
definition like ELF in (2), or correctly handling cyclic data structures in
(3), or being trivial for people to read in (4)).

In general I think people overvalue automatic serialization, because they
overestimate the cost of manual serialization and underestimate the
complexity of the automated systems.


You received this message because you are subscribed to the Google Groups "golang-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-dev+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Search Discussions

Discussion Posts


Follow ups

Related Discussions

Discussion Navigation
viewthread | post
posts ‹ prev | 3 of 4 | next ›
Discussion Overview
groupgolang-dev @
postedNov 23, '15 at 4:12a
activeNov 23, '15 at 5:55a



site design / logo © 2022 Grokbase