I am trying to come up with a binary file format to store GiB (well,
TiB) worth of data.
One particularly important issue is to be able to inspect a file's
content and know:
- for a given "folder" (or rather, "block") what is the type of data
- for a given "folder" what is the type content being stored
so if the user gives the file-reader layer a pointer to a value to be
read into, the unmarshaling can check the types match.
and, the file-reader layer is given a different type but with matching
layouts, the deserialization can still take place.
and, if the file-reader layer is given no type, it can still display
values thanks to the type metadata stored about the type's layout
associated with the "folder".
How would you go about it?
- I know the "gob" (de)serialization package is self-describing but
each gob-stream needs to be primed with the types descriptors, so I
would need to store this information for each file "block" even when
"block-1" and "block-2" contain data of the same type (and I need to
be able to seek to block-1,..-n or just read block-m so I can't lump
also, I need to be able to tell the type's layout just from looking at
some block header (and I don't think "gob" exposes that type of
- "protobuf" is also a self-decribing format (if you serialize the
TypeDescriptors) but it's a pain to use in a pure-go setup.
- "hdf5" is too slow (especially from Go, via the cgo gateway.)
did I forget a very useful (de)serialization package out there?
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to firstname.lastname@example.org.
For more options, visit https://groups.google.com/d/optout.