[go-nuts] [ANN] A reader for files in the dictzip format

Out of curiosity, what is the advantage of your internal 'decode()' over
package "encoding/base64"?

/M.

It is another format.

I should document this.

I'd create the index table from the list in init() so that they are always
consistent.

  huh?

not easy to verify that the table is correct, so if I were you I'd create
the table from "list" in init() function at runtime, instead of writing the
table as static data. In your code "list" is not even referenced at all
which feels odd to me.

Ah, I see. The "list" is used by the encoder, but there is only a decoder.
This is translated from C code.

Uhmh, can I ask what C code? Looking at dictzip manpage you linked
(http://linuxcommand.org/man_pages/dictzip1.html), it seems to read
"dictzip was written by Rik Faith (faith@cs.unc.edu) and is distributed
under the terms of the GNU General Public License. If you need to
distribute under other terms, write to the author." But you seem to be
publishing your code under BSD/MIT?

Sorry for pushing this in various directions which can make you feel it was
more trouble publishing than it's worth... I'm really very grateful you
decided to publish, this fileformat and idea looks very interesting and I'm
very happy I could learn about it, might really come handy one day.

/M.

The C code for decoding the base64 format dictzip uses, it didn't come from
dictzip sources.

But it's not a different format.

From encoding/base64:
const encodeStd =
"ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/"

From your code:

  list = "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/"


Also, why re-implement io.ReadFull?

encoding/base64 endodes strings.

base64 in dictzip encodes an integer value (I don't know why this is
usefull).

Because I couldn't find it.

It's close enough. http://play.golang.org/p/AeLOSRN1Qk

Now a writer as well

Op donderdag 3 juli 2014 11:51:07 UTC+2 schreef Peter Kleiweg:

It might be a good idea if the reader implements one of the io.*
interfaces. My suggestion would be to implement io.ReaderAt, since ReadAt
is essentially the same as your Get method, except that it fills a given
buffer instead of creating a new one (fewer allocs!). Then someone can use
an io.SectionReader to wrap it to get a "Read" method.

Even if I implement io.ReaderAt, I still have to allocate a buffer, because
usually, I need a larger buffer than was requested. The reader has to start
decoding at certain boundaries, and those usually don't coincide with the
start of what the user is requesting.

Ah, I see.

I still think ReadAt is till a good idea for the reason I mentioned, and
since there's no allocations to be saved (without a significant rewrite),
it can easily be written w.r.t. Get().

So users can wrap it to get a Read method? You can use compress/gzip for
that. A dictzip file is just a gzip file with some extra index information.

But you can't Read an arbitrary subsection of a gzip file.

When you make a io.SectionReader you specify which part of the file you
want, via an offset and size, and you get back an object which you can Read
from (starting at that offset).

This would allow you to pipe part of a dictzip file to any method that
wants a Reader. Right now, you have to read the whole chunk at once (via
Get) and then wrap it with a bytes.Reader, with this method it could be
streamed to the target method.

I see. Will do.

Op vrijdag 4 juli 2014 12:15:49 UTC+2 schreef Carlos Castillo: