FAQ
Get it here: https://github.com/pebbe/dictzip

More on dictzip: http://linuxcommand.org/man_pages/dictzip1.html

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Search Discussions

  • Mateusz Czapliński at Jul 3, 2014 at 11:14 am

    On Thursday, July 3, 2014 11:51:07 AM UTC+2, Peter Kleiweg wrote:
    Get it here: https://github.com/pebbe/dictzip
    Out of curiosity, what is the advantage of your internal 'decode()' over
    package "encoding/base64"?

    /M.

    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
  • Peter Kleiweg at Jul 3, 2014 at 3:03 pm

    Mateusz Czapliński schreef op de 3e dag van de hooimaand van het jaar 2014:
    On Thursday, July 3, 2014 11:51:07 AM UTC+2, Peter Kleiweg wrote:

    Get it here: https://github.com/pebbe/dictzip
    Out of curiosity, what is the advantage of your internal 'decode()' over
    package "encoding/base64"?
    It is another format.

    I should document this.



    --
    Peter

    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
  • Rui Ueyama at Jul 3, 2014 at 8:52 pm

    On Thu, Jul 3, 2014 at 4:23 AM, Peter Kleiweg wrote:

    Mateusz Czapliński schreef op de 3e dag van de hooimaand van het jaar 2014:
    On Thursday, July 3, 2014 11:51:07 AM UTC+2, Peter Kleiweg wrote:

    Get it here: https://github.com/pebbe/dictzip
    Out of curiosity, what is the advantage of your internal 'decode()' over
    package "encoding/base64"?
    It is another format.

    I should document this.
    I'd create the index table from the list in init() so that they are always
    consistent.



    --
    Peter

    --
    You received this message because you are subscribed to the Google Groups
    "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an
    email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
  • Peter Kleiweg at Jul 3, 2014 at 10:14 pm

    Op donderdag 3 juli 2014 22:52:47 UTC+2 schreef Rui Ueyama:


    I'd create the index table from the list in init() so that they are always
    consistent.
      huh?

    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
  • Rui Ueyama at Jul 3, 2014 at 10:29 pm

    On Thu, Jul 3, 2014 at 3:14 PM, Peter Kleiweg wrote:

    Op donderdag 3 juli 2014 22:52:47 UTC+2 schreef Rui Ueyama:


    I'd create the index table from the list in init() so that they are
    always consistent.
    huh?

    "index" in your code is the reverse lookup table for "list", right? It's
    not easy to verify that the table is correct, so if I were you I'd create
    the table from "list" in init() function at runtime, instead of writing the
    table as static data. In your code "list" is not even referenced at all
    which feels odd to me.

    --
    You received this message because you are subscribed to the Google Groups
    "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an
    email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
  • Peter Kleiweg at Jul 3, 2014 at 10:49 pm

    Op vrijdag 4 juli 2014 00:30:16 UTC+2 schreef Rui Ueyama:

    On Thu, Jul 3, 2014 at 3:14 PM, Peter Kleiweg <pkle...@xs4all.nl
    <javascript:>> wrote:
    Op donderdag 3 juli 2014 22:52:47 UTC+2 schreef Rui Ueyama:


    I'd create the index table from the list in init() so that they are
    always consistent.
    huh?

    "index" in your code is the reverse lookup table for "list", right? It's
    not easy to verify that the table is correct, so if I were you I'd create
    the table from "list" in init() function at runtime, instead of writing the
    table as static data. In your code "list" is not even referenced at all
    which feels odd to me.
    Ah, I see. The "list" is used by the encoder, but there is only a decoder.
    This is translated from C code.

    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
  • Mateusz Czapliński at Jul 4, 2014 at 9:01 am

    On Friday, July 4, 2014 12:48:59 AM UTC+2, Peter Kleiweg wrote:
    The "list" is used by the encoder, but there is only a decoder. This is
    translated from C code.
    Uhmh, can I ask what C code? Looking at dictzip manpage you linked
    (http://linuxcommand.org/man_pages/dictzip1.html), it seems to read
    "dictzip was written by Rik Faith (faith@cs.unc.edu) and is distributed
    under the terms of the GNU General Public License. If you need to
    distribute under other terms, write to the author." But you seem to be
    publishing your code under BSD/MIT?

    Sorry for pushing this in various directions which can make you feel it was
    more trouble publishing than it's worth... I'm really very grateful you
    decided to publish, this fileformat and idea looks very interesting and I'm
    very happy I could learn about it, might really come handy one day.

    /M.

    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
  • Peter Kleiweg at Jul 4, 2014 at 9:28 am

    Op vrijdag 4 juli 2014 11:01:22 UTC+2 schreef Mateusz Czapliński:
    On Friday, July 4, 2014 12:48:59 AM UTC+2, Peter Kleiweg wrote:

    The "list" is used by the encoder, but there is only a decoder. This is
    translated from C code.
    Uhmh, can I ask what C code? Looking at dictzip manpage you linked (
    http://linuxcommand.org/man_pages/dictzip1.html), it seems to read
    "dictzip was written by Rik Faith (fa...@cs.unc.edu <javascript:>) and is
    distributed under the terms of the GNU General Public License. If you need
    to distribute under other terms, write to the author." But you seem to be
    publishing your code under BSD/MIT?
    The C code for decoding the base64 format dictzip uses, it didn't come from
    dictzip sources.

    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
  • Matt Harden at Jul 4, 2014 at 1:18 am

    On Thu, Jul 3, 2014 at 6:23 AM, Peter Kleiweg wrote:

    Mateusz Czapliński schreef op de 3e dag van de hooimaand van het jaar 2014:
    Out of curiosity, what is the advantage of your internal 'decode()' over
    package "encoding/base64"?
    It is another format.

    I should document this.
    But it's not a different format.

    From encoding/base64:
    const encodeStd =
    "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/"

    From your code:

      list = "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/"


    Also, why re-implement io.ReadFull?


    --
    Peter

    --
    You received this message because you are subscribed to the Google Groups
    "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an
    email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
  • Peter Kleiweg at Jul 4, 2014 at 9:21 am

    Op vrijdag 4 juli 2014 03:19:00 UTC+2 schreef Matt Harden:

    On Thu, Jul 3, 2014 at 6:23 AM, Peter Kleiweg <p.c.j....@rug.nl
    <javascript:>> wrote:
    Mateusz Czapliński schreef op de 3e dag van de hooimaand van het jaar
    2014:
    Out of curiosity, what is the advantage of your internal 'decode()' over
    package "encoding/base64"?
    It is another format.

    I should document this.
    But it's not a different format.

    From encoding/base64:
    const encodeStd =
    "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/"

    From your code:

    list = "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/"
    encoding/base64 endodes strings.

    base64 in dictzip encodes an integer value (I don't know why this is
    usefull).

    Also, why re-implement io.ReadFull?
    Because I couldn't find it.

    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
  • Matt Harden at Jul 4, 2014 at 8:47 pm

    On Fri, Jul 4, 2014 at 4:21 AM, Peter Kleiweg wrote:

    Op vrijdag 4 juli 2014 03:19:00 UTC+2 schreef Matt Harden:
    On Thu, Jul 3, 2014 at 6:23 AM, Peter Kleiweg wrote:

    Mateusz Czapliński schreef op de 3e dag van de hooimaand van het jaar
    2014:
    Out of curiosity, what is the advantage of your internal 'decode()' over
    package "encoding/base64"?
    It is another format.

    I should document this.
    But it's not a different format.

    From encoding/base64:
    const encodeStd = "ABCDEFGHIJKLMNOPQRSTUVWXYZabcd
    efghijklmnopqrstuvwxyz0123456789+/"

    From your code:

    list = "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/"
    encoding/base64 endodes strings.

    base64 in dictzip encodes an integer value (I don't know why this is
    usefull).
    It's close enough. http://play.golang.org/p/AeLOSRN1Qk

    Also, why re-implement io.ReadFull?
    Because I couldn't find it.

    --
    You received this message because you are subscribed to the Google Groups
    "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an
    email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
  • Peter Kleiweg at Jul 3, 2014 at 5:59 pm
    Now a writer as well

    Op donderdag 3 juli 2014 11:51:07 UTC+2 schreef Peter Kleiweg:
    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
  • Carlos Castillo at Jul 4, 2014 at 1:34 am
    It might be a good idea if the reader implements one of the io.*
    interfaces. My suggestion would be to implement io.ReaderAt, since ReadAt
    is essentially the same as your Get method, except that it fills a given
    buffer instead of creating a new one (fewer allocs!). Then someone can use
    an io.SectionReader to wrap it to get a "Read" method.
    On Thursday, July 3, 2014 10:59:11 AM UTC-7, Peter Kleiweg wrote:

    Now a writer as well

    Op donderdag 3 juli 2014 11:51:07 UTC+2 schreef Peter Kleiweg:
    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
  • Peter Kleiweg at Jul 4, 2014 at 9:39 am

    Op vrijdag 4 juli 2014 03:34:00 UTC+2 schreef Carlos Castillo:

    It might be a good idea if the reader implements one of the io.*
    interfaces. My suggestion would be to implement io.ReaderAt, since ReadAt
    is essentially the same as your Get method, except that it fills a given
    buffer instead of creating a new one (fewer allocs!). Then someone can use
    an io.SectionReader to wrap it to get a "Read" method.
    Even if I implement io.ReaderAt, I still have to allocate a buffer, because
    usually, I need a larger buffer than was requested. The reader has to start
    decoding at certain boundaries, and those usually don't coincide with the
    start of what the user is requesting.


    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
  • Carlos Castillo at Jul 4, 2014 at 9:50 am
    Ah, I see.

    I still think ReadAt is till a good idea for the reason I mentioned, and
    since there's no allocations to be saved (without a significant rewrite),
    it can easily be written w.r.t. Get().
    On Friday, July 4, 2014 2:39:32 AM UTC-7, Peter Kleiweg wrote:

    Op vrijdag 4 juli 2014 03:34:00 UTC+2 schreef Carlos Castillo:
    It might be a good idea if the reader implements one of the io.*
    interfaces. My suggestion would be to implement io.ReaderAt, since ReadAt
    is essentially the same as your Get method, except that it fills a given
    buffer instead of creating a new one (fewer allocs!). Then someone can use
    an io.SectionReader to wrap it to get a "Read" method.
    Even if I implement io.ReaderAt, I still have to allocate a buffer,
    because usually, I need a larger buffer than was requested. The reader has
    to start decoding at certain boundaries, and those usually don't coincide
    with the start of what the user is requesting.


    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
  • Peter Kleiweg at Jul 4, 2014 at 10:08 am

    Op vrijdag 4 juli 2014 11:50:45 UTC+2 schreef Carlos Castillo:

    Ah, I see.

    I still think ReadAt is till a good idea for the reason I mentioned, and
    since there's no allocations to be saved (without a significant rewrite),
    it can easily be written w.r.t. Get().
    So users can wrap it to get a Read method? You can use compress/gzip for
    that. A dictzip file is just a gzip file with some extra index information.


    On Friday, July 4, 2014 2:39:32 AM UTC-7, Peter Kleiweg wrote:

    Op vrijdag 4 juli 2014 03:34:00 UTC+2 schreef Carlos Castillo:
    It might be a good idea if the reader implements one of the io.*
    interfaces. My suggestion would be to implement io.ReaderAt, since ReadAt
    is essentially the same as your Get method, except that it fills a given
    buffer instead of creating a new one (fewer allocs!). Then someone can use
    an io.SectionReader to wrap it to get a "Read" method.
    Even if I implement io.ReaderAt, I still have to allocate a buffer,
    because usually, I need a larger buffer than was requested. The reader has
    to start decoding at certain boundaries, and those usually don't coincide
    with the start of what the user is requesting.


    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
  • Carlos Castillo at Jul 4, 2014 at 10:15 am
    But you can't Read an arbitrary subsection of a gzip file.

    When you make a io.SectionReader you specify which part of the file you
    want, via an offset and size, and you get back an object which you can Read
    from (starting at that offset).

    This would allow you to pipe part of a dictzip file to any method that
    wants a Reader. Right now, you have to read the whole chunk at once (via
    Get) and then wrap it with a bytes.Reader, with this method it could be
    streamed to the target method.

    On Fri, Jul 4, 2014 at 3:08 AM, Peter Kleiweg wrote:

    Op vrijdag 4 juli 2014 11:50:45 UTC+2 schreef Carlos Castillo:

    Ah, I see.
    I still think ReadAt is till a good idea for the reason I mentioned, and
    since there's no allocations to be saved (without a significant rewrite),
    it can easily be written w.r.t. Get().
    So users can wrap it to get a Read method? You can use compress/gzip for
    that. A dictzip file is just a gzip file with some extra index information.


    On Friday, July 4, 2014 2:39:32 AM UTC-7, Peter Kleiweg wrote:

    Op vrijdag 4 juli 2014 03:34:00 UTC+2 schreef Carlos Castillo:
    It might be a good idea if the reader implements one of the io.*
    interfaces. My suggestion would be to implement io.ReaderAt, since ReadAt
    is essentially the same as your Get method, except that it fills a given
    buffer instead of creating a new one (fewer allocs!). Then someone can use
    an io.SectionReader to wrap it to get a "Read" method.
    Even if I implement io.ReaderAt, I still have to allocate a buffer,
    because usually, I need a larger buffer than was requested. The reader has
    to start decoding at certain boundaries, and those usually don't coincide
    with the start of what the user is requesting.


    You received this message because you are subscribed to a topic in the
    Google Groups "golang-nuts" group.
    To unsubscribe from this topic, visit
    https://groups.google.com/d/topic/golang-nuts/IRm1I9UzYhA/unsubscribe.
    To unsubscribe from this group and all its topics, send an email to
    golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.


    --
    Carlos Castillo

    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
  • Peter Kleiweg at Jul 4, 2014 at 10:23 am
    I see. Will do.

    Op vrijdag 4 juli 2014 12:15:49 UTC+2 schreef Carlos Castillo:
    But you can't Read an arbitrary subsection of a gzip file.

    When you make a io.SectionReader you specify which part of the file you
    want, via an offset and size, and you get back an object which you can Read
    from (starting at that offset).

    This would allow you to pipe part of a dictzip file to any method that
    wants a Reader. Right now, you have to read the whole chunk at once (via
    Get) and then wrap it with a bytes.Reader, with this method it could be
    streamed to the target method.


    On Fri, Jul 4, 2014 at 3:08 AM, Peter Kleiweg <pkle...@xs4all.nl
    <javascript:>> wrote:
    Op vrijdag 4 juli 2014 11:50:45 UTC+2 schreef Carlos Castillo:

    Ah, I see.
    I still think ReadAt is till a good idea for the reason I mentioned, and
    since there's no allocations to be saved (without a significant rewrite),
    it can easily be written w.r.t. Get().
    So users can wrap it to get a Read method? You can use compress/gzip for
    that. A dictzip file is just a gzip file with some extra index information.


    On Friday, July 4, 2014 2:39:32 AM UTC-7, Peter Kleiweg wrote:

    Op vrijdag 4 juli 2014 03:34:00 UTC+2 schreef Carlos Castillo:
    It might be a good idea if the reader implements one of the io.*
    interfaces. My suggestion would be to implement io.ReaderAt, since ReadAt
    is essentially the same as your Get method, except that it fills a given
    buffer instead of creating a new one (fewer allocs!). Then someone can use
    an io.SectionReader to wrap it to get a "Read" method.
    Even if I implement io.ReaderAt, I still have to allocate a buffer,
    because usually, I need a larger buffer than was requested. The reader has
    to start decoding at certain boundaries, and those usually don't coincide
    with the start of what the user is requesting.


    You received this message because you are subscribed to a topic in the
    Google Groups "golang-nuts" group.
    To unsubscribe from this topic, visit
    https://groups.google.com/d/topic/golang-nuts/IRm1I9UzYhA/unsubscribe.
    To unsubscribe from this group and all its topics, send an email to
    golang-nuts...@googlegroups.com <javascript:>.
    For more options, visit https://groups.google.com/d/optout.


    --
    Carlos Castillo
    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupgolang-nuts @
categoriesgo
postedJul 3, '14 at 9:51a
activeJul 4, '14 at 8:47p
posts19
users6
websitegolang.org

People

Translate

site design / logo © 2022 Grokbase