FAQ
Hi All.

I need to send data chunks over a tcp socket, the size of the chunks are
variable and I want to compress this chunks in stream mode to get the best
compression ratio.
Zlib/deflate algorithm supports streaming compression but I detected this
issue:
In this example: http://play.golang.org/p/Fz8BilOYET the compression stream
using compress/flate, fails after iteration 32 ¿Why?

Tanks.





--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Search Discussions

  • Jasdel at Jun 30, 2014 at 8:31 pm
    From looking at your code, it looks like the problem is because you're read
    from inflate expects the full data. Whereas Read can actually return early,
    and you need to keep reading until there are no more data.

    On Monday, June 30, 2014 12:28:43 PM UTC-7, Pepe Aracil wrote:

    Hi All.

    I need to send data chunks over a tcp socket, the size of the chunks are
    variable and I want to compress this chunks in stream mode to get the best
    compression ratio.
    Zlib/deflate algorithm supports streaming compression but I detected this
    issue:
    In this example: http://play.golang.org/p/Fz8BilOYET the compression
    stream using compress/flate, fails after iteration 32 ¿Why?

    Tanks.



    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
  • Pepe Aracil at Jun 30, 2014 at 8:43 pm
    I think deflate.Flush() MUST send all remaining data and it works well the
    first 32 iterations. I suspect the cause is flate dictionary size is 32K
    and the blocks are 1000 byte length.

    This link explains flush methods and how must work:
    http://www.bolet.org/~pornin/deflate-flush.html



    El lunes, 30 de junio de 2014 22:31:32 UTC+2, jasdel escribió:
    From looking at your code, it looks like the problem is because you're
    read from inflate expects the full data. Whereas Read can actually return
    early, and you need to keep reading until there are no more data.

    On Monday, June 30, 2014 12:28:43 PM UTC-7, Pepe Aracil wrote:

    Hi All.

    I need to send data chunks over a tcp socket, the size of the chunks are
    variable and I want to compress this chunks in stream mode to get the best
    compression ratio.
    Zlib/deflate algorithm supports streaming compression but I detected this
    issue:
    In this example: http://play.golang.org/p/Fz8BilOYET the compression
    stream using compress/flate, fails after iteration 32 ¿Why?

    Tanks.



    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
  • Alex Skinner at Jul 1, 2014 at 2:47 am
    But it doesn't seem to be the Flush() call that's messing up but the read.
      The reader has a 32768B history buffer that is read from...hence why your
    33rd read only has only 768 bytes. If you call read again it fills up
    fine. Honestly, the reader source is too hard for my tired brain to
    follow tonight, but not sure what the 'correct' behavior is on the read
    call in this situation.

    Thanks,
    Alex
    On Monday, June 30, 2014 4:43:49 PM UTC-4, Pepe Aracil wrote:

    I think deflate.Flush() MUST send all remaining data and it works well the
    first 32 iterations. I suspect the cause is flate dictionary size is 32K
    and the blocks are 1000 byte length.

    This link explains flush methods and how must work:
    http://www.bolet.org/~pornin/deflate-flush.html



    El lunes, 30 de junio de 2014 22:31:32 UTC+2, jasdel escribió:
    From looking at your code, it looks like the problem is because you're
    read from inflate expects the full data. Whereas Read can actually return
    early, and you need to keep reading until there are no more data.

    On Monday, June 30, 2014 12:28:43 PM UTC-7, Pepe Aracil wrote:

    Hi All.

    I need to send data chunks over a tcp socket, the size of the chunks are
    variable and I want to compress this chunks in stream mode to get the best
    compression ratio.
    Zlib/deflate algorithm supports streaming compression but I detected
    this issue:
    In this example: http://play.golang.org/p/Fz8BilOYET the compression
    stream using compress/flate, fails after iteration 32 ¿Why?

    Tanks.



    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
  • Nigel Tao at Jul 1, 2014 at 5:34 am

    On Tue, Jul 1, 2014 at 5:28 AM, Pepe Aracil wrote:
    In this example: http://play.golang.org/p/Fz8BilOYET the compression stream
    using compress/flate, fails after iteration 32 ¿Why?
    A flate reader is an io.Reader, and http://golang.org/pkg/io/#Reader
    says "Read reads up to len(p) bytes into p" and note the "up to".
    Also, "If some data is available but not len(p) bytes, Read
    conventionally returns what is available instead of waiting for more."

    If you need a full read, then call io.ReadFull.
    http://golang.org/pkg/io/#ReadFull says that "ReadFull reads exactly
    len(buf) bytes".

    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
  • Pepe Aracil at Jul 1, 2014 at 3:42 pm
    With ReadFull works Ok.

    Thanks Nigel.



    El martes, 1 de julio de 2014 07:34:25 UTC+2, Nigel Tao escribió:
    On Tue, Jul 1, 2014 at 5:28 AM, Pepe Aracil <pepe.ara...@gmail.com
    <javascript:>> wrote:
    In this example: http://play.golang.org/p/Fz8BilOYET the compression stream
    using compress/flate, fails after iteration 32 ¿Why?
    A flate reader is an io.Reader, and http://golang.org/pkg/io/#Reader
    says "Read reads up to len(p) bytes into p" and note the "up to".
    Also, "If some data is available but not len(p) bytes, Read
    conventionally returns what is available instead of waiting for more."

    If you need a full read, then call io.ReadFull.
    http://golang.org/pkg/io/#ReadFull says that "ReadFull reads exactly
    len(buf) bytes".
    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupgolang-nuts @
categoriesgo
postedJun 30, '14 at 7:28p
activeJul 1, '14 at 3:42p
posts6
users4
websitegolang.org

People

Translate

site design / logo © 2022 Grokbase