A couple of responses below, but note that for
github.com/golang/snappy, I have just updated the Go encoder to have
byte-for-byte identical output to the C++ encoder. There were a few
smaller commits recently to set this up, and further optimization work
remains, but the punchline commit at
shows both faster throughput (on GOARCH=amd64) and smaller output.

On Tue, Mar 29, 2016 at 1:54 AM, wrote:
They may offer good progression (again, seeing both the before and the
after for levels 1-3 on the graphs would help inform the discussion),
but in practice, do people use anything other than BestSpeed,
BestCompression, NoCompression or DefaultCompression? I don't know the
answer, but if not, it might not be worth having three copy/pastes of
the new algorithm if nobody uses levels 2 or 3.
You do have a good point. I cannot be the judge of whether it should be
included. I have tried to factor out the common code in the CL now - at
least the things that could be done without sacrificing performance in a
measurable way.

If it is still "too much", I think I would go for adding "encodeL2" as
BestSpeed. It is often very close to "encodeL1" in terms of speed (within
5-10%), but offers 1% better compression. If we should only replace level 1,
I would choose that. Maybe other people have input on whether it should be a
choice, or we should just implement 1 level.
I'd prefer only replacing level 1, and leaving 2 and 3 alone,
especially since, AFAICT,
still has a lot of copy/paste between encodeL1, encodeL2 and encodeL3.

In any case, I suspect that being based on the C++ encoder algorithm
(as mentioned above) will help.

On Thu, Mar 31, 2016 at 11:08 PM, Nigel Tao wrote:
I asked the snappy-compression mailing list (where the C++ snappy code
is discussed) for their thoughts. I'll crunch some C++ numbers
tomorrow (it's getting late here in Sydney):

FWIW, my reading of that discussion thread is that Steinar Gunderson
(one of the C++ snappy authors) reckons that while it's much better
throughput on binary data and slightly worse on textual data, the "1 +
((s - lit) >> 5)" skipping algorithm is a net negative for the C++
implementation, so I'm less inclined to adopt it for the Go
implementation, whether in golang/snappy or in compress/flate.

You received this message because you are subscribed to the Google Groups "golang-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-dev+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Search Discussions

Discussion Posts


Follow ups

Related Discussions

Discussion Navigation
viewthread | post
posts ‹ prev | 16 of 20 | next ›
Discussion Overview
groupgolang-dev @
postedMar 22, '16 at 10:53a
activeApr 10, '16 at 2:06p



site design / logo © 2021 Grokbase