FAQ
Hello gophers,


I am please to announce the LZ4 package (github.com/pierrec/lz4)
implementing the LZ4 compression/decompression algorithm in 100% Go.

The API is similar to the ones in the standard compress packages but it is
such that concurrency is supported.

Any feedback most welcomed,


Pierre

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Search Discussions

  • Aliaksandr Valialkin at Mar 14, 2015 at 8:51 pm
    Hi Pierre,

    I tried substituting compress/flate with your library - see
    https://github.com/valyala/gorpc/tree/pierrec-lz4 .

    Good news: all tests passed.

    Bad news: benchmarks with enabled compression became much slower.

    compress/flate:

    GOMAXPROCS=4 go test -test.bench="RealApp1000"
    BenchmarkRealApp1000Workers-4 50000 32988 ns/op
    BenchmarkRealApp10000Workers-4 30000 36921 ns/op
    BenchmarkRealApp100000Workers-4 50000 40005 ns/op

    pierrec/lz4:

    GOMAXPROCS=4 go test -test.bench="RealApp1000"
    PASS
    BenchmarkRealApp1000Workers-4 10000 207998 ns/op
    BenchmarkRealApp10000Workers-4 5000 223354 ns/op
    BenchmarkRealApp100000Workers-4 10000 181333 ns/op

    Can you look into these performance problems? As I understand, optimized
    lz4 implementation should be much faster comparing to compress/flate .
    On Saturday, March 14, 2015 at 7:39:20 PM UTC+2, pierre...@gmail.com wrote:

    Hello gophers,


    I am please to announce the LZ4 package (github.com/pierrec/lz4)
    implementing the LZ4 compression/decompression algorithm in 100% Go.

    The API is similar to the ones in the standard compress packages but it is
    such that concurrency is supported.

    Any feedback most welcomed,


    Pierre
    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
  • Pierre Curto at Mar 15, 2015 at 8:34 am
    Hi Aliaksandr,


    That's bad results :(.

    I will look into it but I am running into a nil pointer dereference when
    trying to run tests:

    panic: runtime error: invalid memory address or nil pointer dereference
    [signal 0xb code=0x1 addr=0x1 pc=0x80cb35c]

    goroutine 22 [running]:
    sync/atomic.AddUint64(0x18601014, 0x1, 0x0, 0x1, 0x83bba20)
    /usr/local/go/src/sync/atomic/asm_386.s:118 +0xc
    github.com/valyala/gorpc.serverHandler(0x18600fa0, 0x18648cc0)
    /home/pierre/sandbox/go/src/github.com/valyala/gorpc/server.go:178 +0x2ae
    created by github.com/valyala/gorpc.(*Server).Start
    /home/pierre/sandbox/go/src/github.com/valyala/gorpc/server.go:136 +0x404

    goroutine 1 [chan receive]:
    testing.RunTests(0x82fbb18, 0x83ba0a0, 0x19, 0x19, 0x18634301)
    /usr/local/go/src/testing/testing.go:556 +0x89f
    testing.(*M).Run(0x18634000, 0x83c19e0)
    /usr/local/go/src/testing/testing.go:485 +0x5e
    main.main()
    github.com/valyala/gorpc/_test/_testmain.go:202 +0x177

    goroutine 21 [sleep]:
    github.com/valyala/gorpc.TestClientStartStop(0x18642180)
    /home/pierre/sandbox/go/src/github.com/valyala/gorpc/rpc_test.go:69 +0x1fe
    testing.tRunner(0x18642180, 0x83ba0b8)
    /usr/local/go/src/testing/testing.go:447 +0xb1
    created by testing.RunTests
    /usr/local/go/src/testing/testing.go:555 +0x863

    goroutine 12 [sleep]:
    github.com/valyala/gorpc.func·006()
    /home/pierre/sandbox/go/src/github.com/valyala/gorpc/server.go:168 +0x1d0
    created by github.com/valyala/gorpc.serverHandler
    /home/pierre/sandbox/go/src/github.com/valyala/gorpc/server.go:171 +0x1b8

    goroutine 9 [sleep]:
    github.com/valyala/gorpc.func·006()
    /home/pierre/sandbox/go/src/github.com/valyala/gorpc/server.go:168 +0x1d0
    created by github.com/valyala/gorpc.serverHandler
    /home/pierre/sandbox/go/src/github.com/valyala/gorpc/server.go:171 +0x1b8

    goroutine 14 [sleep]:
    github.com/valyala/gorpc.func·006()
    /home/pierre/sandbox/go/src/github.com/valyala/gorpc/server.go:168 +0x1d0
    created by github.com/valyala/gorpc.serverHandler
    /home/pierre/sandbox/go/src/github.com/valyala/gorpc/server.go:171 +0x1b8

    goroutine 16 [sleep]:
    github.com/valyala/gorpc.func·006()
    /home/pierre/sandbox/go/src/github.com/valyala/gorpc/server.go:168 +0x1d0
    created by github.com/valyala/gorpc.serverHandler
    /home/pierre/sandbox/go/src/github.com/valyala/gorpc/server.go:171 +0x1b8

    goroutine 18 [sleep]:
    github.com/valyala/gorpc.func·006()
    /home/pierre/sandbox/go/src/github.com/valyala/gorpc/server.go:168 +0x1d0
    created by github.com/valyala/gorpc.serverHandler
    /home/pierre/sandbox/go/src/github.com/valyala/gorpc/server.go:171 +0x1b8

    goroutine 20 [sleep]:
    github.com/valyala/gorpc.func·006()
    /home/pierre/sandbox/go/src/github.com/valyala/gorpc/server.go:168 +0x1d0
    created by github.com/valyala/gorpc.serverHandler
    /home/pierre/sandbox/go/src/github.com/valyala/gorpc/server.go:171 +0x1b8

    goroutine 23 [select]:
    github.com/valyala/gorpc.clientHandler(0x18650000)
    /home/pierre/sandbox/go/src/github.com/valyala/gorpc/client.go:321 +0x2b8
    created by github.com/valyala/gorpc.(*Client).Start
    /home/pierre/sandbox/go/src/github.com/valyala/gorpc/client.go:136 +0x1c7

    goroutine 24 [select]:
    github.com/valyala/gorpc.clientHandler(0x18650000)
    /home/pierre/sandbox/go/src/github.com/valyala/gorpc/client.go:321 +0x2b8
    created by github.com/valyala/gorpc.(*Client).Start
    /home/pierre/sandbox/go/src/github.com/valyala/gorpc/client.go:136 +0x1c7

    goroutine 25 [select]:
    github.com/valyala/gorpc.clientHandler(0x18650000)
    /home/pierre/sandbox/go/src/github.com/valyala/gorpc/client.go:321 +0x2b8
    created by github.com/valyala/gorpc.(*Client).Start
    /home/pierre/sandbox/go/src/github.com/valyala/gorpc/client.go:136 +0x1c7

    goroutine 27 [IO wait]:
    net.(*pollDesc).Wait(0x18648278, 0x77, 0x0, 0x0)
    /usr/local/go/src/net/fd_poll_runtime.go:84 +0x42
    net.(*pollDesc).WaitWrite(0x18648278, 0x0, 0x0)
    /usr/local/go/src/net/fd_poll_runtime.go:93 +0x40
    net.(*netFD).connect(0x18648240, 0x0, 0x0, 0xb7517090, 0x18637400,
    0xcc973afb, 0xe, 0x594127f, 0x83bba20, 0x0, ...)
    /usr/local/go/src/net/fd_unix.go:114 +0x21c
    net.(*netFD).dial(0x18648240, 0xb7516020, 0x0, 0xb7516020, 0x18636f40,
    0xcc973afb, 0xe, 0x594127f, 0x83bba20, 0x0, ...)
    /usr/local/go/src/net/sock_posix.go:139 +0x2c6
    net.socket(0x8285980, 0x3, 0x2, 0x1, 0x0, 0x18636f00, 0xb7516020, 0x0,
    0xb7516020, 0x18636f40, ...)
    /usr/local/go/src/net/sock_posix.go:91 +0x346
    net.internetSocket(0x8285980, 0x3, 0xb7516020, 0x0, 0xb7516020, 0x18636f40,
    0xcc973afb, 0xe, 0x594127f, 0x83bba20, ...)
    /usr/local/go/src/net/ipsock_posix.go:137 +0xed
    net.dialTCP(0x8285980, 0x3, 0x0, 0x18636f40, 0xcc973afb, 0xe, 0x594127f,
    0x83bba20, 0x8063d1d, 0x0, ...)
    /usr/local/go/src/net/tcpsock_posix.go:156 +0xfc
    net.dialSingle(0x8285980, 0x3, 0x1862e534, 0x6, 0x0, 0x0, 0xb7515fe0,
    0x18636f40, 0xcc973afb, 0xe, ...)
    /usr/local/go/src/net/dial.go:235 +0x35a
    net.func·016(0xcc973afb, 0xe, 0x594127f, 0x83bba20, 0x0, 0x0, 0x0, 0x0)
    /usr/local/go/src/net/dial.go:163 +0xd0
    net.dial(0x8285980, 0x3, 0xb7515fe0, 0x18636f40, 0x18690f50, 0xcc973afb,
    0xe, 0x594127f, 0x83bba20, 0x0, ...)
    /usr/local/go/src/net/fd_unix.go:40 +0x63
    net.(*Dialer).Dial(0x83b9880, 0x8285980, 0x3, 0x1862e534, 0x6, 0x0, 0x0,
    0x0, 0x0)
    /usr/local/go/src/net/dial.go:170 +0x3d8
    github.com/valyala/gorpc.defaultDial(0x1862e534, 0x6, 0x0, 0x0, 0x0, 0x0)
    /home/pierre/sandbox/go/src/github.com/valyala/gorpc/transport.go:51 +0x67
    github.com/valyala/gorpc.func·003()
    /home/pierre/sandbox/go/src/github.com/valyala/gorpc/client.go:314 +0x52
    created by github.com/valyala/gorpc.clientHandler
    /home/pierre/sandbox/go/src/github.com/valyala/gorpc/client.go:319 +0x17b

    goroutine 28 [IO wait]:
    net.(*pollDesc).Wait(0x18648378, 0x77, 0x0, 0x0)
    /usr/local/go/src/net/fd_poll_runtime.go:84 +0x42
    net.(*pollDesc).WaitWrite(0x18648378, 0x0, 0x0)
    /usr/local/go/src/net/fd_poll_runtime.go:93 +0x40
    net.(*netFD).connect(0x18648340, 0x0, 0x0, 0xb7517090, 0x18637440,
    0xcc973afb, 0xe, 0x598d446, 0x83bba20, 0x0, ...)
    /usr/local/go/src/net/fd_unix.go:114 +0x21c
    net.(*netFD).dial(0x18648340, 0xb7516020, 0x0, 0xb7516020, 0x18637420,
    0xcc973afb, 0xe, 0x598d446, 0x83bba20, 0x0, ...)
    /usr/local/go/src/net/sock_posix.go:139 +0x2c6
    net.socket(0x8285980, 0x3, 0x2, 0x1, 0x0, 0x18637400, 0xb7516020, 0x0,
    0xb7516020, 0x18637420, ...)
    /usr/local/go/src/net/sock_posix.go:91 +0x346
    net.internetSocket(0x8285980, 0x3, 0xb7516020, 0x0, 0xb7516020, 0x18637420,
    0xcc973afb, 0xe, 0x598d446, 0x83bba20, ...)
    /usr/local/go/src/net/ipsock_posix.go:137 +0xed
    net.dialTCP(0x8285980, 0x3, 0x0, 0x18637420, 0xcc973afb, 0xe, 0x598d446,
    0x83bba20, 0x8063d1d, 0x0, ...)
    /usr/local/go/src/net/tcpsock_posix.go:156 +0xfc
    net.dialSingle(0x8285980, 0x3, 0x1862e534, 0x6, 0x0, 0x0, 0xb7515fe0,
    0x18637420, 0xcc973afb, 0xe, ...)
    /usr/local/go/src/net/dial.go:235 +0x35a
    net.func·016(0xcc973afb, 0xe, 0x598d446, 0x83bba20, 0x0, 0x0, 0x0, 0x0)
    /usr/local/go/src/net/dial.go:163 +0xd0
    net.dial(0x8285980, 0x3, 0xb7515fe0, 0x18637420, 0x18691f50, 0xcc973afb,
    0xe, 0x598d446, 0x83bba20, 0x0, ...)
    /usr/local/go/src/net/fd_unix.go:40 +0x63
    net.(*Dialer).Dial(0x83b9880, 0x8285980, 0x3, 0x1862e534, 0x6, 0x0, 0x0,
    0x0, 0x0)
    /usr/local/go/src/net/dial.go:170 +0x3d8
    github.com/valyala/gorpc.defaultDial(0x1862e534, 0x6, 0x0, 0x0, 0x0, 0x0)
    /home/pierre/sandbox/go/src/github.com/valyala/gorpc/transport.go:51 +0x67
    github.com/valyala/gorpc.func·003()
    /home/pierre/sandbox/go/src/github.com/valyala/gorpc/client.go:314 +0x52
    created by github.com/valyala/gorpc.clientHandler
    /home/pierre/sandbox/go/src/github.com/valyala/gorpc/client.go:319 +0x17b

    goroutine 29 [runnable]:
    net.(*pollDesc).Wait(0x18648438, 0x77, 0x0, 0x0)
    /usr/local/go/src/net/fd_poll_runtime.go:84 +0x42
    net.(*pollDesc).WaitWrite(0x18648438, 0x0, 0x0)
    /usr/local/go/src/net/fd_poll_runtime.go:93 +0x40
    net.(*netFD).connect(0x18648400, 0x0, 0x0, 0xb7517090, 0x18637480,
    0xcc973afb, 0xe, 0x59ac8ff, 0x83bba20, 0x0, ...)
    /usr/local/go/src/net/fd_unix.go:114 +0x21c
    net.(*netFD).dial(0x18648400, 0xb7516020, 0x0, 0xb7516020, 0x18637460,
    0xcc973afb, 0xe, 0x59ac8ff, 0x83bba20, 0x0, ...)
    /usr/local/go/src/net/sock_posix.go:139 +0x2c6
    net.socket(0x8285980, 0x3, 0x2, 0x1, 0x0, 0x18637400, 0xb7516020, 0x0,
    0xb7516020, 0x18637460, ...)
    /usr/local/go/src/net/sock_posix.go:91 +0x346
    net.internetSocket(0x8285980, 0x3, 0xb7516020, 0x0, 0xb7516020, 0x18637460,
    0xcc973afb, 0xe, 0x59ac8ff, 0x83bba20, ...)
    /usr/local/go/src/net/ipsock_posix.go:137 +0xed
    net.dialTCP(0x8285980, 0x3, 0x0, 0x18637460, 0xcc973afb, 0xe, 0x59ac8ff,
    0x83bba20, 0x8063d1d, 0x0, ...)
    /usr/local/go/src/net/tcpsock_posix.go:156 +0xfc
    net.dialSingle(0x8285980, 0x3, 0x1862e534, 0x6, 0x0, 0x0, 0xb7515fe0,
    0x18637460, 0xcc973afb, 0xe, ...)
    /usr/local/go/src/net/dial.go:235 +0x35a
    net.func·016(0xcc973afb, 0xe, 0x59ac8ff, 0x83bba20, 0x0, 0x0, 0x0, 0x0)
    /usr/local/go/src/net/dial.go:163 +0xd0
    net.dial(0x8285980, 0x3, 0xb7515fe0, 0x18637460, 0x18692f50, 0xcc973afb,
    0xe, 0x59ac8ff, 0x83bba20, 0x0, ...)
    /usr/local/go/src/net/fd_unix.go:40 +0x63
    net.(*Dialer).Dial(0x83b9880, 0x8285980, 0x3, 0x1862e534, 0x6, 0x0, 0x0,
    0x0, 0x0)
    /usr/local/go/src/net/dial.go:170 +0x3d8
    github.com/valyala/gorpc.defaultDial(0x1862e534, 0x6, 0x0, 0x0, 0x0, 0x0)
    /home/pierre/sandbox/go/src/github.com/valyala/gorpc/transport.go:51 +0x67
    github.com/valyala/gorpc.func·003()
    /home/pierre/sandbox/go/src/github.com/valyala/gorpc/client.go:314 +0x52
    created by github.com/valyala/gorpc.clientHandler
    /home/pierre/sandbox/go/src/github.com/valyala/gorpc/client.go:319 +0x17b
    exit status 2
    FAIL github.com/valyala/gorpc 0.132s



    Le samedi 14 mars 2015 21:51:32 UTC+1, Aliaksandr Valialkin a écrit :
    Hi Pierre,

    I tried substituting compress/flate with your library - see
    https://github.com/valyala/gorpc/tree/pierrec-lz4 .

    Good news: all tests passed.

    Bad news: benchmarks with enabled compression became much slower.

    compress/flate:

    GOMAXPROCS=4 go test -test.bench="RealApp1000"
    BenchmarkRealApp1000Workers-4 50000 32988 ns/op
    BenchmarkRealApp10000Workers-4 30000 36921 ns/op
    BenchmarkRealApp100000Workers-4 50000 40005 ns/op

    pierrec/lz4:

    GOMAXPROCS=4 go test -test.bench="RealApp1000"
    PASS
    BenchmarkRealApp1000Workers-4 10000 207998 ns/op
    BenchmarkRealApp10000Workers-4 5000 223354 ns/op
    BenchmarkRealApp100000Workers-4 10000 181333 ns/op

    Can you look into these performance problems? As I understand, optimized
    lz4 implementation should be much faster comparing to compress/flate .

    On Saturday, March 14, 2015 at 7:39:20 PM UTC+2, pierre...@gmail.com
    wrote:
    Hello gophers,


    I am please to announce the LZ4 package (github.com/pierrec/lz4)
    implementing the LZ4 compression/decompression algorithm in 100% Go.

    The API is similar to the ones in the standard compress packages but it
    is such that concurrency is supported.

    Any feedback most welcomed,


    Pierre
    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
  • Tamás Gulácsi at Mar 15, 2015 at 12:30 pm
    On 386, atomic nerds special care for alignment in structs!

    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
  • Aliaksandr Valialkin at Mar 16, 2015 at 12:52 am

    On Sun, Mar 15, 2015 at 2:30 PM, Tamás Gulácsi wrote:

    On 386, atomic nerds special care for alignment in structs!
    Shouldn't golang automatically align struct members in order to prevent
    unaligned memory access? As for 'atomic nerds' I just use
    atomic.AddUint64() for simple counters, which may be incremented from
    arbitrary number of concurrently running goroutines.

    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
  • Caleb Spare at Mar 16, 2015 at 12:57 am

    Shouldn't golang automatically align struct members in order to prevent
    unaligned memory access? As for 'atomic nerds' I just use
    atomic.AddUint64() for simple counters, which may be incremented from
    arbitrary number of concurrently running goroutines.
    ​See the Bugs section at the bottom of http://golang.org/pkg/sync/atomic/:

    "​
    On x86-32, the 64-bit functions use instructions unavailable before the
    Pentium MMX. On non-Linux ARM, the 64-bit functions use instructions
    unavailable before the ARMv6k core. On both ARM and x86-32, it is the
    caller's responsibility to arrange for 64-bit alignment of 64-bit words
    accessed atomically. The first word in a global variable or in an allocated
    struct or slice can be relied upon to be 64-bit aligned.
    ​"​

    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
  • Minux at Mar 16, 2015 at 1:51 am

    On Sun, Mar 15, 2015 at 8:52 PM, Aliaksandr Valialkin wrote:
    On Sun, Mar 15, 2015 at 2:30 PM, Tamás Gulácsi wrote:

    On 386, atomic nerds special care for alignment in structs!
    Shouldn't golang automatically align struct members in order to prevent
    unaligned memory access? As for 'atomic nerds' I just use
    atomic.AddUint64() for simple counters, which may be incremented from
    arbitrary number of concurrently running goroutines.
    The problem is that on 32-bit architectures, the alignment requirement for
    uint64 is just 4-byte,
    however, 64-bit atomic instructions require 8-byte alignment.

    Unfortunately, there is no easy way to get 8-byte alignment except placing
    the fields at the
    beginning of a struct or manually adjust the position on a 32-bit platform,
    which is very tedious
    and error-prone.

    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupgolang-nuts @
categoriesgo
postedMar 14, '15 at 5:39p
activeMar 16, '15 at 1:51a
posts7
users5
websitegolang.org

People

Translate

site design / logo © 2021 Grokbase