[go-nuts] go binary encoding so slowly?

Can you post a usable benchmark? I don't find it slowing me down
excessively in gomemcached doing binary protocol encoding (first
real-world use case I could think of in my groggy state on the train):

BenchmarkEncodingRequest 10000000 243 ns/op 163.97 MB/s
BenchmarkEncodingRequest0CAS 10000000 240 ns/op 166.23 MB/s
BenchmarkEncodingRequest1Extra 10000000 234 ns/op 174.65 MB/s
BenchmarkEncodingResponse 10000000 234 ns/op 170.27 MB/s
BenchmarkEncodingResponseLarge 500000 6280 ns/op 3917.74 MB/s

I just write a simple go1 command project to test. My full code is here.

My test environment:
Linux home 3.2.0-29-generic #46-Ubuntu SMP Fri Jul 27 17:03:23 UTC 2012
x86_64 x86_64 x86_64 GNU/Linux


// binarywrite project main.go
package main

import (
"bufio"
"encoding/binary"
"flag"
"fmt"
"os"
"time"
"unsafe"
"runtime"
"log"
)

// Checkf call os.Exit(1) if err != nil
func Checkf(err error, format string, arg ...interface{}) {
if err != nil {
_, file, line, _ := runtime.Caller(1)
log.Fatalf("%s:%v:%s, %s",
file, line, err, fmt.Sprintf(format, arg))
}
}

type Action struct {
U1 uint64
U2 uint64
U3 uint64
}

func main() {
var (
filePath string
bufSize int
fileSize uint64
)
flag.StringVar(&filePath, "file", "", "file path")
flag.IntVar(&bufSize, "buffer_size", 4096, "buffer size")
flag.Uint64Var(&fileSize, "size", 1024*1024, "file size")

flag.Usage = func() {
fmt.Printf("Usage: %s [Options]\n", os.Args[0])
flag.PrintDefaults()
}
flag.Parse()

if filePath == "" {
flag.Usage()
return
}

f, err := os.Create(filePath)
Checkf(err, "create file %s failed", filePath)
defer f.Close()

var totalSize uint64
w := bufio.NewWriter(f)
tStart := time.Now()
var action Action
for totalSize < fileSize {
Checkf(binary.Write(w, binary.LittleEndian, &action), "write %v", totalSize)
totalSize += uint64(unsafe.Sizeof(action))
}
tStop := time.Now()

tDuration := tStop.Sub(tStart)
fmt.Printf("Write over, totalSize: %v, time elapsed: %v(s), speed: %v MB/s
\n",
totalSize,
tDuration.Seconds(),
float64(totalSize/1048576)/tDuration.Seconds())
}

You really should use testing.B for this. Your implementation will
not be very accurate.

If you implement io.WriterTo, you can test a couple implementations,
including this one below (with reflection) and just manually doing the
encoding into a buffer and sending it out.

i rewrite my code here and run it, the output is :

Write over, totalSize: 104857608, time elapsed: 7.297325(s), speed:
13.703651680581583 MB/s

i use ioutil.Discard to avoid the write influence

///////////////////////////////////////////////////////////////////////////////////////////////////////////
package main

import (
"encoding/binary"
"fmt"
"time"
"io/ioutil"
)

type Action struct {
U1 uint64
U2 uint64
U3 uint64
}

func main() {
tStart := time.Now()
var action Action
fileSize := uint64(1024 * 1024 * 100) // 100M
var totalSize uint64
for totalSize < fileSize {
binary.Write(ioutil.Discard, binary.LittleEndian, &action)
totalSize += uint64(24)
}
tStop := time.Now()

tDuration := tStop.Sub(tStart)
fmt.Printf("Write over, totalSize: %v, time elapsed: %v(s), speed: %v MB/s
\n",
totalSize,
tDuration.Seconds(),
float64(totalSize/1048576)/tDuration.Seconds())
}

While I question the accuracy of your test less, you really don't need
to reinvent testing.B. It's quite good, and will likely be a huge help
when you do the next optimization step. Read the docs for the testing
package for testing.B. Then contain your encoding code in a function.
Then benchmark that function. Then write a better version of function
manipulating a []byte directly. See which trade-off you'd like to
make.

Here ya' go. Now you can use the benchmarking, and more importantly,
profiling tools, to improve the performance of this package.

package write

import (
"encoding/binary"
"io/ioutil"
"testing"
)

type Action struct {
U1 uint64
U2 uint64
U3 uint64
}

const fileSize = 1 << 16

func BenchmarkBinaryWrite64k(b *testing.B) {
var action Action
b.SetBytes(fileSize)
for i := 0; i < b.N; i++ {
for totalSize := 0; totalSize < fileSize; {
binary.Write(ioutil.Discard, binary.LittleEndian, &action)
totalSize += 24
}
}
}

% go test -bench=.
testing: warning: no tests to run
PASS
BenchmarkBinaryWrite64k 100 19805230 ns/op 3.31 MB/s
ok write 2.020s

Ok, i use binary.LittleEndian to get the fastest write speed, full code here

Write over, totalSize: 104857600, time elapsed: 0.33172300000000005(s),
speed: 301.4563355570762 MB/s

///////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////
package main

import (
"encoding/binary"
"fmt"
"time"
"io/ioutil"
)

func main() {
tStart := time.Now()
fileSize := uint64(1024 * 1024 * 100) // 100M
var totalSize uint64
b := make([]byte, 8)
for totalSize < fileSize {
binary.LittleEndian.PutUint64(b, uint64(totalSize))
ioutil.Discard.Write(b)
totalSize += uint64(8)
}
tStop := time.Now()

tDuration := tStop.Sub(tStart)
fmt.Printf("Write over, totalSize: %v, time elapsed: %v(s), speed: %v MB/s
\n",
totalSize,
tDuration.Seconds(),
float64(totalSize/1048576)/tDuration.Seconds())
}

--