FAQ
i use encoding/binary package for testing the binary write/read speed, but
it reach 12MB/s at most.
What is the fastest way to read/write binary file using go?

f, _ := os.Create("test")
defer f.Close()

w := bufio.NewWriter(f)
var action Action
for {
binary.Write(w, binary.LittleEndian, &action)
}

--

Search Discussions

  • Dustin Sallings at Nov 7, 2012 at 5:34 pm

    go-fish writes:

    i use encoding/binary package for testing the binary write/read speed,
    but it reach 12MB/s at most.
    What is the fastest way to read/write binary file using go?

    f, _ := os.Create("test")
    defer f.Close()

    w := bufio.NewWriter(f)
    var action Action
    for  {
    binary.Write(w, binary.LittleEndian, &action)
    }
    Can you post a usable benchmark? I don't find it slowing me down
    excessively in gomemcached doing binary protocol encoding (first
    real-world use case I could think of in my groggy state on the train):

    BenchmarkEncodingRequest 10000000 243 ns/op 163.97 MB/s
    BenchmarkEncodingRequest0CAS 10000000 240 ns/op 166.23 MB/s
    BenchmarkEncodingRequest1Extra 10000000 234 ns/op 174.65 MB/s
    BenchmarkEncodingResponse 10000000 234 ns/op 170.27 MB/s
    BenchmarkEncodingResponseLarge 500000 6280 ns/op 3917.74 MB/s

    --
    dustin

    --
  • Zhang bin at Nov 8, 2012 at 12:57 am
    I just write a simple go1 command project to test. My full code is here.

    My test environment:
    Linux home 3.2.0-29-generic #46-Ubuntu SMP Fri Jul 27 17:03:23 UTC 2012
    x86_64 x86_64 x86_64 GNU/Linux


    // binarywrite project main.go
    package main

    import (
    "bufio"
    "encoding/binary"
    "flag"
    "fmt"
    "os"
    "time"
    "unsafe"
    "runtime"
    "log"
    )

    // Checkf call os.Exit(1) if err != nil
    func Checkf(err error, format string, arg ...interface{}) {
    if err != nil {
    _, file, line, _ := runtime.Caller(1)
    log.Fatalf("%s:%v:%s, %s",
    file, line, err, fmt.Sprintf(format, arg))
    }
    }

    type Action struct {
    U1 uint64
    U2 uint64
    U3 uint64
    }

    func main() {
    var (
    filePath string
    bufSize int
    fileSize uint64
    )
    flag.StringVar(&filePath, "file", "", "file path")
    flag.IntVar(&bufSize, "buffer_size", 4096, "buffer size")
    flag.Uint64Var(&fileSize, "size", 1024*1024, "file size")

    flag.Usage = func() {
    fmt.Printf("Usage: %s [Options]\n", os.Args[0])
    flag.PrintDefaults()
    }
    flag.Parse()

    if filePath == "" {
    flag.Usage()
    return
    }

    f, err := os.Create(filePath)
    Checkf(err, "create file %s failed", filePath)
    defer f.Close()

    var totalSize uint64
    w := bufio.NewWriter(f)
    tStart := time.Now()
    var action Action
    for totalSize < fileSize {
    Checkf(binary.Write(w, binary.LittleEndian, &action), "write %v", totalSize)
    totalSize += uint64(unsafe.Sizeof(action))
    }
    tStop := time.Now()

    tDuration := tStop.Sub(tStart)
    fmt.Printf("Write over, totalSize: %v, time elapsed: %v(s), speed: %v MB/s
    \n",
    totalSize,
    tDuration.Seconds(),
    float64(totalSize/1048576)/tDuration.Seconds())
    }
    On Thu, Nov 8, 2012 at 1:04 AM, Dustin Sallings wrote:

    go-fish <gdmmzb@gmail.com> writes:
    i use encoding/binary package for testing the binary write/read speed,
    but it reach 12MB/s at most.
    What is the fastest way to read/write binary file using go?

    f, _ := os.Create("test")
    defer f.Close()

    w := bufio.NewWriter(f)
    var action Action
    for {
    binary.Write(w, binary.LittleEndian, &action)
    }
    Can you post a usable benchmark? I don't find it slowing me down
    excessively in gomemcached doing binary protocol encoding (first
    real-world use case I could think of in my groggy state on the train):

    BenchmarkEncodingRequest 10000000 243 ns/op 163.97
    MB/s
    BenchmarkEncodingRequest0CAS 10000000 240 ns/op 166.23
    MB/s
    BenchmarkEncodingRequest1Extra 10000000 234 ns/op 174.65
    MB/s
    BenchmarkEncodingResponse 10000000 234 ns/op 170.27
    MB/s
    BenchmarkEncodingResponseLarge 500000 6280 ns/op 3917.74
    MB/s

    --
    dustin

    --

    --
  • Dustin Sallings at Nov 8, 2012 at 1:13 am

    zhang bin writes:

    I just write a simple go1 command project to test. My full code is
    here.
    You really should use testing.B for this. Your implementation will
    not be very accurate.

    If you implement io.WriterTo, you can test a couple implementations,
    including this one below (with reflection) and just manually doing the
    encoding into a buffer and sending it out.

    --
    dustin

    --
  • Zhang bin at Nov 8, 2012 at 2:37 am
    i rewrite my code here and run it, the output is :

    Write over, totalSize: 104857608, time elapsed: 7.297325(s), speed:
    13.703651680581583 MB/s

    i use ioutil.Discard to avoid the write influence

    ///////////////////////////////////////////////////////////////////////////////////////////////////////////
    package main

    import (
    "encoding/binary"
    "fmt"
    "time"
    "io/ioutil"
    )

    type Action struct {
    U1 uint64
    U2 uint64
    U3 uint64
    }

    func main() {
    tStart := time.Now()
    var action Action
    fileSize := uint64(1024 * 1024 * 100) // 100M
    var totalSize uint64
    for totalSize < fileSize {
    binary.Write(ioutil.Discard, binary.LittleEndian, &action)
    totalSize += uint64(24)
    }
    tStop := time.Now()

    tDuration := tStop.Sub(tStart)
    fmt.Printf("Write over, totalSize: %v, time elapsed: %v(s), speed: %v MB/s
    \n",
    totalSize,
    tDuration.Seconds(),
    float64(totalSize/1048576)/tDuration.Seconds())
    }
    On Thu, Nov 8, 2012 at 9:12 AM, Dustin Sallings wrote:

    zhang bin <gdmmzb@gmail.com> writes:
    I just write a simple go1 command project to test. My full code is
    here.
    You really should use testing.B for this. Your implementation will
    not be very accurate.

    If you implement io.WriterTo, you can test a couple implementations,
    including this one below (with reflection) and just manually doing the
    encoding into a buffer and sending it out.

    --
    dustin

    --

    --
  • Dustin Sallings at Nov 8, 2012 at 2:42 am

    zhang bin writes:

    i rewrite my code here and run it, the output is :

    Write over, totalSize: 104857608, time elapsed: 7.297325(s), speed:
    13.703651680581583 MB/s

    i use ioutil.Discard to avoid the write influence
    While I question the accuracy of your test less, you really don't need
    to reinvent testing.B. It's quite good, and will likely be a huge help
    when you do the next optimization step. Read the docs for the testing
    package for testing.B. Then contain your encoding code in a function.
    Then benchmark that function. Then write a better version of function
    manipulating a []byte directly. See which trade-off you'd like to
    make.

    --
    dustin

    --
  • Dave Cheney at Nov 8, 2012 at 2:47 am
    Here ya' go. Now you can use the benchmarking, and more importantly,
    profiling tools, to improve the performance of this package.

    package write

    import (
    "encoding/binary"
    "io/ioutil"
    "testing"
    )

    type Action struct {
    U1 uint64
    U2 uint64
    U3 uint64
    }

    const fileSize = 1 << 16

    func BenchmarkBinaryWrite64k(b *testing.B) {
    var action Action
    b.SetBytes(fileSize)
    for i := 0; i < b.N; i++ {
    for totalSize := 0; totalSize < fileSize; {
    binary.Write(ioutil.Discard, binary.LittleEndian, &action)
    totalSize += 24
    }
    }
    }

    % go test -bench=.
    testing: warning: no tests to run
    PASS
    BenchmarkBinaryWrite64k 100 19805230 ns/op 3.31 MB/s
    ok write 2.020s
    On Thu, Nov 8, 2012 at 1:42 PM, Dustin Sallings wrote:
    zhang bin <gdmmzb@gmail.com> writes:
    i rewrite my code here and run it, the output is :

    Write over, totalSize: 104857608, time elapsed: 7.297325(s), speed:
    13.703651680581583 MB/s

    i use ioutil.Discard to avoid the write influence
    While I question the accuracy of your test less, you really don't need
    to reinvent testing.B. It's quite good, and will likely be a huge help
    when you do the next optimization step. Read the docs for the testing
    package for testing.B. Then contain your encoding code in a function.
    Then benchmark that function. Then write a better version of function
    manipulating a []byte directly. See which trade-off you'd like to
    make.

    --
    dustin

    --
    --
  • Zhang bin at Nov 8, 2012 at 3:14 am
    Ok, i use binary.LittleEndian to get the fastest write speed, full code here

    Write over, totalSize: 104857600, time elapsed: 0.33172300000000005(s),
    speed: 301.4563355570762 MB/s

    ///////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////
    package main

    import (
    "encoding/binary"
    "fmt"
    "time"
    "io/ioutil"
    )

    func main() {
    tStart := time.Now()
    fileSize := uint64(1024 * 1024 * 100) // 100M
    var totalSize uint64
    b := make([]byte, 8)
    for totalSize < fileSize {
    binary.LittleEndian.PutUint64(b, uint64(totalSize))
    ioutil.Discard.Write(b)
    totalSize += uint64(8)
    }
    tStop := time.Now()

    tDuration := tStop.Sub(tStart)
    fmt.Printf("Write over, totalSize: %v, time elapsed: %v(s), speed: %v MB/s
    \n",
    totalSize,
    tDuration.Seconds(),
    float64(totalSize/1048576)/tDuration.Seconds())
    }

    --

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupgolang-nuts @
categoriesgo
postedNov 7, '12 at 1:52p
activeNov 8, '12 at 3:14a
posts8
users3
websitegolang.org

People

Translate

site design / logo © 2022 Grokbase