FAQ
I am reading a TCP byte streaming connection and to start the decoding I
need to find the ID in the stream.

I solve the issue with the next function, but the performance is not very
good.

func searchUID(r *bufio.Reader, uid [16]byte) (n int, Size int32, Type
int32, err error) {
var seq [16]byte
n = 0
for seq != uid {
copy(seq[:15], seq[1:])
seq[15], err = r.ReadByte()
if err != nil {
fmt.Println("Error during ReadByte (UID Search)", err)
return n, 0, 0, err
}
// fmt.Printf("%x, %x, %v \n", uid, seq, n) // debug
if n > 4096 {
// fmt.Printf("UID not found, Err :Out of search range \n")
return n, 0, 0, errors.New("UID not found, Out of search range")
}
n = n + 1
}

// Read the other parts of the header
err = binary.Read(r, binary.LittleEndian, &Size)
err = binary.Read(r, binary.LittleEndian, &Type)
return n, Size, Type, nil
}

Is there a better way to solve the problem?


--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Search Discussions

  • Konstantin Shaposhnikov at Jul 9, 2015 at 4:48 pm
    I would suggest to profile the app to identify the slow code path first. Go
    makes profiling really easy, have a look at
    http://blog.golang.org/profiling-go-programs
    On Thursday, 9 July 2015 23:37:24 UTC+8, Marcel Farrés Franch wrote:

    I am reading a TCP byte streaming connection and to start the decoding I
    need to find the ID in the stream.

    I solve the issue with the next function, but the performance is not very
    good.

    func searchUID(r *bufio.Reader, uid [16]byte) (n int, Size int32, Type
    int32, err error) {
    var seq [16]byte
    n = 0
    for seq != uid {
    copy(seq[:15], seq[1:])
    seq[15], err = r.ReadByte()
    if err != nil {
    fmt.Println("Error during ReadByte (UID Search)", err)
    return n, 0, 0, err
    }
    // fmt.Printf("%x, %x, %v \n", uid, seq, n) // debug
    if n > 4096 {
    // fmt.Printf("UID not found, Err :Out of search range \n")
    return n, 0, 0, errors.New("UID not found, Out of search range")
    }
    n = n + 1
    }

    // Read the other parts of the header
    err = binary.Read(r, binary.LittleEndian, &Size)
    err = binary.Read(r, binary.LittleEndian, &Type)
    return n, Size, Type, nil
    }

    Is there a better way to solve the problem?
    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
  • Marcel Farrés Franch at Jul 9, 2015 at 5:01 pm
    I had done that, it consumes 26%. That's why I asked if there is a better
    way to implement it.


    Total: 430ms 800ms (flat, cum) 26.40%
         151 . 10ms func searchUID(r *bufio.Reader, uid [16]byte) (n int, Size int32, Type int32, err error) {
          152 . . var seq [16]byte
          153 . .
          154 . . n = 0
          155 280ms 410ms for seq != uid {
          156 50ms 100ms copy(seq[:15], seq[1:])
          157 30ms 190ms seq[15], err = r.ReadByte()
          158 50ms 50ms if err != nil {
          159 . . fmt.Println("Error during ReadByte (UID Search)", err)
          160 . . return n, 0, 0, err
          161 . . }
          162 . .
          163 . . // fmt.Printf("%x, %x, %v \n", uid, seq, n)
          164 . . if n > 4096 {
          165 . . fmt.Printf("UID not found, Err :Out of search range \n")
          166 . . return n, 0, 0, errors.New("UID not found, Out of search range")
          167 . . }
          168 . . n = n + 1
          169 . . }
          170 . .
          171 . . // fmt.Printf("Search end successfully in %d iterations! (%x vs %x)\n", n, uid, seq)
          172 . . // Read the other parts of the header
          173 . 10ms err = binary.Read(r, binary.LittleEndian, &Size)
          174 20ms 20ms err = binary.Read(r, binary.LittleEndian, &Type)
          175 . .
          176 . . return n, Size, Type, nil
          177 . 10ms }

    On Thursday, 9 July 2015 18:48:34 UTC+2, Konstantin Shaposhnikov wrote:

    I would suggest to profile the app to identify the slow code path first.
    Go makes profiling really easy, have a look at
    http://blog.golang.org/profiling-go-programs
    On Thursday, 9 July 2015 23:37:24 UTC+8, Marcel Farrés Franch wrote:

    I am reading a TCP byte streaming connection and to start the decoding I
    need to find the ID in the stream.

    I solve the issue with the next function, but the performance is not very
    good.

    func searchUID(r *bufio.Reader, uid [16]byte) (n int, Size int32, Type
    int32, err error) {
    var seq [16]byte
    n = 0
    for seq != uid {
    copy(seq[:15], seq[1:])
    seq[15], err = r.ReadByte()
    if err != nil {
    fmt.Println("Error during ReadByte (UID Search)", err)
    return n, 0, 0, err
    }
    // fmt.Printf("%x, %x, %v \n", uid, seq, n) // debug
    if n > 4096 {
    // fmt.Printf("UID not found, Err :Out of search range \n")
    return n, 0, 0, errors.New("UID not found, Out of search range")
    }
    n = n + 1
    }

    // Read the other parts of the header
    err = binary.Read(r, binary.LittleEndian, &Size)
    err = binary.Read(r, binary.LittleEndian, &Type)
    return n, Size, Type, nil
    }

    Is there a better way to solve the problem?
    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
  • Konstantin Shaposhnikov at Jul 9, 2015 at 5:26 pm
    You might want to try a few simple to implement possible optimizations:
    - use circular buffer to avoid copying (i.e. seq[k] = ReadByte(), k = (k +
    1) % 16)
    - custom function to compare arrays: compare first byte in uid and seq and
    if they are the same compare the rest (taking into account circular buffer).

    Alternatively you can use more optimal algorithm to search a sub-string in
    the stream.
    E.g. https://en.wikipedia.org/wiki/Knuth%E2%80%93Morris%E2%80%93Pratt_algorithm

    Another idea: use two uint64 instead of [16]byte inside the function. The
    comparison should be much faster. Construct these two numbers from the last
    16 bytes using bit shift operations.
    On Friday, 10 July 2015 01:00:53 UTC+8, Marcel Farrés Franch wrote:

    I had done that, it consumes 26%. That's why I asked if there is a better
    way to implement it.


    Total: 430ms 800ms (flat, cum) 26.40%
    151 . 10ms func searchUID(r *bufio.Reader, uid [16]byte) (n int, Size int32, Type int32, err error) {
    152 . . var seq [16]byte
    153 . .
    154 . . n = 0
    155 280ms 410ms for seq != uid {
    156 50ms 100ms copy(seq[:15], seq[1:])
    157 30ms 190ms seq[15], err = r.ReadByte()
    158 50ms 50ms if err != nil {
    159 . . fmt.Println("Error during ReadByte (UID Search)", err)
    160 . . return n, 0, 0, err
    161 . . }
    162 . .
    163 . . // fmt.Printf("%x, %x, %v \n", uid, seq, n)
    164 . . if n > 4096 {
    165 . . fmt.Printf("UID not found, Err :Out of search range \n")
    166 . . return n, 0, 0, errors.New("UID not found, Out of search range")
    167 . . }
    168 . . n = n + 1
    169 . . }
    170 . .
    171 . . // fmt.Printf("Search end successfully in %d iterations! (%x vs %x)\n", n, uid, seq)
    172 . . // Read the other parts of the header
    173 . 10ms err = binary.Read(r, binary.LittleEndian, &Size)
    174 20ms 20ms err = binary.Read(r, binary.LittleEndian, &Type)
    175 . .
    176 . . return n, Size, Type, nil
    177 . 10ms }

    On Thursday, 9 July 2015 18:48:34 UTC+2, Konstantin Shaposhnikov wrote:

    I would suggest to profile the app to identify the slow code path first.
    Go makes profiling really easy, have a look at
    http://blog.golang.org/profiling-go-programs
    On Thursday, 9 July 2015 23:37:24 UTC+8, Marcel Farrés Franch wrote:

    I am reading a TCP byte streaming connection and to start the decoding I
    need to find the ID in the stream.

    I solve the issue with the next function, but the performance is not
    very good.

    func searchUID(r *bufio.Reader, uid [16]byte) (n int, Size int32, Type
    int32, err error) {
    var seq [16]byte
    n = 0
    for seq != uid {
    copy(seq[:15], seq[1:])
    seq[15], err = r.ReadByte()
    if err != nil {
    fmt.Println("Error during ReadByte (UID Search)", err)
    return n, 0, 0, err
    }
    // fmt.Printf("%x, %x, %v \n", uid, seq, n) // debug
    if n > 4096 {
    // fmt.Printf("UID not found, Err :Out of search range \n")
    return n, 0, 0, errors.New("UID not found, Out of search range")
    }
    n = n + 1
    }

    // Read the other parts of the header
    err = binary.Read(r, binary.LittleEndian, &Size)
    err = binary.Read(r, binary.LittleEndian, &Type)
    return n, Size, Type, nil
    }

    Is there a better way to solve the problem?
    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
  • Konstantin Shaposhnikov at Jul 9, 2015 at 5:39 pm
    Also you can try using Peek (http://golang.org/pkg/bufio/#Reader.Peek) and
    Buffered (http://golang.org/pkg/bufio/#Reader.Buffered) methods to avoid
    reading one byte at a time.
    On Friday, 10 July 2015 01:26:37 UTC+8, Konstantin Shaposhnikov wrote:

    You might want to try a few simple to implement possible optimizations:
    - use circular buffer to avoid copying (i.e. seq[k] = ReadByte(), k = (k +
    1) % 16)
    - custom function to compare arrays: compare first byte in uid and seq and
    if they are the same compare the rest (taking into account circular buffer).

    Alternatively you can use more optimal algorithm to search a sub-string in
    the stream. E.g.
    https://en.wikipedia.org/wiki/Knuth%E2%80%93Morris%E2%80%93Pratt_algorithm

    Another idea: use two uint64 instead of [16]byte inside the function. The
    comparison should be much faster. Construct these two numbers from the last
    16 bytes using bit shift operations.
    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupgolang-nuts @
categoriesgo
postedJul 9, '15 at 3:37p
activeJul 9, '15 at 5:39p
posts5
users2
websitegolang.org

People

Translate

site design / logo © 2021 Grokbase