FAQ
Hi, I'm writing an STL parser for a project, and I like some idiomatic, but
more over performance opinions. An STL binary file is a triangle file and
its pretty simple (http://en.wikipedia.org/wiki/STL_(file_format)), I
defined something like this...

type Model struct {
   Header [80]byte
   Count uint32
   Triangles []Triangle
}

type Triangle struct {
   Vertex1 [3]float32
   Vertex2 [3]float32
   Vertex3 [3]float32
   Attribute uint16
}

My problem as always is that Go has never liked byte arrays, it encourages
byte slices for everything, and I'm used to parsing with *bytes.buffer* and
*encoding/binary* libraries, but they have issues with byte arrays as in
"invalid type" issues as far as I have tested, usually all my structs when
dealing with this have ints, floats, and all that "defined size" stuff, I
though a [80]byte would be the same. I like byte arrays cause you can be
sure how much memory you are using (no memory leaks, garbage collection,
memory copying and stuff) and I need this part to perform as tight as
possible. Right now I'm using *ioutil *to read the whole file and get my
structs, but I am pretty sure the optimal way would be using *bufio *in
case of large files.

So... what would be the optimal (as in less memory and parse time) method
to store my model using slices? Do i preallocate slice capacity in every
single one? Buffer size? (cause im pretty sure reading a triangle (14
bytes) at a time would be painfully slow. And then again... would using
slices would not incur in the usual memory copy operations when trying to
expand them? Good thing is I have the Triangle count on bytes [80:84] so I
can proceed then to allocate length/capacity of my Triangle slice. Any
method is welcome.

Ty Gophers,

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Search Discussions

  • Guillermo Estrada at Oct 1, 2013 at 2:57 am
    I screwed the endianess on one try, this seems to work though:

       data, err := ioutil.ReadFile(filename)
       if err != nil {
         panic(err)
       }
       fmt.Println(len(data))
       buf := new(bytes.Buffer)
       m := new(Model)
       buf.Write(data[0:84])
       err = binary.Read(buf, binary.LittleEndian, m)
       if err != nil {
         panic(err)
       }
       fmt.Println("Triangles:", m.Count)

    The triangle number seems to work, but I had to modify my Model struct
    because of the Triangle Slice, I guess I'll have to read that on another
    one. Any other thoughts...
    On Monday, September 30, 2013 9:41:59 PM UTC-5, Guillermo Estrada wrote:

    Hi, I'm writing an STL parser for a project, and I like some idiomatic,
    but more over performance opinions. An STL binary file is a triangle file
    and its pretty simple (http://en.wikipedia.org/wiki/STL_(file_format)), I
    defined something like this...

    type Model struct {
    Header [80]byte
    Count uint32
    Triangles []Triangle
    }

    type Triangle struct {
    Vertex1 [3]float32
    Vertex2 [3]float32
    Vertex3 [3]float32
    Attribute uint16
    }

    My problem as always is that Go has never liked byte arrays, it encourages
    byte slices for everything, and I'm used to parsing with *bytes.buffer*and
    *encoding/binary* libraries, but they have issues with byte arrays as in
    "invalid type" issues as far as I have tested, usually all my structs when
    dealing with this have ints, floats, and all that "defined size" stuff, I
    though a [80]byte would be the same. I like byte arrays cause you can be
    sure how much memory you are using (no memory leaks, garbage collection,
    memory copying and stuff) and I need this part to perform as tight as
    possible. Right now I'm using *ioutil *to read the whole file and get my
    structs, but I am pretty sure the optimal way would be using *bufio *in
    case of large files.

    So... what would be the optimal (as in less memory and parse time) method
    to store my model using slices? Do i preallocate slice capacity in every
    single one? Buffer size? (cause im pretty sure reading a triangle (14
    bytes) at a time would be painfully slow. And then again... would using
    slices would not incur in the usual memory copy operations when trying to
    expand them? Good thing is I have the Triangle count on bytes [80:84] so I
    can proceed then to allocate length/capacity of my Triangle slice. Any
    method is welcome.

    Ty Gophers,
    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.
  • Elliott Polk at Oct 1, 2013 at 3:05 am
    Unrelated to Go, just remember there are 2 types of STL formatting. I work for a 3D printer manufacture and we're also working with STL files. I wish you the best if you're writing a Go based slicer. This is currently on my list as well.

    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.
  • Guillermo Estrada at Oct 1, 2013 at 3:10 am
    Eliott, now that you mention it... It is a Go based slicer for a 3D printer
    preview program. What two types are you referring to? ASCII and Binary?
    On Monday, September 30, 2013 10:04:33 PM UTC-5, Elliott Polk wrote:

    Unrelated to Go, just remember there are 2 types of STL formatting. I work
    for a 3D printer manufacture and we're also working with STL files. I wish
    you the best if you're writing a Go based slicer. This is currently on my
    list as well.
    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.
  • Erwin at Oct 1, 2013 at 11:12 am
    I think for optimal read performance you'll need to use package unsafe and
    avoid encoding/binary. After you have read in the header (which you can do
    with encoding/binary because it's not a lot), you can allocate a byte slice
    the size of triangle data that will follow. Just read this with one call to
    os File.Read(), which should be fast. Once you have your byte slice, you
    can use package unsafe to convert the slice to a []Triangle without copying
    anything. It's like C where you cast a void * to the expected data type.

    something like this (untested):

    const triSize = int(unsafe.Sizeof(Triangle{}))

    bytes = make([]byte, model.Count*triSize)
    _, err := file.Read()
    ...

    // create a []Triangle from []byte
    var slice = struct {
         addr uintptr
         len int
         cap int
    }{
         addr: uintptr(unsafe.Pointer(&bytes[0])
         len: len(bytes) / triSize
         cap: cap(bytes) / triSize
    }
    triangles := *(*[]Triangle)(unsafe.Pointer(&slice))


    I haven't done the above with .stl files yet, but with other types of files
    that have large chunks of similar data, and it turns out to be much faster
    than using encoding/binary. Ideally encoding/binary would be optimized to
    use a similar technique (when the file byte order matches the machine byte
    order), so one could simply use encoding/binary and read the []Triangle in
    a single call as well and it would fly like the code above.









    On 1 October 2013 05:10, Guillermo Estrada wrote:

    Eliott, now that you mention it... It is a Go based slicer for a 3D
    printer preview program. What two types are you referring to? ASCII and
    Binary?

    On Monday, September 30, 2013 10:04:33 PM UTC-5, Elliott Polk wrote:

    Unrelated to Go, just remember there are 2 types of STL formatting. I
    work for a 3D printer manufacture and we're also working with STL files. I
    wish you the best if you're writing a Go based slicer. This is currently on
    my list as well.
    --
    You received this message because you are subscribed to the Google Groups
    "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an
    email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.
    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.
  • Guillermo Estrada at Oct 1, 2013 at 8:58 pm
    Ok I did what you proposed! I used nitro to get a simple profiling of the
    data, I exported both methods to an ASCII version of the STL and both are
    identical so... Here is what I got.

    Parsing STL:
             7.0004ms (8.0005ms) 1.24 MB 33347 Allocs
    Parsing Unsafe STL:
             1.0001ms (13.0008ms) 0.26 MB 27 Allocs

    That is a pretty small file with just 4.7K Triangles, I will do more
    testing with different file sizes. On the other hand...

    @Elliott what is the biggest size you have seen on a STL for a 3D Printer?
    with *unsafe *method, *bufio *might not be needed after all...

    *I think for optimal read performance you'll need to use package unsafe and
    avoid encoding/binary. After you have read in the header (which you can do
    with encoding/binary because it's not a lot), you can allocate a byte slice
    the size of triangle data that will follow. Just read this with one call to
    os File.Read(), which should be fast. Once you have your byte slice, you
    can use package unsafe to convert the slice to a []Triangle without copying
    anything. It's like C where you cast a void * to the expected data type.
    * * *
    *I haven't done the above with .stl files yet, but with other types of
    files that have large chunks of similar data, and it turns out to be much
    faster than using encoding/binary. Ideally encoding/binary would be
    optimized to use a similar technique (when the file byte order matches the
    machine byte order), so one could simply use encoding/binary and read the
    []Triangle in a single call as well and it would fly like the code above.*










    On 1 October 2013 05:10, Guillermo Estrada <phro...@gmail.com<javascript:>
    wrote:
    Eliott, now that you mention it... It is a Go based slicer for a 3D
    printer preview program. What two types are you referring to? ASCII and
    Binary?

    On Monday, September 30, 2013 10:04:33 PM UTC-5, Elliott Polk wrote:

    Unrelated to Go, just remember there are 2 types of STL formatting. I
    work for a 3D printer manufacture and we're also working with STL files. I
    wish you the best if you're writing a Go based slicer. This is currently on
    my list as well.
    --
    You received this message because you are subscribed to the Google Groups
    "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an
    email to golang-nuts...@googlegroups.com <javascript:>.
    For more options, visit https://groups.google.com/groups/opt_out.
    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.
  • Erwin at Oct 1, 2013 at 9:59 pm


    @Elliott what is the biggest size you have seen on a STL for a 3D Printer?
    with *unsafe *method, *bufio *might not be needed after all...
    I've been printing files up the 350MB, larger than that, the printer
    software balks... I suppose you'll find gigabyte .stl files in the wild.

    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.
  • 布艾德 - Elliott Polk at Oct 2, 2013 at 1:03 am
    The biggest we're playing with is around 9.5MB, tho we're just looking at
    things from thingiverse at the moment.

    Have a look for a large desk tidy or even the Yoda is a decent stress test.

    Is this a personal project? Feel free to ping me outside of this channel.
    Would be interested to follow the progress.
    On Oct 2, 2013 6:00 AM, "Erwin" wrote:

    @Elliott what is the biggest size you have seen on a STL for a 3D
    Printer? with *unsafe *method, *bufio *might not be needed after all...
    I've been printing files up the 350MB, larger than that, the printer
    software balks... I suppose you'll find gigabyte .stl files in the wild.


    --
    You received this message because you are subscribed to a topic in the
    Google Groups "golang-nuts" group.
    To unsubscribe from this topic, visit
    https://groups.google.com/d/topic/golang-nuts/eqtmH2WGSKE/unsubscribe.
    To unsubscribe from this group and all its topics, send an email to
    golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.
    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.
  • Ian Lance Taylor at Oct 1, 2013 at 3:03 am

    On Mon, Sep 30, 2013 at 7:41 PM, Guillermo Estrada wrote:
    My problem as always is that Go has never liked byte arrays, it encourages
    byte slices for everything, and I'm used to parsing with bytes.buffer and
    encoding/binary libraries, but they have issues with byte arrays as in
    "invalid type" issues as far as I have tested, usually all my structs when
    dealing with this have ints, floats, and all that "defined size" stuff, I
    though a [80]byte would be the same.
    If a function takes a slice, and you have an array, you can always
    pass the array to the function by using [:] to slice the array.

    http://play.golang.org/p/ZEZObSrqyW

    Ian

    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.
  • Guillermo Estrada at Oct 1, 2013 at 3:28 pm
    I still have to test using unsafe library to read all the data. But wgile
    using encoding/binary I have a problem like this

    Reading from the buffer data[0:84] works while using encoding/binary
    type Model struct {
    Header [80]byte
    Count uint32
    }
    Reading from the buffer data[84:98] panics with an unexpected EOF, I think
    it has something to do with the 8 byte allocation on 64 bit systems, but I
    have not had this problem before. Panics the same with the float arrays or
    9 floats, just in case you were wondering.
    type Triangle struct {
    Vertex1 [3]float32
    Vertex2 [3]float32
    Vertex3 [3]float32
    Attribute uint16
    }
    Any ideas on this? Why is this happening?

    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.
  • Ian Lance Taylor at Oct 1, 2013 at 3:40 pm

    On Tue, Oct 1, 2013 at 8:28 AM, Guillermo Estrada wrote:
    Reading from the buffer data[0:84] works while using encoding/binary
    type Model struct {
    Header [80]byte
    Count uint32
    }

    Reading from the buffer data[84:98] panics with an unexpected EOF, I think
    it has something to do with the 8 byte allocation on 64 bit systems, but I
    have not had this problem before. Panics the same with the float arrays or 9
    floats, just in case you were wondering.
    type Triangle struct {
    Vertex1 [3]float32
    Vertex2 [3]float32
    Vertex3 [3]float32
    Attribute uint16
    }

    Any ideas on this? Why is this happening?
    Show us the code.

    Ian

    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.
  • Guillermo Estrada at Oct 1, 2013 at 3:44 pm
    @Ian

    *// READ THE WHOLE FILE*
       data, err := ioutil.ReadFile(filename)
       if err != nil {
         panic(err)
       }

    *//CREATE A BUFFER*
       buf := new(bytes.Buffer)
       m := new(Model)

    *//WRITING 84 bytes OF DATA TO A BUFFER AND DECODING IN MODEL WORKS*
       buf.Write(data[0:84])
       err = binary.Read(buf, binary.LittleEndian,m)
       if err != nil {
         panic(err)
       }
       buf.Reset()

       tri := new(Triangle)

    *//WRITING 14 bytes OF DATA AND DECODING INTO TRIANGLE PANICS WITH
    UNEXPECTED EOF*
       buf.Write(data[84:98])
       err = binary.Read(buf, binary.LittleEndian, tri)
       if err != nil {
         panic(err)
       }
       buf.Reset()

    Show us the code.

    Ian
    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.
  • Guillermo Estrada at Oct 1, 2013 at 3:55 pm
    I am realizing Triangle struct is actually 38 bytes long.. Silly me.
    Anyway, I'll do a quick benchmark using *ioutil*, *bytes.buffer*, and *
    encoding/binary* to check for speed and memory usage, and I'll proceed to
    do use *unsafe,* to compare the options. Also, I guess *bufio *is a better
    way to read the file. Any recommended approach for large files for optimal
    parsing?

    *//WRITING 14 bytes OF DATA AND DECODING INTO TRIANGLE PANICS WITH
    UNEXPECTED EOF*
    buf.Write(data[84:98])
    err = binary.Read(buf, binary.LittleEndian, tri)
    if err != nil {
    panic(err)
    }
    buf.Reset()

    Show us the code.

    Ian
    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.
  • Guillermo Estrada at Oct 2, 2013 at 3:17 am
    Apparently although everything "looked" fine, I am unable to get it right
    using the *unsafe *method described by *@notnot*
    The first normal and vertex are all good, but I'm having different data
    after the first one...
    Does anyone have an example?

    My code looks like this...

    type Triangle struct {
       Normal [3]float32
       Vertex1 [3]float32
       Vertex2 [3]float32
       Vertex3 [3]float32
       Attribute uint16
    }

    func ParseUnsafeSTL(filename string) *Model {
       data, err := ioutil.ReadFile(filename)
       if err != nil {
         panic(err)
       }
       m := new(Model)
       binary.Read(bytes.NewBuffer(data[80:84]), binary.LittleEndian, &m.Length)
       // create a []Triangle from []byte
       triSize := int(unsafe.Sizeof(Triangle{}))
       var slice = struct {
           addr uintptr
           len int
           cap int
       }{
           addr: uintptr(unsafe.Pointer(&data[84])),
           len: len(data[84:]) / triSize,
           cap: cap(data[84:]) / triSize,
       }
       m.Triangles = *(*[]Triangle)(unsafe.Pointer(&slice))
       return m
    }

    Only the first one read is fine, I suppose it has something to do with the
    sizes, but I'm unsure as I have never used unsafe before. Any help is
    welcome.
    On Tuesday, October 1, 2013 10:55:36 AM UTC-5, Guillermo Estrada wrote:

    I am realizing Triangle struct is actually 38 bytes long.. Silly me.
    Anyway, I'll do a quick benchmark using *ioutil*, *bytes.buffer*, and *
    encoding/binary* to check for speed and memory usage, and I'll proceed to
    do use *unsafe,* to compare the options. Also, I guess *bufio *is a
    better way to read the file. Any recommended approach for large files for
    optimal parsing
    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.
  • Erwin at Oct 2, 2013 at 4:09 am


    Only the first one read is fine, I suppose it has something to do with the
    sizes, but I'm unsure as I have never used unsafe before. Any help is
    welcome.

    What is the value of triSize exactly? I suspect it is 52 instead of 50,
    for alignment purposes. Too bad, because then the above unsafe conversion
    from []byte to []Triangle won't work. In memory a Triangle is 52 bytes, in
    the file it is 50 bytes. So it looks like you'll have to decode triangle
    after triangle...

    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.
  • Guillermo Estrada at Oct 2, 2013 at 5:08 am
    Yeah I was expecting something like that to be troublesome...
    This is the Best Implementation I got yet. It loads everything in memory on
    a single go, guess larger files will be troublesome and I must do a *bufio *implementation,
    but I'm already expecting it to be kinda slow, cause I guess I will be
    reading a triangle at a time? Anyway, thoughts on optimizing this code, or
    on a performant bufio implementation are always welcome.

    func ParseSTL(filename string) *Model {

       // Reading entire STL file into memory
       data, err := ioutil.ReadFile(filename)
       if err != nil {
         panic(err)
       }

       m := new(Model)

       // Parsing Header first 80 bytes.
       err = binary.Read(bytes.NewBuffer(data[0:80]), binary.LittleEndian,
    &m.Header)
       if err != nil {
         panic(err)
       }

       // Parsing triangle count uint32 at byte 80
       err = binary.Read(bytes.NewBuffer(data[80:84]), binary.LittleEndian,
    &m.Length)
       if err != nil {
         panic(err)
       }

       // Allocating enough memory for all the triangles in the slice
       m.Triangles = make([]Triangle, m.Length)

       // Parsing the Triangle slice on byte 84 onwards, 50 bytes per triangle
       err = binary.Read(bytes.NewBuffer(data[84:]), binary.LittleEndian,
    &m.Triangles)
       if err != nil {
         panic(err)
       }

       return m
    }

    On Tuesday, October 1, 2013 11:09:50 PM UTC-5, notnot wrote:

    Only the first one read is fine, I suppose it has something to do with
    the sizes, but I'm unsure as I have never used unsafe before. Any help is
    welcome.

    What is the value of triSize exactly? I suspect it is 52 instead of 50,
    for alignment purposes. Too bad, because then the above unsafe conversion
    from []byte to []Triangle won't work. In memory a Triangle is 52 bytes, in
    the file it is 50 bytes. So it looks like you'll have to decode triangle
    after triangle...
    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.
  • Jesse van den Kieboom at Oct 2, 2013 at 7:14 am
    Wrt the alignment problem, you might be able to make a special io.Reader which emits bytes for the padding as you read from an underlying io.Reader. Not sure if it will be faster though.

    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.
  • Guillermo Estrada at Oct 2, 2013 at 7:18 pm

    On Wednesday, October 2, 2013 2:14:25 AM UTC-5, Jesse van den Kieboom wrote:
    Wrt the alignment problem, you might be able to make a special io.Reader
    which emits bytes for the padding as you read from an underlying io.Reader.
    Not sure if it will be faster though.

    Yeah although the idea behind using *ioutil *to read the entire file at
    once rely on the possibility to use that already reserved memory and data
    with *unsafe *to cast it to my Triangle slice to make it as efficient as
    possible.

    Once I have to actually check the data read it becomes the exercise of
    optimizing the allocations in memory of data read into my structs, either
    way I'm better off there with *bufio *because I cannot use in my struct
    slice the same memory space I used to read the file in the first place.
    I'll post the *bufio *implementation and some numbers to get ideas on
    optimizations. Thnx!

    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.
  • Erwin at Oct 2, 2013 at 7:54 pm
    Perhaps a reasonable compromise is to read the file in smaller chunks,
    using a smaller read buffer (say 50000 bytes -> 1000 triangles) and
    allocate the full triangle slice once you know the number of them, then
    decode the triangles one by one into your triangle slice. That way you are
    keeping the number of file read calls low, and you don't have to allocate
    twice the memory that you need for the model?

    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.
  • Guillermo Estrada at Oct 2, 2013 at 8:05 pm
    That not only sounds plausible, but looks like it is the way to go. Problem
    is... I already did!! This is the profiling of the loading of the XYZ
    Stanford Dragon (352MB STL binary file)

    Parsing STL:
             3.8892224s (3.8892224s) 1097.35 MB 57 Allocs
    File has 7219045 triangles.

    Parsing STL (buffered):
             3.8702214s (7.7594438s) 1097.32 MB 41 Allocs
    File has 7219045 triangles.

    Numbers are almost identical! And I found no gain in the buffered reading!
    Something must be wrong, but I'm pretty sure I'm reading a triangle at a
    time with a 50 bytes buffer! Can someone help me with this?

    Code:
    https://gist.github.com/phrozen/6799694

    Seems like memory usage in the buffered version is the same, either that or
    the profiler is reporting it wrong... Can someone else runs tests?
    On Wednesday, October 2, 2013 2:54:17 PM UTC-5, notnot wrote:

    Perhaps a reasonable compromise is to read the file in smaller chunks,
    using a smaller read buffer (say 50000 bytes -> 1000 triangles) and
    allocate the full triangle slice once you know the number of them, then
    decode the triangles one by one into your triangle slice. That way you are
    keeping the number of file read calls low, and you don't have to allocate
    twice the memory that you need for the model?
    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.
  • Guillermo Estrada at Oct 2, 2013 at 8:18 pm
    Sorry wrong posting of data, this is the real one, and its even worse.
    Seems like that memory allocation alone while doing triangle per triangle
    (I could read 1000 at a time with the buffer but would be kinda the same)
    slows the process and consumes more memory that loading the whole file with
    *ioutil*. I'm pretty sure my *bufio *version is not great, but I didn't
    suppose it would be soo bad. Everytime you call *encoding/binary* Read on a
    Triangle, more memory is used, and as far as I know, Garbage collector
    won't work until the function returns making the *ioutil *reader a lot
    better in almost all cases.

    Parsing STL:
             4.0122294s (4.0122294s) 1097.35 MB 57 Allocs
    File has 7219045 triangles.

    Parsing STL (buffered):
             11.1656386s (15.1788681s) 2315.18 MB 58100620 Allocs
    File has 7219045 triangles.


    Anyway, I hope someone can come with a better solution. Help and code
    examples appreciated.

    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.
  • Guillermo Estrada at Oct 2, 2013 at 8:53 pm
    Ok, getting some progress with the buffered version...

    Parsing STL:
             3.8682212s (3.8692213s) 1097.35 MB 57 Allocs
    File has 7219045 triangles.

    Parsing STL (buffered) 100:
             566.0324ms (4.4352537s) 395.42 MB 652648 Allocs
    File has 7219045 triangles.

    Parsing STL (buffered) 1000:
             142.0081ms (4.5782618s) 377.46 MB 65311 Allocs
    File has 7219045 triangles.

    Parsing STL (buffered) 10000:
             102.0058ms (4.6812677s) 376.12 MB 6547 Allocs
    File has 7219045 triangles

    Parsing STL (buffered) 100000:
             110.0063ms (4.8582779s) 380.42 MB 699 Allocs
    File has 7219045 triangles.

    Now this is a lot of progress, tests are run with a 100, 1K, 10K, and 100K
    triangles (*50bytes) buffers. And assigning them like in a go. As we can
    see the difference in memory usage between 1K and a 10K is not much, but
    in this case the speed is. 376 MB of RAM used in parsing a 352MB file works
    for me. also 100ms on the parsing of the file is great!

    I hope people trying to parse binary file formats in an optimal way find
    this thread. Anyway. comments are still welcome! Ty Gophers.

    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.
  • Erwin at Oct 2, 2013 at 9:22 pm
    excellent progress! what exactly did you change from the previous version?

    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.
  • Guillermo Estrada at Oct 2, 2013 at 10:09 pm
    First of all I incremented the buffer size, but checking the result files
    converted to STL ASCII format, only the first Triangle is fine. So I guess
    I'll have to find a way to read a 1000 triangles and assign them to the
    slice in a single memory operation because binary.Read() only plays well
    with fixed size data, I guess I will have to create a secondary slice of
    the size and copy the data over to the Triangle slice (copy built in func
    maybe), that will surely put more overhead on memory, but at least won't
    put memory allocation calls to the roof.

    I'll keep testing code, profiling and I'll post back the results with the
    code. If anyone knows a way to use a buffered read of binary file and do
    large chunks assignment to a slice would be GREAT!
    On Wednesday, October 2, 2013 4:22:03 PM UTC-5, notnot wrote:

    excellent progress! what exactly did you change from the previous version?
    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.
  • Guillermo Estrada at Oct 2, 2013 at 10:49 pm
    OK, I'm officially lost, I'm trying the copy slice form and it kinda works
    (and performs OK), But I'm getting sometimes the same values, and some
    other different values. No errors at all. Dunno if padding has something to
    do with it. Here is he code if someone wants to try it out. Just grab any
    STL file out there.

    https://gist.github.com/phrozen/6799694
    On Wednesday, October 2, 2013 5:08:49 PM UTC-5, Guillermo Estrada wrote:

    First of all I incremented the buffer size, but checking the result files
    converted to STL ASCII format, only the first Triangle is fine. So I guess
    I'll have to find a way to read a 1000 triangles and assign them to the
    slice in a single memory operation because binary.Read() only plays well
    with fixed size data, I guess I will have to create a secondary slice of
    the size and copy the data over to the Triangle slice (copy built in func
    maybe), that will surely put more overhead on memory, but at least won't
    put memory allocation calls to the roof.

    I'll keep testing code, profiling and I'll post back the results with the
    code. If anyone knows a way to use a buffered read of binary file and do
    large chunks assignment to a slice would be GREAT!
    On Wednesday, October 2, 2013 4:22:03 PM UTC-5, notnot wrote:

    excellent progress! what exactly did you change from the previous version?
    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.
  • Guillermo Estrada at Oct 2, 2013 at 10:53 pm
    Here is a simple Output of the program if helps, I still dunno why the
    first read is always length 80! (I already read 84 bytes out of the reader)
    I dunno if that is the problem, but as you can see, sometimes I get good
    results and sometimes bad. Maybe I have to set the buffer size of the
    reader? Ty.
    STL -file="GEAR.STL" -stepAnalysis
    GEAR.STL
    Parsing STL:
             5.0042ms (8.0069ms) 0.76 MB 56 Allocs
    File has 4748 triangles.
    true 0 4012 80 80
    true 80 5000 100 100
    true 180 5000 100 100
    true 280 5000 100 100
    true 380 5000 100 100
    false 480 5000 100 100
    true 580 5000 100 100
    true 680 5000 100 100
    false 780 5000 100 100
    true 880 5000 100 100
    false 980 5000 100 100
    true 1080 5000 100 100
    true 1180 5000 100 100
    false 1280 5000 100 100
    false 1380 5000 100 100
    false 1480 5000 100 100
    true 1580 5000 100 100
    true 1680 5000 100 100
    true 1780 5000 100 100
    false 1880 5000 100 100
    true 1980 5000 100 100
    true 2080 5000 100 100
    false 2180 5000 100 100
    true 2280 5000 100 100
    true 2380 5000 100 100
    false 2480 5000 100 100
    true 2580 5000 100 100
    false 2680 5000 100 100
    true 2780 5000 100 100
    true 2880 5000 100 100
    true 2980 5000 100 100
    true 3080 5000 100 100
    false 3180 5000 100 100
    true 3280 5000 100 100
    false 3380 5000 100 100
    true 3480 5000 100 100
    true 3580 5000 100 100
    true 3680 5000 100 100
    true 3780 5000 100 100
    true 3880 5000 100 100
    true 3980 5000 100 100
    true 4080 5000 100 100
    true 4180 5000 100 100
    true 4280 5000 100 100
    false 4380 5000 100 100
    true 4480 5000 100 100
    false 4580 5000 100 100
    true 4680 3388 67 67
    Parsing STL (buffered) 100:
             18.0157ms (26.0226ms) 0.61 MB 666 Allocs
    File has 4748 triangles.
    true 0 4012 80 80
    false 80 50000 1000 1000
    false 1080 50000 1000 1000
    false 2080 50000 1000 1000
    false 3080 50000 1000 1000
    false 4080 33388 667 667
    Parsing STL (buffered) 1000:
             5.0034ms (34.029ms) 0.69 MB 111 Allocs
    File has 4748 triangles.
    Saving ASCII:
             81.068ms (115.097ms) 0.28 MB 33263 Allocs
    Saving ASCII:
             77.0633ms (193.1612ms) 0.27 MB 33254 Allocs
    Saving ASCII:
             77.065ms (270.2262ms) 0.27 MB 33256 Allocs
    On Wednesday, October 2, 2013 5:49:09 PM UTC-5, Guillermo Estrada wrote:

    OK, I'm officially lost, I'm trying the copy slice form and it kinda works
    (and performs OK), But I'm getting sometimes the same values, and some
    other different values. No errors at all. Dunno if padding has something to
    do with it. Here is he code if someone wants to try it out. Just grab any
    STL file out there.

    https://gist.github.com/phrozen/6799694
    On Wednesday, October 2, 2013 5:08:49 PM UTC-5, Guillermo Estrada wrote:

    First of all I incremented the buffer size, but checking the result files
    converted to STL ASCII format, only the first Triangle is fine. So I guess
    I'll have to find a way to read a 1000 triangles and assign them to the
    slice in a single memory operation because binary.Read() only plays well
    with fixed size data, I guess I will have to create a secondary slice of
    the size and copy the data over to the Triangle slice (copy built in func
    maybe), that will surely put more overhead on memory, but at least won't
    put memory allocation calls to the roof.

    I'll keep testing code, profiling and I'll post back the results with the
    code. If anyone knows a way to use a buffered read of binary file and do
    large chunks assignment to a slice would be GREAT!
    On Wednesday, October 2, 2013 4:22:03 PM UTC-5, notnot wrote:

    excellent progress! what exactly did you change from the previous
    version?
    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.
  • Guillermo Estrada at Oct 2, 2013 at 11:16 pm
    Ok, so I wish there was a more "elegant" way of doing this, and maybe
    someone can point it out for me. Here is what I added to the code.

       rs := io.ReadSeeker(file)
       rs.Seek(84,0)
       b = bufio.NewReaderSize(rs, 50*size)

    1) Open File and create a bufio.Reader out of it to read the first 84
    bytes, with a 84 byte buffer ([]byte slice)
    2) Afterwards I had to create a new io.ReadSeeker of the same file, and
    seek to position 84 from the start of the file.
    3) Create a new bufio.Reader with buffer size of at least 50 bytes *
    TriangleBufferSize, no ensure I'll always read full triangles.
    4) Profit...

    Thing is to optimize as possible I'm reading 1000 Triangles at a time, But
    That buffer size is variable depending on the number of triangles in the
    file. But for that... I have to red the 84 first bytes of the file. And
    then create another 2 Readers of it? If I use the first bufio.Reader, I
    always start with a used buffer of X size and problems start. Is there a
    way to cleanup this mess in an idiomatic, elegant, non-c-monkey-hack way?

    Still, numbers aren't THAT great, but certainly are better than the
    alternatives. One thing that pops out is that 1000 Triangles seems to be
    the best choice in almost any situation. That checks out with something I
    read somewhere in this group about keeping buffers as close to but below
    65,535 bytes (50*1000 is close) to optimize them, something to do with
    register size on CPU and RAM and stuff.

    STL -file="Yoda_fixed.stl" -stepAnalysis
    Yoda_fixed.stl
    Parsing STL:
             220.1787ms (222.1846ms) 73.15 MB 60 Allocs
    File has 480844 triangles.
    Parsing STL (buffered) 100:
             225.2346ms (447.4192ms) 50.70 MB 43513 Allocs
    File has 480844 triangles.
    Parsing STL (buffered) 1000:
             213.1773ms (660.5965ms) 50.90 MB 4375 Allocs
    File has 480844 triangles.
    Parsing STL (buffered) 10000:
             216.1788ms (876.7753ms) 51.25 MB 483 Allocs
    File has 480844 triangles.

    Thanxs everyone, code is still here: https://gist.github.com/phrozen/6799694
    If someone with more experience with Readers and Buffers can give it a try
    and make observations on it, I will appreciate it a lot.

    On Wednesday, October 2, 2013 5:53:41 PM UTC-5, Guillermo Estrada wrote:

    Here is a simple Output of the program if helps, I still dunno why the
    first read is always length 80! (I already read 84 bytes out of the reader)
    I dunno if that is the problem, but as you can see, sometimes I get good
    results and sometimes bad. Maybe I have to set the buffer size of the
    reader? Ty.
    STL -file="GEAR.STL" -stepAnalysis
    GEAR.STL
    Parsing STL:
    5.0042ms (8.0069ms) 0.76 MB 56 Allocs
    File has 4748 triangles.
    true 0 4012 80 80
    true 80 5000 100 100
    true 180 5000 100 100
    true 280 5000 100 100
    true 380 5000 100 100
    false 480 5000 100 100
    true 580 5000 100 100
    true 680 5000 100 100
    false 780 5000 100 100
    true 880 5000 100 100
    false 980 5000 100 100
    true 1080 5000 100 100
    true 1180 5000 100 100
    false 1280 5000 100 100
    false 1380 5000 100 100
    false 1480 5000 100 100
    true 1580 5000 100 100
    true 1680 5000 100 100
    true 1780 5000 100 100
    false 1880 5000 100 100
    true 1980 5000 100 100
    true 2080 5000 100 100
    false 2180 5000 100 100
    true 2280 5000 100 100
    true 2380 5000 100 100
    false 2480 5000 100 100
    true 2580 5000 100 100
    false 2680 5000 100 100
    true 2780 5000 100 100
    true 2880 5000 100 100
    true 2980 5000 100 100
    true 3080 5000 100 100
    false 3180 5000 100 100
    true 3280 5000 100 100
    false 3380 5000 100 100
    true 3480 5000 100 100
    true 3580 5000 100 100
    true 3680 5000 100 100
    true 3780 5000 100 100
    true 3880 5000 100 100
    true 3980 5000 100 100
    true 4080 5000 100 100
    true 4180 5000 100 100
    true 4280 5000 100 100
    false 4380 5000 100 100
    true 4480 5000 100 100
    false 4580 5000 100 100
    true 4680 3388 67 67
    Parsing STL (buffered) 100:
    18.0157ms (26.0226ms) 0.61 MB 666 Allocs
    File has 4748 triangles.
    true 0 4012 80 80
    false 80 50000 1000 1000
    false 1080 50000 1000 1000
    false 2080 50000 1000 1000
    false 3080 50000 1000 1000
    false 4080 33388 667 667
    Parsing STL (buffered) 1000:
    5.0034ms (34.029ms) 0.69 MB 111 Allocs
    File has 4748 triangles.
    Saving ASCII:
    81.068ms (115.097ms) 0.28 MB 33263 Allocs
    Saving ASCII:
    77.0633ms (193.1612ms) 0.27 MB 33254 Allocs
    Saving ASCII:
    77.065ms (270.2262ms) 0.27 MB 33256 Allocs
    On Wednesday, October 2, 2013 5:49:09 PM UTC-5, Guillermo Estrada wrote:

    OK, I'm officially lost, I'm trying the copy slice form and it kinda
    works (and performs OK), But I'm getting sometimes the same values, and
    some other different values. No errors at all. Dunno if padding has
    something to do with it. Here is he code if someone wants to try it out.
    Just grab any STL file out there.

    https://gist.github.com/phrozen/6799694
    On Wednesday, October 2, 2013 5:08:49 PM UTC-5, Guillermo Estrada wrote:

    First of all I incremented the buffer size, but checking the result
    files converted to STL ASCII format, only the first Triangle is fine. So I
    guess I'll have to find a way to read a 1000 triangles and assign them to
    the slice in a single memory operation because binary.Read() only plays
    well with fixed size data, I guess I will have to create a secondary slice
    of the size and copy the data over to the Triangle slice (copy built in
    func maybe), that will surely put more overhead on memory, but at least
    won't put memory allocation calls to the roof.

    I'll keep testing code, profiling and I'll post back the results with
    the code. If anyone knows a way to use a buffered read of binary file and
    do large chunks assignment to a slice would be GREAT!
    On Wednesday, October 2, 2013 4:22:03 PM UTC-5, notnot wrote:

    excellent progress! what exactly did you change from the previous
    version?
    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.
  • Kyle Lemons at Oct 3, 2013 at 2:27 am
    a few things. Sorry I'm so late to the game. If I'm suggesting a repeat,
    let me know.

    https://gist.github.com/phrozen/6799694#file-main-go-L46
    Just start out by reading from the file directly. You don't know your
    buffer size yet, so hold off on the bufio.

    https://gist.github.com/phrozen/6799694#file-main-go-L48
    use io.ReadFull

    https://gist.github.com/phrozen/6799694#file-main-go-L51
    bytes.NewReader

    however, you should be able to read these straight out of the file, you
    shouldn't need to read it in and then make a bytes.Reader

    https://gist.github.com/phrozen/6799694#file-main-go-L70
    if you don't make a bufio.Reader above, you don't need to seek, and can
    just make the ReaderSize here.

    https://gist.github.com/phrozen/6799694#file-main-go-L89
    You've got some crazy double-buffering going on here. Get rid of "buf" and
    just read straight out of the bufio.Reader. When it empties, it will fill
    itself up, and then you can binary.Read straight out of it.

    https://gist.github.com/phrozen/6799694#file-main-go-L94
    I would be very surprised if this is helping you. Read straight into the
    triangle you're currently decoding. If your profile shows that you're
    spending a lot of time in reflect, unroll that binary.Read into a ReadFull
    and then the right sequence of calls to binary.LittleEndian.*

    On Wed, Oct 2, 2013 at 4:16 PM, Guillermo Estrada wrote:

    Ok, so I wish there was a more "elegant" way of doing this, and maybe
    someone can point it out for me. Here is what I added to the code.

    rs := io.ReadSeeker(file)
    rs.Seek(84,0)
    b = bufio.NewReaderSize(rs, 50*size)

    1) Open File and create a bufio.Reader out of it to read the first 84
    bytes, with a 84 byte buffer ([]byte slice)
    2) Afterwards I had to create a new io.ReadSeeker of the same file, and
    seek to position 84 from the start of the file.
    3) Create a new bufio.Reader with buffer size of at least 50 bytes *
    TriangleBufferSize, no ensure I'll always read full triangles.
    4) Profit...

    Thing is to optimize as possible I'm reading 1000 Triangles at a time, But
    That buffer size is variable depending on the number of triangles in the
    file. But for that... I have to red the 84 first bytes of the file. And
    then create another 2 Readers of it? If I use the first bufio.Reader, I
    always start with a used buffer of X size and problems start. Is there a
    way to cleanup this mess in an idiomatic, elegant, non-c-monkey-hack way?

    Still, numbers aren't THAT great, but certainly are better than the
    alternatives. One thing that pops out is that 1000 Triangles seems to be
    the best choice in almost any situation. That checks out with something I
    read somewhere in this group about keeping buffers as close to but below
    65,535 bytes (50*1000 is close) to optimize them, something to do with
    register size on CPU and RAM and stuff.

    STL -file="Yoda_fixed.stl" -stepAnalysis
    Yoda_fixed.stl
    Parsing STL:
    220.1787ms (222.1846ms) 73.15 MB 60 Allocs
    File has 480844 triangles.
    Parsing STL (buffered) 100:
    225.2346ms (447.4192ms) 50.70 MB 43513 Allocs
    File has 480844 triangles.
    Parsing STL (buffered) 1000:
    213.1773ms (660.5965ms) 50.90 MB 4375 Allocs
    File has 480844 triangles.
    Parsing STL (buffered) 10000:
    216.1788ms (876.7753ms) 51.25 MB 483 Allocs
    File has 480844 triangles.

    Thanxs everyone, code is still here:
    https://gist.github.com/phrozen/6799694
    If someone with more experience with Readers and Buffers can give it a try
    and make observations on it, I will appreciate it a lot.

    On Wednesday, October 2, 2013 5:53:41 PM UTC-5, Guillermo Estrada wrote:

    Here is a simple Output of the program if helps, I still dunno why the
    first read is always length 80! (I already read 84 bytes out of the reader)
    I dunno if that is the problem, but as you can see, sometimes I get good
    results and sometimes bad. Maybe I have to set the buffer size of the
    reader? Ty.
    STL -file="GEAR.STL" -stepAnalysis
    GEAR.STL
    Parsing STL:
    5.0042ms (8.0069ms) 0.76 MB 56 Allocs
    File has 4748 triangles.
    true 0 4012 80 80
    true 80 5000 100 100
    true 180 5000 100 100
    true 280 5000 100 100
    true 380 5000 100 100
    false 480 5000 100 100
    true 580 5000 100 100
    true 680 5000 100 100
    false 780 5000 100 100
    true 880 5000 100 100
    false 980 5000 100 100
    true 1080 5000 100 100
    true 1180 5000 100 100
    false 1280 5000 100 100
    false 1380 5000 100 100
    false 1480 5000 100 100
    true 1580 5000 100 100
    true 1680 5000 100 100
    true 1780 5000 100 100
    false 1880 5000 100 100
    true 1980 5000 100 100
    true 2080 5000 100 100
    false 2180 5000 100 100
    true 2280 5000 100 100
    true 2380 5000 100 100
    false 2480 5000 100 100
    true 2580 5000 100 100
    false 2680 5000 100 100
    true 2780 5000 100 100
    true 2880 5000 100 100
    true 2980 5000 100 100
    true 3080 5000 100 100
    false 3180 5000 100 100
    true 3280 5000 100 100
    false 3380 5000 100 100
    true 3480 5000 100 100
    true 3580 5000 100 100
    true 3680 5000 100 100
    true 3780 5000 100 100
    true 3880 5000 100 100
    true 3980 5000 100 100
    true 4080 5000 100 100
    true 4180 5000 100 100
    true 4280 5000 100 100
    false 4380 5000 100 100
    true 4480 5000 100 100
    false 4580 5000 100 100
    true 4680 3388 67 67
    Parsing STL (buffered) 100:
    18.0157ms (26.0226ms) 0.61 MB 666 Allocs
    File has 4748 triangles.
    true 0 4012 80 80
    false 80 50000 1000 1000
    false 1080 50000 1000 1000
    false 2080 50000 1000 1000
    false 3080 50000 1000 1000
    false 4080 33388 667 667
    Parsing STL (buffered) 1000:
    5.0034ms (34.029ms) 0.69 MB 111 Allocs
    File has 4748 triangles.
    Saving ASCII:
    81.068ms (115.097ms) 0.28 MB 33263 Allocs
    Saving ASCII:
    77.0633ms (193.1612ms) 0.27 MB 33254 Allocs
    Saving ASCII:
    77.065ms (270.2262ms) 0.27 MB 33256 Allocs
    On Wednesday, October 2, 2013 5:49:09 PM UTC-5, Guillermo Estrada wrote:

    OK, I'm officially lost, I'm trying the copy slice form and it kinda
    works (and performs OK), But I'm getting sometimes the same values, and
    some other different values. No errors at all. Dunno if padding has
    something to do with it. Here is he code if someone wants to try it out.
    Just grab any STL file out there.

    https://gist.github.com/**phrozen/6799694<https://gist.github.com/phrozen/6799694>
    On Wednesday, October 2, 2013 5:08:49 PM UTC-5, Guillermo Estrada wrote:

    First of all I incremented the buffer size, but checking the result
    files converted to STL ASCII format, only the first Triangle is fine. So I
    guess I'll have to find a way to read a 1000 triangles and assign them to
    the slice in a single memory operation because binary.Read() only plays
    well with fixed size data, I guess I will have to create a secondary slice
    of the size and copy the data over to the Triangle slice (copy built in
    func maybe), that will surely put more overhead on memory, but at least
    won't put memory allocation calls to the roof.

    I'll keep testing code, profiling and I'll post back the results with
    the code. If anyone knows a way to use a buffered read of binary file and
    do large chunks assignment to a slice would be GREAT!
    On Wednesday, October 2, 2013 4:22:03 PM UTC-5, notnot wrote:

    excellent progress! what exactly did you change from the previous
    version?

    --
    You received this message because you are subscribed to the Google Groups
    "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an
    email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.
    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.
  • Jesse van den Kieboom at Oct 3, 2013 at 9:00 am
    I tried writing some code to parse binary STL files. The biggest bottleneck
    (on my PC) is reading from the harddisk. Reading it all at once seems to
    give the best performance I can measure. Assuming you are on a little
    endian machine, and that alignment causes the STL triangle struct to be 52
    bytes, you can try this code:

    http://play.golang.org/p/XjQbBuBeV7

    It's unsafe, of course, but should be pretty fast:

    63.628341ms (63.79001ms) 49.16 MB 15 Allocs

    That's of course on my PC and the disk is hot, so can't really compare
    those numbers with yours (it's on the Yoda_fixed.stl model).

    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.
  • Guillermo Estrada at Oct 3, 2013 at 2:41 pm
    Thanks Kyle! I was expecting your take on this, I have yet to implement
    those changes once I read about them again. Menwhile I tried Jesse's code
    and... here are some results. Uses the same memory as the buffered version,
    but its faster! (without Kyle's corrections yet...) I'll post more results
    once I correct the buffered one. Kudos!
    STL -file="xyzrgb_dragon.stl" -stepAnalysis
    C:\Users\highurz1\Desktop\dragon_recon\xyzrgb_dragon.stl

    Parsing STL:
             4.0142296s (4.0142296s) 1097.35 MB 57 Allocs
    File has 7219045 triangles.

    Parsing STL (buffered) 10000:
             3.8372194s (7.851449s) 740.84 MB 6561 Allocs
    File has 7219045 triangles.

    Parsing STL (unsafe):
             460.0264ms (8.3114754s) 736.37 MB 37 Allocs
    File has 7219045 triangles.
    On Thursday, October 3, 2013 4:00:18 AM UTC-5, Jesse van den Kieboom wrote:

    I tried writing some code to parse binary STL files. The biggest
    bottleneck (on my PC) is reading from the harddisk. Reading it all at once
    seems to give the best performance I can measure. Assuming you are on a
    little endian machine, and that alignment causes the STL triangle struct to
    be 52 bytes, you can try this code:

    http://play.golang.org/p/XjQbBuBeV7

    It's unsafe, of course, but should be pretty fast:

    63.628341ms (63.79001ms) 49.16 MB 15 Allocs

    That's of course on my PC and the disk is hot, so can't really compare
    those numbers with yours (it's on the Yoda_fixed.stl model).
    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.
  • Roger peppe at Oct 3, 2013 at 3:48 pm
    That will give the wrong result if run on a big-endian machine, no?
    Or if the Go compiler starts to use different alignment rules.

    Please let's not go this way

    On 3 October 2013 10:00, Jesse van den Kieboom wrote:
    I tried writing some code to parse binary STL files. The biggest bottleneck
    (on my PC) is reading from the harddisk. Reading it all at once seems to
    give the best performance I can measure. Assuming you are on a little endian
    machine, and that alignment causes the STL triangle struct to be 52 bytes,
    you can try this code:

    http://play.golang.org/p/XjQbBuBeV7

    It's unsafe, of course, but should be pretty fast:

    63.628341ms (63.79001ms) 49.16 MB 15 Allocs

    That's of course on my PC and the disk is hot, so can't really compare those
    numbers with yours (it's on the Yoda_fixed.stl model).

    --
    You received this message because you are subscribed to the Google Groups
    "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an
    email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.
    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.
  • Jesse van den Kieboom at Oct 3, 2013 at 4:30 pm

    On Thursday, October 3, 2013 5:48:30 PM UTC+2, rog wrote:

    That will give the wrong result if run on a big-endian machine, no?
    Or if the Go compiler starts to use different alignment rules.

    Please let's not go this way
    The standard library does the same thing where performance matters. What I
    do when I really care enough to use unsafe is to test at runtime for
    endianness and alignment and choose the fast path only when possible.
    What's wrong with that?

    On 3 October 2013 10:00, Jesse van den Kieboom wrote:
    I tried writing some code to parse binary STL files. The biggest
    bottleneck
    (on my PC) is reading from the harddisk. Reading it all at once seems to
    give the best performance I can measure. Assuming you are on a little endian
    machine, and that alignment causes the STL triangle struct to be 52 bytes,
    you can try this code:

    http://play.golang.org/p/XjQbBuBeV7

    It's unsafe, of course, but should be pretty fast:

    63.628341ms (63.79001ms) 49.16 MB 15 Allocs

    That's of course on my PC and the disk is hot, so can't really compare those
    numbers with yours (it's on the Yoda_fixed.stl model).

    --
    You received this message because you are subscribed to the Google Groups
    "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an
    email to golang-nuts...@googlegroups.com <javascript:>.
    For more options, visit https://groups.google.com/groups/opt_out.
    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.
  • Jan Mercl at Oct 3, 2013 at 4:49 pm

    On Thu, Oct 3, 2013 at 6:30 PM, Jesse van den Kieboom wrote:
    The standard library does the same thing where performance matters.
    I'm not aware of any single place where any stdlib code is not
    endianess agnostic. Can you please point out such place? I'll fill a
    bug in that case.

    -j

    PS: Really good reading is
    http://commandcenter.blogspot.de/2012/04/byte-order-fallacy.html

    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.
  • Jesse van den Kieboom at Oct 3, 2013 at 4:55 pm

    On Thursday, October 3, 2013 6:48:59 PM UTC+2, Jan Mercl wrote:

    On Thu, Oct 3, 2013 at 6:30 PM, Jesse van den Kieboom
    <jess...@gmail.com <javascript:>> wrote:
    The standard library does the same thing where performance matters.
    I'm not aware of any single place where any stdlib code is not
    endianess agnostic. Can you please point out such place? I'll fill a
    bug in that case.
    crypto/md5/gen.go

    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.
  • Jan Mercl at Oct 3, 2013 at 5:13 pm

    On Thu, Oct 3, 2013 at 6:55 PM, Jesse van den Kieboom wrote:
    crypto/md5/gen.go
    gen.go produces the same result regardless of endianess. The file this
    tool generates, md5block.go produces the same results regardless of
    endianess.

    Your code works for one specific endianess and fails for the other one.

    -j

    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.
  • Roger peppe at Oct 3, 2013 at 5:19 pm

    On 3 October 2013 18:12, Jan Mercl wrote:
    On Thu, Oct 3, 2013 at 6:55 PM, Jesse van den Kieboom
    wrote:
    crypto/md5/gen.go
    gen.go produces the same result regardless of endianess. The file this
    tool generates, md5block.go produces the same results regardless of
    endianess.

    Your code works for one specific endianess and fails for the other one.
    Not even mentioning potential alignment issues, which md5block.go
    copes with ok. Doing this right is awkward.

    It would be nice to have a version of the encoding/binary package
    that could optionally generate custom Go code for marshalling
    and unmarshalling given types. Hopefully the API would not
    need to change at all.

    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.
  • Jesse van den Kieboom at Oct 3, 2013 at 5:26 pm

    On Thursday, October 3, 2013 7:12:54 PM UTC+2, Jan Mercl wrote:

    On Thu, Oct 3, 2013 at 6:55 PM, Jesse van den Kieboom
    <jess...@gmail.com <javascript:>> wrote:
    crypto/md5/gen.go
    gen.go produces the same result regardless of endianess. The file this
    tool generates, md5block.go produces the same results regardless of
    endianess.

    Your code works for one specific endianess and fails for the other one.
    I also said, and I quote "What I do when I really care enough to use unsafe
    is to test at runtime for endianness and alignment and choose the fast path
    only when possible"

    -j
    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.
  • Jesse van den Kieboom at Oct 3, 2013 at 5:03 pm

    On Thursday, October 3, 2013 6:48:59 PM UTC+2, Jan Mercl wrote:

    PS: Really good reading is
    http://commandcenter.blogspot.de/2012/04/byte-order-fallacy.html
    Sure, I read that already, and I don't disagree that you should normally
    not care about endianness. But when your code ends up processing a file 4
    or 5 times faster and you _care_ about this performance, then in my opinion
    you can justify caring.

    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.
  • Guillermo Estrada at Oct 4, 2013 at 12:15 am
    Hey Kyle, I'm trying to go through all this changes, but I'm not used to
    the *Reader* Interface. Should I create a *bytes.NewReader* to feed into
    the *io.Reader* with a set size? One other thing, you mention the copy
    operation is useless and that I should read directly to the Triangle
    slice... thing is, how do I use a slice of my slice as a param in the *
    binary.Read* function. I already tried that, and only the first Triangle
    was copied correctly when I sent the pointer to the first element of the
    subslice.

    If you have an example of code with a *bytes.Reader* to send into my *
    io.ReaderSize* with a set sized buffer, that would be awesome! (and I can
    get rid of buf, and the *bytes.Buffer* I'm used to) Then I can just figure
    out the best way to copy the data to the Triangle slice with *binary.Read* or
    else. Thing is, I'm concerned how parsing a file requires at least twice
    the memory even while buffering the read... I suppose that should not
    happen and I hope you changes help me fix that.
    On Wednesday, October 2, 2013 9:27:11 PM UTC-5, Kyle Lemons wrote:

    a few things. Sorry I'm so late to the game. If I'm suggesting a repeat,
    let me know.

    https://gist.github.com/phrozen/6799694#file-main-go-L46
    Just start out by reading from the file directly. You don't know your
    buffer size yet, so hold off on the bufio.

    https://gist.github.com/phrozen/6799694#file-main-go-L48
    use io.ReadFull

    https://gist.github.com/phrozen/6799694#file-main-go-L51
    bytes.NewReader

    however, you should be able to read these straight out of the file, you
    shouldn't need to read it in and then make a bytes.Reader

    https://gist.github.com/phrozen/6799694#file-main-go-L70
    if you don't make a bufio.Reader above, you don't need to seek, and can
    just make the ReaderSize here.

    https://gist.github.com/phrozen/6799694#file-main-go-L89
    You've got some crazy double-buffering going on here. Get rid of "buf"
    and just read straight out of the bufio.Reader. When it empties, it will
    fill itself up, and then you can binary.Read straight out of it.

    https://gist.github.com/phrozen/6799694#file-main-go-L94
    I would be very surprised if this is helping you. Read straight into the
    triangle you're currently decoding. If your profile shows that you're
    spending a lot of time in reflect, unroll that binary.Read into a ReadFull
    and then the right sequence of calls to binary.LittleEndian.*
    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.
  • Kyle Lemons at Oct 4, 2013 at 6:40 am

    On Thu, Oct 3, 2013 at 5:15 PM, Guillermo Estrada wrote:

    Hey Kyle, I'm trying to go through all this changes, but I'm not used to
    the *Reader* Interface. Should I create a *bytes.NewReader* to feed into
    the *io.Reader* with a set size? One other thing, you mention the copy
    operation is useless and that I should read directly to the Triangle
    slice... thing is, how do I use a slice of my slice as a param in the *
    binary.Read* function. I already tried that, and only the first Triangle
    was copied correctly when I sent the pointer to the first element of the
    subslice.
    To feed into which io.Reader? The one for binary.Read? I didn't look at
    the code again, but I think it should look like this:

    f, err := os.Open(filename)
    if err != nil { ... }
    defer f.Close()

    var header Header
    if err := binary.Read(f, binary.LittleEndian, &header); err != nil { ... }

    triangles := make([]Triangle, header.Triangles)

    r := bufio.NewReaderSize(f, BufferTriangles*binary.Size(Triangle{}))
    for i := range triangles {
       if err := binary.Read(f, binary.LittleEndian, &triangles[i]); err != nil
    { ... }
    }

    If you have an example of code with a *bytes.Reader* to send into my *
    io.ReaderSize* with a set sized buffer, that would be awesome! (and I can
    get rid of buf, and the *bytes.Buffer* I'm used to) Then I can just
    figure out the best way to copy the data to the Triangle slice with *
    binary.Read* or else. Thing is, I'm concerned how parsing a file requires
    at least twice the memory even while buffering the read... I suppose that
    should not happen and I hope you changes help me fix that.

    On Wednesday, October 2, 2013 9:27:11 PM UTC-5, Kyle Lemons wrote:

    a few things. Sorry I'm so late to the game. If I'm suggesting a
    repeat, let me know.

    https://gist.github.com/**phrozen/6799694#file-main-go-**L46<https://gist.github.com/phrozen/6799694#file-main-go-L46>
    Just start out by reading from the file directly. You don't know your
    buffer size yet, so hold off on the bufio.

    https://gist.github.com/**phrozen/6799694#file-main-go-**L48<https://gist.github.com/phrozen/6799694#file-main-go-L48>
    use io.ReadFull

    https://gist.github.com/**phrozen/6799694#file-main-go-**L51<https://gist.github.com/phrozen/6799694#file-main-go-L51>
    bytes.NewReader

    however, you should be able to read these straight out of the file, you
    shouldn't need to read it in and then make a bytes.Reader

    https://gist.github.com/**phrozen/6799694#file-main-go-**L70<https://gist.github.com/phrozen/6799694#file-main-go-L70>
    if you don't make a bufio.Reader above, you don't need to seek, and can
    just make the ReaderSize here.

    https://gist.github.com/**phrozen/6799694#file-main-go-**L89<https://gist.github.com/phrozen/6799694#file-main-go-L89>
    You've got some crazy double-buffering going on here. Get rid of "buf"
    and just read straight out of the bufio.Reader. When it empties, it will
    fill itself up, and then you can binary.Read straight out of it.

    https://gist.github.com/**phrozen/6799694#file-main-go-**L94<https://gist.github.com/phrozen/6799694#file-main-go-L94>
    I would be very surprised if this is helping you. Read straight into the
    triangle you're currently decoding. If your profile shows that you're
    spending a lot of time in reflect, unroll that binary.Read into a ReadFull
    and then the right sequence of calls to binary.LittleEndian.*
    --
    You received this message because you are subscribed to the Google Groups
    "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an
    email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.
    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.
  • Kyle Lemons at Oct 4, 2013 at 6:41 am

    On Thu, Oct 3, 2013 at 11:40 PM, Kyle Lemons wrote:
    On Thu, Oct 3, 2013 at 5:15 PM, Guillermo Estrada wrote:

    Hey Kyle, I'm trying to go through all this changes, but I'm not used to
    the *Reader* Interface. Should I create a *bytes.NewReader* to feed into
    the *io.Reader* with a set size? One other thing, you mention the copy
    operation is useless and that I should read directly to the Triangle
    slice... thing is, how do I use a slice of my slice as a param in the *
    binary.Read* function. I already tried that, and only the first Triangle
    was copied correctly when I sent the pointer to the first element of the
    subslice.
    To feed into which io.Reader? The one for binary.Read? I didn't look at
    the code again, but I think it should look like this:

    f, err := os.Open(filename)
    if err != nil { ... }
    defer f.Close()

    var header Header
    if err := binary.Read(f, binary.LittleEndian, &header); err != nil { ... }

    triangles := make([]Triangle, header.Triangles)

    r := bufio.NewReaderSize(f, BufferTriangles*binary.Size(Triangle{}))
    for i := range triangles {
    if err := binary.Read(f, binary.LittleEndian, &triangles[i]); err != nil
    { ... }
    Whoops, obviously that should be "r" instead of "f"

    }

    If you have an example of code with a *bytes.Reader* to send into my *
    io.ReaderSize* with a set sized buffer, that would be awesome! (and I
    can get rid of buf, and the *bytes.Buffer* I'm used to) Then I can just
    figure out the best way to copy the data to the Triangle slice with *
    binary.Read* or else. Thing is, I'm concerned how parsing a file
    requires at least twice the memory even while buffering the read... I
    suppose that should not happen and I hope you changes help me fix that.

    On Wednesday, October 2, 2013 9:27:11 PM UTC-5, Kyle Lemons wrote:

    a few things. Sorry I'm so late to the game. If I'm suggesting a
    repeat, let me know.

    https://gist.github.com/**phrozen/6799694#file-main-go-**L46<https://gist.github.com/phrozen/6799694#file-main-go-L46>
    Just start out by reading from the file directly. You don't know your
    buffer size yet, so hold off on the bufio.

    https://gist.github.com/**phrozen/6799694#file-main-go-**L48<https://gist.github.com/phrozen/6799694#file-main-go-L48>
    use io.ReadFull

    https://gist.github.com/**phrozen/6799694#file-main-go-**L51<https://gist.github.com/phrozen/6799694#file-main-go-L51>
    bytes.NewReader

    however, you should be able to read these straight out of the file, you
    shouldn't need to read it in and then make a bytes.Reader

    https://gist.github.com/**phrozen/6799694#file-main-go-**L70<https://gist.github.com/phrozen/6799694#file-main-go-L70>
    if you don't make a bufio.Reader above, you don't need to seek, and can
    just make the ReaderSize here.

    https://gist.github.com/**phrozen/6799694#file-main-go-**L89<https://gist.github.com/phrozen/6799694#file-main-go-L89>
    You've got some crazy double-buffering going on here. Get rid of "buf"
    and just read straight out of the bufio.Reader. When it empties, it will
    fill itself up, and then you can binary.Read straight out of it.

    https://gist.github.com/**phrozen/6799694#file-main-go-**L94<https://gist.github.com/phrozen/6799694#file-main-go-L94>
    I would be very surprised if this is helping you. Read straight into
    the triangle you're currently decoding. If your profile shows that you're
    spending a lot of time in reflect, unroll that binary.Read into a ReadFull
    and then the right sequence of calls to binary.LittleEndian.*
    --
    You received this message because you are subscribed to the Google Groups
    "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an
    email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.
    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.
  • Jesse van den Kieboom at Oct 4, 2013 at 9:52 am
    Just to put some measurements on my claims, I compared both reading unsafe
    and reading using standard reflect. The two functions I used are
    on http://play.golang.org/p/HlAFRnxQ-K

    STL unsafe:
    49.527777ms (49.69955ms) 25.05 MB 10 Allocs

    STL reflect:
    974.764166ms (974.900398ms) 99.02 MB 3371574 Allocs

    This is on the Yoda model. Barring that I didn't make any mistakes in the
    implementation, using reflect is about 20 times slower on my machine and
    uses about 4 times more memory.
    On Friday, October 4, 2013 8:40:57 AM UTC+2, Kyle Lemons wrote:




    On Thu, Oct 3, 2013 at 11:40 PM, Kyle Lemons <kev...@google.com<javascript:>
    wrote:
    On Thu, Oct 3, 2013 at 5:15 PM, Guillermo Estrada <phro...@gmail.com<javascript:>
    wrote:
    Hey Kyle, I'm trying to go through all this changes, but I'm not used to
    the *Reader* Interface. Should I create a *bytes.NewReader* to feed
    into the *io.Reader* with a set size? One other thing, you mention the
    copy operation is useless and that I should read directly to the Triangle
    slice... thing is, how do I use a slice of my slice as a param in the *
    binary.Read* function. I already tried that, and only the first
    Triangle was copied correctly when I sent the pointer to the first element
    of the subslice.
    To feed into which io.Reader? The one for binary.Read? I didn't look at
    the code again, but I think it should look like this:

    f, err := os.Open(filename)
    if err != nil { ... }
    defer f.Close()

    var header Header
    if err := binary.Read(f, binary.LittleEndian, &header); err != nil { ... }

    triangles := make([]Triangle, header.Triangles)

    r := bufio.NewReaderSize(f, BufferTriangles*binary.Size(Triangle{}))
    for i := range triangles {
    if err := binary.Read(f, binary.LittleEndian, &triangles[i]); err !=
    nil { ... }
    Whoops, obviously that should be "r" instead of "f"

    }

    If you have an example of code with a *bytes.Reader* to send into my *
    io.ReaderSize* with a set sized buffer, that would be awesome! (and I
    can get rid of buf, and the *bytes.Buffer* I'm used to) Then I can just
    figure out the best way to copy the data to the Triangle slice with *
    binary.Read* or else. Thing is, I'm concerned how parsing a file
    requires at least twice the memory even while buffering the read... I
    suppose that should not happen and I hope you changes help me fix that.

    On Wednesday, October 2, 2013 9:27:11 PM UTC-5, Kyle Lemons wrote:

    a few things. Sorry I'm so late to the game. If I'm suggesting a
    repeat, let me know.

    https://gist.github.com/**phrozen/6799694#file-main-go-**L46<https://gist.github.com/phrozen/6799694#file-main-go-L46>
    Just start out by reading from the file directly. You don't know your
    buffer size yet, so hold off on the bufio.

    https://gist.github.com/**phrozen/6799694#file-main-go-**L48<https://gist.github.com/phrozen/6799694#file-main-go-L48>
    use io.ReadFull

    https://gist.github.com/**phrozen/6799694#file-main-go-**L51<https://gist.github.com/phrozen/6799694#file-main-go-L51>
    bytes.NewReader

    however, you should be able to read these straight out of the file, you
    shouldn't need to read it in and then make a bytes.Reader

    https://gist.github.com/**phrozen/6799694#file-main-go-**L70<https://gist.github.com/phrozen/6799694#file-main-go-L70>
    if you don't make a bufio.Reader above, you don't need to seek, and can
    just make the ReaderSize here.

    https://gist.github.com/**phrozen/6799694#file-main-go-**L89<https://gist.github.com/phrozen/6799694#file-main-go-L89>
    You've got some crazy double-buffering going on here. Get rid of "buf"
    and just read straight out of the bufio.Reader. When it empties, it will
    fill itself up, and then you can binary.Read straight out of it.

    https://gist.github.com/**phrozen/6799694#file-main-go-**L94<https://gist.github.com/phrozen/6799694#file-main-go-L94>
    I would be very surprised if this is helping you. Read straight into
    the triangle you're currently decoding. If your profile shows that you're
    spending a lot of time in reflect, unroll that binary.Read into a ReadFull
    and then the right sequence of calls to binary.LittleEndian.*
    --
    You received this message because you are subscribed to the Google
    Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send
    an email to golang-nuts...@googlegroups.com <javascript:>.
    For more options, visit https://groups.google.com/groups/opt_out.
    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.
  • Jan Mercl at Oct 4, 2013 at 4:38 pm

    On Fri, Oct 4, 2013 at 11:52 AM, Jesse van den Kieboom wrote:
    Just to put some measurements on my claims, I compared both reading unsafe
    and reading using standard reflect. The two functions I used are on
    http://play.golang.org/p/HlAFRnxQ-K

    STL unsafe:
    49.527777ms (49.69955ms) 25.05 MB 10 Allocs

    STL reflect:
    974.764166ms (974.900398ms) 99.02 MB 3371574 Allocs
    Comparing the performance against a solution using reflect makes no
    sense as you can write code running correctly regardless of endianess
    without using whatsoever from the reflect package, which is okay per
    se, but inevitably principally slow in many, if not most cases.

    -j

    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.
  • Jesse van den Kieboom at Oct 4, 2013 at 8:11 pm

    On Friday, October 4, 2013 6:38:30 PM UTC+2, Jan Mercl wrote:
    On Fri, Oct 4, 2013 at 11:52 AM, Jesse van den Kieboom
    <jess...@gmail.com <javascript:>> wrote:
    Just to put some measurements on my claims, I compared both reading unsafe
    and reading using standard reflect. The two functions I used are on
    http://play.golang.org/p/HlAFRnxQ-K

    STL unsafe:
    49.527777ms (49.69955ms) 25.05 MB 10 Allocs

    STL reflect:
    974.764166ms (974.900398ms) 99.02 MB 3371574 Allocs
    Comparing the performance against a solution using reflect makes no
    sense

    Oh come on, of course it makes sense. Using reflect would be the cleanest
    way to implement this, and it's also been suggested in this topic. It is
    both valid and valuable to evaluate how much performance difference there
    is between the cleanest and fastest code to accomplish reading a binary
    file.

    as you can write code running correctly regardless of endianess
    without using whatsoever from the reflect package, which is okay per
    se, but inevitably principally slow in many, if not most cases

    Less talk, more numbers


    .

    -j
    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.
  • Jan Mercl at Oct 4, 2013 at 8:18 pm

    On Fri, Oct 4, 2013 at 10:11 PM, Jesse van den Kieboom wrote:
    Oh come on, of course it makes sense. Using reflect would be the cleanest
    way to implement this, and it's also been suggested in this topic.
    Please define what you mean by "cleanest". Reflect is guaranteed to be
    the slowest option in your usage case. Some careless programmers abuse
    both computer's resources and the reflect package - instead of writing
    sometimes a bit more code.

    Everyone is free to use what she wants, but if the topic is reaching
    maximum possible speed/performance then using reflect is the wrong
    answer in about every case.

    That said, it's not reflect's fault when used where not appropriate.

    -j

    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.
  • Jesse van den Kieboom at Oct 4, 2013 at 8:44 pm
    Fine, so I also implemented using just binary.LittleEndian.* and here are
    the results:

    STL reflect:
    981.812881ms (981.975427ms) 99.02 MB 3371579 Allocs
    STL binary:
    886.004605ms (1.868050023s) 25.02 MB 14 Allocs
    STL unsafe:
    42.3406ms (1.910452496s) 25.04 MB 9 Allocs

    The memory usage is now down, so no more allocations (the second case), but
    the time for parsing hasn't decreased much. The unsafe method is still an
    order of magnitude faster...
    On Friday, October 4, 2013 10:18:31 PM UTC+2, Jan Mercl wrote:

    On Fri, Oct 4, 2013 at 10:11 PM, Jesse van den Kieboom
    <jess...@gmail.com <javascript:>> wrote:
    Oh come on, of course it makes sense. Using reflect would be the cleanest
    way to implement this, and it's also been suggested in this topic.
    Please define what you mean by "cleanest". Reflect is guaranteed to be
    the slowest option in your usage case. Some careless programmers abuse
    both computer's resources and the reflect package - instead of writing
    sometimes a bit more code.

    Everyone is free to use what she wants, but if the topic is reaching
    maximum possible speed/performance then using reflect is the wrong
    answer in about every case.

    That said, it's not reflect's fault when used where not appropriate.

    -j
    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.
  • Jesse van den Kieboom at Oct 4, 2013 at 8:58 pm
    I realized that my implementation of the binary case was not completely
    comparable because it didn't use the same batching of reading from the
    io.Reader. I thought that since I'm using a buffered io, it wouldn't make a
    big difference, but it seems doing small io.ReadFull can hurt a lot. In any
    case, with the same batching the results are:

    STL reflect:
    983.321431ms (983.47328ms) 99.02 MB 3371579 Allocs
    STL binary:
    115.144052ms (1.098696975s) 25.10 MB 18 Allocs
    STL unsafe:
    39.730116ms (1.138491161s) 25.04 MB 9 Allocs

    So no longer an order of magnitude of difference, but unsafe is still ~3
    times faster.
    On Friday, October 4, 2013 10:44:43 PM UTC+2, Jesse van den Kieboom wrote:

    Fine, so I also implemented using just binary.LittleEndian.* and here are
    the results:

    STL reflect:
    981.812881ms (981.975427ms) 99.02 MB 3371579 Allocs
    STL binary:
    886.004605ms (1.868050023s) 25.02 MB 14 Allocs
    STL unsafe:
    42.3406ms (1.910452496s) 25.04 MB 9 Allocs

    The memory usage is now down, so no more allocations (the second case),
    but the time for parsing hasn't decreased much. The unsafe method is still
    an order of magnitude faster...
    On Friday, October 4, 2013 10:18:31 PM UTC+2, Jan Mercl wrote:

    On Fri, Oct 4, 2013 at 10:11 PM, Jesse van den Kieboom
    wrote:
    Oh come on, of course it makes sense. Using reflect would be the cleanest
    way to implement this, and it's also been suggested in this topic.
    Please define what you mean by "cleanest". Reflect is guaranteed to be
    the slowest option in your usage case. Some careless programmers abuse
    both computer's resources and the reflect package - instead of writing
    sometimes a bit more code.

    Everyone is free to use what she wants, but if the topic is reaching
    maximum possible speed/performance then using reflect is the wrong
    answer in about every case.

    That said, it's not reflect's fault when used where not appropriate.

    -j
    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.
  • Guillermo Estrada at Oct 4, 2013 at 9:05 pm
    Hey Jesse, I'm keeping an eye in that binary implementation, I will try
    what Kyle suggests and write my own Read func with ReadFull and
    binary.LittleEndian, and compare results.
    Just one thing.. are you checking the data being parsed? One of my
    implementations ran like hell with almost no memory overhead (much like
    unsafe), until I realized... not all memory was being copied correctly. I
    mean, less ram, more speed, etc... but in the end you need the right data...
    On Friday, October 4, 2013 3:58:46 PM UTC-5, Jesse van den Kieboom wrote:

    I realized that my implementation of the binary case was not completely
    comparable because it didn't use the same batching of reading from the
    io.Reader. I thought that since I'm using a buffered io, it wouldn't make a
    big difference, but it seems doing small io.ReadFull can hurt a lot. In any
    case, with the same batching the results are:

    STL reflect:
    983.321431ms (983.47328ms) 99.02 MB 3371579 Allocs
    STL binary:
    115.144052ms (1.098696975s) 25.10 MB 18 Allocs
    STL unsafe:
    39.730116ms (1.138491161s) 25.04 MB 9 Allocs

    So no longer an order of magnitude of difference, but unsafe is still ~3
    times faster.
    On Friday, October 4, 2013 10:44:43 PM UTC+2, Jesse van den Kieboom wrote:

    Fine, so I also implemented using just binary.LittleEndian.* and here are
    the results:

    STL reflect:
    981.812881ms (981.975427ms) 99.02 MB 3371579 Allocs
    STL binary:
    886.004605ms (1.868050023s) 25.02 MB 14 Allocs
    STL unsafe:
    42.3406ms (1.910452496s) 25.04 MB 9 Allocs

    The memory usage is now down, so no more allocations (the second case),
    but the time for parsing hasn't decreased much. The unsafe method is still
    an order of magnitude faster...
    On Friday, October 4, 2013 10:18:31 PM UTC+2, Jan Mercl wrote:

    On Fri, Oct 4, 2013 at 10:11 PM, Jesse van den Kieboom
    wrote:
    Oh come on, of course it makes sense. Using reflect would be the cleanest
    way to implement this, and it's also been suggested in this topic.
    Please define what you mean by "cleanest". Reflect is guaranteed to be
    the slowest option in your usage case. Some careless programmers abuse
    both computer's resources and the reflect package - instead of writing
    sometimes a bit more code.

    Everyone is free to use what she wants, but if the topic is reaching
    maximum possible speed/performance then using reflect is the wrong
    answer in about every case.

    That said, it's not reflect's fault when used where not appropriate.

    -j
    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.
  • Paul Gruenbacher at Aug 7, 2015 at 2:17 am
    Hi Guillermo,
    Came across this thread, looking to implement an stl binary/ascii
    converter. What implementation did you end up using?
    Paul
    On Friday, October 4, 2013 at 5:05:04 PM UTC-4, Guillermo Estrada wrote:

    Hey Jesse, I'm keeping an eye in that binary implementation, I will try
    what Kyle suggests and write my own Read func with ReadFull and
    binary.LittleEndian, and compare results.
    Just one thing.. are you checking the data being parsed? One of my
    implementations ran like hell with almost no memory overhead (much like
    unsafe), until I realized... not all memory was being copied correctly. I
    mean, less ram, more speed, etc... but in the end you need the right data...
    On Friday, October 4, 2013 3:58:46 PM UTC-5, Jesse van den Kieboom wrote:

    I realized that my implementation of the binary case was not completely
    comparable because it didn't use the same batching of reading from the
    io.Reader. I thought that since I'm using a buffered io, it wouldn't make a
    big difference, but it seems doing small io.ReadFull can hurt a lot. In any
    case, with the same batching the results are:

    STL reflect:
    983.321431ms (983.47328ms) 99.02 MB 3371579 Allocs
    STL binary:
    115.144052ms (1.098696975s) 25.10 MB 18 Allocs
    STL unsafe:
    39.730116ms (1.138491161s) 25.04 MB 9 Allocs

    So no longer an order of magnitude of difference, but unsafe is still ~3
    times faster.
    On Friday, October 4, 2013 10:44:43 PM UTC+2, Jesse van den Kieboom wrote:

    Fine, so I also implemented using just binary.LittleEndian.* and here
    are the results:

    STL reflect:
    981.812881ms (981.975427ms) 99.02 MB 3371579 Allocs
    STL binary:
    886.004605ms (1.868050023s) 25.02 MB 14 Allocs
    STL unsafe:
    42.3406ms (1.910452496s) 25.04 MB 9 Allocs

    The memory usage is now down, so no more allocations (the second case),
    but the time for parsing hasn't decreased much. The unsafe method is still
    an order of magnitude faster...
    On Friday, October 4, 2013 10:18:31 PM UTC+2, Jan Mercl wrote:

    On Fri, Oct 4, 2013 at 10:11 PM, Jesse van den Kieboom
    wrote:
    Oh come on, of course it makes sense. Using reflect would be the cleanest
    way to implement this, and it's also been suggested in this topic.
    Please define what you mean by "cleanest". Reflect is guaranteed to be
    the slowest option in your usage case. Some careless programmers abuse
    both computer's resources and the reflect package - instead of writing
    sometimes a bit more code.

    Everyone is free to use what she wants, but if the topic is reaching
    maximum possible speed/performance then using reflect is the wrong
    answer in about every case.

    That said, it's not reflect's fault when used where not appropriate.

    -j
    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
  • Guillermo Estrada at Oct 4, 2013 at 2:22 pm
    Hey Kyle, I implemented your ideas and I really like themthe code looks
    awesome, but after testing, it seems kinda suboptimal. I think that
    iterating over each triangle is pretty slow compared over copying chunks of
    triangle slices. Hope to hear your thoughts on this cause your method is
    hurting on RAM a LOT.

    C:\Users\highurz1\Desktop\dragon_recon\xyzrgb_dragon.stl
    Parsing STL:
             3.9622266s (3.9622266s) 1097.35 MB 57 Allocs
    File has 7219045 triangles.

    Parsing STL (buffered) Kyle:
             9.7515577s (13.7137843s) 1486.41 MB 50625332 Allocs
    File has 7219045 triangles.

    Parsing STL (buffered):
             3.986228s (17.7000123s) 752.17 MB 70691 Allocs
    File has 7219045 triangles.

    Parsing STL (unsafe):
             447.0256ms (18.148038s) 736.35 MB 26 Allocs
    File has 7219045 triangles.

    And here is the code I used (love the code though, clean and simple)...

    func ParseBufferedSTLKyle(filename string) *Model {
       f, err := os.Open(filename)
       if err != nil { panic(err) }
       defer f.Close()

       m := new(Model)

       if err := binary.Read(f, binary.LittleEndian, &m.Header); err != nil {
    panic(err) }
       if err := binary.Read(f, binary.LittleEndian, &m.Length); err != nil {
    panic(err) }

       m.Triangles = make([]Triangle, m.Length)

       r := bufio.NewReaderSize(f, 1024*binary.Size(Triangle{}))

       for i := range m.Triangles {
         if err := binary.Read(r, binary.LittleEndian, &m.Triangles[i]); err !=
    nil { panic(err) }
       }

       return m
    }


    To feed into which io.Reader? The one for binary.Read? I didn't look at
    the code again, but I think it should look like this:

    f, err := os.Open(filename)
    if err != nil { ... }
    defer f.Close()

    var header Header
    if err := binary.Read(f, binary.LittleEndian, &header); err != nil { ... }

    triangles := make([]Triangle, header.Triangles)

    r := bufio.NewReaderSize(f, BufferTriangles*binary.Size(Triangle{}))
    for i := range triangles {
    if err := binary.Read(f, binary.LittleEndian, &triangles[i]); err != nil
    { ... }
    }
    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.
  • Guillermo Estrada at Oct 4, 2013 at 2:42 pm
    Switched the main loop for something like this:

    for i := 0; i < length; i = i + bufferSize {
       if length - i < bufferSize {
         bufferSize = length - i
       }
       if err := binary.Read(r, binary.LittleEndian,
    m.Triangles[i:i+bufferSize]); err != nil { panic(err) }
    }

    And got WAY better results!!!

    Parsing STL (buffered) Kyle:
             3.9282246s (8.0194587s) 751.77 MB 63565 Allocs
    File has 7219045 triangles.

    Pretty much the same as my buffered version but using less RAM! I have to
    play with the buffer size to determine what is the best size depending on
    the file. But I don't see any other way to optimize this further.

    Jesse's version with unsafe pkg is FAST! but uses a lot of RAM too. I would
    like to parse it slow and then do something like binary.Marshal and save
    the optimized format for future use.

    Does anybody know if a Marshalling system for binary structs exist? I think
    notnot or someone did mentioned it here earlier.
    On Friday, October 4, 2013 9:22:20 AM UTC-5, Guillermo Estrada wrote:

    Hey Kyle, I implemented your ideas and I really like themthe code looks
    awesome, but after testing, it seems kinda suboptimal. I think that
    iterating over each triangle is pretty slow compared over copying chunks of
    triangle slices. Hope to hear your thoughts on this cause your method is
    hurting on RAM a LOT.

    C:\Users\highurz1\Desktop\dragon_recon\xyzrgb_dragon.stl
    Parsing STL:
    3.9622266s (3.9622266s) 1097.35 MB 57 Allocs
    File has 7219045 triangles.

    Parsing STL (buffered) Kyle:
    9.7515577s (13.7137843s) 1486.41 MB 50625332 Allocs
    File has 7219045 triangles.

    Parsing STL (buffered):
    3.986228s (17.7000123s) 752.17 MB 70691 Allocs
    File has 7219045 triangles.

    Parsing STL (unsafe):
    447.0256ms (18.148038s) 736.35 MB 26 Allocs
    File has 7219045 triangles.

    And here is the code I used (love the code though, clean and simple)...

    func ParseBufferedSTLKyle(filename string) *Model {
    f, err := os.Open(filename)
    if err != nil { panic(err) }
    defer f.Close()

    m := new(Model)

    if err := binary.Read(f, binary.LittleEndian, &m.Header); err != nil {
    panic(err) }
    if err := binary.Read(f, binary.LittleEndian, &m.Length); err != nil {
    panic(err) }

    m.Triangles = make([]Triangle, m.Length)

    r := bufio.NewReaderSize(f, 1024*binary.Size(Triangle{}))

    for i := range m.Triangles {
    if err := binary.Read(r, binary.LittleEndian, &m.Triangles[i]); err !=
    nil { panic(err) }
    }

    return m
    }


    To feed into which io.Reader? The one for binary.Read? I didn't look at
    the code again, but I think it should look like this:

    f, err := os.Open(filename)
    if err != nil { ... }
    defer f.Close()

    var header Header
    if err := binary.Read(f, binary.LittleEndian, &header); err != nil { ... }

    triangles := make([]Triangle, header.Triangles)

    r := bufio.NewReaderSize(f, BufferTriangles*binary.Size(Triangle{}))
    for i := range triangles {
    if err := binary.Read(f, binary.LittleEndian, &triangles[i]); err !=
    nil { ... }
    }
    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupgolang-nuts @
categoriesgo
postedOct 1, '13 at 2:42a
activeAug 7, '15 at 2:17a
posts52
users9
websitegolang.org

People

Translate

site design / logo © 2021 Grokbase