FAQ
Hi Dave,

I use net.Conn.Read(data []byte) in a for LOOP, such as
for {
          data := make([]byte, buffer_size)
          n, err := http.conn.Read(data)
          if err != nil {
return
}
          f.WriteAt(data[:n], int64(http.offset))
          if http.Callback != nil {
http.Callback(n)
}
          http.offset += n
      }
And I tested it using 3 connctions number (with/without) limited
download speed, net.Conn.Read() in a for LOOP worked well :)

To io.Copy(chunkFile, resp.Body), because it is using multiply
connections for one file, it need to create several (depend on
connection number) chunk files, when download finished, combined chunks
into one output file
https://github.com/xiangzhai/goaxel/blob/25e604de2e0c016bf8bd759f7e7bcd8468b9cf5e/goaxel.go#L119
If directly io.Copy(outputFile, resp.Body) without chunk files, the
goroutines might io.Copy the output file with wrong file position
indicator, for example, downloaded a PNG image, the output file might be
disordered.

Leslie
You need to re-read the contact for io.Reader, specifically your code
may loose the last block of data read.
On Thu, Nov 28, 2013 at 12:13 PM, Leslie Zhai wrote:
Hi Jesse,

Thanks for your reply :)

I tried io.Copy(file, resp.Body), it might be blocked too when limited the
download speed by setting limit_rate 10k for nginx`s configuration.
So I use net.Conn.Read(data []byte) in a for LOOP
https://github.com/xiangzhai/goaxel/blob/master/conn/http.go#L129

Leslie
On Wed, Nov 27, 2013 at 9:15 PM, Zhai Xiang wrote:
To HTTP protocol, there is already http package provides HTTP client and
server implementations in GoLang, but ioutil.ReadAll(resp.Body) will be
blocked when downloading SUPER ARGE file with VERY SLOW down speed
https://github.com/tuxcanfly/godown/blob/master/godown.go#L68
Don't use ioutil.ReadAll() will allocate space in memory for the whole of
resp.Body. This will use a lot of memory for a large file and if the file is
large enough you might run out of memory.

If you want to handle large input then you shouldn't use ioutil.ReadAll(),
you should stream the resp.Body to disk by creating a file using os.Create()
and io.Copy(file, resp.Body)

--
=====================
http://jessta.id.au


--
You received this message because you are subscribed to the Google Groups
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to golang-nuts+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Search Discussions

Discussion Posts

Previous

Follow ups

Related Discussions

Discussion Navigation
viewthread | post
posts ‹ prev | 5 of 13 | next ›
Discussion Overview
groupgolang-nuts @
categoriesgo
postedNov 27, '13 at 10:15a
activeApr 15, '14 at 10:08a
posts13
users5
websitegolang.org

People

Translate

site design / logo © 2022 Grokbase