FAQ
Hi Dave,

Thanks for your advice, it helps me to learn more APIs about GoLang :)

I tried io.LimitReader in a for LOOP
https://github.com/xiangzhai/goaxel/blob/master/test/http.go#L71

for {
          lr := io.LimitReader(resp.Body, buffer_size)
          n, err := io.ReadAtLeast(lr, data, int(buffer_size))
          if err != nil { return }
          f.WriteAt(data, int64(h.offset))
          if h.Callback != nil {
              h.Callback(n)
          }
          h.offset += n
      }

But if buffer_size was 102400, io.LimitReader might be blocked (I hop it
is my misunderstanding of io.LimitReader), then set buffer_size to
10240, it worked...
And output file was diff from the original one, run command diff -a
output /usr/share/nginx/html/original in my Linux box, it is able to
copy the test/http.go to conn/http.go to run the test by go run
goaxel.go http://localhost/original

Leslie
On Thu, Nov 28, 2013 at 12:41 PM, Leslie Zhai wrote:
Hi Dave,

I use net.Conn.Read(data []byte) in a for LOOP, such as
for {
data := make([]byte, buffer_size)
n, err := http.conn.Read(data)
if err != nil {
return
}
f.WriteAt(data[:n], int64(http.offset))
if http.Callback != nil {
http.Callback(n)
}
http.offset += n
}
And I tested it using 3 connctions number (with/without) limited download
speed, net.Conn.Read() in a for LOOP worked well :)

To io.Copy(chunkFile, resp.Body), because it is using multiply connections
for one file, it need to create several (depend on connection number) chunk
files, when download finished, combined chunks into one output file
https://github.com/xiangzhai/goaxel/blob/25e604de2e0c016bf8bd759f7e7bcd8468b9cf5e/goaxel.go#L119
If directly io.Copy(outputFile, resp.Body) without chunk files, the
goroutines might io.Copy the output file with wrong file position indicator,
for example, downloaded a PNG image, the output file might be disordered.
Use an io.LimitedReader
Leslie

You need to re-read the contact for io.Reader, specifically your code
may loose the last block of data read.

On Thu, Nov 28, 2013 at 12:13 PM, Leslie Zhai wrote:

Hi Jesse,

Thanks for your reply :)

I tried io.Copy(file, resp.Body), it might be blocked too when limited the
download speed by setting limit_rate 10k for nginx`s configuration.
So I use net.Conn.Read(data []byte) in a for LOOP
https://github.com/xiangzhai/goaxel/blob/master/conn/http.go#L129

Leslie

On Wed, Nov 27, 2013 at 9:15 PM, Zhai Xiang wrote:

To HTTP protocol, there is already http package provides HTTP client and
server implementations in GoLang, but ioutil.ReadAll(resp.Body) will be
blocked when downloading SUPER ARGE file with VERY SLOW down speed
https://github.com/tuxcanfly/godown/blob/master/godown.go#L68

Don't use ioutil.ReadAll() will allocate space in memory for the whole of
resp.Body. This will use a lot of memory for a large file and if the file is
large enough you might run out of memory.

If you want to handle large input then you shouldn't use ioutil.ReadAll(),
you should stream the resp.Body to disk by creating a file using os.Create()
and io.Copy(file, resp.Body)

--
=====================
http://jessta.id.au


--
You received this message because you are subscribed to the Google Groups
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to golang-nuts+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Search Discussions

Discussion Posts

Previous

Follow ups

Related Discussions

Discussion Navigation
viewthread | post
posts ‹ prev | 10 of 13 | next ›
Discussion Overview
groupgolang-nuts @
categoriesgo
postedNov 27, '13 at 10:15a
activeApr 15, '14 at 10:08a
posts13
users5
websitegolang.org

People

Translate

site design / logo © 2022 Grokbase