Hi all,

   I've hit an interesting problem, and I was a bit surprised that there
isn't anything in the standard libs that could have solved it easily. It
isn't too complicated to write, but it isn't trivial either. If by any
chance it's already in the libs, please enlighten me :), otherwise would
anyone be interested in including it?

   The thing I was solving is fairly trivial: download a file from the
internet, and stream-upload it somewhere else (Google Cloud Storage
specifically, but it doesn't really matter). The naive solution is pretty
straightforward: wire together the downloader's reader with the uploader's
writer, and voila, magic... until you look at the network usage: x1 secs
download, y1 secs upload, x2 secs download, y2 secs upload.

   The problem is, that the uploader will read some fixed amount of data,
buffer it up and then start the upload. But while the upload is in
progress, it doesn't read any more data from the reader, essentially
pausing it until it finishes. By that time the reader could have filled up
the next buffer to send, but alas, it was blocked so it didn't download

   Note, buffered readers/writers won't really solve this issue, since even
though they have the buffers in place to store arriving data, those buffers
cannot be simultaneously filled and flushed too. As far as I figured it
out, the only way to solve this streaming problem is to have to concurrent
go routines, one reading and another writing, with a data buffer in between.

   If there is indeed no such thing, would it be worthwhile to add something

bufio.Copy(dst io.Writer, src io.Reader, buffer int) (written int64, err

   Which essentially does what io.Copy does, but starts up a separate writer
go routine and passes everything through a user definable buffer. Then it
could handle both data bursts as well batching readers/writers.

   Comments/feedback? :)


You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Search Discussions

Discussion Posts

Follow ups

Related Discussions

Discussion Navigation
viewthread | post
posts ‹ prev | 1 of 22 | next ›
Discussion Overview
groupgolang-nuts @
postedJan 28, '15 at 8:18a
activeJan 30, '15 at 1:02p



site design / logo © 2021 Grokbase