An initial attempt at getting a buffered concurrent copy. It's a tad more
than a dozen lines, more like 200 :) I haven't tested it too extensively,
nor cleaned it up too much, just a rather quick hack together to see if the
concept works.


Opinions? :)

On Wed, Jan 28, 2015 at 3:09 PM, Nick Craig-Wood wrote:
On 28/01/15 08:18, Péter Szilágyi wrote:
I've hit an interesting problem, and I was a bit surprised that there
isn't anything in the standard libs that could have solved it easily. It
isn't too complicated to write, but it isn't trivial either. If by any
chance it's already in the libs, please enlighten me :), otherwise would
anyone be interested in including it?

The thing I was solving is fairly trivial: download a file from the
internet, and stream-upload it somewhere else (Google Cloud Storage
specifically, but it doesn't really matter). The naive solution is
pretty straightforward: wire together the downloader's reader with the
uploader's writer, and voila, magic... until you look at the network
usage: x1 secs download, y1 secs upload, x2 secs download, y2 secs

I came across exactly the same problem yesterday!

I was reading, gzipping and uploading, but I had exactly the same
problem - 10 seconds of 100% CPU for gzipping into a 64 MB block, then
20 seconds of upload at 0% CPU.

Here was my solution


Which is an annoying amount of code. The error handling is tricky too.
bufio.Copy(dst io.Writer, src io.Reader, buffer int) (written int64, err

Which essentially does what io.Copy does, but starts up a separate
writer go routine and passes everything through a user definable buffer.
Then it could handle both data bursts as well batching readers/writers.
I like it!

Nick Craig-Wood <nick@craig-wood.com> -- http://www.craig-wood.com/nick
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Search Discussions

Discussion Posts


Follow ups

Related Discussions

Discussion Navigation
viewthread | post
posts ‹ prev | 9 of 22 | next ›
Discussion Overview
groupgolang-nuts @
postedJan 28, '15 at 8:18a
activeJan 30, '15 at 1:02p



site design / logo © 2021 Grokbase