than a dozen lines, more like 200 :) I haven't tested it too extensively,
nor cleaned it up too much, just a rather quick hack together to see if the
concept works.
https://gist.github.com/karalabe/6de57007034d972b9ab6
Opinions? :)
On Wed, Jan 28, 2015 at 3:09 PM, Nick Craig-Wood wrote:
I came across exactly the same problem yesterday!
I was reading, gzipping and uploading, but I had exactly the same
problem - 10 seconds of 100% CPU for gzipping into a 64 MB block, then
20 seconds of upload at 0% CPU.
Here was my solution
https://github.com/Memset/snapshot-manager/blob/master/snapshot/snapshot.go#L166
Which is an annoying amount of code. The error handling is tricky too.
--
Nick Craig-Wood <nick@craig-wood.com> -- http://www.craig-wood.com/nick
--On 28/01/15 08:18, Péter Szilágyi wrote:
I've hit an interesting problem, and I was a bit surprised that there
isn't anything in the standard libs that could have solved it easily. It
isn't too complicated to write, but it isn't trivial either. If by any
chance it's already in the libs, please enlighten me :), otherwise would
anyone be interested in including it?
The thing I was solving is fairly trivial: download a file from the
internet, and stream-upload it somewhere else (Google Cloud Storage
specifically, but it doesn't really matter). The naive solution is
pretty straightforward: wire together the downloader's reader with the
uploader's writer, and voila, magic... until you look at the network
usage: x1 secs download, y1 secs upload, x2 secs download, y2 secs
upload.I've hit an interesting problem, and I was a bit surprised that there
isn't anything in the standard libs that could have solved it easily. It
isn't too complicated to write, but it isn't trivial either. If by any
chance it's already in the libs, please enlighten me :), otherwise would
anyone be interested in including it?
The thing I was solving is fairly trivial: download a file from the
internet, and stream-upload it somewhere else (Google Cloud Storage
specifically, but it doesn't really matter). The naive solution is
pretty straightforward: wire together the downloader's reader with the
uploader's writer, and voila, magic... until you look at the network
usage: x1 secs download, y1 secs upload, x2 secs download, y2 secs
I came across exactly the same problem yesterday!
I was reading, gzipping and uploading, but I had exactly the same
problem - 10 seconds of 100% CPU for gzipping into a 64 MB block, then
20 seconds of upload at 0% CPU.
Here was my solution
https://github.com/Memset/snapshot-manager/blob/master/snapshot/snapshot.go#L166
Which is an annoying amount of code. The error handling is tricky too.
bufio.Copy(dst io.Writer, src io.Reader, buffer int) (written int64, err
error)
Which essentially does what io.Copy does, but starts up a separate
writer go routine and passes everything through a user definable buffer.
Then it could handle both data bursts as well batching readers/writers.
I like it!error)
Which essentially does what io.Copy does, but starts up a separate
writer go routine and passes everything through a user definable buffer.
Then it could handle both data bursts as well batching readers/writers.
--
Nick Craig-Wood <nick@craig-wood.com> -- http://www.craig-wood.com/nick
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
