FAQ
http://play.golang.org/p/owO96v5See demonstrates what I was saying.
On Wed Jan 28 2015 at 8:08:58 PM Matt Harden wrote:

Why not just run two io.Copy() in two goroutines - the first copying from
a bufio.Reader to an io.PipeWriter, and the second copying from an
io.PipeReader to a bufio.Writer? This way you get two buffers so that
reading and writing can occur simultaneously, and data will be copied from
one buffer to the other whenever both buffers need flushing. Seems pretty
simple; what am I missing?
On Wed Jan 28 2015 at 1:04:30 PM Péter Szilágyi wrote:

Just to reply to Donovan, pipes are not buffered. That's the hard part of
the equation, not the threading.
On Jan 28, 2015 8:39 PM, "Péter Szilágyi" wrote:

An initial attempt at getting a buffered concurrent copy. It's a tad
more than a dozen lines, more like 200 :) I haven't tested it too
extensively, nor cleaned it up too much, just a rather quick hack together
to see if the concept works.

https://gist.github.com/karalabe/6de57007034d972b9ab6

Opinions? :)


On Wed, Jan 28, 2015 at 3:09 PM, Nick Craig-Wood <nick@craig-wood.com>
wrote:
On 28/01/15 08:18, Péter Szilágyi wrote:
I've hit an interesting problem, and I was a bit surprised that there
isn't anything in the standard libs that could have solved it easily. It
isn't too complicated to write, but it isn't trivial either. If by any
chance it's already in the libs, please enlighten me :), otherwise would
anyone be interested in including it?

The thing I was solving is fairly trivial: download a file from the
internet, and stream-upload it somewhere else (Google Cloud Storage
specifically, but it doesn't really matter). The naive solution is
pretty straightforward: wire together the downloader's reader with the
uploader's writer, and voila, magic... until you look at the network
usage: x1 secs download, y1 secs upload, x2 secs download, y2 secs
upload.

I came across exactly the same problem yesterday!

I was reading, gzipping and uploading, but I had exactly the same
problem - 10 seconds of 100% CPU for gzipping into a 64 MB block, then
20 seconds of upload at 0% CPU.

Here was my solution

https://github.com/Memset/snapshot-manager/blob/master/snaps
hot/snapshot.go#L166

Which is an annoying amount of code. The error handling is tricky too.
bufio.Copy(dst io.Writer, src io.Reader, buffer int) (written int64, err
error)

Which essentially does what io.Copy does, but starts up a separate
writer go routine and passes everything through a user definable buffer.
Then it could handle both data bursts as well batching
readers/writers.

I like it!

--
Nick Craig-Wood <nick@craig-wood.com> -- http://www.craig-wood.com/nick
--
You received this message because you are subscribed to the Google Groups
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to golang-nuts+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Search Discussions

Discussion Posts

Previous

Follow ups

Related Discussions

Discussion Navigation
viewthread | post
posts ‹ prev | 12 of 22 | next ›
Discussion Overview
groupgolang-nuts @
categoriesgo
postedJan 28, '15 at 8:18a
activeJan 30, '15 at 1:02p
posts22
users7
websitegolang.org

People

Translate

site design / logo © 2021 Grokbase