What about something like this: http://play.golang.org/p/g0-g6x-Kp_

The idea is to allocate a large buffer and read into it in
smaller chunks (at buffersize/N boundaries). For each read
chunk the reader sends its size in a chan. The writer waits
until it gets a valid count and writes it out and then sends
back an ack on another chan. The reader can be upto N reads
ahead of the writer. On a recent MBP I get about 2.9GB/s with

  dd < /dev/zero bs=1m count=10000 | bufcopy > /dev/null

The sample interface in this quick hack is:

  func bufcopy(w io.Writer, r io.Reader) error

Shouldn't be hard to turn it into what you want.

[Aside: The current IO interfaces force extra copying.
Copying could've been made faster when reading from a file or
writing to one if read(n) returned a filled-in buffer &/or
writes were done lazily -- that is, instead of
writer.write(buf) one does buf := writer.alloc_buf(n) to grab
a buffer to write into. Ultimately followed by a sync. See
"The Alloc Stream facility" paper by Krieger, Strum and
On Wed, 28 Jan 2015 10:18:40 +0200 =?UTF-8?B?UMOpdGVyIFN6aWzDoWd5aQ==?= wrote:

Hi all,

I've hit an interesting problem, and I was a bit surprised that there
isn't anything in the standard libs that could have solved it easily. It
isn't too complicated to write, but it isn't trivial either. If by any
chance it's already in the libs, please enlighten me :), otherwise would
anyone be interested in including it?

The thing I was solving is fairly trivial: download a file from the
internet, and stream-upload it somewhere else (Google Cloud Storage
specifically, but it doesn't really matter). The naive solution is pretty
straightforward: wire together the downloader's reader with the uploader's
writer, and voila, magic... until you look at the network usage: x1 secs
download, y1 secs upload, x2 secs download, y2 secs upload.

The problem is, that the uploader will read some fixed amount of data,
buffer it up and then start the upload. But while the upload is in
progress, it doesn't read any more data from the reader, essentially
pausing it until it finishes. By that time the reader could have filled up
the next buffer to send, but alas, it was blocked so it didn't download

Note, buffered readers/writers won't really solve this issue, since even
though they have the buffers in place to store arriving data, those buffers
cannot be simultaneously filled and flushed too. As far as I figured it
out, the only way to solve this streaming problem is to have to concurrent
go routines, one reading and another writing, with a data buffer in between.

If there is indeed no such thing, would it be worthwhile to add something

bufio.Copy(dst io.Writer, src io.Reader, buffer int) (written int64, err
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Search Discussions

Discussion Posts


Follow ups

Related Discussions

Discussion Navigation
viewthread | post
posts ‹ prev | 18 of 22 | next ›
Discussion Overview
groupgolang-nuts @
postedJan 28, '15 at 8:18a
activeJan 30, '15 at 1:02p



site design / logo © 2021 Grokbase