I've added your implementation to the shootout, but it's currently
panicing. Please investigate,
To the others too in general, I've openen a more formal proposal thread
(Subject: [proposal] bufio.Copy) in which various implementations and
discussions are going. I think we should move from this thread to that one
to keep it under a single thread of discussion and not multiplex between
two.
Cheers :),
Peter
On Fri, Jan 30, 2015 at 12:16 AM, Bakul Shah wrote:
What about something like this: http://play.golang.org/p/g0-g6x-Kp_
The idea is to allocate a large buffer and read into it in
smaller chunks (at buffersize/N boundaries). For each read
chunk the reader sends its size in a chan. The writer waits
until it gets a valid count and writes it out and then sends
back an ack on another chan. The reader can be upto N reads
ahead of the writer. On a recent MBP I get about 2.9GB/s with
dd < /dev/zero bs=1m count=10000 | bufcopy > /dev/null
The sample interface in this quick hack is:
func bufcopy(w io.Writer, r io.Reader) error
Shouldn't be hard to turn it into what you want.
[Aside: The current IO interfaces force extra copying.
Copying could've been made faster when reading from a file or
writing to one if read(n) returned a filled-in buffer &/or
writes were done lazily -- that is, instead of
writer.write(buf) one does buf := writer.alloc_buf(n) to grab
a buffer to write into. Ultimately followed by a sync. See
"The Alloc Stream facility" paper by Krieger, Strum and
Unrau.]
On Wed, 28 Jan 2015 10:18:40 +0200 =?UTF-8?B?UMOpdGVyIFN6aWzDoWd5aQ==?= <
peterke@gmail.com> wrote:
--What about something like this: http://play.golang.org/p/g0-g6x-Kp_
The idea is to allocate a large buffer and read into it in
smaller chunks (at buffersize/N boundaries). For each read
chunk the reader sends its size in a chan. The writer waits
until it gets a valid count and writes it out and then sends
back an ack on another chan. The reader can be upto N reads
ahead of the writer. On a recent MBP I get about 2.9GB/s with
dd < /dev/zero bs=1m count=10000 | bufcopy > /dev/null
The sample interface in this quick hack is:
func bufcopy(w io.Writer, r io.Reader) error
Shouldn't be hard to turn it into what you want.
[Aside: The current IO interfaces force extra copying.
Copying could've been made faster when reading from a file or
writing to one if read(n) returned a filled-in buffer &/or
writes were done lazily -- that is, instead of
writer.write(buf) one does buf := writer.alloc_buf(n) to grab
a buffer to write into. Ultimately followed by a sync. See
"The Alloc Stream facility" paper by Krieger, Strum and
Unrau.]
On Wed, 28 Jan 2015 10:18:40 +0200 =?UTF-8?B?UMOpdGVyIFN6aWzDoWd5aQ==?= <
peterke@gmail.com> wrote:
Hi all,
I've hit an interesting problem, and I was a bit surprised that there
isn't anything in the standard libs that could have solved it easily. It
isn't too complicated to write, but it isn't trivial either. If by any
chance it's already in the libs, please enlighten me :), otherwise would
anyone be interested in including it?
The thing I was solving is fairly trivial: download a file from the
internet, and stream-upload it somewhere else (Google Cloud Storage
specifically, but it doesn't really matter). The naive solution is pretty
straightforward: wire together the downloader's reader with the
uploader's
writer, and voila, magic... until you look at the network usage: x1 secs
download, y1 secs upload, x2 secs download, y2 secs upload.
The problem is, that the uploader will read some fixed amount of data,
buffer it up and then start the upload. But while the upload is in
progress, it doesn't read any more data from the reader, essentially
pausing it until it finishes. By that time the reader could have filled up
the next buffer to send, but alas, it was blocked so it didn't download
anything.
Note, buffered readers/writers won't really solve this issue, since even
though they have the buffers in place to store arriving data, those buffers
cannot be simultaneously filled and flushed too. As far as I figured it
out, the only way to solve this streaming problem is to have to
concurrent
go routines, one reading and another writing, with a data buffer in between.
If there is indeed no such thing, would it be worthwhile to add something
like?
bufio.Copy(dst io.Writer, src io.Reader, buffer int) (written int64, err
error)
I've hit an interesting problem, and I was a bit surprised that there
isn't anything in the standard libs that could have solved it easily. It
isn't too complicated to write, but it isn't trivial either. If by any
chance it's already in the libs, please enlighten me :), otherwise would
anyone be interested in including it?
The thing I was solving is fairly trivial: download a file from the
internet, and stream-upload it somewhere else (Google Cloud Storage
specifically, but it doesn't really matter). The naive solution is pretty
straightforward: wire together the downloader's reader with the
uploader's
writer, and voila, magic... until you look at the network usage: x1 secs
download, y1 secs upload, x2 secs download, y2 secs upload.
The problem is, that the uploader will read some fixed amount of data,
buffer it up and then start the upload. But while the upload is in
progress, it doesn't read any more data from the reader, essentially
pausing it until it finishes. By that time the reader could have filled up
the next buffer to send, but alas, it was blocked so it didn't download
anything.
Note, buffered readers/writers won't really solve this issue, since even
though they have the buffers in place to store arriving data, those buffers
cannot be simultaneously filled and flushed too. As far as I figured it
out, the only way to solve this streaming problem is to have to
concurrent
go routines, one reading and another writing, with a data buffer in between.
If there is indeed no such thing, would it be worthwhile to add something
like?
bufio.Copy(dst io.Writer, src io.Reader, buffer int) (written int64, err
error)
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
