FAQ
Hi all,

   A few modifications went into the benchmarks:

    - Roger added a repeating reader in order not to need a giant
    pre-allocated test data blob, makes startup faster, also should better
    reflect algo performance.
    - To prevent measuring occasional hiccups, the throughput benchmarks not
    use best-out-of-three. Scores are much stabler.

Implementation wise the updates are:

    - Roger sent in a full Pipe based solution, arguing that it's way more
    flexible than a simple copy (I agree).
    - Jan sent in a nice optimization that seems to beat other algos in the
    case of very large buffers.
    - I've also ported my solution to a pipe version (both coexisting
    currently). That codebase needs to be cleaned up, it was to see if it works
    ok.

With these, the current standing is:

Latency benchmarks (GOMAXPROCS = 1):
       [!] bufio.Copy: 4.322µs 37 allocs 2736 B.
   [!] bufio.PipeCopy: 4.396µs 22 allocs 2224 B.
      rogerpeppe.Copy: 4.666µs 20 allocs 2032 B.
       egonelbre.Copy: 4.681µs 29 allocs 2368 B.
            jnml.Copy: 4.964µs 18 allocs 1936 B.

Latency benchmarks (GOMAXPROCS = 8):
       [!] bufio.Copy: 4.525µs 398 allocs 25840 B.
   [!] bufio.PipeCopy: 4.58µs 632 allocs 41264 B.
      rogerpeppe.Copy: 4.918µs 325 allocs 21552 B.
       egonelbre.Copy: 4.779µs 518 allocs 33664 B.
            jnml.Copy: 5.187µs 321 allocs 21328 B.

Throughput (GOMAXPROCS = 1) (256 MB):

+--------------------+--------+---------+---------+---------+----------+
THROUGHPUT | 333 | 4155 | 65359 | 1048559 | 16777301 |
+--------------------+--------+---------+---------+---------+----------+
[!] bufio.Copy | 524.45 | 3137.31 | 4765.79 | 4922.01 | 2083.84 |
[!] bufio.PipeCopy | 523.99 | 3115.08 | 4767.71 | 4924.59 | 2083.15 |
rogerpeppe.Copy | 231.94 | 1942.33 | 4499.48 | 4906.88 | 2085.31 |
egonelbre.Copy | 252.79 | 1865.77 | 4482.94 | 4832.15 | 2053.85 |
jnml.Copy | 233.75 | 1947.74 | 4500.00 | 4914.93 | 6055.04 |
+--------------------+--------+---------+---------+---------+----------+

+--------------------+-----------------------+-----------------------+-----------------------+-----------------------+-----------------------+
ALLOCS/BYTES | 333 | 4155 |
     65359 | 1048559 | 16777301 |
+--------------------+-----------------------+-----------------------+-----------------------+-----------------------+-----------------------+
[!] bufio.Copy | ( 24 / 1280) | ( 24 / 5536) | (
    24 / 66464) | ( 24 / 1049504) | ( 24 / 16786336) |
[!] bufio.PipeCopy | ( 13 / 1024) | ( 13 / 5280) | (
    13 / 66208) | ( 13 / 1049248) | ( 13 / 16786080) |
rogerpeppe.Copy | ( 12 / 896) | ( 12 / 5152) | (
    12 / 66080) | ( 12 / 1049120) | ( 12 / 16785952) |
egonelbre.Copy | ( 21 / 1248) | ( 21 / 5504) | (
    21 / 66432) | ( 21 / 1049472) | ( 21 / 16786304) |
jnml.Copy | ( 13 / 1008) | ( 13 / 5264) | (
    13 / 66192) | ( 13 / 1049232) | ( 13 / 16787424) |
+--------------------+-----------------------+-----------------------+-----------------------+-----------------------+-----------------------+

Throughput (GOMAXPROCS = 8) (256 MB):

+--------------------+--------+---------+---------+---------+----------+
THROUGHPUT | 333 | 4155 | 65359 | 1048559 | 16777301 |
+--------------------+--------+---------+---------+---------+----------+
[!] bufio.Copy | 510.38 | 2935.96 | 3899.06 | 4593.50 | 2067.04 |
[!] bufio.PipeCopy | 503.73 | 2928.34 | 4204.07 | 4602.74 | 2073.62 |
rogerpeppe.Copy | 206.61 | 1809.26 | 3770.79 | 4608.11 | 2069.03 |
egonelbre.Copy | 350.66 | 2439.46 | 3946.51 | 4377.70 | 1917.33 |
jnml.Copy | 214.09 | 1683.43 | 3773.63 | 4621.34 | 5749.54 |
+--------------------+--------+---------+---------+---------+----------+

+--------------------+-----------------------+-----------------------+-----------------------+-----------------------+-----------------------+
ALLOCS/BYTES | 333 | 4155 |
     65359 | 1048559 | 16777301 |
+--------------------+-----------------------+-----------------------+-----------------------+-----------------------+-----------------------+
[!] bufio.Copy | ( 82 / 4992) | ( 46 / 6944) | (
   108 / 71840) | ( 536 / 1082272) | ( 56 / 16788384) |
[!] bufio.PipeCopy | ( 65 / 4352) | ( 26 / 6336) | (
    66 / 69824) | ( 526 / 1082304) | ( 46 / 16788416) |
rogerpeppe.Copy | ( 14 / 1248) | ( 14 / 5504) | (
    14 / 66432) | ( 14 / 1049472) | ( 14 / 16786304) |
egonelbre.Copy | ( 89 / 5600) | ( 49 / 7744) | (
    51 / 68800) | ( 39 / 1051072) | ( 29 / 16787264) |
jnml.Copy | ( 13 / 1008) | ( 14 / 5552) | (
    13 / 66192) | ( 13 / 1049232) | ( 13 / 16787424) |
+--------------------+-----------------------+-----------------------+-----------------------+-----------------------+-----------------------+

@Jan, Egon: would you guys agree that a pipe based solution would be
better/more flexible? If so, we could rework the shootout to use pipe's as
the underlying implementation and a pre-baked copy function (i.e. I
blatantly copied mine from Roger).

Cheers,
   Peter
On Fri, Jan 30, 2015 at 6:08 PM, Péter Szilágyi wrote:

I've been also playing with this idea. One concrete instance where a
buffered pipe would be more useful/flexible than a buffered copy is if you
want to keep the buffer state, but maybe intermittently do something else
too (e.g. progress report after every N bytes copied). It would play nicely
with constructs such as (pseudo go):

for !done {
io.CopyN(bufDst, src, chunk)
fmt.Println("Chunk done")
}

If we put this functionality directly into Copy, then I guess it's much
more painful to do such a composable thing. I'm open to this idea of
converting the proposal into a bufio.Pipe(). Though I'll check how it would
fit into my design (too see if something's non-obviously wrong with the
idea).

In the meanwhile, running Jan's latest fix: all lately updated contenders
are converging to the same speed, though the internal buffering ones
outperforming the channels on small ops where the threading overhead hits
the channel.

Latency benchmarks (GOMAXPROCS = 1):
[!] bufio.Copy: 4.282µs 34 allocs 2544 B.
rogerpeppe.Copy: 5.304µs 20 allocs 67440 B.
egonelbre.Copy: 4.438µs 30 allocs 2432 B.
jnml.Copy: 4.812µs 17 allocs 1904 B.

Latency benchmarks (GOMAXPROCS = 8):
[!] bufio.Copy: 4.54µs 557 allocs 36016 B.
rogerpeppe.Copy: 5.547µs 150 allocs 75760 B.
egonelbre.Copy: 4.738µs 632 allocs 42240 B.
jnml.Copy: 5.05µs 249 allocs 16752 B.

Throughput (GOMAXPROCS = 1) (256 MB):

+-----------------+--------+---------+---------+---------+----------+
THROUGHPUT | 333 | 4155 | 65359 | 1048559 | 16777301 |
+-----------------+--------+---------+---------+---------+----------+
[!] bufio.Copy | 522.82 | 2880.92 | 4230.01 | 4360.93 | 1639.96 |
rogerpeppe.Copy | 203.08 | 1526.76 | 2609.97 | 2783.97 | 1198.19 |
egonelbre.Copy | 255.64 | 1791.27 | 4012.22 | 4278.48 | 1624.37 |
jnml.Copy | 229.16 | 1784.35 | 4024.98 | 4374.86 | 1716.81 |
+-----------------+--------+---------+---------+---------+----------+


+-----------------+-----------------------+-----------------------+-----------------------+-----------------------+-----------------------+
ALLOCS/BYTES | 333 | 4155 |
65359 | 1048559 | 16777301 |

+-----------------+-----------------------+-----------------------+-----------------------+-----------------------+-----------------------+
[!] bufio.Copy | ( 24 / 1280) | ( 24 / 5536) | (
24 / 66464) | ( 24 / 1049504) | ( 24 / 16786336) |
rogerpeppe.Copy | ( 17 / 9368) | ( 16 / 13560) | (
16 / 74488) | ( 16 / 1057528) | ( 16 / 16794360) |
egonelbre.Copy | ( 21 / 1248) | ( 21 / 5504) | (
21 / 66432) | ( 21 / 1049472) | ( 21 / 16786304) |
jnml.Copy | ( 12 / 976) | ( 12 / 5232) | (
12 / 66160) | ( 12 / 1049200) | ( 27 / 16778848) |

+-----------------+-----------------------+-----------------------+-----------------------+-----------------------+-----------------------+

Throughput (GOMAXPROCS = 8) (256 MB):

+-----------------+--------+---------+---------+---------+----------+
THROUGHPUT | 333 | 4155 | 65359 | 1048559 | 16777301 |
+-----------------+--------+---------+---------+---------+----------+
[!] bufio.Copy | 507.82 | 2689.22 | 3768.93 | 4071.21 | 1637.15 |
rogerpeppe.Copy | 186.95 | 1403.50 | 1752.47 | 1816.09 | 1147.60 |
egonelbre.Copy | 345.16 | 2196.97 | 3671.74 | 3919.41 | 1533.35 |
jnml.Copy | 204.41 | 1712.08 | 3176.28 | 4036.61 | 1664.63 |
+-----------------+--------+---------+---------+---------+----------+


+-----------------+-----------------------+-----------------------+-----------------------+-----------------------+-----------------------+
ALLOCS/BYTES | 333 | 4155 |
65359 | 1048559 | 16777301 |

+-----------------+-----------------------+-----------------------+-----------------------+-----------------------+-----------------------+
[!] bufio.Copy | ( 88 / 5376) | ( 40 / 6560) | (
88 / 70560) | ( 536 / 1082272) | ( 56 / 16788384) |
rogerpeppe.Copy | ( 18 / 10664) | ( 18 / 14920) | (
23 / 76168) | ( 21 / 1059080) | ( 20 / 16796072) |
egonelbre.Copy | ( 96 / 6496) | ( 47 / 7616) | (
46 / 68256) | ( 31 / 1050560) | ( 27 / 16786688) |
jnml.Copy | ( 12 / 976) | ( 13 / 5520) | (
13 / 66448) | ( 13 / 1049488) | ( 28 / 16779136) |

+-----------------+-----------------------+-----------------------+-----------------------+-----------------------+-----------------------+

On Fri, Jan 30, 2015 at 5:44 PM, roger peppe wrote:

My take is that all these proposals are essentially creating a data
pipe and using it for the copy. If we're going to do that, we may
as well implement an actual buffered pipe, because that's useful
for more than just Copy (you can pass the write end directly
to a Go function that expects an io.Writer for example).

I can't see that approach necessarily entails any significant
performance sacrifice.

On 30 January 2015 at 15:19, Jan Mercl wrote:
On Fri Jan 30 2015 at 14:13:07 Péter Szilágyi <peterke@gmail.com>
wrote:
On the other hand, Jan's solution manages to allocate a massive 64MB
internal buffers during the latency benchmarks, which requested 1KB
buffers
only.
Memory overallocation fixed, please pull[0].

[0]: https://github.com/karalabe/bufioprop/pull/10

-j

--
You received this message because you are subscribed to the Google Groups
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to golang-nuts+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to golang-nuts+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Search Discussions

Discussion Posts

Previous

Follow ups

Related Discussions

Discussion Navigation
viewthread | post
posts ‹ prev | 41 of 50 | next ›
Discussion Overview
groupgolang-nuts @
categoriesgo
postedJan 29, '15 at 11:01a
activeFeb 3, '15 at 11:21a
posts50
users8
websitegolang.org

People

Translate

site design / logo © 2021 Grokbase