FAQ
If we have 100 go routines what would be a good size for a buffered channel
which all go routines receive data from (if filling the channel is not
considered the bottle neck)?

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Search Discussions

  • James Bardin at Sep 10, 2014 at 5:40 pm
    An unbuffered chan would be good size.

    If you don't want to always block the producer when all goroutines are
    busy, *then* use a buffered chan with an appropriate capacity. That
    capacity is completely up to you and its context in your code. All it is is
    a buffer -- it's like asking "How large should my slice be when I'm reading
    100 files?".

    On Wednesday, September 10, 2014 1:24:48 PM UTC-4, i...@bodokaiser.io wrote:

    If we have 100 go routines what would be a good size for a buffered
    channel which all go routines receive data from (if filling the channel is
    not considered the bottle neck)?
    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
  • James Bardin at Sep 10, 2014 at 7:18 pm
    +gonag-nuts

    On Wed, Sep 10, 2014 at 2:18 PM, Bodo Kaiser wrote:

    Let me double my precision of the question :)

    How can I determine the best size of my buffered channel for my specific
    use case?
    What are some „rules“ I could follow?
    You may be able to empirically determine some specific requirement, If you
    can state "I need to always accept a minimum of 100 items before blocking",
    then 100 would be a good starting point.

    You could test your program with a very large buffered channel, and track
    the actual size of the channel to see what gets used (just a form of
    profiling)

    If you're only worried about performance, and have no numbers to back it
    up, then profile with various buffer sizes and see what happens.

    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
  • Dave Cheney at Sep 10, 2014 at 8:55 pm
    I believe there are only two values for a channel's buffer, 0 and 1.
    Setting a value higher than 1 is really saying "well, I sort of want to
    buffer as much as I can, I hope this number is enough", and that is not a
    stable way to design systems.
    On Thursday, 11 September 2014 03:24:48 UTC+10, i...@bodokaiser.io wrote:

    If we have 100 go routines what would be a good size for a buffered
    channel which all go routines receive data from (if filling the channel is
    not considered the bottle neck)?
    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
  • Andrewchamberss at Sep 10, 2014 at 11:54 pm
    Its true that you can't predict the producer hitting the end of the buffer
    and stalling the send, which means so a buffer isn't good for stability in
    high load cases, but you seem to be ignoring that its good for performance
    when you actually need parallelism and not just concurrency in the face of
    sporadic loads and different workloads between goroutines.
    On Thursday, September 11, 2014 8:54:54 AM UTC+12, Dave Cheney wrote:

    I believe there are only two values for a channel's buffer, 0 and 1.
    Setting a value higher than 1 is really saying "well, I sort of want to
    buffer as much as I can, I hope this number is enough", and that is not a
    stable way to design systems.
    On Thursday, 11 September 2014 03:24:48 UTC+10, i...@bodokaiser.io wrote:

    If we have 100 go routines what would be a good size for a buffered
    channel which all go routines receive data from (if filling the channel is
    not considered the bottle neck)?
    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
  • Caleb Spare at Sep 10, 2014 at 11:56 pm
    Two cases where buffer sizes > 1 seem fine to me:

    - Using a channel as quick'n'easy FIFO queue of size N
    - There is some necessary performance advantage that comes from using
    a larger buffer size. (For instance, the consumer at the end of the
    buffer might have spiky performance and buffering up extra items
    improves throughput.) In this case, the magical size value ought to be
    accompanied by some benchmark to justify it.

    -Caleb
    On Wed, Sep 10, 2014 at 1:54 PM, Dave Cheney wrote:
    I believe there are only two values for a channel's buffer, 0 and 1. Setting
    a value higher than 1 is really saying "well, I sort of want to buffer as
    much as I can, I hope this number is enough", and that is not a stable way
    to design systems.

    On Thursday, 11 September 2014 03:24:48 UTC+10, i...@bodokaiser.io wrote:

    If we have 100 go routines what would be a good size for a buffered
    channel which all go routines receive data from (if filling the channel is
    not considered the bottle neck)?
    --
    You received this message because you are subscribed to the Google Groups
    "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an
    email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
  • Bodo Kaiser at Sep 14, 2014 at 12:37 pm
    I done a small bench on the crawler (http://github.com/bodokaiser/crawler <http://github.com/bodokaiser/crawler>) with different channel sizes.

    For benchmarking I output all urls with there referenced links to stdout and measure the average rate with „pipe viewer“.

    My results (always measured after 10s):
    Channel size of 0: 13 KiB/s, 10 KiB/s, 8.5 KiB/s, 7.4 KiB/s
    Channel size of 10: 10.9 KiB/s, 9.8 KiB/s, 8.3 KiB/s, 7 KiB/s
    Channel size of 100: 9.16 KiB/s, 7.8 KiB/s, 6.7 KiB/s, 5.8 KiB/s

    I interpret the results in that way that the throughput gets lower the higher the channel capacity is.

    Though I must admit that the there are a lot of other points which could have influenced the test (e.g. order of sites which were crawled, socket available from the OS, …).
    Am 11.09.2014 um 01:55 schrieb Caleb Spare <cespare@gmail.com>:

    Two cases where buffer sizes > 1 seem fine to me:

    - Using a channel as quick'n'easy FIFO queue of size N
    - There is some necessary performance advantage that comes from using
    a larger buffer size. (For instance, the consumer at the end of the
    buffer might have spiky performance and buffering up extra items
    improves throughput.) In this case, the magical size value ought to be
    accompanied by some benchmark to justify it.

    -Caleb
    On Wed, Sep 10, 2014 at 1:54 PM, Dave Cheney wrote:
    I believe there are only two values for a channel's buffer, 0 and 1. Setting
    a value higher than 1 is really saying "well, I sort of want to buffer as
    much as I can, I hope this number is enough", and that is not a stable way
    to design systems.

    On Thursday, 11 September 2014 03:24:48 UTC+10, i...@bodokaiser.io wrote:

    If we have 100 go routines what would be a good size for a buffered
    channel which all go routines receive data from (if filling the channel is
    not considered the bottle neck)?
    --
    You received this message because you are subscribed to the Google Groups
    "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an
    email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupgolang-nuts @
categoriesgo
postedSep 10, '14 at 5:24p
activeSep 14, '14 at 12:37p
posts7
users5
websitegolang.org

People

Translate

site design / logo © 2021 Grokbase