FAQ
Hi,

Already searched through previous posts, but could not find optimal
solution.

Suppose I have the following:
- 1 goroutine generating work
- N worker goroutines
- 1 go routine receiving results

Amount of work is arbitrary, does not fit single buffered channel, nor can
I know in advance how many units of work will there be.
Current implementation works when there is fixed number of units, and is
implemented as such:
- 1 work channel
- 1 data channel
- 1 done channel
- 1 error channel

All work is (N units) generated up-front and put into buffer channel
(work). Each worker takes 1 unit from work channel, does work and sends
data or error through respective channels. In any case, worker sends 1 done
to done channel for each unit of work.
There is another goroutine collecting data/errors and checking how many
units have completed (it expects N reads from done channel).

This obviously works when number of units (N) is known up-front, as its
simply a matter of receiving form done channel N times.

I was thinking how to apply this to where N is not known. There would be
another goroutine generating work (instead of generating it up-front).
Simplest solution would be to mutex protect a counter, increase it for each
work added and decrease it for each processed (on receive from done).
Obviously some checks need to be done to make sure no more units are being
generated before ending the whole thing when counter reaches 0 (if workers
are faster than producer).

Is there a more idiomatic way to solve this? Seems like mutex is simplest
to use. Would be also simple to solve if channels had .isEmpty() function ;)

Thx,
Igor

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Search Discussions

  • Egon at Feb 19, 2014 at 10:58 am
    http://play.golang.org/p/CcoX6rL-6K

    Just close the producer channel and let the consumers terminate. And then
    wait for the consumers.

    + egon
    On Wednesday, February 19, 2014 12:21:22 PM UTC+2, igor....@gmail.com wrote:

    Hi,

    Already searched through previous posts, but could not find optimal
    solution.

    Suppose I have the following:
    - 1 goroutine generating work
    - N worker goroutines
    - 1 go routine receiving results

    Amount of work is arbitrary, does not fit single buffered channel, nor can
    I know in advance how many units of work will there be.
    Current implementation works when there is fixed number of units, and is
    implemented as such:
    - 1 work channel
    - 1 data channel
    - 1 done channel
    - 1 error channel

    All work is (N units) generated up-front and put into buffer channel
    (work). Each worker takes 1 unit from work channel, does work and sends
    data or error through respective channels. In any case, worker sends 1 done
    to done channel for each unit of work.
    There is another goroutine collecting data/errors and checking how many
    units have completed (it expects N reads from done channel).

    This obviously works when number of units (N) is known up-front, as its
    simply a matter of receiving form done channel N times.

    I was thinking how to apply this to where N is not known. There would be
    another goroutine generating work (instead of generating it up-front).
    Simplest solution would be to mutex protect a counter, increase it for each
    work added and decrease it for each processed (on receive from done).
    Obviously some checks need to be done to make sure no more units are being
    generated before ending the whole thing when counter reaches 0 (if workers
    are faster than producer).

    Is there a more idiomatic way to solve this? Seems like mutex is simplest
    to use. Would be also simple to solve if channels had .isEmpty() function ;)

    Thx,
    Igor
    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.
  • Roger peppe at Feb 19, 2014 at 1:37 pm

    On 19 February 2014 10:58, egon wrote:
    http://play.golang.org/p/CcoX6rL-6K
    That's not quite right, as it doesn't guarantee that all the
    results get processed before the program exits (there's
    nothing waiting for the monitor goroutine to finish).

    Something like this is slightly more general:

    http://play.golang.org/p/sfe_ohiQZd

    and enables you to chain concurrent processing stages;
    for instance:

    http://play.golang.org/p/YpEg8Z3caG

    Things can become slightly more complex in the presence of errors.

    I wrote a little package to make that a little easier, although
    it might not help much in this particular scenario:

        http://godoc.org/code.google.com/p/rog-go/parallel

    but it really depends what you want to do on error - that
    package is based around a workflow where you continue
    on processing in the presence of errors, but need to collect them
    to report them afterwards. In other circumstances, you might wish
    to stop everything on the first error received.

       cheers,
         rog.

    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.
  • Egon at Feb 19, 2014 at 2:28 pm

    On Wednesday, February 19, 2014 3:37:41 PM UTC+2, rog wrote:
    On 19 February 2014 10:58, egon <egon...@gmail.com <javascript:>> wrote:
    http://play.golang.org/p/CcoX6rL-6K
    That's not quite right, as it doesn't guarantee that all the
    results get processed before the program exits (there's
    nothing waiting for the monitor goroutine to finish).
    As usual... tiny bugs :).

    Something like this is slightly more general:

    http://play.golang.org/p/sfe_ohiQZd

    and enables you to chain concurrent processing stages;
    for instance:

    http://play.golang.org/p/YpEg8Z3caG

    Things can become slightly more complex in the presence of errors.

    But, not too much: http://play.golang.org/p/I1s4dOAGyC

    (I use the atomic int instead of WaitGroup to avoid the need for an extra
    goroutine.)
    (The waiting for closing of multiple channels is explained here
    http://stackoverflow.com/questions/13666253/breaking-out-of-a-select-statement-when-all-channels-are-closed
    )

    + egon


    I wrote a little package to make that a little easier, although
    it might not help much in this particular scenario:

    http://godoc.org/code.google.com/p/rog-go/parallel<http://www.google.com/url?q=http%3A%2F%2Fgodoc.org%2Fcode.google.com%2Fp%2Frog-go%2Fparallel&sa=D&sntz=1&usg=AFQjCNERx86LdV-DgnK1O1Y2pHGgU3iEUg>

    but it really depends what you want to do on error - that
    package is based around a workflow where you continue
    on processing in the presence of errors, but need to collect them
    to report them afterwards. In other circumstances, you might wish
    to stop everything on the first error received.

    cheers,
    rog.
    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.
  • Roger peppe at Feb 19, 2014 at 3:34 pm

    On 19 February 2014 14:28, egon wrote:
    On Wednesday, February 19, 2014 3:37:41 PM UTC+2, rog wrote:
    On 19 February 2014 10:58, egon wrote:
    http://play.golang.org/p/CcoX6rL-6K
    That's not quite right, as it doesn't guarantee that all the
    results get processed before the program exits (there's
    nothing waiting for the monitor goroutine to finish).

    As usual... tiny bugs :).
    Something like this is slightly more general:

    http://play.golang.org/p/sfe_ohiQZd

    and enables you to chain concurrent processing stages;
    for instance:

    http://play.golang.org/p/YpEg8Z3caG

    Things can become slightly more complex in the presence of errors.

    But, not too much: http://play.golang.org/p/I1s4dOAGyC

    (I use the atomic int instead of WaitGroup to avoid the need for an extra
    goroutine.)
    (The waiting for closing of multiple channels is explained here
    http://stackoverflow.com/questions/13666253/breaking-out-of-a-select-statement-when-all-channels-are-closed
    )
    A slight simplification, there's actually no need for an extra channel there:

    http://play.golang.org/p/OuY6P9jWOe

    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.
  • Igor Lautar at Feb 19, 2014 at 3:15 pm

    On Wednesday, February 19, 2014 2:37:41 PM UTC+1, rog wrote:

    Something like this is slightly more general:

    http://play.golang.org/p/sfe_ohiQZd
    and enables you to chain concurrent processing stages;
    for instance:

    http://play.golang.org/p/YpEg8Z3caG

    Things can become slightly more complex in the presence of errors.
    Yes, exactly what I was thinking.

    I wrote a little package to make that a little easier, although
    it might not help much in this particular scenario:

    http://godoc.org/code.google.com/p/rog-go/parallel
    Sounds interesting, will check it out.

    Error handling is simple reporting, but even if not just use a separate
    channel as egon pointed out.

    Thx to both.

    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupgolang-nuts @
categoriesgo
postedFeb 19, '14 at 10:21a
activeFeb 19, '14 at 3:34p
posts6
users3
websitegolang.org

3 users in discussion

Egon: 2 posts Igor Lautar: 2 posts Roger peppe: 2 posts

People

Translate

site design / logo © 2022 Grokbase