FAQ
Hi All,

I'm running into this WaitGroup vs. signal channel debate with a slightly
different type of problem.

I have a main function that spins off goroutines for processing and uses a
signal channel to track when the goroutines should return. If the signal
channel is closed, then the goroutines should return. Once all have
returned, I do additional processing in the main function.

However, I have an additional requirement, that if certain conditions are
met, all processing in goroutines should stop and we should immediately
move on in the main function.

I can trigger the channel close, but some goroutines are still processing,
so sometimes I get panics as they try to write to channels that had been
closed in the meantime. (I'm noticing panics even if I check the signal
channel right before writing to any other channel.)

I can't use a WaitGroup because I might not want to wait until all
goroutines are done.

Any ideas on what I'm missing?

Thank you,
Aki

On Wednesday, June 6, 2012 at 10:24:42 PM UTC-4, Jim Robinson wrote:

Hi folks,

I've been finding myself reaching for sync.WaitGroup for a program
I'm writing, and it feels wrong... I'm hoping for some feedback
from you experts.

I'm writing a program that walks through a deep directory tree, looking
for particular
file names. When it finds a match it chucks the path over the wall to
another
goroutine, which is responsible for cracking the file open and performing
some
analysis. Once that goroutine is done it chucks the analysis over the wall
to a goroutine that is responsbile for writing the results out to a log
file.

The goroutines are using buffered channels, and I've got multiple instances
of goroutines reading from the channels. I did this because I did want to
have some parallel execution.

All this works great, as long as the main() function is kept alive long
enough
to allow any submitted work to make its way through the goroutines. If my
program just waited in main() until it got an interrupt, it would
finish processing
the files w/o a problem. But I want the program to exit on its own, when
all
the work has completed.

To accomplish this I reached for sync.WaitGroup, but I'm wondering
if there is a more elegant way to accomplish this "wait until the channels
have cleared" problem.

Jim
--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Search Discussions

  • Nick Craig-Wood at Jan 12, 2015 at 5:15 pm

    On 12/01/15 15:25, akibalogh@gmail.com wrote:
    I have a main function that spins off goroutines for processing and uses
    a signal channel to track when the goroutines should return. If the
    signal channel is closed, then the goroutines should return. Once all
    have returned, I do additional processing in the main function.

    However, I have an additional requirement, that if certain conditions
    are met, all processing in goroutines should stop and we should
    immediately move on in the main function.

    I can trigger the channel close, but some goroutines are still
    processing, so sometimes I get panics as they try to write to channels
    that had been closed in the meantime. (I'm noticing panics even if I
    check the signal channel right before writing to any other channel.)

    I can't use a WaitGroup because I might not want to wait until all
    goroutines are done.

    Any ideas on what I'm missing?
    Here is an idea for you

    http://play.golang.org/p/AoH_n3UDWY

    Note how I've separated the done test out of the select in the writer

    func writer(out chan<- int, finished chan bool) {
    outer:
      for {
       out <- 1
       select {
       case <-finished:
        break outer
       }

      }
      fmt.Println("Ending writer")
    }

    No waitgroups required.

    --
    Nick Craig-Wood <nick@craig-wood.com> -- http://www.craig-wood.com/nick

    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
  • Aki Balogh at Jan 12, 2015 at 5:49 pm
    Hi Nick,

    Thanks for the prompt response.

    I understand that, in your code, you're breaking the loop if the timeout is
    hit. This does indeed take away the need for WaitGroups.

    However, in my code, there's an event that happens that requires me to halt
    all goroutines and move on, to save time.

    In your code, you're sleeping at the end for 1 second, presumably to
    'mop-up' any running goroutines, but I can't do that, since I'm optimizing
    for speed and need the code to run and return as quickly as possible.

    Here's my code: http://play.golang.org/p/6SchAM3-sU

    (Please do let me know if I'm missing the point of how your code works.)


    On Mon, Jan 12, 2015 at 12:15 PM, Nick Craig-Wood wrote:
    On 12/01/15 15:25, akibalogh@gmail.com wrote:
    I have a main function that spins off goroutines for processing and uses
    a signal channel to track when the goroutines should return. If the
    signal channel is closed, then the goroutines should return. Once all
    have returned, I do additional processing in the main function.

    However, I have an additional requirement, that if certain conditions
    are met, all processing in goroutines should stop and we should
    immediately move on in the main function.

    I can trigger the channel close, but some goroutines are still
    processing, so sometimes I get panics as they try to write to channels
    that had been closed in the meantime. (I'm noticing panics even if I
    check the signal channel right before writing to any other channel.)

    I can't use a WaitGroup because I might not want to wait until all
    goroutines are done.

    Any ideas on what I'm missing?
    Here is an idea for you

    http://play.golang.org/p/AoH_n3UDWY

    Note how I've separated the done test out of the select in the writer

    func writer(out chan<- int, finished chan bool) {
    outer:
    for {
    out <- 1
    select {
    case <-finished:
    break outer
    }

    }
    fmt.Println("Ending writer")
    }

    No waitgroups required.

    --
    Nick Craig-Wood <nick@craig-wood.com> -- http://www.craig-wood.com/nick
    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
  • Jesper Louis Andersen at Jan 12, 2015 at 5:58 pm
    If you need to stop in the middle of some work and move on, then you need
    to regularly check for that event in your goroutines and you also need to
    make sure that the "channel network" can handle such abrupt closes.
    Usually, there is a need to handle some corner-cases for this to work.

    Another possiblity, I would definitely look at, would be to use something
    where you "tombstone" processes and then having the tombstone's still
    process "structural" events and passing them on, but stopping processing.
    This allows a more controlled closedown of the channel network. I believe
    there are several packages out there which tries to abstract these ideas.

    There is also the 'context' package, of which there is a blog post here:

    https://blog.golang.org/context

    it may be possible to use that idiom. Set up your computation in the same
    context and then terminate that context when you decide to abort the
    computation. If each goroutine in the context does the "right thing(tm)",
    then it should be possible to avoid the panic.

    On Mon Jan 12 2015 at 6:49:04 PM Aki Balogh wrote:

    Hi Nick,

    Thanks for the prompt response.

    I understand that, in your code, you're breaking the loop if the timeout
    is hit. This does indeed take away the need for WaitGroups.

    However, in my code, there's an event that happens that requires me to
    halt all goroutines and move on, to save time.

    In your code, you're sleeping at the end for 1 second, presumably to
    'mop-up' any running goroutines, but I can't do that, since I'm optimizing
    for speed and need the code to run and return as quickly as possible.

    Here's my code: http://play.golang.org/p/6SchAM3-sU

    (Please do let me know if I'm missing the point of how your code works.)


    On Mon, Jan 12, 2015 at 12:15 PM, Nick Craig-Wood wrote:
    On 12/01/15 15:25, akibalogh@gmail.com wrote:
    I have a main function that spins off goroutines for processing and uses
    a signal channel to track when the goroutines should return. If the
    signal channel is closed, then the goroutines should return. Once all
    have returned, I do additional processing in the main function.

    However, I have an additional requirement, that if certain conditions
    are met, all processing in goroutines should stop and we should
    immediately move on in the main function.

    I can trigger the channel close, but some goroutines are still
    processing, so sometimes I get panics as they try to write to channels
    that had been closed in the meantime. (I'm noticing panics even if I
    check the signal channel right before writing to any other channel.)

    I can't use a WaitGroup because I might not want to wait until all
    goroutines are done.

    Any ideas on what I'm missing?
    Here is an idea for you

    http://play.golang.org/p/AoH_n3UDWY

    Note how I've separated the done test out of the select in the writer

    func writer(out chan<- int, finished chan bool) {
    outer:
    for {
    out <- 1
    select {
    case <-finished:
    break outer
    }

    }
    fmt.Println("Ending writer")
    }

    No waitgroups required.

    --
    Nick Craig-Wood <nick@craig-wood.com> -- http://www.craig-wood.com/nick
    --
    You received this message because you are subscribed to the Google Groups
    "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an
    email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
  • Aki Balogh at Jan 12, 2015 at 6:10 pm
    Thanks, Jesper (and Benjamin).

    At this point, I'm checking whether the done channel is closed at every
    step of processing. But I'm still encountering corner cases where I see
    panics. I recover from the panics -- not a problem since those goroutines
    were going to be thrown away anyway -- but it's messy code and fills up my
    error log.

    I will check out the tomb package, contexts and the blog post on pipelines.


    Akos (Aki) Balogh
    Co-Founder, MarketMuse
    https://www.marketmuse.com
    On Mon, Jan 12, 2015 at 12:57 PM, Jesper Louis Andersen wrote:

    If you need to stop in the middle of some work and move on, then you need
    to regularly check for that event in your goroutines and you also need to
    make sure that the "channel network" can handle such abrupt closes.
    Usually, there is a need to handle some corner-cases for this to work.

    Another possiblity, I would definitely look at, would be to use something
    where you "tombstone" processes and then having the tombstone's still
    process "structural" events and passing them on, but stopping processing.
    This allows a more controlled closedown of the channel network. I believe
    there are several packages out there which tries to abstract these ideas.

    There is also the 'context' package, of which there is a blog post here:

    https://blog.golang.org/context

    it may be possible to use that idiom. Set up your computation in the same
    context and then terminate that context when you decide to abort the
    computation. If each goroutine in the context does the "right thing(tm)",
    then it should be possible to avoid the panic.

    On Mon Jan 12 2015 at 6:49:04 PM Aki Balogh wrote:

    Hi Nick,

    Thanks for the prompt response.

    I understand that, in your code, you're breaking the loop if the timeout
    is hit. This does indeed take away the need for WaitGroups.

    However, in my code, there's an event that happens that requires me to
    halt all goroutines and move on, to save time.

    In your code, you're sleeping at the end for 1 second, presumably to
    'mop-up' any running goroutines, but I can't do that, since I'm optimizing
    for speed and need the code to run and return as quickly as possible.

    Here's my code: http://play.golang.org/p/6SchAM3-sU

    (Please do let me know if I'm missing the point of how your code works.)



    On Mon, Jan 12, 2015 at 12:15 PM, Nick Craig-Wood <nick@craig-wood.com>
    wrote:
    On 12/01/15 15:25, akibalogh@gmail.com wrote:
    I have a main function that spins off goroutines for processing and uses
    a signal channel to track when the goroutines should return. If the
    signal channel is closed, then the goroutines should return. Once all
    have returned, I do additional processing in the main function.

    However, I have an additional requirement, that if certain conditions
    are met, all processing in goroutines should stop and we should
    immediately move on in the main function.

    I can trigger the channel close, but some goroutines are still
    processing, so sometimes I get panics as they try to write to channels
    that had been closed in the meantime. (I'm noticing panics even if I
    check the signal channel right before writing to any other channel.)

    I can't use a WaitGroup because I might not want to wait until all
    goroutines are done.

    Any ideas on what I'm missing?
    Here is an idea for you

    http://play.golang.org/p/AoH_n3UDWY

    Note how I've separated the done test out of the select in the writer

    func writer(out chan<- int, finished chan bool) {
    outer:
    for {
    out <- 1
    select {
    case <-finished:
    break outer
    }

    }
    fmt.Println("Ending writer")
    }

    No waitgroups required.

    --
    Nick Craig-Wood <nick@craig-wood.com> -- http://www.craig-wood.com/nick
    --
    You received this message because you are subscribed to the Google Groups
    "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an
    email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
  • Matt Harden at Jan 15, 2015 at 2:21 am
    Perhaps the code you posted to the playground is incomplete, but nothing
    ever reads from the timeChan channel, and nothing closes the done channel
    until all processing is complete. Perhaps in line 47, you intended to have
    "case <-timeChan" instead of "case <-done".

    Assuming that change was made, I would avoid a race condition in function f
    by performing your send to the a channel in the same select as you check
    the done channel. In this case nothing should panic due to channel issues.
    Don't close the "a" channel, just let it be GC'd once all references go
    away; you might want to assign a = nil in the main goroutine so it won't
    hold a reference.

    http://play.golang.org/p/8G6UothCud
    On Mon Jan 12 2015 at 12:10:20 PM Aki Balogh wrote:

    Thanks, Jesper (and Benjamin).

    At this point, I'm checking whether the done channel is closed at every
    step of processing. But I'm still encountering corner cases where I see
    panics. I recover from the panics -- not a problem since those goroutines
    were going to be thrown away anyway -- but it's messy code and fills up my
    error log.

    I will check out the tomb package, contexts and the blog post on pipelines.


    Akos (Aki) Balogh
    Co-Founder, MarketMuse
    https://www.marketmuse.com

    On Mon, Jan 12, 2015 at 12:57 PM, Jesper Louis Andersen <
    jesper.louis.andersen@gmail.com> wrote:
    If you need to stop in the middle of some work and move on, then you need
    to regularly check for that event in your goroutines and you also need to
    make sure that the "channel network" can handle such abrupt closes.
    Usually, there is a need to handle some corner-cases for this to work.

    Another possiblity, I would definitely look at, would be to use something
    where you "tombstone" processes and then having the tombstone's still
    process "structural" events and passing them on, but stopping processing.
    This allows a more controlled closedown of the channel network. I believe
    there are several packages out there which tries to abstract these ideas.

    There is also the 'context' package, of which there is a blog post here:

    https://blog.golang.org/context

    it may be possible to use that idiom. Set up your computation in the same
    context and then terminate that context when you decide to abort the
    computation. If each goroutine in the context does the "right thing(tm)",
    then it should be possible to avoid the panic.

    On Mon Jan 12 2015 at 6:49:04 PM Aki Balogh wrote:

    Hi Nick,

    Thanks for the prompt response.

    I understand that, in your code, you're breaking the loop if the timeout
    is hit. This does indeed take away the need for WaitGroups.

    However, in my code, there's an event that happens that requires me to
    halt all goroutines and move on, to save time.

    In your code, you're sleeping at the end for 1 second, presumably to
    'mop-up' any running goroutines, but I can't do that, since I'm optimizing
    for speed and need the code to run and return as quickly as possible.

    Here's my code: http://play.golang.org/p/6SchAM3-sU

    (Please do let me know if I'm missing the point of how your code works.)



    On Mon, Jan 12, 2015 at 12:15 PM, Nick Craig-Wood <nick@craig-wood.com>
    wrote:
    On 12/01/15 15:25, akibalogh@gmail.com wrote:
    I have a main function that spins off goroutines for processing and uses
    a signal channel to track when the goroutines should return. If the
    signal channel is closed, then the goroutines should return. Once all
    have returned, I do additional processing in the main function.

    However, I have an additional requirement, that if certain conditions
    are met, all processing in goroutines should stop and we should
    immediately move on in the main function.

    I can trigger the channel close, but some goroutines are still
    processing, so sometimes I get panics as they try to write to channels
    that had been closed in the meantime. (I'm noticing panics even if I
    check the signal channel right before writing to any other channel.)

    I can't use a WaitGroup because I might not want to wait until all
    goroutines are done.

    Any ideas on what I'm missing?
    Here is an idea for you

    http://play.golang.org/p/AoH_n3UDWY

    Note how I've separated the done test out of the select in the writer

    func writer(out chan<- int, finished chan bool) {
    outer:
    for {
    out <- 1
    select {
    case <-finished:
    break outer
    }

    }
    fmt.Println("Ending writer")
    }

    No waitgroups required.

    --
    Nick Craig-Wood <nick@craig-wood.com> -- http://www.craig-wood.com/nick
    --
    You received this message because you are subscribed to the Google
    Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send
    an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
    --
    You received this message because you are subscribed to the Google Groups
    "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an
    email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
  • Aki Balogh at Jan 16, 2015 at 8:08 pm
    Thanks, Matt. It was just an oversight that I wasn't checking timeChan in
    the select. In my code, I was checking it there.

    Interesting idea not to close the "a" channel, and let GC clean it up. I
    thought that this could lead to a memory leak. I am now testing this -- so
    far so good, no leaks.

    Where in the main goroutine do you suggest I assign a = nil ?

    Thanks,
    Aki

    On Wed, Jan 14, 2015 at 9:21 PM, Matt Harden wrote:

    Perhaps the code you posted to the playground is incomplete, but nothing
    ever reads from the timeChan channel, and nothing closes the done channel
    until all processing is complete. Perhaps in line 47, you intended to have
    "case <-timeChan" instead of "case <-done".

    Assuming that change was made, I would avoid a race condition in function
    f by performing your send to the a channel in the same select as you check
    the done channel. In this case nothing should panic due to channel issues.
    Don't close the "a" channel, just let it be GC'd once all references go
    away; you might want to assign a = nil in the main goroutine so it won't
    hold a reference.

    http://play.golang.org/p/8G6UothCud
    On Mon Jan 12 2015 at 12:10:20 PM Aki Balogh wrote:

    Thanks, Jesper (and Benjamin).

    At this point, I'm checking whether the done channel is closed at every
    step of processing. But I'm still encountering corner cases where I see
    panics. I recover from the panics -- not a problem since those goroutines
    were going to be thrown away anyway -- but it's messy code and fills up my
    error log.

    I will check out the tomb package, contexts and the blog post on
    pipelines.


    Akos (Aki) Balogh
    Co-Founder, MarketMuse
    https://www.marketmuse.com

    On Mon, Jan 12, 2015 at 12:57 PM, Jesper Louis Andersen <
    jesper.louis.andersen@gmail.com> wrote:
    If you need to stop in the middle of some work and move on, then you
    need to regularly check for that event in your goroutines and you also need
    to make sure that the "channel network" can handle such abrupt closes.
    Usually, there is a need to handle some corner-cases for this to work.

    Another possiblity, I would definitely look at, would be to use
    something where you "tombstone" processes and then having the tombstone's
    still process "structural" events and passing them on, but stopping
    processing. This allows a more controlled closedown of the channel network.
    I believe there are several packages out there which tries to abstract
    these ideas.

    There is also the 'context' package, of which there is a blog post here:

    https://blog.golang.org/context

    it may be possible to use that idiom. Set up your computation in the
    same context and then terminate that context when you decide to abort the
    computation. If each goroutine in the context does the "right thing(tm)",
    then it should be possible to avoid the panic.

    On Mon Jan 12 2015 at 6:49:04 PM Aki Balogh wrote:

    Hi Nick,

    Thanks for the prompt response.

    I understand that, in your code, you're breaking the loop if the
    timeout is hit. This does indeed take away the need for WaitGroups.

    However, in my code, there's an event that happens that requires me to
    halt all goroutines and move on, to save time.

    In your code, you're sleeping at the end for 1 second, presumably to
    'mop-up' any running goroutines, but I can't do that, since I'm optimizing
    for speed and need the code to run and return as quickly as possible.

    Here's my code: http://play.golang.org/p/6SchAM3-sU

    (Please do let me know if I'm missing the point of how your code works.)



    On Mon, Jan 12, 2015 at 12:15 PM, Nick Craig-Wood <nick@craig-wood.com>
    wrote:
    On 12/01/15 15:25, akibalogh@gmail.com wrote:
    I have a main function that spins off goroutines for processing and uses
    a signal channel to track when the goroutines should return. If the
    signal channel is closed, then the goroutines should return. Once all
    have returned, I do additional processing in the main function.

    However, I have an additional requirement, that if certain conditions
    are met, all processing in goroutines should stop and we should
    immediately move on in the main function.

    I can trigger the channel close, but some goroutines are still
    processing, so sometimes I get panics as they try to write to channels
    that had been closed in the meantime. (I'm noticing panics even if I
    check the signal channel right before writing to any other channel.)

    I can't use a WaitGroup because I might not want to wait until all
    goroutines are done.

    Any ideas on what I'm missing?
    Here is an idea for you

    http://play.golang.org/p/AoH_n3UDWY

    Note how I've separated the done test out of the select in the writer

    func writer(out chan<- int, finished chan bool) {
    outer:
    for {
    out <- 1
    select {
    case <-finished:
    break outer
    }

    }
    fmt.Println("Ending writer")
    }

    No waitgroups required.

    --
    Nick Craig-Wood <nick@craig-wood.com> --
    http://www.craig-wood.com/nick
    --
    You received this message because you are subscribed to the Google
    Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send
    an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
    --
    You received this message because you are subscribed to the Google Groups
    "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an
    email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
  • Matt Harden at Jan 17, 2015 at 3:31 am
    As soon as you aren't using the channel stored in "a" anymore, assign a =
    nil. That way a reference won't be held by that goroutine and the GC will
    be able to collect the channel. There is no need to do this if "a" will go
    out of scope immediately anyway, like at the end of a function.
    On Fri Jan 16 2015 at 2:08:14 PM Aki Balogh wrote:

    Thanks, Matt. It was just an oversight that I wasn't checking timeChan in
    the select. In my code, I was checking it there.

    Interesting idea not to close the "a" channel, and let GC clean it up. I
    thought that this could lead to a memory leak. I am now testing this -- so
    far so good, no leaks.

    Where in the main goroutine do you suggest I assign a = nil ?

    Thanks,
    Aki

    On Wed, Jan 14, 2015 at 9:21 PM, Matt Harden wrote:

    Perhaps the code you posted to the playground is incomplete, but nothing
    ever reads from the timeChan channel, and nothing closes the done channel
    until all processing is complete. Perhaps in line 47, you intended to have
    "case <-timeChan" instead of "case <-done".

    Assuming that change was made, I would avoid a race condition in function
    f by performing your send to the a channel in the same select as you check
    the done channel. In this case nothing should panic due to channel issues.
    Don't close the "a" channel, just let it be GC'd once all references go
    away; you might want to assign a = nil in the main goroutine so it won't
    hold a reference.

    http://play.golang.org/p/8G6UothCud
    On Mon Jan 12 2015 at 12:10:20 PM Aki Balogh wrote:

    Thanks, Jesper (and Benjamin).

    At this point, I'm checking whether the done channel is closed at every
    step of processing. But I'm still encountering corner cases where I see
    panics. I recover from the panics -- not a problem since those goroutines
    were going to be thrown away anyway -- but it's messy code and fills up my
    error log.

    I will check out the tomb package, contexts and the blog post on
    pipelines.


    Akos (Aki) Balogh
    Co-Founder, MarketMuse
    https://www.marketmuse.com

    On Mon, Jan 12, 2015 at 12:57 PM, Jesper Louis Andersen <
    jesper.louis.andersen@gmail.com> wrote:
    If you need to stop in the middle of some work and move on, then you
    need to regularly check for that event in your goroutines and you also need
    to make sure that the "channel network" can handle such abrupt closes.
    Usually, there is a need to handle some corner-cases for this to work.

    Another possiblity, I would definitely look at, would be to use
    something where you "tombstone" processes and then having the tombstone's
    still process "structural" events and passing them on, but stopping
    processing. This allows a more controlled closedown of the channel network.
    I believe there are several packages out there which tries to abstract
    these ideas.

    There is also the 'context' package, of which there is a blog post here:

    https://blog.golang.org/context

    it may be possible to use that idiom. Set up your computation in the
    same context and then terminate that context when you decide to abort the
    computation. If each goroutine in the context does the "right thing(tm)",
    then it should be possible to avoid the panic.


    On Mon Jan 12 2015 at 6:49:04 PM Aki Balogh <akibalogh@gmail.com>
    wrote:
    Hi Nick,

    Thanks for the prompt response.

    I understand that, in your code, you're breaking the loop if the
    timeout is hit. This does indeed take away the need for WaitGroups.

    However, in my code, there's an event that happens that requires me to
    halt all goroutines and move on, to save time.

    In your code, you're sleeping at the end for 1 second, presumably to
    'mop-up' any running goroutines, but I can't do that, since I'm optimizing
    for speed and need the code to run and return as quickly as possible.

    Here's my code: http://play.golang.org/p/6SchAM3-sU

    (Please do let me know if I'm missing the point of how your code
    works.)



    On Mon, Jan 12, 2015 at 12:15 PM, Nick Craig-Wood <nick@craig-wood.com
    wrote:
    On 12/01/15 15:25, akibalogh@gmail.com wrote:
    I have a main function that spins off goroutines for processing and uses
    a signal channel to track when the goroutines should return. If the
    signal channel is closed, then the goroutines should return. Once all
    have returned, I do additional processing in the main function.

    However, I have an additional requirement, that if certain
    conditions
    are met, all processing in goroutines should stop and we should
    immediately move on in the main function.

    I can trigger the channel close, but some goroutines are still
    processing, so sometimes I get panics as they try to write to channels
    that had been closed in the meantime. (I'm noticing panics even if I
    check the signal channel right before writing to any other channel.)

    I can't use a WaitGroup because I might not want to wait until all
    goroutines are done.

    Any ideas on what I'm missing?
    Here is an idea for you

    http://play.golang.org/p/AoH_n3UDWY

    Note how I've separated the done test out of the select in the writer

    func writer(out chan<- int, finished chan bool) {
    outer:
    for {
    out <- 1
    select {
    case <-finished:
    break outer
    }

    }
    fmt.Println("Ending writer")
    }

    No waitgroups required.

    --
    Nick Craig-Wood <nick@craig-wood.com> --
    http://www.craig-wood.com/nick
    --
    You received this message because you are subscribed to the Google
    Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send
    an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
    --
    You received this message because you are subscribed to the Google
    Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send
    an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
  • Nick Craig-Wood at Jan 12, 2015 at 6:22 pm

    On 12/01/15 17:48, Aki Balogh wrote:
    I understand that, in your code, you're breaking the loop if the timeout
    is hit. This does indeed take away the need for WaitGroups.

    However, in my code, there's an event that happens that requires me to
    halt all goroutines and move on, to save time.

    In your code, you're sleeping at the end for 1 second, presumably to
    'mop-up' any running goroutines, but I can't do that, since I'm
    optimizing for speed and need the code to run and return as quickly as
    possible.
    You can just return and the running go routines will finish in the
    background. You've asked the running go routines to halt and they will,
    just not immediately.

    If you want to wait for running go routines to finish then you should
    use a WaitGroup.
    If you get your timeout to close the done channel it will all work

    http://play.golang.org/p/dGY_WFuogh


    --
    Nick Craig-Wood <nick@craig-wood.com> -- http://www.craig-wood.com/nick

    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
  • Aki Balogh at Jan 12, 2015 at 6:27 pm
    I thought that my timeout would "just work", but in practice I'm getting
    panics. Probably some edge cases that we're not aware of.

    context might be the way to go (https://blog.golang.org/context)


    Akos (Aki) Balogh
    Co-Founder, MarketMuse
    https://www.marketmuse.com
    On Mon, Jan 12, 2015 at 1:21 PM, Nick Craig-Wood wrote:
    On 12/01/15 17:48, Aki Balogh wrote:
    I understand that, in your code, you're breaking the loop if the timeout
    is hit. This does indeed take away the need for WaitGroups.

    However, in my code, there's an event that happens that requires me to
    halt all goroutines and move on, to save time.

    In your code, you're sleeping at the end for 1 second, presumably to
    'mop-up' any running goroutines, but I can't do that, since I'm
    optimizing for speed and need the code to run and return as quickly as
    possible.
    You can just return and the running go routines will finish in the
    background. You've asked the running go routines to halt and they will,
    just not immediately.

    If you want to wait for running go routines to finish then you should
    use a WaitGroup.
    If you get your timeout to close the done channel it will all work

    http://play.golang.org/p/dGY_WFuogh


    --
    Nick Craig-Wood <nick@craig-wood.com> -- http://www.craig-wood.com/nick
    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
  • Peter Waller at Jan 12, 2015 at 9:39 pm
    You might find time.After useful [1]. It removes the need for your
    timeout goroutine.

    So let's say you have a hard loop. You can cancel it like so:

    timeout := time.After(2*time.Seconds)

    for record := range millionsOfRecords {
       select {
       case <-timeout:
         return
       case <-cancelled:
         return
       default:
       }
       process(record)
    }

    The select on the timeout channel is pretty cheap. I've put it in
    critical processing loops and found it only cost me something like 10%
    of the runtime in my case. YMMV.

    [1] http://golang.org/pkg/time/#After
    On 12 January 2015 at 18:26, Aki Balogh wrote:
    I thought that my timeout would "just work", but in practice I'm getting
    panics. Probably some edge cases that we're not aware of.

    context might be the way to go (https://blog.golang.org/context)


    Akos (Aki) Balogh
    Co-Founder, MarketMuse
    https://www.marketmuse.com
    On Mon, Jan 12, 2015 at 1:21 PM, Nick Craig-Wood wrote:
    On 12/01/15 17:48, Aki Balogh wrote:
    I understand that, in your code, you're breaking the loop if the timeout
    is hit. This does indeed take away the need for WaitGroups.

    However, in my code, there's an event that happens that requires me to
    halt all goroutines and move on, to save time.

    In your code, you're sleeping at the end for 1 second, presumably to
    'mop-up' any running goroutines, but I can't do that, since I'm
    optimizing for speed and need the code to run and return as quickly as
    possible.
    You can just return and the running go routines will finish in the
    background. You've asked the running go routines to halt and they will,
    just not immediately.

    If you want to wait for running go routines to finish then you should
    use a WaitGroup.
    If you get your timeout to close the done channel it will all work

    http://play.golang.org/p/dGY_WFuogh


    --
    Nick Craig-Wood <nick@craig-wood.com> -- http://www.craig-wood.com/nick

    --
    You received this message because you are subscribed to the Google Groups
    "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an
    email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
  • Matt Harden at Jan 15, 2015 at 1:54 am

    On Mon Jan 12 2015 at 12:22:11 PM Nick Craig-Wood wrote:
    You can just return and the running go routines will finish in the
    background. You've asked the running go routines to halt and they will,
    just not immediately.
    Actually, when the main goroutine ends, the process exits without giving
    other goroutines a chance to finish or clean up. You can see this by
    commenting out the last Sleep in your code; none of the goroutines writes
    their "Ending" message. This is one of the main reasons WaitGroups are used
    in the first place.

    Regards,
    Matt

    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
  • Benjamin Measures at Jan 12, 2015 at 6:06 pm

    On Monday, 12 January 2015 15:25:52 UTC, akib...@gmail.com wrote:
    I'm running into this WaitGroup vs. signal channel debate with a slightly
    different type of problem.
    Since then there's been a good blog entry
    written: http://blog.golang.org/pipelines

    I have a main function that spins off goroutines for processing and uses a
    signal channel to track when the goroutines should return. If the signal
    channel is closed, then the goroutines should return. Once all have
    returned, I do additional processing in the main function.

    However, I have an additional requirement, that if certain conditions are
    met, all processing in goroutines should stop and we should immediately
    move on in the main function.

    I can trigger the channel close, but some goroutines are still processing,
    so sometimes I get panics as they try to write to channels that had been
    closed in the meantime. (I'm noticing panics even if I check the signal
    channel right before writing to any other channel.)

    I can't use a WaitGroup because I might not want to wait until all
    goroutines are done.

    Any ideas on what I'm missing?
    Channels should only be closed by the sender otherwise (without additional
    synchronisation) there'll be the race condition you seem to be
    experiencing. Dedicate a cancellation channel that the goroutines do not
    write to (but check during work periodically).

    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupgolang-nuts @
categoriesgo
postedJan 12, '15 at 4:28p
activeJan 17, '15 at 3:31a
posts13
users6
websitegolang.org

People

Translate

site design / logo © 2022 Grokbase