[go-nuts] Re: reaching for sync.WaitGroup feels wrong...

Nick Craig-Wood

at Jan 12, 2015 at 5:15 pm

On 12/01/15 15:25, akibalogh@gmail.com wrote:
I have a main function that spins off goroutines for processing and uses
a signal channel to track when the goroutines should return. If the
signal channel is closed, then the goroutines should return. Once all
have returned, I do additional processing in the main function.

However, I have an additional requirement, that if certain conditions
are met, all processing in goroutines should stop and we should
immediately move on in the main function.

I can trigger the channel close, but some goroutines are still
processing, so sometimes I get panics as they try to write to channels
that had been closed in the meantime. (I'm noticing panics even if I
check the signal channel right before writing to any other channel.)

I can't use a WaitGroup because I might not want to wait until all
goroutines are done.

Any ideas on what I'm missing?

Here is an idea for you

http://play.golang.org/p/AoH_n3UDWY

Note how I've separated the done test out of the select in the writer

func writer(out chan<- int, finished chan bool) {
outer:
  for {
   out <- 1
   select {
   case <-finished:
    break outer
   }

  }
  fmt.Println("Ending writer")
}

No waitgroups required.

--
Nick Craig-Wood <nick@craig-wood.com> -- http://www.craig-wood.com/nick

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

reply | permalink

Aki Balogh

at Jan 12, 2015 at 5:49 pm

⇧

Hi Nick,

Thanks for the prompt response.

I understand that, in your code, you're breaking the loop if the timeout is
hit. This does indeed take away the need for WaitGroups.

However, in my code, there's an event that happens that requires me to halt
all goroutines and move on, to save time.

In your code, you're sleeping at the end for 1 second, presumably to
'mop-up' any running goroutines, but I can't do that, since I'm optimizing
for speed and need the code to run and return as quickly as possible.

Here's my code: http://play.golang.org/p/6SchAM3-sU

(Please do let me know if I'm missing the point of how your code works.)

On Mon, Jan 12, 2015 at 12:15 PM, Nick Craig-Wood wrote:

On 12/01/15 15:25, akibalogh@gmail.com wrote:
I have a main function that spins off goroutines for processing and uses
a signal channel to track when the goroutines should return. If the
signal channel is closed, then the goroutines should return. Once all
have returned, I do additional processing in the main function.

However, I have an additional requirement, that if certain conditions
are met, all processing in goroutines should stop and we should
immediately move on in the main function.

I can trigger the channel close, but some goroutines are still
processing, so sometimes I get panics as they try to write to channels
that had been closed in the meantime. (I'm noticing panics even if I
check the signal channel right before writing to any other channel.)

I can't use a WaitGroup because I might not want to wait until all
goroutines are done.

Any ideas on what I'm missing?

Here is an idea for you

http://play.golang.org/p/AoH_n3UDWY

Note how I've separated the done test out of the select in the writer

func writer(out chan<- int, finished chan bool) {
outer:
for {
out <- 1
select {
case <-finished:
break outer
}

}
fmt.Println("Ending writer")
}

No waitgroups required.

--
Nick Craig-Wood <nick@craig-wood.com> -- http://www.craig-wood.com/nick

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

reply | permalink

Jesper Louis Andersen

at Jan 12, 2015 at 5:58 pm

⇧

If you need to stop in the middle of some work and move on, then you need
to regularly check for that event in your goroutines and you also need to
make sure that the "channel network" can handle such abrupt closes.
Usually, there is a need to handle some corner-cases for this to work.

Another possiblity, I would definitely look at, would be to use something
where you "tombstone" processes and then having the tombstone's still
process "structural" events and passing them on, but stopping processing.
This allows a more controlled closedown of the channel network. I believe
there are several packages out there which tries to abstract these ideas.

There is also the 'context' package, of which there is a blog post here:

https://blog.golang.org/context

it may be possible to use that idiom. Set up your computation in the same
context and then terminate that context when you decide to abort the
computation. If each goroutine in the context does the "right thing(tm)",
then it should be possible to avoid the panic.

On Mon Jan 12 2015 at 6:49:04 PM Aki Balogh wrote:

Hi Nick,

Thanks for the prompt response.

I understand that, in your code, you're breaking the loop if the timeout
is hit. This does indeed take away the need for WaitGroups.

However, in my code, there's an event that happens that requires me to
halt all goroutines and move on, to save time.

In your code, you're sleeping at the end for 1 second, presumably to
'mop-up' any running goroutines, but I can't do that, since I'm optimizing
for speed and need the code to run and return as quickly as possible.

Here's my code: http://play.golang.org/p/6SchAM3-sU

(Please do let me know if I'm missing the point of how your code works.)

On Mon, Jan 12, 2015 at 12:15 PM, Nick Craig-Wood wrote:

On 12/01/15 15:25, akibalogh@gmail.com wrote:
I have a main function that spins off goroutines for processing and uses
a signal channel to track when the goroutines should return. If the
signal channel is closed, then the goroutines should return. Once all
have returned, I do additional processing in the main function.

However, I have an additional requirement, that if certain conditions
are met, all processing in goroutines should stop and we should
immediately move on in the main function.

I can trigger the channel close, but some goroutines are still
processing, so sometimes I get panics as they try to write to channels
that had been closed in the meantime. (I'm noticing panics even if I
check the signal channel right before writing to any other channel.)

I can't use a WaitGroup because I might not want to wait until all
goroutines are done.

Any ideas on what I'm missing?

Here is an idea for you

http://play.golang.org/p/AoH_n3UDWY

Note how I've separated the done test out of the select in the writer

func writer(out chan<- int, finished chan bool) {
outer:
for {
out <- 1
select {
case <-finished:
break outer
}

}
fmt.Println("Ending writer")
}

No waitgroups required.

--
Nick Craig-Wood <nick@craig-wood.com> -- http://www.craig-wood.com/nick

--
You received this message because you are subscribed to the Google Groups
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to golang-nuts+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

reply | permalink

Aki Balogh

at Jan 12, 2015 at 6:10 pm

⇧

Thanks, Jesper (and Benjamin).

At this point, I'm checking whether the done channel is closed at every
step of processing. But I'm still encountering corner cases where I see
panics. I recover from the panics -- not a problem since those goroutines
were going to be thrown away anyway -- but it's messy code and fills up my
error log.

I will check out the tomb package, contexts and the blog post on pipelines.

Akos (Aki) Balogh
Co-Founder, MarketMuse
https://www.marketmuse.com

On Mon, Jan 12, 2015 at 12:57 PM, Jesper Louis Andersen wrote:

If you need to stop in the middle of some work and move on, then you need
to regularly check for that event in your goroutines and you also need to
make sure that the "channel network" can handle such abrupt closes.
Usually, there is a need to handle some corner-cases for this to work.

Another possiblity, I would definitely look at, would be to use something
where you "tombstone" processes and then having the tombstone's still
process "structural" events and passing them on, but stopping processing.
This allows a more controlled closedown of the channel network. I believe
there are several packages out there which tries to abstract these ideas.

There is also the 'context' package, of which there is a blog post here:

https://blog.golang.org/context

it may be possible to use that idiom. Set up your computation in the same
context and then terminate that context when you decide to abort the
computation. If each goroutine in the context does the "right thing(tm)",
then it should be possible to avoid the panic.

On Mon Jan 12 2015 at 6:49:04 PM Aki Balogh wrote:

Hi Nick,

Thanks for the prompt response.

I understand that, in your code, you're breaking the loop if the timeout
is hit. This does indeed take away the need for WaitGroups.

However, in my code, there's an event that happens that requires me to
halt all goroutines and move on, to save time.

In your code, you're sleeping at the end for 1 second, presumably to
'mop-up' any running goroutines, but I can't do that, since I'm optimizing
for speed and need the code to run and return as quickly as possible.

Here's my code: http://play.golang.org/p/6SchAM3-sU

(Please do let me know if I'm missing the point of how your code works.)

On Mon, Jan 12, 2015 at 12:15 PM, Nick Craig-Wood <nick@craig-wood.com>
wrote:

On 12/01/15 15:25, akibalogh@gmail.com wrote:
I have a main function that spins off goroutines for processing and uses
a signal channel to track when the goroutines should return. If the
signal channel is closed, then the goroutines should return. Once all
have returned, I do additional processing in the main function.

However, I have an additional requirement, that if certain conditions
are met, all processing in goroutines should stop and we should
immediately move on in the main function.

I can trigger the channel close, but some goroutines are still
processing, so sometimes I get panics as they try to write to channels
that had been closed in the meantime. (I'm noticing panics even if I
check the signal channel right before writing to any other channel.)

I can't use a WaitGroup because I might not want to wait until all
goroutines are done.

Any ideas on what I'm missing?

Here is an idea for you

http://play.golang.org/p/AoH_n3UDWY

Note how I've separated the done test out of the select in the writer

func writer(out chan<- int, finished chan bool) {
outer:
for {
out <- 1
select {
case <-finished:
break outer
}

}
fmt.Println("Ending writer")
}

No waitgroups required.

--
Nick Craig-Wood <nick@craig-wood.com> -- http://www.craig-wood.com/nick

--
You received this message because you are subscribed to the Google Groups
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to golang-nuts+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

reply | permalink

Matt Harden

at Jan 15, 2015 at 2:21 am

⇧

Perhaps the code you posted to the playground is incomplete, but nothing
ever reads from the timeChan channel, and nothing closes the done channel
until all processing is complete. Perhaps in line 47, you intended to have
"case <-timeChan" instead of "case <-done".

Assuming that change was made, I would avoid a race condition in function f
by performing your send to the a channel in the same select as you check
the done channel. In this case nothing should panic due to channel issues.
Don't close the "a" channel, just let it be GC'd once all references go
away; you might want to assign a = nil in the main goroutine so it won't
hold a reference.

http://play.golang.org/p/8G6UothCud

On Mon Jan 12 2015 at 12:10:20 PM Aki Balogh wrote:

Thanks, Jesper (and Benjamin).

At this point, I'm checking whether the done channel is closed at every
step of processing. But I'm still encountering corner cases where I see
panics. I recover from the panics -- not a problem since those goroutines
were going to be thrown away anyway -- but it's messy code and fills up my
error log.

I will check out the tomb package, contexts and the blog post on pipelines.

Akos (Aki) Balogh
Co-Founder, MarketMuse
https://www.marketmuse.com

On Mon, Jan 12, 2015 at 12:57 PM, Jesper Louis Andersen <
jesper.louis.andersen@gmail.com> wrote:

If you need to stop in the middle of some work and move on, then you need
to regularly check for that event in your goroutines and you also need to
make sure that the "channel network" can handle such abrupt closes.
Usually, there is a need to handle some corner-cases for this to work.

Another possiblity, I would definitely look at, would be to use something
where you "tombstone" processes and then having the tombstone's still
process "structural" events and passing them on, but stopping processing.
This allows a more controlled closedown of the channel network. I believe
there are several packages out there which tries to abstract these ideas.

There is also the 'context' package, of which there is a blog post here:

https://blog.golang.org/context

it may be possible to use that idiom. Set up your computation in the same
context and then terminate that context when you decide to abort the
computation. If each goroutine in the context does the "right thing(tm)",
then it should be possible to avoid the panic.

On Mon Jan 12 2015 at 6:49:04 PM Aki Balogh wrote:

Hi Nick,

Thanks for the prompt response.

I understand that, in your code, you're breaking the loop if the timeout
is hit. This does indeed take away the need for WaitGroups.

However, in my code, there's an event that happens that requires me to
halt all goroutines and move on, to save time.

In your code, you're sleeping at the end for 1 second, presumably to
'mop-up' any running goroutines, but I can't do that, since I'm optimizing
for speed and need the code to run and return as quickly as possible.

Here's my code: http://play.golang.org/p/6SchAM3-sU

(Please do let me know if I'm missing the point of how your code works.)

On Mon, Jan 12, 2015 at 12:15 PM, Nick Craig-Wood <nick@craig-wood.com>
wrote:

On 12/01/15 15:25, akibalogh@gmail.com wrote:
I have a main function that spins off goroutines for processing and uses
a signal channel to track when the goroutines should return. If the
signal channel is closed, then the goroutines should return. Once all
have returned, I do additional processing in the main function.

However, I have an additional requirement, that if certain conditions
are met, all processing in goroutines should stop and we should
immediately move on in the main function.

I can trigger the channel close, but some goroutines are still
processing, so sometimes I get panics as they try to write to channels
that had been closed in the meantime. (I'm noticing panics even if I
check the signal channel right before writing to any other channel.)

I can't use a WaitGroup because I might not want to wait until all
goroutines are done.

Any ideas on what I'm missing?

Here is an idea for you

http://play.golang.org/p/AoH_n3UDWY

Note how I've separated the done test out of the select in the writer

func writer(out chan<- int, finished chan bool) {
outer:
for {
out <- 1
select {
case <-finished:
break outer
}

}
fmt.Println("Ending writer")
}

No waitgroups required.

--
Nick Craig-Wood <nick@craig-wood.com> -- http://www.craig-wood.com/nick

--
You received this message because you are subscribed to the Google
Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to golang-nuts+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to golang-nuts+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

reply | permalink

Aki Balogh

at Jan 16, 2015 at 8:08 pm

⇧

Thanks, Matt. It was just an oversight that I wasn't checking timeChan in
the select. In my code, I was checking it there.

Interesting idea not to close the "a" channel, and let GC clean it up. I
thought that this could lead to a memory leak. I am now testing this -- so
far so good, no leaks.

Where in the main goroutine do you suggest I assign a = nil ?

Thanks,
Aki

On Wed, Jan 14, 2015 at 9:21 PM, Matt Harden wrote:

Perhaps the code you posted to the playground is incomplete, but nothing
ever reads from the timeChan channel, and nothing closes the done channel
until all processing is complete. Perhaps in line 47, you intended to have
"case <-timeChan" instead of "case <-done".

Assuming that change was made, I would avoid a race condition in function
f by performing your send to the a channel in the same select as you check
the done channel. In this case nothing should panic due to channel issues.
Don't close the "a" channel, just let it be GC'd once all references go
away; you might want to assign a = nil in the main goroutine so it won't
hold a reference.

http://play.golang.org/p/8G6UothCud

On Mon Jan 12 2015 at 12:10:20 PM Aki Balogh wrote:

Thanks, Jesper (and Benjamin).

At this point, I'm checking whether the done channel is closed at every
step of processing. But I'm still encountering corner cases where I see
panics. I recover from the panics -- not a problem since those goroutines
were going to be thrown away anyway -- but it's messy code and fills up my
error log.

I will check out the tomb package, contexts and the blog post on
pipelines.

Akos (Aki) Balogh
Co-Founder, MarketMuse
https://www.marketmuse.com

On Mon, Jan 12, 2015 at 12:57 PM, Jesper Louis Andersen <
jesper.louis.andersen@gmail.com> wrote:

If you need to stop in the middle of some work and move on, then you
need to regularly check for that event in your goroutines and you also need
to make sure that the "channel network" can handle such abrupt closes.
Usually, there is a need to handle some corner-cases for this to work.

Another possiblity, I would definitely look at, would be to use
something where you "tombstone" processes and then having the tombstone's
still process "structural" events and passing them on, but stopping
processing. This allows a more controlled closedown of the channel network.
I believe there are several packages out there which tries to abstract
these ideas.

There is also the 'context' package, of which there is a blog post here:

https://blog.golang.org/context

it may be possible to use that idiom. Set up your computation in the
same context and then terminate that context when you decide to abort the
computation. If each goroutine in the context does the "right thing(tm)",
then it should be possible to avoid the panic.

On Mon Jan 12 2015 at 6:49:04 PM Aki Balogh wrote:

Hi Nick,

Thanks for the prompt response.

I understand that, in your code, you're breaking the loop if the
timeout is hit. This does indeed take away the need for WaitGroups.

However, in my code, there's an event that happens that requires me to
halt all goroutines and move on, to save time.

In your code, you're sleeping at the end for 1 second, presumably to
'mop-up' any running goroutines, but I can't do that, since I'm optimizing
for speed and need the code to run and return as quickly as possible.

Here's my code: http://play.golang.org/p/6SchAM3-sU

(Please do let me know if I'm missing the point of how your code works.)

On Mon, Jan 12, 2015 at 12:15 PM, Nick Craig-Wood <nick@craig-wood.com>
wrote:

On 12/01/15 15:25, akibalogh@gmail.com wrote:
I have a main function that spins off goroutines for processing and uses
a signal channel to track when the goroutines should return. If the
signal channel is closed, then the goroutines should return. Once all
have returned, I do additional processing in the main function.

However, I have an additional requirement, that if certain conditions
are met, all processing in goroutines should stop and we should
immediately move on in the main function.

I can trigger the channel close, but some goroutines are still
processing, so sometimes I get panics as they try to write to channels
that had been closed in the meantime. (I'm noticing panics even if I
check the signal channel right before writing to any other channel.)

I can't use a WaitGroup because I might not want to wait until all
goroutines are done.

Any ideas on what I'm missing?

Here is an idea for you

http://play.golang.org/p/AoH_n3UDWY

Note how I've separated the done test out of the select in the writer

func writer(out chan<- int, finished chan bool) {
outer:
for {
out <- 1
select {
case <-finished:
break outer
}

}
fmt.Println("Ending writer")
}

No waitgroups required.

--
Nick Craig-Wood <nick@craig-wood.com> --
http://www.craig-wood.com/nick

--
You received this message because you are subscribed to the Google
Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to golang-nuts+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to golang-nuts+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

reply | permalink

Matt Harden

at Jan 17, 2015 at 3:31 am

⇧

As soon as you aren't using the channel stored in "a" anymore, assign a =
nil. That way a reference won't be held by that goroutine and the GC will
be able to collect the channel. There is no need to do this if "a" will go
out of scope immediately anyway, like at the end of a function.

On Fri Jan 16 2015 at 2:08:14 PM Aki Balogh wrote:

Thanks, Matt. It was just an oversight that I wasn't checking timeChan in
the select. In my code, I was checking it there.

Interesting idea not to close the "a" channel, and let GC clean it up. I
thought that this could lead to a memory leak. I am now testing this -- so
far so good, no leaks.

Where in the main goroutine do you suggest I assign a = nil ?

Thanks,
Aki

On Wed, Jan 14, 2015 at 9:21 PM, Matt Harden wrote:

Perhaps the code you posted to the playground is incomplete, but nothing
ever reads from the timeChan channel, and nothing closes the done channel
until all processing is complete. Perhaps in line 47, you intended to have
"case <-timeChan" instead of "case <-done".

Assuming that change was made, I would avoid a race condition in function
f by performing your send to the a channel in the same select as you check
the done channel. In this case nothing should panic due to channel issues.
Don't close the "a" channel, just let it be GC'd once all references go
away; you might want to assign a = nil in the main goroutine so it won't
hold a reference.

http://play.golang.org/p/8G6UothCud

On Mon Jan 12 2015 at 12:10:20 PM Aki Balogh wrote:

Thanks, Jesper (and Benjamin).

At this point, I'm checking whether the done channel is closed at every
step of processing. But I'm still encountering corner cases where I see
panics. I recover from the panics -- not a problem since those goroutines
were going to be thrown away anyway -- but it's messy code and fills up my
error log.

I will check out the tomb package, contexts and the blog post on
pipelines.

Akos (Aki) Balogh
Co-Founder, MarketMuse
https://www.marketmuse.com

On Mon, Jan 12, 2015 at 12:57 PM, Jesper Louis Andersen <
jesper.louis.andersen@gmail.com> wrote:

If you need to stop in the middle of some work and move on, then you
need to regularly check for that event in your goroutines and you also need
to make sure that the "channel network" can handle such abrupt closes.
Usually, there is a need to handle some corner-cases for this to work.

Another possiblity, I would definitely look at, would be to use
something where you "tombstone" processes and then having the tombstone's
still process "structural" events and passing them on, but stopping
processing. This allows a more controlled closedown of the channel network.
I believe there are several packages out there which tries to abstract
these ideas.

There is also the 'context' package, of which there is a blog post here:

https://blog.golang.org/context

it may be possible to use that idiom. Set up your computation in the
same context and then terminate that context when you decide to abort the
computation. If each goroutine in the context does the "right thing(tm)",
then it should be possible to avoid the panic.

On Mon Jan 12 2015 at 6:49:04 PM Aki Balogh <akibalogh@gmail.com>
wrote:

Hi Nick,

Thanks for the prompt response.

I understand that, in your code, you're breaking the loop if the
timeout is hit. This does indeed take away the need for WaitGroups.

However, in my code, there's an event that happens that requires me to
halt all goroutines and move on, to save time.

In your code, you're sleeping at the end for 1 second, presumably to
'mop-up' any running goroutines, but I can't do that, since I'm optimizing
for speed and need the code to run and return as quickly as possible.

Here's my code: http://play.golang.org/p/6SchAM3-sU

(Please do let me know if I'm missing the point of how your code
works.)

On Mon, Jan 12, 2015 at 12:15 PM, Nick Craig-Wood <nick@craig-wood.com

wrote:

On 12/01/15 15:25, akibalogh@gmail.com wrote:
I have a main function that spins off goroutines for processing and uses
a signal channel to track when the goroutines should return. If the
signal channel is closed, then the goroutines should return. Once all
have returned, I do additional processing in the main function.

However, I have an additional requirement, that if certain
conditions
are met, all processing in goroutines should stop and we should
immediately move on in the main function.

I can trigger the channel close, but some goroutines are still
processing, so sometimes I get panics as they try to write to channels
that had been closed in the meantime. (I'm noticing panics even if I
check the signal channel right before writing to any other channel.)

I can't use a WaitGroup because I might not want to wait until all
goroutines are done.

Any ideas on what I'm missing?

Here is an idea for you

http://play.golang.org/p/AoH_n3UDWY

Note how I've separated the done test out of the select in the writer

func writer(out chan<- int, finished chan bool) {
outer:
for {
out <- 1
select {
case <-finished:
break outer
}

}
fmt.Println("Ending writer")
}

No waitgroups required.

--
Nick Craig-Wood <nick@craig-wood.com> --
http://www.craig-wood.com/nick

--
You received this message because you are subscribed to the Google
Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to golang-nuts+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

reply | permalink

Nick Craig-Wood

at Jan 12, 2015 at 6:22 pm

⇧

On 12/01/15 17:48, Aki Balogh wrote:
I understand that, in your code, you're breaking the loop if the timeout
is hit. This does indeed take away the need for WaitGroups.

However, in my code, there's an event that happens that requires me to
halt all goroutines and move on, to save time.

In your code, you're sleeping at the end for 1 second, presumably to
'mop-up' any running goroutines, but I can't do that, since I'm
optimizing for speed and need the code to run and return as quickly as
possible.

You can just return and the running go routines will finish in the
background. You've asked the running go routines to halt and they will,
just not immediately.

If you want to wait for running go routines to finish then you should
use a WaitGroup.

Here's my code: http://play.golang.org/p/6SchAM3-sU

If you get your timeout to close the done channel it will all work

http://play.golang.org/p/dGY_WFuogh

--
Nick Craig-Wood <nick@craig-wood.com> -- http://www.craig-wood.com/nick

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

reply | permalink

Aki Balogh

at Jan 12, 2015 at 6:27 pm

⇧

I thought that my timeout would "just work", but in practice I'm getting
panics. Probably some edge cases that we're not aware of.

context might be the way to go (https://blog.golang.org/context)

Akos (Aki) Balogh
Co-Founder, MarketMuse
https://www.marketmuse.com

On Mon, Jan 12, 2015 at 1:21 PM, Nick Craig-Wood wrote:

On 12/01/15 17:48, Aki Balogh wrote:
I understand that, in your code, you're breaking the loop if the timeout
is hit. This does indeed take away the need for WaitGroups.

However, in my code, there's an event that happens that requires me to
halt all goroutines and move on, to save time.

In your code, you're sleeping at the end for 1 second, presumably to
'mop-up' any running goroutines, but I can't do that, since I'm
optimizing for speed and need the code to run and return as quickly as
possible.

You can just return and the running go routines will finish in the
background. You've asked the running go routines to halt and they will,
just not immediately.

If you want to wait for running go routines to finish then you should
use a WaitGroup.

Here's my code: http://play.golang.org/p/6SchAM3-sU

If you get your timeout to close the done channel it will all work

http://play.golang.org/p/dGY_WFuogh

--
Nick Craig-Wood <nick@craig-wood.com> -- http://www.craig-wood.com/nick

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

reply | permalink

Peter Waller

at Jan 12, 2015 at 9:39 pm

⇧

You might find time.After useful [1]. It removes the need for your
timeout goroutine.

So let's say you have a hard loop. You can cancel it like so:

timeout := time.After(2*time.Seconds)

for record := range millionsOfRecords {
   select {
   case <-timeout:
     return
   case <-cancelled:
     return
   default:
   }
   process(record)
}

The select on the timeout channel is pretty cheap. I've put it in
critical processing loops and found it only cost me something like 10%
of the runtime in my case. YMMV.

[1] http://golang.org/pkg/time/#After

On 12 January 2015 at 18:26, Aki Balogh wrote:
I thought that my timeout would "just work", but in practice I'm getting
panics. Probably some edge cases that we're not aware of.

context might be the way to go (https://blog.golang.org/context)

Akos (Aki) Balogh
Co-Founder, MarketMuse
https://www.marketmuse.com

On Mon, Jan 12, 2015 at 1:21 PM, Nick Craig-Wood wrote:

On 12/01/15 17:48, Aki Balogh wrote:
I understand that, in your code, you're breaking the loop if the timeout
is hit. This does indeed take away the need for WaitGroups.

However, in my code, there's an event that happens that requires me to
halt all goroutines and move on, to save time.

In your code, you're sleeping at the end for 1 second, presumably to
'mop-up' any running goroutines, but I can't do that, since I'm
optimizing for speed and need the code to run and return as quickly as
possible.

You can just return and the running go routines will finish in the
background. You've asked the running go routines to halt and they will,
just not immediately.

If you want to wait for running go routines to finish then you should
use a WaitGroup.

Here's my code: http://play.golang.org/p/6SchAM3-sU

If you get your timeout to close the done channel it will all work

http://play.golang.org/p/dGY_WFuogh

--
Nick Craig-Wood <nick@craig-wood.com> -- http://www.craig-wood.com/nick

--
You received this message because you are subscribed to the Google Groups
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to golang-nuts+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

reply | permalink

Matt Harden

at Jan 15, 2015 at 1:54 am

⇧

On Mon Jan 12 2015 at 12:22:11 PM Nick Craig-Wood wrote:
You can just return and the running go routines will finish in the
background. You've asked the running go routines to halt and they will,
just not immediately.

Actually, when the main goroutine ends, the process exits without giving
other goroutines a chance to finish or clean up. You can see this by
commenting out the last Sleep in your code; none of the goroutines writes
their "Ending" message. This is one of the main reasons WaitGroups are used
in the first place.

Regards,
Matt

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

reply | permalink

Benjamin Measures

at Jan 12, 2015 at 6:06 pm

⇧

On Monday, 12 January 2015 15:25:52 UTC, akib...@gmail.com wrote:
I'm running into this WaitGroup vs. signal channel debate with a slightly
different type of problem.

Since then there's been a good blog entry
written: http://blog.golang.org/pipelines

I have a main function that spins off goroutines for processing and uses a

signal channel to track when the goroutines should return. If the signal
channel is closed, then the goroutines should return. Once all have
returned, I do additional processing in the main function.

However, I have an additional requirement, that if certain conditions are
met, all processing in goroutines should stop and we should immediately
move on in the main function.

I can trigger the channel close, but some goroutines are still processing,
so sometimes I get panics as they try to write to channels that had been
closed in the meantime. (I'm noticing panics even if I check the signal
channel right before writing to any other channel.)

I can't use a WaitGroup because I might not want to wait until all
goroutines are done.

Any ideas on what I'm missing?

Channels should only be closed by the sender otherwise (without additional
synchronisation) there'll be the race condition you seem to be
experiencing. Dedicate a cancellation channel that the goroutines do not
write to (but check during work periodically).

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

reply | permalink

group	golang-nuts @
categories	go
posted	Jan 12, '15 at 4:28p
active	Jan 17, '15 at 3:31a
posts	13
users	6
website	golang.org

responses	expanded	Hotkey: s
font	variable	Hotkey: f
user style	avatars	Hotkey: a

[go-nuts] Re: reaching for sync.WaitGroup feels wrong...

Search Discussions

Related Discussions

6 users in discussion