FAQ
I'm a system administrator doing golang development, part as a perl
replacement, part because is fun.

I need to know the number of goroutines available in a WaitGroup, but there
are no Count() method or any other way to obtain such information from the
WaitGroup structure.

I've changed the base libraries to add the method I need, it's fast and
simple , but It's not elegant and I don't believe that this must be the way
to do it.

The other option is build a wrapper around WaitGroup so I can create the
method Count(), but this seem as inelegant as modifying the base libraries
and a burden on the system for such small change.

// in sync/waitgroup.go
func (wg *WaitGroup) Count() uint32 {
         return uint32(*(wg.state()) >> 32)
}


The programm I'm doing has a kind of goroutine spool, but the number of
goroutines is not fixed, sometimes when channel (that feeds the goroutines)
capacity is full you push more goroutines to work to relieve some channel
capacity. Goroutines dies when channel capacity is empty. And yes I know I
can develop a more sophisticated architecture that takes care of the spool,
but as it is now is extremely simple.

Congrats to the Go Team, 1.5 works wonders in my 32 core machine.

Anyone able to suggest a more right approach to have a WaiGroup.Count() ?
How to suggest to Go team to include the Count() method ? does it make
sense ?

Thanks.

Enric

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Search Discussions

  • Egon at Aug 27, 2015 at 2:05 pm

    On Thursday, 27 August 2015 16:49:57 UTC+3, Enric Lafont wrote:
    I'm a system administrator doing golang development, part as a perl
    replacement, part because is fun.

    I need to know the number of goroutines available in a WaitGroup, but
    there are no Count() method or any other way to obtain such information
    from the WaitGroup structure.
    I've changed the base libraries to add the method I need, it's fast and
    simple , but It's not elegant and I don't believe that this must be the way
    to do it.

    The other option is build a wrapper around WaitGroup so I can create the
    method Count(), but this seem as inelegant as modifying the base libraries
    and a burden on the system for such small change.

    // in sync/waitgroup.go
    func (wg *WaitGroup) Count() uint32 {
    return uint32(*(wg.state()) >> 32)
    }


    The programm I'm doing has a kind of goroutine spool, but the number of
    goroutines is not fixed, sometimes when channel (that feeds the goroutines)
    capacity is full you push more goroutines to work to relieve some channel
    capacity. Goroutines dies when channel capacity is empty. And yes I know I
    can develop a more sophisticated architecture that takes care of the spool,
    but as it is now is extremely simple.

    Congrats to the Go Team, 1.5 works wonders in my 32 core machine.

    Anyone able to suggest a more right approach to have a WaiGroup.Count() ?
    How to suggest to Go team to include the Count() method ? does it make
    sense ?
    Could you provide some more explanation how/when they get the items and how
    long is the whole system run, how long is it run, etc.? It's hard to say
    what the correct approach is, because some might work better than others in
    particular cases.

    *(Note: following code untested, might contain bugs)*

    E.g. could you just leave the goroutines let hang around and kill all at
    once:

    func process(v int){
    v = v * 2
    time.Sleep(1*time.Second)
    println(v)
    }

    func processqueue(queue chan int){
    mux := make(chan int, 4)
    processor := func(){
    for v := range mux {
    process(v)
    }
    }

    for v := range queue {
    select {
    case mux <- v:
    // do nothing when was able to send
    default:
    // add new processor
    go processor()
    mux <- v
    }
    }

    close(mux)
    }

    Or an alternative approach:

    func process(v int) {
    v = v * 2
    time.Sleep(1 * time.Second)
    println(v)
    }

    func processqueue(queue chan int) {
    mux := make(chan request, 4)

    total := uint64(0)

    type request struct {
    v uint64
    die bool
    }

    processor := func() {
    atomic.AddUint64(&total, 1)
    defer atomic.AddUint64(&total, -1)

    for v := range mux {
    if v.die {
    break
    }
    process(v)
    }
    }

    for v := range queue {
    select {
    case mux <- request{v}:
    // do nothing when was able to send
    default:
    // add new processor
    go processor()
    mux <- request{v}
    }

    if atomic.LoadUint64(&total) > 10 {
    mux <- request{die: true}
    }
    }

    close(mux)
    }

    + Egon

    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
  • Enric Lafont at Aug 28, 2015 at 8:59 am
    Thanks for answering me.

    The solution you propose is quite similar to the one I'm using, well you do
    a manual count when such number must be equal to the one wg.Count() returns.

    What I'm doing is to process a directory in our NAS server badly organized,
    it has 60 million files and 350.000 subdirectories

    The program I'm refactoring is an old PHP script that takes more than 4
    days to complete operation, it's badly written and executed slowly, so, as
    the nature of the problem is a very concurrent one, Golang seem a perfect
    fit for me, well its the same I would have rewrite it in go even if the
    problem were not concurrent ... :)

    The main problem is that a goroutine for every file overloads the machine
    and at the end golang complains about too many open files, and "fs.file-max
    = 4033344" when the goroutines aproach the milion the program crashes.

    I can not overload the machine, it's a server doing business procedures and
    our NAS too, so I need to limit the number of simultaneous goroutines, but,
    when I limit the number I find that some subdirectories are deeper than the
    number of max goroutines I'm allowed to run, and the feeder channel gets
    full, so the program stops working, I needed a way to keep the goroutines
    on order, and this way is limiting the number but allowing more when the
    deep of the tree structure needs it, something like
                 I:
                         for { // Repeat until fullFile name has space in dirch
                             select {
                             case dirch <- fullFile:
                                 fmt.Printf("Queue: %3d, %3d File:
    %v \r", len(dirch), wg.Count(), fullFile)
                                 if wg.Count() < (uint32)(len(dirch)) { //
    len(dirch) is 256 now
                                     go readDirectory(dirch, wr, wg) // The
    goroutine accepts an input read channel, an output write channel , and a
    waitgroup
                                     runtime.Gosched() // Give time to other
    goroutines
                                 }
                                 break I
                             default: // Done when channel capacity is full
                                 runtime.Gosched() // Give time to
    reduce channel pressure
                                 if len(dirch) == cap(dirch) { // if none of the
    existing routines solved the problem, then
                                     go readDirectory(dirch, wr, wg) // run
    extra routine to relieve channel capacity
                                 }
                             }
                         }

    This currently works, being able to count the number of goroutines lets me
    open more goroutines when the need arises, the goroutines dies when the
    queue (dirch) becomes empty, so I can keep the load on the system as low as
    possible, using this approach I have a peak of ~900 goroutines for a while
    and an average of 150 routines running concurrently, the load on the system
    is about 40%.

    Thanks again for your time.



    El jueves, 27 de agosto de 2015, 16:05:03 (UTC+2), Egon escribió:
    On Thursday, 27 August 2015 16:49:57 UTC+3, Enric Lafont wrote:

    I'm a system administrator doing golang development, part as a perl
    replacement, part because is fun.

    I need to know the number of goroutines available in a WaitGroup, but
    there are no Count() method or any other way to obtain such information
    from the WaitGroup structure.
    I've changed the base libraries to add the method I need, it's fast and
    simple , but It's not elegant and I don't believe that this must be the way
    to do it.

    The other option is build a wrapper around WaitGroup so I can create the
    method Count(), but this seem as inelegant as modifying the base libraries
    and a burden on the system for such small change.

    // in sync/waitgroup.go
    func (wg *WaitGroup) Count() uint32 {
    return uint32(*(wg.state()) >> 32)
    }


    The programm I'm doing has a kind of goroutine spool, but the number of
    goroutines is not fixed, sometimes when channel (that feeds the goroutines)
    capacity is full you push more goroutines to work to relieve some channel
    capacity. Goroutines dies when channel capacity is empty. And yes I know I
    can develop a more sophisticated architecture that takes care of the spool,
    but as it is now is extremely simple.

    Congrats to the Go Team, 1.5 works wonders in my 32 core machine.

    Anyone able to suggest a more right approach to have a WaiGroup.Count() ?
    How to suggest to Go team to include the Count() method ? does it make
    sense ?
    Could you provide some more explanation how/when they get the items and
    how long is the whole system run, how long is it run, etc.? It's hard to
    say what the correct approach is, because some might work better than
    others in particular cases.

    *(Note: following code untested, might contain bugs)*

    E.g. could you just leave the goroutines let hang around and kill all at
    once:

    func process(v int){
    v = v * 2
    time.Sleep(1*time.Second)
    println(v)
    }

    func processqueue(queue chan int){
    mux := make(chan int, 4)
    processor := func(){
    for v := range mux {
    process(v)
    }
    }

    for v := range queue {
    select {
    case mux <- v:
    // do nothing when was able to send
    default:
    // add new processor
    go processor()
    mux <- v
    }
    }

    close(mux)
    }

    Or an alternative approach:

    func process(v int) {
    v = v * 2
    time.Sleep(1 * time.Second)
    println(v)
    }

    func processqueue(queue chan int) {
    mux := make(chan request, 4)

    total := uint64(0)

    type request struct {
    v uint64
    die bool
    }

    processor := func() {
    atomic.AddUint64(&total, 1)
    defer atomic.AddUint64(&total, -1)

    for v := range mux {
    if v.die {
    break
    }
    process(v)
    }
    }

    for v := range queue {
    select {
    case mux <- request{v}:
    // do nothing when was able to send
    default:
    // add new processor
    go processor()
    mux <- request{v}
    }

    if atomic.LoadUint64(&total) > 10 {
    mux <- request{die: true}
    }
    }

    close(mux)
    }

    + Egon
    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
  • Egon at Aug 28, 2015 at 9:26 am

    On Friday, 28 August 2015 11:59:41 UTC+3, Enric Lafont wrote:
    Thanks for answering me.

    The solution you propose is quite similar to the one I'm using, well you
    do a manual count when such number must be equal to the one wg.Count()
    returns.

    What I'm doing is to process a directory in our NAS server badly
    organized, it has 60 million files and 350.000 subdirectories

    The program I'm refactoring is an old PHP script that takes more than 4
    days to complete operation, it's badly written and executed slowly, so, as
    the nature of the problem is a very concurrent one, Golang seem a perfect
    fit for me, well its the same I would have rewrite it in go even if the
    problem were not concurrent ... :)

    The main problem is that a goroutine for every file overloads the machine
    and at the end golang complains about too many open files, and "fs.file-max
    = 4033344" when the goroutines aproach the milion the program crashes.

    I can not overload the machine, it's a server doing business procedures
    and our NAS too, so I need to limit the number of simultaneous goroutines,
    but, when I limit the number I find that some subdirectories are deeper
    than the number of max goroutines I'm allowed to run, and the feeder
    channel gets full, so the program stops working, I needed a way to keep the
    goroutines on order, and this way is limiting the number but allowing more
    when the deep of the tree structure needs it, something like
    I:
    for { // Repeat until fullFile name has space in dirch
    select {
    case dirch <- fullFile:
    fmt.Printf("Queue: %3d, %3d File:
    %v \r", len(dirch), wg.Count(), fullFile)
    if wg.Count() < (uint32)(len(dirch)) { //
    len(dirch) is 256 now
    go readDirectory(dirch, wr, wg) // The
    goroutine accepts an input read channel, an output write channel , and a
    waitgroup
    runtime.Gosched() // Give time to other
    goroutines
    }
    break I
    default: // Done when channel capacity is full
    runtime.Gosched() // Give time to
    reduce channel pressure
    if len(dirch) == cap(dirch) { // if none of
    the existing routines solved the problem, then
    go readDirectory(dirch, wr, wg) // run
    extra routine to relieve channel capacity
    }
    }
    }

    This currently works, being able to count the number of goroutines lets me
    open more goroutines when the need arises, the goroutines dies when the
    queue (dirch) becomes empty, so I can keep the load on the system as low as
    possible, using this approach I have a peak of ~900 goroutines for a while
    and an average of 150 routines running concurrently, the load on the system
    is about 40%.

    Thanks again for your time.
    Can you separate your file feeding from the processing?
    Only one goroutine iterates the directories and rest do the processing:
    https://github.com/loov/qloc/blob/master/main.go#L105

    Recursive decomposition (a solution by Dmitry from another thread
    https://groups.google.com/d/msg/golang-nuts/o2mkC1xsmC4/lDUjP2c1V6gJ):
    http://play.golang.org/p/epvbA0vjZX

    Solution with busy waiting:
    https://play.golang.org/p/uH5fwlw0qp

    Without busy waiting:
    https://github.com/egonelbre/spexs2/blob/dev/search/spexs.go#L64
    https://github.com/egonelbre/spexs2/blob/dev/search/spexs.go#L162
    (explanation in
      https://github.com/egonelbre/spexs2/raw/dev/_doc/Thesis.pdf page 45).


    + Egon

    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
  • Roger peppe at Feb 24, 2016 at 2:41 pm
    Assuming that it's your business logic that takes the time,
    why don't you separate the directory walking from the
    processing itself, using one goroutine for traversing the
    file tree, and all the others for doing the actual file processing?

    Something like this, perhaps, although you'd probably want
    to be a bit more careful about the error handling:

    http://play.golang.org/p/Lp-Us_At_z

    If the directory traversal is also a bottleneck, you could do
    a similar thing using the concurrent filepath.Walk implementation
    in github.com/MichaelTJones/walk instead of filepath.Walk
    itself.

       cheers,
         rog.
    On 28 August 2015 at 09:59, Enric Lafont wrote:
    Thanks for answering me.

    The solution you propose is quite similar to the one I'm using, well you do
    a manual count when such number must be equal to the one wg.Count() returns.

    What I'm doing is to process a directory in our NAS server badly organized,
    it has 60 million files and 350.000 subdirectories

    The program I'm refactoring is an old PHP script that takes more than 4 days
    to complete operation, it's badly written and executed slowly, so, as the
    nature of the problem is a very concurrent one, Golang seem a perfect fit
    for me, well its the same I would have rewrite it in go even if the problem
    were not concurrent ... :)

    The main problem is that a goroutine for every file overloads the machine
    and at the end golang complains about too many open files, and "fs.file-max
    = 4033344" when the goroutines aproach the milion the program crashes.

    I can not overload the machine, it's a server doing business procedures and
    our NAS too, so I need to limit the number of simultaneous goroutines, but,
    when I limit the number I find that some subdirectories are deeper than the
    number of max goroutines I'm allowed to run, and the feeder channel gets
    full, so the program stops working, I needed a way to keep the goroutines on
    order, and this way is limiting the number but allowing more when the deep
    of the tree structure needs it, something like
    I:
    for { // Repeat until fullFile name has space in dirch
    select {
    case dirch <- fullFile:
    fmt.Printf("Queue: %3d, %3d File: %v
    \r", len(dirch), wg.Count(), fullFile)
    if wg.Count() < (uint32)(len(dirch)) { //
    len(dirch) is 256 now
    go readDirectory(dirch, wr, wg) // The
    goroutine accepts an input read channel, an output write channel , and a
    waitgroup
    runtime.Gosched() // Give time to other
    goroutines
    }
    break I
    default: // Done when channel capacity is full
    runtime.Gosched() // Give time to
    reduce channel pressure
    if len(dirch) == cap(dirch) { // if none of the
    existing routines solved the problem, then
    go readDirectory(dirch, wr, wg) // run extra
    routine to relieve channel capacity
    }
    }
    }

    This currently works, being able to count the number of goroutines lets me
    open more goroutines when the need arises, the goroutines dies when the
    queue (dirch) becomes empty, so I can keep the load on the system as low as
    possible, using this approach I have a peak of ~900 goroutines for a while
    and an average of 150 routines running concurrently, the load on the system
    is about 40%.

    Thanks again for your time.




    El jueves, 27 de agosto de 2015, 16:05:03 (UTC+2), Egon escribió:
    On Thursday, 27 August 2015 16:49:57 UTC+3, Enric Lafont wrote:

    I'm a system administrator doing golang development, part as a perl
    replacement, part because is fun.

    I need to know the number of goroutines available in a WaitGroup, but
    there are no Count() method or any other way to obtain such information from
    the WaitGroup structure.


    I've changed the base libraries to add the method I need, it's fast and
    simple , but It's not elegant and I don't believe that this must be the way
    to do it.

    The other option is build a wrapper around WaitGroup so I can create the
    method Count(), but this seem as inelegant as modifying the base libraries
    and a burden on the system for such small change.

    // in sync/waitgroup.go
    func (wg *WaitGroup) Count() uint32 {
    return uint32(*(wg.state()) >> 32)
    }


    The programm I'm doing has a kind of goroutine spool, but the number of
    goroutines is not fixed, sometimes when channel (that feeds the goroutines)
    capacity is full you push more goroutines to work to relieve some channel
    capacity. Goroutines dies when channel capacity is empty. And yes I know I
    can develop a more sophisticated architecture that takes care of the spool,
    but as it is now is extremely simple.

    Congrats to the Go Team, 1.5 works wonders in my 32 core machine.

    Anyone able to suggest a more right approach to have a WaiGroup.Count() ?
    How to suggest to Go team to include the Count() method ? does it make
    sense ?

    Could you provide some more explanation how/when they get the items and
    how long is the whole system run, how long is it run, etc.? It's hard to say
    what the correct approach is, because some might work better than others in
    particular cases.

    (Note: following code untested, might contain bugs)

    E.g. could you just leave the goroutines let hang around and kill all at
    once:

    func process(v int){
    v = v * 2
    time.Sleep(1*time.Second)
    println(v)
    }

    func processqueue(queue chan int){
    mux := make(chan int, 4)
    processor := func(){
    for v := range mux {
    process(v)
    }
    }

    for v := range queue {
    select {
    case mux <- v:
    // do nothing when was able to send
    default:
    // add new processor
    go processor()
    mux <- v
    }
    }

    close(mux)
    }

    Or an alternative approach:

    func process(v int) {
    v = v * 2
    time.Sleep(1 * time.Second)
    println(v)
    }

    func processqueue(queue chan int) {
    mux := make(chan request, 4)

    total := uint64(0)

    type request struct {
    v uint64
    die bool
    }

    processor := func() {
    atomic.AddUint64(&total, 1)
    defer atomic.AddUint64(&total, -1)

    for v := range mux {
    if v.die {
    break
    }
    process(v)
    }
    }

    for v := range queue {
    select {
    case mux <- request{v}:
    // do nothing when was able to send
    default:
    // add new processor
    go processor()
    mux <- request{v}
    }

    if atomic.LoadUint64(&total) > 10 {
    mux <- request{die: true}
    }
    }

    close(mux)
    }

    + Egon
    --
    You received this message because you are subscribed to the Google Groups
    "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an
    email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
  • Jan Mercl at Aug 27, 2015 at 2:16 pm
    On Thu, Aug 27, 2015, 15:50 Enric Lafont wrote:


    I need to know the number of goroutines available in a WaitGroup, but there
    are no Count() method or any other way to obtain such information from the
    WaitGroup structure.

    WaitGroup has no Count method because, in the general case, there's no
    non-racy way to provide that number - it may get invalidated the moment you
    know it. The racy value possibly obtained in any other way is thus not much
    useful.

    --

    -j

    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
  • Enric Lafont at Aug 28, 2015 at 9:04 am
    Thanks Jon for answering

    Yes I'm aware that wg.Count() is going to give me a past value the very
    moment I read it, but as I need to verify an extreme case (max number of
    goroutines) it is right even if a nanosecond later the value becomes
    untrue, Count() is more a guide on what should I do that a deterministic
    value on some computation.

    El jueves, 27 de agosto de 2015, 16:16:56 (UTC+2), Jan Mercl escribió:

    On Thu, Aug 27, 2015, 15:50 Enric Lafont <enr...@gmail.com <javascript:>>
    wrote:


    I need to know the number of goroutines available in a WaitGroup, but
    there are no Count() method or any other way to obtain such information
    from the WaitGroup structure.

    WaitGroup has no Count method because, in the general case, there's no
    non-racy way to provide that number - it may get invalidated the moment you
    know it. The racy value possibly obtained in any other way is thus not much
    useful.

    --

    -j
    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
  • Jan Mercl at Aug 28, 2015 at 9:33 am

    On Fri, Aug 28, 2015, 11:04 Enric Lafont <enricl@ <enricl@gmail.com> gmail.com wrote:
    If the need for Count() is actually just a need to limit a number of
    things, for example goroutines, then use a buffered channel.

             limiter := make(chan struct{}, limit)

    Before creating a new thing (starting a new goroutine)

             limiter <- struct{}{}

    Before destroying a thing (returning from a goroutine)

             <-limiter

    Such limiter is synchronous and safe for concurrent use.

    --

    -j

    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
  • Mike at Feb 23, 2016 at 11:06 pm
    I too wanted a way to see the number and was ok to accept nonsafe count.
    Just wanted to log the count from time to time and on SIGTERM ("Signal
    received, waiting on %d jobs.")


    type CountWG struct {
       sync.WaitGroup
       Count int // Race conditions, only for info logging.
    }


    // Increment on Add
    func (cg CountWG) Add(delta int) {
       cg.Count += delta
       cg.WaitGroup.Add(delta)
    }


    // Decrement on Done
    func (cg CountWG) Done() {
       cg.Count--
       cg.WaitGroup.Done()
    }



    On Thursday, August 27, 2015 at 6:49:57 AM UTC-7, Enric Lafont wrote:

    I'm a system administrator doing golang development, part as a perl
    replacement, part because is fun.

    I need to know the number of goroutines available in a WaitGroup, but
    there are no Count() method or any other way to obtain such information
    from the WaitGroup structure.

    I've changed the base libraries to add the method I need, it's fast and
    simple , but It's not elegant and I don't believe that this must be the way
    to do it.

    The other option is build a wrapper around WaitGroup so I can create the
    method Count(), but this seem as inelegant as modifying the base libraries
    and a burden on the system for such small change.

    // in sync/waitgroup.go
    func (wg *WaitGroup) Count() uint32 {
    return uint32(*(wg.state()) >> 32)
    }


    The programm I'm doing has a kind of goroutine spool, but the number of
    goroutines is not fixed, sometimes when channel (that feeds the goroutines)
    capacity is full you push more goroutines to work to relieve some channel
    capacity. Goroutines dies when channel capacity is empty. And yes I know I
    can develop a more sophisticated architecture that takes care of the spool,
    but as it is now is extremely simple.

    Congrats to the Go Team, 1.5 works wonders in my 32 core machine.

    Anyone able to suggest a more right approach to have a WaiGroup.Count() ?
    How to suggest to Go team to include the Count() method ? does it make
    sense ?

    Thanks.

    Enric
    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
  • Egon at Feb 23, 2016 at 11:31 pm

    On Wednesday, 24 February 2016 01:06:18 UTC+2, mi...@leftronic.com wrote:
    I too wanted a way to see the number and was ok to accept nonsafe count.
    All bets are off with data races in the code.
    https://software.intel.com/en-us/blogs/2013/01/06/benign-data-races-what-could-possibly-go-wrong

    Just wanted to log the count from time to time and on SIGTERM ("Signal
    received, waiting on %d jobs.")


    type CountWG struct {
    sync.WaitGroup
    Count int // Race conditions, only for info logging.
    atomic.AddInt32(&cg.count, 1)

    }


    // Increment on Add
    func (cg CountWG) Add(delta int) {
    cg.Count += delta
    cg.WaitGroup.Add(delta)
    }


    // Decrement on Done
    func (cg CountWG) Done() {
    cg.Count--
    atomic.AddInt32(&cg.count, -1)

    cg.WaitGroup.Done()
    }



    On Thursday, August 27, 2015 at 6:49:57 AM UTC-7, Enric Lafont wrote:

    I'm a system administrator doing golang development, part as a perl
    replacement, part because is fun.

    I need to know the number of goroutines available in a WaitGroup, but
    there are no Count() method or any other way to obtain such information
    from the WaitGroup structure.

    I've changed the base libraries to add the method I need, it's fast and
    simple , but It's not elegant and I don't believe that this must be the way
    to do it.

    The other option is build a wrapper around WaitGroup so I can create the
    method Count(), but this seem as inelegant as modifying the base libraries
    and a burden on the system for such small change.

    // in sync/waitgroup.go
    func (wg *WaitGroup) Count() uint32 {
    return uint32(*(wg.state()) >> 32)
    }


    The programm I'm doing has a kind of goroutine spool, but the number of
    goroutines is not fixed, sometimes when channel (that feeds the goroutines)
    capacity is full you push more goroutines to work to relieve some channel
    capacity. Goroutines dies when channel capacity is empty. And yes I know I
    can develop a more sophisticated architecture that takes care of the spool,
    but as it is now is extremely simple.

    Congrats to the Go Team, 1.5 works wonders in my 32 core machine.

    Anyone able to suggest a more right approach to have a WaiGroup.Count() ?
    How to suggest to Go team to include the Count() method ? does it make
    sense ?

    Thanks.

    Enric
    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupgolang-nuts @
categoriesgo
postedAug 27, '15 at 1:49p
activeFeb 24, '16 at 2:41p
posts10
users5
websitegolang.org

People

Translate

site design / logo © 2021 Grokbase