FAQ
Hello, I am dealing with go and I try to parallelize each instance of a for
loop.

My final goal is to do a ping test for 255 IP addresses. So basically my
for loop is something like

i := 0

for i < 255 {

    i++

    commandLinePing := fmt.Sprint("ping 10.11.204." + string(i))

        err := exec.Command("bash", "-c", commandLinePing).Run()

    if err != nil {
        }

}



So, the loops works well. But I would like to parallelize every 255
iterations, so that all 255 execute in the time of 1 (or so).


Any good way to do so ? To considerably reduce the execution time ?

Of course, this is a ping test so I need to listen for the reply from these
255 IPs. So, at the end, I need to wait until the last thread is finished.


Thank you very much.

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Search Discussions

  • Sebastien Binet at Jul 29, 2015 at 7:54 pm
    here is a first stab:

    http://play.golang.org/p/wZW0dRnq-F

    note that one should still limit the number of in-flight commands,
    because "file descriptors".

    -s

    On Wed, Jul 29, 2015 at 9:21 PM, Stéphane phenetas wrote:
    Hello, I am dealing with go and I try to parallelize each instance of a for
    loop.

    My final goal is to do a ping test for 255 IP addresses. So basically my for
    loop is something like

    i := 0

    for i < 255 {

    i++

    commandLinePing := fmt.Sprint("ping 10.11.204." + string(i))

    err := exec.Command("bash", "-c", commandLinePing).Run()

    if err != nil {
    }

    }



    So, the loops works well. But I would like to parallelize every 255
    iterations, so that all 255 execute in the time of 1 (or so).


    Any good way to do so ? To considerably reduce the execution time ?

    Of course, this is a ping test so I need to listen for the reply from these
    255 IPs. So, at the end, I need to wait until the last thread is finished.


    Thank you very much.

    --
    You received this message because you are subscribed to the Google Groups
    "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an
    email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
  • Roberto Zanotto at Jul 29, 2015 at 8:00 pm
    The canonical way of parallelizing a loop and waiting for the results is
    this:
    https://play.golang.org/p/7jea3O0-_j
    For pinging you may want to find a library that does it inside your process
    instead of launching 255.
    Cheers
    On Wednesday, July 29, 2015 at 9:21:09 PM UTC+2, Stéphane phenetas wrote:

    Hello, I am dealing with go and I try to parallelize each instance of a
    for loop.

    My final goal is to do a ping test for 255 IP addresses. So basically my
    for loop is something like

    i := 0

    for i < 255 {

    i++

    commandLinePing := fmt.Sprint("ping 10.11.204." + string(i))

    err := exec.Command("bash", "-c", commandLinePing).Run()

    if err != nil {
    }

    }



    So, the loops works well. But I would like to parallelize every 255
    iterations, so that all 255 execute in the time of 1 (or so).


    Any good way to do so ? To considerably reduce the execution time ?

    Of course, this is a ping test so I need to listen for the reply from
    these 255 IPs. So, at the end, I need to wait until the last thread is
    finished.


    Thank you very much.
    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
  • Stéphane phenetas at Jul 29, 2015 at 11:15 pm
    Hi Roberto, I have tried your solution, using a function called in the
    main, like this :

    func main() {
          testfunc()
    }

    func testfunc() {

             var w sync.WaitGroup

         n := 255 // do you want to ping #255 or not?

         w.Add(n)

         for i := 1; i < n; i++ {

             go func(j int) {

             defer w.Done()


                               commandLinePing := fmt.Sprint("ping 10.11.204." + string(j))

                 err := exec.Command("bash", "-c", commandLinePing).Run()


                 if err != nil {

                 }

             }(i)

         }

         w.Wait()

    }



    This code just do nothing, I can run it but do nothing.

    Is I use .Start() instead of .Run(), then I get an error :

    fatal error: all goroutines are asleep - deadlock!

    goroutine 1 [semacquire]:

    sync.(*WaitGroup).Wait(0x2081e4020)

    /usr/local/go/src/sync/waitgroup.go:132 +0x169
    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
  • Roberto Zanotto at Jul 29, 2015 at 11:37 pm
    You changed it :P
    You are waiting for 255 items to complete (w.Add(255)), but you are
    executing only 254 of them (for i := 1; i < 255; i++).
    On Thursday, July 30, 2015 at 1:15:38 AM UTC+2, Stéphane phenetas wrote:

    Hi Roberto, I have tried your solution, using a function called in the
    main, like this :

    func main() {
    testfunc()
    }

    func testfunc() {

    var w sync.WaitGroup

    n := 255 // do you want to ping #255 or not?

    w.Add(n)

    for i := 1; i < n; i++ {

    go func(j int) {

    defer w.Done()


    commandLinePing := fmt.Sprint("ping 10.11.204." + string(j))

    err := exec.Command("bash", "-c", commandLinePing).Run()


    if err != nil {

    }

    }(i)

    }

    w.Wait()

    }



    This code just do nothing, I can run it but do nothing.

    Is I use .Start() instead of .Run(), then I get an error :

    fatal error: all goroutines are asleep - deadlock!

    goroutine 1 [semacquire]:

    sync.(*WaitGroup).Wait(0x2081e4020)

    /usr/local/go/src/sync/waitgroup.go:132 +0x169
    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
  • Stéphane phenetas at Jul 30, 2015 at 12:24 am
    Hi Roberto, you'r right, this is so true. I changed it back to a working
    code (same code but with *w.Add(n-1)*).

    However I may have played too much with my new toy, because now It appears
    that I have reached the limit of process running. I have blocked my
    terminal and now I have a "*fork: Resource temporarily unavailable*" message

    I am not sure the process are released when they are done.

    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
  • Roberto Zanotto at Jul 30, 2015 at 12:50 am
    Looks like launching 255 processes at once is a bad idea after all :D
    You could try with the code Sebastien wrote, it should limit the number of
    concurrent things.
    I'd also suggest something like go-fastping
    <https://github.com/tatsushid/go-fastping> to do the pings, instead of
    exec.Command.
    On Thursday, July 30, 2015 at 2:24:25 AM UTC+2, Stéphane phenetas wrote:

    Hi Roberto, you'r right, this is so true. I changed it back to a working
    code (same code but with *w.Add(n-1)*).

    However I may have played too much with my new toy, because now It appears
    that I have reached the limit of process running. I have blocked my
    terminal and now I have a "*fork: Resource temporarily unavailable*"
    message

    I am not sure the process are released when they are done.
    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
  • Stéphane phenetas at Jul 30, 2015 at 1:10 am
    The thing is, it worked for several times. It worked well let say maybe 4/5
    times.

    Then the ping never stop, and even if the "ping" is done, the thread
    remains there and I can see hundreds of Pings thread in the task manager.
    How to really kill them when they are done in the loop ?

    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
  • Roberto Zanotto at Jul 30, 2015 at 1:33 am
    I actually don't know why they stay alive. When Cmd.Run() returns, the
    process should be done already, there would be no point in trying to kill
    it again. I don't want to make hypothesis on what is happening exactly, but
    trying to start so many processes is certainly bad. This is not a program,
    it's a stress test :D
    On Thursday, July 30, 2015 at 3:10:26 AM UTC+2, Stéphane phenetas wrote:

    The thing is, it worked for several times. It worked well let say maybe
    4/5 times.

    Then the ping never stop, and even if the "ping" is done, the thread
    remains there and I can see hundreds of Pings thread in the task manager.
    How to really kill them when they are done in the loop ?
    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
  • Roberto Zanotto at Jul 30, 2015 at 1:01 am
    step 1) go get -u github.com/tatsushid/go-fastping
    step 2) use this code https://play.golang.org/p/el8ZsaqM6N
    Should be as fast as the parallel one, as the API seems to be asynchronous.
    On Thursday, July 30, 2015 at 2:24:25 AM UTC+2, Stéphane phenetas wrote:

    Hi Roberto, you'r right, this is so true. I changed it back to a working
    code (same code but with *w.Add(n-1)*).

    However I may have played too much with my new toy, because now It appears
    that I have reached the limit of process running. I have blocked my
    terminal and now I have a "*fork: Resource temporarily unavailable*"
    message

    I am not sure the process are released when they are done.
    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
  • Dustin at Jul 29, 2015 at 11:38 pm
    Your loop runs only 254 times, i should start at 0. Your program is waiting
    forever for the 255th call to Done(), which never comes.

    Also, the exec command is probably running, but you're not doing anything
    with its output.
    Try reading up on https://golang.org/pkg/os/exec/#Cmd.Run and see how you
    want to handle its output.

    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
  • Tamás Gulácsi at Jul 30, 2015 at 4:14 am
    Please at least print out your err! And are you sure string(j) does what you want? strconv.Itoa is what you need.

    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
  • Marcus Franke at Jul 30, 2015 at 5:26 am
    The command "ping IP" will ping for ever, maybe your processes do not
    return due to this.

    Have a look at the man page of ping, there is a flag for this. I think its
    -c count_pings. But this may differ running under different OS.

    -mf

    Tamás Gulácsi <tgulacsi78@gmail.com> schrieb am Do., 30. Juli 2015 06:13:
    Please at least print out your err! And are you sure string(j) does what
    you want? strconv.Itoa is what you need.

    --
    You received this message because you are subscribed to the Google Groups
    "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an
    email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
  • Stéphane phenetas at Jul 30, 2015 at 5:31 pm
    And, YES ! You got it right Marcus ! I was so into my program that I did
    not see that.

    The command "ping x.x.x.x" runs forever, so adding the option "-c 1", makes
    it run only once.

    Thank you very much all of you !

    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
  • Stéphane phenetas at Aug 4, 2015 at 4:47 am
    Hi, I am back with my problems :/

    I can not really use parallelism because every time after a while I get an
    error message "*fork/exec /bin/bash: resource temporarily unavailable*"

    This seams to come from the terminal, and I think that all threads are not
    correctly closes or something. For exemple if I execute the same code we
    were discussing about, it returns me this error after a while. Then the
    terminal is blocked ( I can not execute anything else) and I have to close
    the window and reopen a new terminal.)

    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
  • Gulácsi Tamás at Aug 4, 2015 at 4:51 am
    Maybe you've invented a half fork bomb? I mean maybe you start too
    many bash scripts at once.

    2015-08-04 6:47 GMT+02:00 Stéphane phenetas <phenetas@gmail.com>:
    Hi, I am back with my problems :/

    I can not really use parallelism because every time after a while I get an
    error message "fork/exec /bin/bash: resource temporarily unavailable"

    This seams to come from the terminal, and I think that all threads are not
    correctly closes or something. For exemple if I execute the same code we
    were discussing about, it returns me this error after a while. Then the
    terminal is blocked ( I can not execute anything else) and I have to close
    the window and reopen a new terminal.)
    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
  • Stéphane phenetas at Aug 4, 2015 at 7:05 am
    That is what I thought, and it could be understandable. However, this error
    also appears if I start less thread at once.

    I tried to start 20 thread at once, then wait for 10 seconds using the
    time.sleep package. (so I thought during these 10 seconds, any thread would
    have time to finish and release the memory assigned)
    And even with doing that, after 10 or 12 loops, the error happens again.

    So it seems to appears if I launch more than 250/300 process *IN TOTAL. *Even
    with lots of sleep, with less threads at a time, it will end with the error
    :/

    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupgolang-nuts @
categoriesgo
postedJul 29, '15 at 7:21p
activeAug 4, '15 at 7:05a
posts17
users6
websitegolang.org

People

Translate

site design / logo © 2022 Grokbase