FAQ
Hi there,

I have developed an utility (very simple) to run multiple commands in
parallel:

https://github.com/gdm85/coshell/

I needed this for customization of initramfs scripts; I originally tried
to use GNU Parallel, but perl dependencies can get up to 25M :s
Then I tried moreutils-parallel but the command-line syntax would force
me to use hacks; too bad a GNU Parallel command is not available in
busybox, but I quickly coded coshell and achieved my goal with 1.6M of
(stripped) executable. I think we have here another great use-case for
Go-built binaries :)

I think coshell might be quite useful for distro makers and/or
minimalistic Linux distributions; feedback is welcome.

Kind regards,
--
   Giuseppe M (gdm85)

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Search Discussions

  • Andreas Klauer at Oct 5, 2014 at 6:41 pm
    The (busybox) shell can execute multiple commands in parallel just fine,
    though.

    What's wrong with this?

    echo test1 &
    echo test2 &
    echo test3 &
    wait

    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
  • Giuseppe Mazzotta at Oct 5, 2014 at 7:50 pm

    On 05/10/14 20:41, Andreas Klauer wrote:
    The (busybox) shell can execute multiple commands in parallel just fine,
    though.

    What's wrong with this?

    echo test1 &
    echo test2 &
    echo test3 &
    wait
    Exit code tracking. Now try to use the above to keep track of which one
    failed...I tried and was not elegant nor easy for people to read.

    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
  • Benjamin Measures at Oct 5, 2014 at 8:07 pm
    For one, commands with multi-line output are randomly interlaced in bash.

    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
  • James Aguilar at Sep 6, 2015 at 8:40 pm

    On Sunday, October 5, 2014 at 1:07:48 PM UTC-7, Benjamin Measures wrote:
    For one, commands with multi-line output are randomly interlaced in bash.

    That can be fixed trivially by a shell function to prepend a tag to each
    output line. Something like

    network_setup | tag "net"


    Where tag is something like

    tag() {
       while read line; do
         echo "$1: $line"
       done
    }

    Then, if you want to see "deinterlaced" output, just grep for ^net: in the
    output. And you don't get the problem of the output being delayed until the
    command terminates.

    Just a heads up. That's no reason not to make a tool in go, it's just
    something to help anyone who thinks this is a difficult problem to solve in
    sh. I have not personally tested this code, but something like it should
    work.

    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
  • Ole Tange at Sep 8, 2015 at 12:08 am

    On Sunday, September 6, 2015 at 10:40:47 PM UTC+2, James Aguilar wrote:
    On Sunday, October 5, 2014 at 1:07:48 PM UTC-7, Benjamin Measures wrote:

    For one, commands with multi-line output are randomly interlaced in bash.

    That can be fixed trivially by a shell function to prepend a tag to each
    output line. Something like

    network_setup | tag "net"


    Where tag is something like

    tag() {
    while read line; do
    echo "$1: $line"
    done
    }

    Then, if you want to see "deinterlaced" output, just grep for ^net: in the
    output. And you don't get the problem of the output being delayed until the
    command terminates.

    Just a heads up. That's no reason not to make a tool in go, it's just
    something to help anyone who thinks this is a difficult problem to solve in
    sh. I have not personally tested this code, but something like it should
    work.
    Just a heads up: The above does not work in general. It may work in certain
    limited situations, but relying it for production systems will be dangerous.


    To see see a failing example try this:



    tag() {

       while read line; do

         echo "$1: $line"

       done

    }

    export -f tag



    parallel perl -e '\$a=\"1{}\"x10000000\;print\ \$a,\"\\n\"' '>' {} ::: a b
    c d e f

    ls -l a b c d e f

    echo a b c d e f | xargs -n1 bash -c "grep 1 \$0 | tag \$0" |sort >
    out.xargs-serial

    md5sum out.xargs-serial

    parallel -kP6 -n1 'grep 1 {} | tag {}' ::: a b c d e f | sort > out.par

    md5sum out.par

    echo a b c d e f | xargs -P6 -n1 bash -c "grep 1 \$0 | tag \$0" |sort >
    out.xargs-P

    md5sum out.xargs-P

    Had 'tag' worked reliably, then the md5sums would have been the same.


    /Ole

    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
  • James Aguilar at Sep 8, 2015 at 2:14 am
    I'm not a Unix expert -- maybe why I thought that that simple approach
    could work. Can you explain what the problem here is? Is it just that
    something is refusing to buffer the full lines, because they are too large?

    In similar situations, I would think that coshell's deinterlacing approach
    would be at least as bad of a result -- it seems to require memory in the
    order of the size of the sum of the outputs of the child programs. I guess
    it depends on whether your logs are so important that you'd prefer to OOM
    rather than see them in a bad state.

    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
  • Ole Tange at Sep 8, 2015 at 7:03 am

    On Tuesday, September 8, 2015 at 4:14:42 AM UTC+2, James Aguilar wrote:
    I'm not a Unix expert -- maybe why I thought that that simple approach
    could work. Can you explain what the problem here is? Is it just that
    something is refusing to buffer the full lines, because they are too large?
    Yes.

    In similar situations, I would think that coshell's deinterlacing approach
    would be at least as bad of a result -- it seems to require memory in the
    order of the size of the sum of the outputs of the child programs. I guess
    it depends on whether your logs are so important that you'd prefer to OOM
    rather than see them in a bad state.
    That is one way to do it. GNU Parallel (which you can see deals correctly
    with the long lines) uses temporary files that are immediately deleted, but
    kept open. This way even if GNU Parallel crashes, there will be no files to
    clean up.

    None of the ways are ideal: coshell can force OOM, GNU Parallel can force
    disk full; but I reckon most people have more free disk space than they
    have RAM.


    /Ole

    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
  • Kishore Kumar Vaishnav at Oct 5, 2014 at 6:59 pm
    Just curious to understand. What is the logic behind naming coshell instead
    of something like GoParallel or GoCMDParallel, etc.,
    On Sunday, October 5, 2014 11:18:49 AM UTC-7, Giuseppe Mazzotta wrote:

    Hi there,

    I have developed an utility (very simple) to run multiple commands in
    parallel:

    https://github.com/gdm85/coshell/

    I needed this for customization of initramfs scripts; I originally tried
    to use GNU Parallel, but perl dependencies can get up to 25M :s
    Then I tried moreutils-parallel but the command-line syntax would force
    me to use hacks; too bad a GNU Parallel command is not available in
    busybox, but I quickly coded coshell and achieved my goal with 1.6M of
    (stripped) executable. I think we have here another great use-case for
    Go-built binaries :)

    I think coshell might be quite useful for distro makers and/or
    minimalistic Linux distributions; feedback is welcome.

    Kind regards,
    --
    Giuseppe M (gdm85)
    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
  • Giuseppe Mazzotta at Oct 5, 2014 at 7:53 pm

    On 05/10/14 20:59, Kishore Kumar Vaishnav wrote:
    Just curious to understand. What is the logic behind naming coshell
    instead of something like GoParallel or GoCMDParallel, etc.,
    Mainly 2:
    - a name that is not already associated to different projects, trying to
    be unique and easily searchable online
    - although it is a trend to name Go-something all software written in
    Go, I wouldn't want to predate this trend for all utilities or code I
    write in this language :P

    Plus I think it's still short and meaningful.
    On Sunday, October 5, 2014 11:18:49 AM UTC-7, Giuseppe Mazzotta wrote:

    Hi there,

    I have developed an utility (very simple) to run multiple commands in
    parallel:

    https://github.com/gdm85/coshell/ <https://github.com/gdm85/coshell/>

    I needed this for customization of initramfs scripts; I originally
    tried
    to use GNU Parallel, but perl dependencies can get up to 25M :s
    Then I tried moreutils-parallel but the command-line syntax would force
    me to use hacks; too bad a GNU Parallel command is not available in
    busybox, but I quickly coded coshell and achieved my goal with 1.6M of
    (stripped) executable. I think we have here another great use-case for
    Go-built binaries :)

    I think coshell might be quite useful for distro makers and/or
    minimalistic Linux distributions; feedback is welcome.

    Kind regards,
    --
    Giuseppe M (gdm85)
    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
  • Giuseppe Mazzotta at Sep 6, 2015 at 3:33 pm
    I have recently added an option, --deinterlace, that works around the
    problem of interleaved output lines. I think this makes the utility more
    useful and it's not something you can already trivially do with other
    non-monolithic binaries.

    --
       gdm85

    On 05/10/14 14:54, Giuseppe Mazzotta wrote:
    Hi there,

    I have developed an utility (very simple) to run multiple commands in
    parallel:

    https://github.com/gdm85/coshell/

    I needed this for customization of initramfs scripts; I originally tried
    to use GNU Parallel, but perl dependencies can get up to 25M :s
    Then I tried moreutils-parallel but the command-line syntax would force
    me to use hacks; too bad a GNU Parallel command is not available in
    busybox, but I quickly coded coshell and achieved my goal with 1.6M of
    (stripped) executable. I think we have here another great use-case for
    Go-built binaries :)

    I think coshell might be quite useful for distro makers and/or
    minimalistic Linux distributions; feedback is welcome.

    Kind regards,
    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupgolang-nuts @
categoriesgo
postedOct 5, '14 at 6:18p
activeSep 8, '15 at 7:03a
posts11
users6
websitegolang.org

People

Translate

site design / logo © 2022 Grokbase