[go-nuts] coshell: no-dependencies GNU parallel clone for initramfs

The (busybox) shell can execute multiple commands in parallel just fine,
though.

What's wrong with this?

echo test1 &
echo test2 &
echo test3 &
wait

Exit code tracking. Now try to use the above to keep track of which one
failed...I tried and was not elegant nor easy for people to read.

For one, commands with multi-line output are randomly interlaced in bash.

That can be fixed trivially by a shell function to prepend a tag to each
output line. Something like

network_setup | tag "net"


Where tag is something like

tag() {
   while read line; do
     echo "$1: $line"
   done
}

Then, if you want to see "deinterlaced" output, just grep for ^net: in the
output. And you don't get the problem of the output being delayed until the
command terminates.

Just a heads up. That's no reason not to make a tool in go, it's just
something to help anyone who thinks this is a difficult problem to solve in
sh. I have not personally tested this code, but something like it should
work.

Just a heads up: The above does not work in general. It may work in certain
limited situations, but relying it for production systems will be dangerous.


To see see a failing example try this:



tag() {

   while read line; do

     echo "$1: $line"

   done

}

export -f tag



parallel perl -e '\$a=\"1{}\"x10000000\;print\ \$a,\"\\n\"' '>' {} ::: a b
c d e f

ls -l a b c d e f

echo a b c d e f | xargs -n1 bash -c "grep 1 \$0 | tag \$0" |sort >
out.xargs-serial

md5sum out.xargs-serial

parallel -kP6 -n1 'grep 1 {} | tag {}' ::: a b c d e f | sort > out.par

md5sum out.par

echo a b c d e f | xargs -P6 -n1 bash -c "grep 1 \$0 | tag \$0" |sort >
out.xargs-P

md5sum out.xargs-P

Had 'tag' worked reliably, then the md5sums would have been the same.


/Ole

I'm not a Unix expert -- maybe why I thought that that simple approach
could work. Can you explain what the problem here is? Is it just that
something is refusing to buffer the full lines, because they are too large?

In similar situations, I would think that coshell's deinterlacing approach
would be at least as bad of a result -- it seems to require memory in the
order of the size of the sum of the outputs of the child programs. I guess
it depends on whether your logs are so important that you'd prefer to OOM
rather than see them in a bad state.

Yes.

That is one way to do it. GNU Parallel (which you can see deals correctly
with the long lines) uses temporary files that are immediately deleted, but
kept open. This way even if GNU Parallel crashes, there will be no files to
clean up.

None of the ways are ideal: coshell can force OOM, GNU Parallel can force
disk full; but I reckon most people have more free disk space than they
have RAM.


/Ole

Just curious to understand. What is the logic behind naming coshell instead
of something like GoParallel or GoCMDParallel, etc.,

Mainly 2:
- a name that is not already associated to different projects, trying to
be unique and easily searchable online
- although it is a trend to name Go-something all software written in
Go, I wouldn't want to predate this trend for all utilities or code I
write in this language :P

Plus I think it's still short and meaningful.

I have recently added an option, --deinterlace, that works around the
problem of interleaved output lines. I think this makes the utility more
useful and it's not something you can already trivially do with other
non-monolithic binaries.