Dan Bongert (dbo...@wisc.edu)

Ok, it's definitely getting an error from somewhere:

thoth(3) /tmp> ls

thoth(4) /tmp> echo $?
141

Although:

thoth(31) ~> top


thoth(32) ~> echo $?


> What is your output device? A serial terminal? If so, could be simple
> flow control issues. In fact, any serial connection (even a PC emulating
> a terminal) could suffer from flow control problems. And they would tend
> to be erratic in nature.

I'm usually sshing into the machine, but I've also experienced the problem
on the console.

> If you are on a normal console, try running the commands similart to
> this (trying to determine if *something* else is receiving output or
> not)
>
>     <your command> &> /dev/tty
>
> if this works reliably, maybe that's a starting point.

Nope, that fails intermittently as well.

> There's a couple kernel guys who frequent this list. Maybe one of them
> will have a clue as to what could go wrong. Corrupted libraries and
> whatnot.
>
> You might try that rpm -V command earlier against all packages (add a
> "a" IIRC). Maybe some library accessed by the coreutils, but which is
> not itself part of coreutils, is corrupt.

Hmm....when I do a 'rpm -Va', I get lots of "at least one of file's
dependencies has changed since prelinking" errors. Even if I run prelink
manually, and then do a 'rpm -Va' immediately afterwards.

It's a local disk:

thoth(97) /tmp> df -h .
Filesystem            Size  Used Avail Use% Mounted on
/dev/md4               16G   77M   15G   1% /tmp

Though 'ls' was just an example -- just about any program will fail. The 'w'
command will fail too:

thoth(118) /tmp> w
   16:06:51 up  5:34,  1 user,  load average: 0.94, 1.46, 2.04
USER     TTY      FROM              LOGIN@   IDLE   JCPU   PCPU WHAT
dbongert pts/0    copland.ssc.wisc 14:16    0.00s  0.22s  0.05s w

thoth(119) /tmp> w
   16:06:52 up  5:34,  1 user,  load average: 0.94, 1.46, 2.04
USER     TTY      FROM              LOGIN@   IDLE   JCPU   PCPU WHAT
dbongert pts/0    copland.ssc.wisc 14:16    0.00s  0.22s  0.05s w

thoth(120) /tmp> w

thoth(121) /tmp> w

Everything seems OK there:


thoth(96) /tmp> sudo rpm -V coreutils procps util-linux

> You can also try running ``strace /bin/ls'' to see what is going on.

Funnily enough, running strace will work just fine. Though, as I said, just
about any command will fail -- 'ls' was just for testing purposes.

Hello all:

I have a couple CentOS 4 servers (all up-to-date) that are having strange
command failures. I first noticed this with a perl script that uses lots of
system calls.

Basically, sometimes a command just won't run:

thoth(52) /tmp> ls

thoth(53) /tmp> ls

thoth(54) /tmp> ls

thoth(55) /tmp> ls
learner  lost+found/

thoth(56) /tmp> ls
learner  lost+found/

thoth(57) /tmp> ls
learner  lost+found/

thoth(58) /tmp> ls
learner  lost+found/

thoth(59) /tmp> ls
learner  lost+found/

thoth(60) /tmp> ls
learner  lost+found/

thoth(61) /tmp> ls
learner  lost+found/

thoth(62) /tmp> ls

thoth(63) /tmp> ls

thoth(64) /tmp> ls

thoth(65) /tmp> ls

thoth(66) /tmp> uname -a
Linux thoth.ssc.wisc.edu 2.6.9-67.0.7.ELsmp #1 SMP Sat Mar 15 06:54:55 EDT
2008 i686 i686 i386 GNU/Linux

Nothing in either dmesg or /var/log/messages seems to indicate any problems.
It also doesn't seem to matter what the command is -- ls is the quickest
test, but sshd will sometimes to fail to spawn children, etc. There aren't a
large amount of processes on the machine either -- only 122 at the moment.

Has anyone seen this behavior before? Have I been hit with some sort of
cunning rootkit? This machine shouldn't be publicly accessible; it's behind
our firewall.

Thanks.