Status field in ps output for threaded programs in 2.6 kernel

Status field in ps output for threaded programs in 2.6 kernel

[Nikita V. Youshchenko]

Hello.

Today I faced a situation that confused me.

One moment I found that on our server there is almost 100% CPU load in 
system state. However, ps and top and other tools could not locate any 
process eating CPU.

The situation was permanent - several minutes passed, still almost 100% CPU 
in system state, still no visible running processes.

I executed chkrootkit and it found lots of "hidden processes" - ones for 
which /proc/N/ exists (i.e. it is possible to cd there), but N is not in 
ps output (and even not in ls /proc output).

I checked /proc/N/cmdline for some of such "hidden" pids; it looks like 
they represent threads. 
Well, maybe it is how NPTL threads work: for each thread a hidden /proc 
entry exists.
And it is chkrootkit's problem that is reports this (normal) case as 
possible LKM installed.

However, in my case, CPU-eating entities were found among thise hidden 
pids! They belonged to the same process (locally developed app, probably 
hanged), and this process was listed in ps output with status "S". After 
process was killed, CPU load went to 0%.

Looks like kernel bug: displayed status of threaded process should not be 
"S" if some threads are active. And CPU usage statistics should correspond 
to all threads.

This happened on a Debian testing/unstable system, running kernel package 
kernel-image-2.6.5-1-k7-smp 2.6.5-4.

Re: Status field in ps output for threaded programs in 2.6 kernel

[Mihai Rusu]

On Wed, 12 May 2004, Nikita V. Youshchenko wrote:

> Hello.
Hi

> Looks like kernel bug: displayed status of threaded process should not be 
> "S" if some threads are active. And CPU usage statistics should correspond 
> to all threads.

Hmm I dont know about that. A "solution" here is to have top also display
threads (like ps -m or ps -H of a recent procps distro).

> This happened on a Debian testing/unstable system, running kernel package 
> kernel-image-2.6.5-1-k7-smp 2.6.5-4.

-- 
Mihai RUSU                                    Email: dizzy@roedu.net
GPG : http://dizzy.roedu.net/dizzy-gpg.txt    WWW: http://dizzy.roedu.net
                       "Linux is obsolete" -- AST

Re: Status field in ps output for threaded programs in 2.6 kernel

[Albert Cahalan]

Mihai Rusu writes:
> On Wed, 12 May 2004, Nikita V. Youshchenko wrote:

>> Looks like kernel bug: displayed status of threaded process
>> should not be "S" if some threads are active. And CPU usage
>> statistics should correspond to all threads.
>
> Hmm I dont know about that. A "solution" here is to have top also
> display threads (like ps -m or ps -H of a recent procps distro).

First of all, "ps -H" does a process tree display.
You want one of these:

ps m    Tru64 BSD style, with threads after process
ps -m   Tru64 SysV style, with threads after process
ps H    FreeBSD style, with loose threads (independent sorting)
ps -L   Sun style, with grouped threads and LWP column
ps -T   SGI style, with grouped threads and TID column

For process state, maybe have the kernel spit out a '*'
if not all threads are in the same state?

For CPU usage, Linux has been totally broken since day 1.
The SUSv3 and POSIX standard requires that the OS report
some notion of recent CPU usage. Linux doesn't bother.
BSD provides a decaying average, like a per-process
load average. AIX provides a simple average over a few
seconds. SysV provides something similar.

Here is what Solaris does:

You get both per-process and per-thread data. It is
provided as a fixed-point decaying average, with a
15-bit fraction. Thus:

0x8000  100%
0x6000   75%
0x4000   50%
0x2000   25%
0x0000    0%

With Linux, you get only the %CPU over the whole life
of the process. So a process that runs for months using
only 1% of the CPU will still appear to use 1% of the
CPU even after it gets stuck in a CPU-eating loop. You
can get around this by using "top", which is able to
compute a real %CPU from one refresh to the next.