Detecting threads vs processes with ps or /proc

Detecting threads vs processes with ps or /proc

[Nick LeRoy]

Hello, all.

>From searching through mail archives, etc., I'm pretty sure I know the answer 
already, but I'm going to post it anyway.

Our software (Condor) and some related software (Globus) is running on a 
number of systems around the world.  Condor attempts to monitor the RAM usage 
of it's "user" (maybe "client" is a better word here) processes.  If the 
client forks, we need to monitor the client and all of it's children, which 
really isn't difficult.  The _problem_ is that if the client creates threads, 
it's impossible, from what we can tell, to tell the difference between 
separate threads and processes.

So my question, I guess, is this.  How can you tell, from user space, whether 
a group of processes are related as threads or through a "normal" child / 
parent relationship?  The systems that we're running on currently are 2.2.19 
and 2.4.18/19.

>From what else I've read, it seems that the new threading model in 2.5/2.6 is 
changing to a more POSIX friendly model, which will effect this answer, but 
we're not running 2.5 and really can't force such an upgrade -- hell, right 
now we're having problems getting a switch from 2.2 pushed through.

Thanks _very_ much in advance.  I'd be tickled pink if the answer is something 
like "just look at the foo flag in ps", or "upgrade to version 1.2.3.4 of 
procps and do xyzzy", but my intuition tells me otherwise.

Thanks,

-Nick

Re: Detecting threads vs processes with ps or /proc

[Robert Love]

On Fri, 2002-12-06 at 10:24, Nick LeRoy wrote:

> >From what else I've read, it seems that the new threading model in 2.5/2.6 is 
> changing to a more POSIX friendly model, which will effect this answer, but 
> we're not running 2.5 and really can't force such an upgrade -- hell, right 
> now we're having problems getting a switch from 2.2 pushed through.

Yep, you should get what you want with 2.5 + NPTL.  We need to add a few
bits, though, to make it complete.

> Thanks _very_ much in advance.  I'd be tickled pink if the answer is something 
> like "just look at the foo flag in ps", or "upgrade to version 1.2.3.4 of 
> procps and do xyzzy", but my intuition tells me otherwise.

See http://tech9.net/rml/procps

and "upgrade to version 2.0.8 or later of procps" :)

It is just a heuristic, though.  A hack in fact.  We look at a process's
children and compare RSS, VM size, and the process image they are
running.  If they are the same, we label them threads.

It is the default behavior.  Flag `-m' turns it off.

See thread_group() and flag_threads().

	Robert Love

Re: Detecting threads vs processes with ps or /proc

[Nick LeRoy]

On Friday 06 December 2002 1:48 pm, Robert Love wrote:
> On Fri, 2002-12-06 at 10:24, Nick LeRoy wrote:
> > >From what else I've read, it seems that the new threading model in
> > > 2.5/2.6 is
> >
> > changing to a more POSIX friendly model, which will effect this answer,
> > but we're not running 2.5 and really can't force such an upgrade -- hell,
> > right now we're having problems getting a switch from 2.2 pushed through.
>
> Yep, you should get what you want with 2.5 + NPTL.  We need to add a few
> bits, though, to make it complete.
>
> > Thanks _very_ much in advance.  I'd be tickled pink if the answer is
> > something like "just look at the foo flag in ps", or "upgrade to version
> > 1.2.3.4 of procps and do xyzzy", but my intuition tells me otherwise.
>
> See http://tech9.net/rml/procps
>
> and "upgrade to version 2.0.8 or later of procps" :)
>
> It is just a heuristic, though.  A hack in fact.  We look at a process's
> children and compare RSS, VM size, and the process image they are
> running.  If they are the same, we label them threads.

I was considerring doing something like this as well.  From your experience, 
does it work reliably?  Do you need to apply a small 'fudge factor' (aka 
VMsize.1 ~= VMsize.2)?

> It is the default behavior.  Flag `-m' turns it off.
>
> See thread_group() and flag_threads().

I assume these are functions in the tools themselves?

> 	Robert Love

Thanks

-Nick

Re: Detecting threads vs processes with ps or /proc

[Robert Love]

On Fri, 2002-12-06 at 14:56, Nick LeRoy wrote:

> I was considerring doing something like this as well.  From your
> experience,  does it work reliably?

It never fails to detect threads (no false negatives).

It does sometimes detect child processes that forked but did not exec as
threads (false positives).  The failure case is when a program forks,
does not call exec, and the children go on to execute the exact same
code (so they look and act just like threads, but they have unique
addresses spaces).

One thing to note: if you can modify the kernel and procps, you can just
export the value of task->mm out of /proc.  It is a gross hack, and
perhaps a security issue, but that will work 100%.  Same ->mm implies
thread.

> Do you need to apply a small 'fudge factor' (aka 
> VMsize.1 ~= VMsize.2)?

No, they need to be exact.  The stats are pulled from the same structure
in the kernel so they need to be perfect.

> > It is the default behavior.  Flag `-m' turns it off.
> >
> > See thread_group() and flag_threads().
> 
> I assume these are functions in the tools themselves?

Yep.  Both ps and top have support.  There is a patch on the website
that added the feature, so you can take a look at that.

Red Hat 8.0 has support in their version of procps 2.0.7, too. 
Otherwise it was added in 2.0.8.  The current version is 2.0.10.

We have a mailing list at procps-list@redhat.com where this sort of
discussion is welcome.

	Robert Love

Re: Detecting threads vs processes with ps or /proc

[Gilad Ben-Yossef]

On Fri, 2002-12-06 at 17:24, Nick LeRoy wrote:

> 
> Our software (Condor) and some related software (Globus) is running on a 
> number of systems around the world.  Condor attempts to monitor the RAM usage 
> of it's "user" (maybe "client" is a better word here) processes.  If the 
> client forks, we need to monitor the client and all of it's children, which 
> really isn't difficult.  The _problem_ is that if the client creates threads, 
> it's impossible, from what we can tell, to tell the difference between 
> separate threads and processes.
> 
> So my question, I guess, is this.  How can you tell, from user space, whether 
> a group of processes are related as threads or through a "normal" child / 
> parent relationship?  The systems that we're running on currently are 2.2.19 
> and 2.4.18/19.

There is another approach save for the one already discussed here, which
I have no idea how applicable it is in your case, but will produce 100%
reliable results without additional kernel support - track the processes
forks.

There are several ways you can go about it - there's the expensive (in
terms of CPU cycles) approach of using ptrace(2), the relativly painless
way of overriding the "default" calls to fork and friends via ld.so(8)
magical LD_PRELOAD  that nevertheless requires you control the execution
of the "client" programs and even using a system call tracking module
such as syscall-tracker (http://syscalltrack.sf.net) which might not be
quite ready for use on a production system as of yet.

Hope this helps,
Gilad

-- 
 Gilad Ben-Yossef <gilad@benyossef.com> 
 http://benyossef.com 
 "Denial really is a river in Eygept."

Re: Detecting threads vs processes with ps or /proc

[Albert D. Cahalan]

Robert Love writes:
> On Fri, 2002-12-06 at 14:56, Nick LeRoy wrote:
>
>> I was considerring doing something like this as well.  From your
>> experience,  does it work reliably?
>
> It never fails to detect threads (no false negatives).

Testing the algorithm on idle test processes won't show
this damn-obvious race condition:

1. you read the first thread's info
2. the second thread calls mmap() and/or takes a page fault
3. you read the second thread's info

> It does sometimes detect child processes that forked but did not exec as
> threads (false positives).  The failure case is when a program forks,
> does not call exec, and the children go on to execute the exact same
> code (so they look and act just like threads, but they have unique
> addresses spaces).
>
> One thing to note: if you can modify the kernel and procps, you can just
> export the value of task->mm out of /proc.  It is a gross hack, and
> perhaps a security issue, but that will work 100%.  Same ->mm implies
> thread.

This forces sorting, which is slow and eats memory.
It's better to list the ->mm peers in a proc file.
For example, a /proc/42/mm-peers file containing
a simple list of the peers.

There's still a race condition, but at least it only
involves threads that get created during the /proc scan.

There's also the problem with new-style threads simply not
being listed anywhere. Crackers are going to love it if
the Linux 2.6 kernel has a built-in way to hide things.

Re: Detecting threads vs processes with ps or /proc

[Nick LeRoy]

On Sunday 08 December 2002 06:24 pm, Albert D. Cahalan wrote:
> Robert Love writes:
> > On Fri, 2002-12-06 at 14:56, Nick LeRoy wrote:
> >> I was considerring doing something like this as well.  From your
> >> experience,  does it work reliably?
> >
> > It never fails to detect threads (no false negatives).
>
> Testing the algorithm on idle test processes won't show
> this damn-obvious race condition:
>
> 1. you read the first thread's info
> 2. the second thread calls mmap() and/or takes a page fault
> 3. you read the second thread's info

Yeah, I thought of that, too.  Probably not too likely, of course, but it 
still there.  Murphy rules.

Robert:  Any thoughts on this?

-Nick

Re: Detecting threads vs processes with ps or /proc

[Robert Love]

On Sun, 2002-12-08 at 22:18, Nick LeRoy wrote:

> Yeah, I thought of that, too.  Probably not too likely, of course, but it 
> still there.  Murphy rules.
> 
> Robert:  Any thoughts on this?

Sure, it can happen.  The original poster asked me this in private the
other day.  The race is pretty improbable but it certainly can happen. 
The heuristic is imperfect.  Then again, this same sort of thing can
occur in other places too since all the reads in /proc are non-atomic
wrt each other.

	Robert Love

Re: Detecting threads vs processes with ps or /proc

[Jeremy Fitzhardinge]

On Fri, 2002-12-06 at 12:09, Robert Love wrote:
> One thing to note: if you can modify the kernel and procps, you can just
> export the value of task->mm out of /proc.  It is a gross hack, and
> perhaps a security issue, but that will work 100%.  Same ->mm implies
> thread.

It isn't a terribly gross hack.  I have a patch (somewhere...) which
adds an ASID: field to /proc/<pid>/status, which simply contains the mm
pointer (as an opaque identifier token).  If you were worried about
exposing (yet another) kernel pointer value, I suppose you could mush it
about a bit, but I think that would give the illusion of obscurity
rather than any actual increase in security.

	J