Re: v3.4-rc2 out-of-memory problems (was Re: 3.4-rc1 sticks-and-crashs)

Re: v3.4-rc2 out-of-memory problems (was Re: 3.4-rc1 sticks-and-crashs)

[David Rientjes]

On Sun, 8 Apr 2012, Linus Torvalds wrote:

> Or does anybody see anything that keeps thread counts raised so that
> "free_task()" doesn't get done. kernel/profoe.c does that
> "profile_handoff_task()" thing - but only oprofile and the android
> low-memory-killer logic seems to use it though. But that's exactly the
> kind of thing that Werner's "configure everything" might enable -
> Werner?
> 

I think you nailed it.

I suspect the problem is 1eda5166c764 ("staging: android/lowmemorykiller: 
Don't unregister notifier from atomic context") merged during the 3.4 
merge window and, unfortunately, backported to stable.

Werner's config has CONFIG_ANDROID_LOW_MEMORY_KILLER=y so we never 
actually unregister the callback for the task handoff as a result of the 
patch.  It's supposed to take responsibility for doing free_task() itself 
when it's good and ready, usually by putting it into a list to free, but 
now we're just doing this:

	struct task_struct *task = data;

	if (task == lowmem_deathpending)
		lowmem_deathpending = NULL;

	return NOTIFY_OK;

whenever put_task_struct() decrements the refcount to 0 and thus they get 
leaked and bad things happen.

This is confirmed by Werner's oom log that shows extremely small values 
for the oom score of the task chosen to oom kill.  His first log showed X 
being killed with a score of 29.  That means it is the most memory-hogging 
task on the system and is only using 2.9% of total system memory.

I can't actually see how the lowmemorykiller actually ever freed any 
task_struct after unregistering the notifier during the callback.  It 
seems like this has always leaked memory but it used to happen much more 
slowly because, prior to the patch, we did task_handoff_unregister() in 
the callback.  So I think the code was always wrong but now it's out of 
control because the notifier remains enabled indefinitely.  I can't say 
the 1eda5166c764 ("staging: android/lowmemorykiller: Don't unregister 
notifier from atomic context") commit is fully to blame, it just made the 
error much more egregious.

As it sits in 3.4-rc2, this whole lowmem_deathpending business seems to be 
storing a pointer to the task_struct of something sent a SIGKILL and it 
remains that way until the lowmem_deathpending_timeout expires and 
something else is killed instead.  lowmem_deathpending gets cleared on the 
task handoff if the task selected for kill just exited.  This ensures we 
only kill one thread at a time.

That's all fine and good but it seems like we're never freeing the 
task_struct itself on exit.  This seems like the most obvious fix but it 
would be really nice to revisit this and remove the dependency on 
CONFIG_PROFILING and just check if the lowmem_deathpending thread is found 
in the iteration for lowmem_shrink() prior to killing.


android, lowmemorykiller: free task struct on profiling handoff

The lowmemorykiller stores a pointer to a killed thread's task_struct in 
lowmem_deathpending when profiling is enabled.  When put_task_struct() 
results in the refcount going to 0, the task_notify_func() callback clears 
lowmem_deathpending if it is the thread that was killed last.  This 
prevents additional killing until lowmem_deathpending_timeout elapses.

The responsibility of every task handoff notifier is to free the tasks 
handed off to it, however, and this was being neglected, which results 
in a massive memory leak since no task_struct ever gets freed.

Fix that by freeing the task_struct since we no longer need a reference to 
it.

Reported-by: werner <w.landgraf@ru.ru>
Cc: stable@vger.kernel.org
Signed-off-by: David Rientjes <rientjes@google.com>
---
 drivers/staging/android/lowmemorykiller.c |    1 +
 1 file changed, 1 insertion(+)

diff --git a/drivers/staging/android/lowmemorykiller.c b/drivers/staging/android/lowmemorykiller.c
--- a/drivers/staging/android/lowmemorykiller.c
+++ b/drivers/staging/android/lowmemorykiller.c
@@ -78,6 +78,7 @@ task_notify_func(struct notifier_block *self, unsigned long val, void *data)
 
 	if (task == lowmem_deathpending)
 		lowmem_deathpending = NULL;
+	free_task(task);
 
 	return NOTIFY_OK;
 }

Re: v3.4-rc2 out-of-memory problems (was Re: 3.4-rc1 sticks-and-crashs)

[Linus Torvalds]

On Mon, Apr 9, 2012 at 3:15 AM, David Rientjes <rientjes@google.com> wrote:
>
> I think you nailed it.
>
> I suspect the problem is 1eda5166c764 ("staging: android/lowmemorykiller:
> Don't unregister notifier from atomic context") merged during the 3.4
> merge window and, unfortunately, backported to stable.

Ok. That does seem to match everything.

However, I think your patch is the wrong one.

The real bug is actually that those notifiers are a f*cking joke, and
the return value from the notifier is a mistake.

So I personally think that the real problem is this code in
profile_handoff_task:

        return (ret == NOTIFY_OK) ? 1 : 0;

and ask yourself two questions:

 - what the hell does NOTIFY_OK/NOTIFY_DONE mean?
 - what happens if there are multiple notifiers that all (or some)
return NOTIFY_OK?

I'll tell you what my answers are:

 (a) NOTIFY_DONE is the "ok, everything is fine, you can free the
task-struct". It's also what that handoff notifier thing returns if
there are no notifiers registered at all.

     So the fix to the Android lowmemorykiller is as simple as just
changing NOTIFY_OK to NOTIFY_DONE, which will mean that the caller
will properly free the task struct.

     The NOTIFY_OK/NOTIFY_DONE difference really does seem to be just
"NOTIFY_OK means that I will free the task myself later". That's what
the oprofile uses, and it frees the task.

 (b) But the whole interface is a total f*cking mess. If *multiple*
people return NOTIFY_OK, they're royally fucked. And the whole (and
only) point of notifiers is that you can register multiple different
ones independently.

So quite frankly, the *real* bug is not in that android driver
(although I'd say that we should just make it return NOTIFY_DONE and
be done with it). The real bug is that the whole f*cking notifier is a
mistake, and checking the error return was the biggest mistake of all.

Werner: just test David's patch (do *not* change both the error value
*and* apply David's patch - that would free the task-struct twice). I
don't think his patch is what I want to apply eventually, but it
should fix the issue.

Sadly, I don't think we have anybody who really "owns"
kernel/profile.c - the thing is broken, it was misdesigned, and nobody
really cares. Which is why we'll probably have to fix this by just
making that Android thing return NOTIFY_DONE, and just accept that the
whole thing is a f*cking joke.

                         Linus

Re: v3.4-rc2 out-of-memory problems (was Re: 3.4-rc1 sticks-and-crashs)

[David Rientjes]

On Mon, 9 Apr 2012, Linus Torvalds wrote:

> The real bug is actually that those notifiers are a f*cking joke, and
> the return value from the notifier is a mistake.
> 
> So I personally think that the real problem is this code in
> profile_handoff_task:
> 
>         return (ret == NOTIFY_OK) ? 1 : 0;
> 
> and ask yourself two questions:
> 
>  - what the hell does NOTIFY_OK/NOTIFY_DONE mean?
>  - what happens if there are multiple notifiers that all (or some)
> return NOTIFY_OK?
> 

NOTIFY_OK should never be a valid response for this notifier the way it's 
currently implemented, it should be NOTIFY_STOP to stop iterating the call 
chain to avoid a double free.  Right now it doesn't matter because only 
oprofile is actually freeing the task_struct and lowmemorykiller should be 
using NOTIFY_DONE.

Then we have a completeness issue if multiple callbacks want to return 
NOTIFY_STOP and an ordering issue if the oprofile callback is invoked 
before lowmemorykiller.

> I'll tell you what my answers are:
> 
>  (a) NOTIFY_DONE is the "ok, everything is fine, you can free the
> task-struct". It's also what that handoff notifier thing returns if
> there are no notifiers registered at all.
> 
>      So the fix to the Android lowmemorykiller is as simple as just
> changing NOTIFY_OK to NOTIFY_DONE, which will mean that the caller
> will properly free the task struct.
> 

I don't think so for Werner's config who also has CONFIG_OPROFILE=y, so 
oprofile would return NOTIFY_OK and queue the task_struct for free, then 
the second notifier callback to the lowmemorykiller would return 
NOTIFY_DONE which would result in put_task_struct() doing free_task() 
itself for a double free.

>      The NOTIFY_OK/NOTIFY_DONE difference really does seem to be just
> "NOTIFY_OK means that I will free the task myself later". That's what
> the oprofile uses, and it frees the task.
> 
>  (b) But the whole interface is a total f*cking mess. If *multiple*
> people return NOTIFY_OK, they're royally fucked. And the whole (and
> only) point of notifiers is that you can register multiple different
> ones independently.
> 
> So quite frankly, the *real* bug is not in that android driver
> (although I'd say that we should just make it return NOTIFY_DONE and
> be done with it). The real bug is that the whole f*cking notifier is a
> mistake, and checking the error return was the biggest mistake of all.
> 

Right, we can't handoff the freeing of the task_struct to more than one 
notifier.  It seems misdesigned from the beginning and what we really want 
is to hijack task->usage for __put_task_struct(task) if we have such a 
notifier callchain and require each one (currently just oprofile) to take 
a reference on task->usage for NOTIFY_OK and then be responsible for 
dropping the reference when it's done with it later instead of requiring 
it to free the task_struct itself.

That's _if_ we want to continue to have such an interface in the first 
place where it's only really necessary right now for oprofile (and, hence, 
wasn't implemented in an extendable way).  I'm thinking the 
lowmemorykiller, as I eluded to, could be written in a way where we can 
detect if a thread we've already killed has exited yet before killing 
another one.  We can't just store a pointer to the task_struct of the 
killed task since it could be reused for a fork later, but we could use 
TIF_MEMDIE like the oom killer does.

Re: v3.4-rc2 out-of-memory problems (was Re: 3.4-rc1 sticks-and-crashs)

[Linus Torvalds]

[-- Attachment #1: Type: text/plain, Size: 2126 bytes --]

On Mon, Apr 9, 2012 at 2:22 PM, David Rientjes <rientjes@google.com> wrote:
>
> NOTIFY_OK should never be a valid response for this notifier the way it's
> currently implemented, it should be NOTIFY_STOP to stop iterating the call
> chain to avoid a double free.

No, that's no good either. That would mean that some people wouldn't
be notified about the death of the task at all.

So NOTIFY_STOP just implies *another* bug.

> Right, we can't handoff the freeing of the task_struct to more than one
> notifier.  It seems misdesigned from the beginning and what we really want
> is to hijack task->usage for __put_task_struct(task) if we have such a
> notifier callchain and require each one (currently just oprofile) to take
> a reference on task->usage for NOTIFY_OK and then be responsible for
> dropping the reference when it's done with it later instead of requiring
> it to free the task_struct itself.

We could make notifier.c just "or" all the return values together, and
then it's ok if *one* person returns NOTIFY_OK.

Of course, that's not how notifiers are documented to work, but quite
frankly, notifiers with non-zero values that don't sat STOP are broken
as-is anyway, so you might we well do a logical "or" of the return
values and at least make things like this work.

I personally think every single notifier interface we have ever had in
the kernel has been a total f*cking disaster. The whole concept of
"run these random functions at this random event" is a broken concept
that just makes people do crazy broken things.

Oh well. So my suggestion right now would be something like the
attached. It's still horribly broken, it actively breaks documented
notifier behavior, but dammit, if the notifier people don't like
'or'ing return values together they should damn well return zero from
the notifier that doesn't do anything. And returning an error will
exit out, so..

Hmm? Who cares about that kernel/notifier.c code? Andrew? Ingo? We
don't have any actual maintainer for that crap, but judging by the
commits, it's one of you two.

                  Linus

[-- Attachment #2: patch.diff --]
[-- Type: application/octet-stream, Size: 1047 bytes --]

 drivers/staging/android/lowmemorykiller.c |    2 +-
 kernel/notifier.c                         |    2 +-
 2 files changed, 2 insertions(+), 2 deletions(-)

diff --git a/drivers/staging/android/lowmemorykiller.c b/drivers/staging/android/lowmemorykiller.c
index 052b43e4e505..142bfc2f84db 100644
--- a/drivers/staging/android/lowmemorykiller.c
+++ b/drivers/staging/android/lowmemorykiller.c
@@ -79,7 +79,7 @@ task_notify_func(struct notifier_block *self, unsigned long val, void *data)
 	if (task == lowmem_deathpending)
 		lowmem_deathpending = NULL;
 
-	return NOTIFY_OK;
+	return NOTIFY_DONE;
 }
 
 static int lowmem_shrink(struct shrinker *s, struct shrink_control *sc)
diff --git a/kernel/notifier.c b/kernel/notifier.c
index 2d5cc4ccff7f..11fe956e8daf 100644
--- a/kernel/notifier.c
+++ b/kernel/notifier.c
@@ -90,7 +90,7 @@ static int __kprobes notifier_call_chain(struct notifier_block **nl,
 			continue;
 		}
 #endif
-		ret = nb->notifier_call(nb, val, v);
+		ret |= nb->notifier_call(nb, val, v);
 
 		if (nr_calls)
 			(*nr_calls)++;

Re: v3.4-rc2 out-of-memory problems (was Re: 3.4-rc1 sticks-and-crashs)

[Colin Cross]

On Mon, Apr 9, 2012 at 2:22 PM, David Rientjes <rientjes@google.com> wrote:
> On Mon, 9 Apr 2012, Linus Torvalds wrote:
>
>> The real bug is actually that those notifiers are a f*cking joke, and
>> the return value from the notifier is a mistake.
>>
>> So I personally think that the real problem is this code in
>> profile_handoff_task:
>>
>> � � � � return (ret == NOTIFY_OK) ? 1 : 0;
>>
>> and ask yourself two questions:
>>
>> �- what the hell does NOTIFY_OK/NOTIFY_DONE mean?
>> �- what happens if there are multiple notifiers that all (or some)
>> return NOTIFY_OK?
>>
> NOTIFY_OK should never be a valid response for this notifier the way it's
> currently implemented, it should be NOTIFY_STOP to stop iterating the call
> chain to avoid a double free. �Right now it doesn't matter because only
> oprofile is actually freeing the task_struct and lowmemorykiller should be
> using NOTIFY_DONE.
>
> Then we have a completeness issue if multiple callbacks want to return
> NOTIFY_STOP and an ordering issue if the oprofile callback is invoked
> before lowmemorykiller.
>
>> I'll tell you what my answers are:
>>
>> �(a) NOTIFY_DONE is the "ok, everything is fine, you can free the
>> task-struct". It's also what that handoff notifier thing returns if
>> there are no notifiers registered at all.
>>
>> � � �So the fix to the Android lowmemorykiller is as simple as just
>> changing NOTIFY_OK to NOTIFY_DONE, which will mean that the caller
>> will properly free the task struct.
>>
>
> I don't think so for Werner's config who also has CONFIG_OPROFILE=y, so
> oprofile would return NOTIFY_OK and queue the task_struct for free, then
> the second notifier callback to the lowmemorykiller would return
> NOTIFY_DONE which would result in put_task_struct() doing free_task()
> itself for a double free.
>
>> � � �The NOTIFY_OK/NOTIFY_DONE difference really does seem to be just
>> "NOTIFY_OK means that I will free the task myself later". That's what
>> the oprofile uses, and it frees the task.
>>
>> �(b) But the whole interface is a total f*cking mess. If *multiple*
>> people return NOTIFY_OK, they're royally fucked. And the whole (and
>> only) point of notifiers is that you can register multiple different
>> ones independently.
>>
>> So quite frankly, the *real* bug is not in that android driver
>> (although I'd say that we should just make it return NOTIFY_DONE and
>> be done with it). The real bug is that the whole f*cking notifier is a
>> mistake, and checking the error return was the biggest mistake of all.
>>
>
> Right, we can't handoff the freeing of the task_struct to more than one
> notifier. �It seems misdesigned from the beginning and what we really want
> is to hijack task->usage for __put_task_struct(task) if we have such a
> notifier callchain and require each one (currently just oprofile) to take
> a reference on task->usage for NOTIFY_OK and then be responsible for
> dropping the reference when it's done with it later instead of requiring
> it to free the task_struct itself.
>
> That's _if_ we want to continue to have such an interface in the first
> place where it's only really necessary right now for oprofile (and, hence,
> wasn't implemented in an extendable way). �I'm thinking the
> lowmemorykiller, as I eluded to, could be written in a way where we can
> detect if a thread we've already killed has exited yet before killing
> another one. �We can't just store a pointer to the task_struct of the
> killed task since it could be reused for a fork later, but we could use
> TIF_MEMDIE like the oom killer does.

This was a known issue in 2010, in the android tree the use of
task_handoff_register was dropped one day after it was added and
replaced with a new task_free_register hook.  I assume Greg dropped
the fix during the android tree refresh in 3.0 because it depended on
a change to kernel/fork.c.  The two relevant patches are (using
codeaurora's gitweb becase we don't have one right now):

sched: Add a generic notifier when a task struct is about to be freed
https://www.codeaurora.org/gitweb/quic/la/?p=kernel/common.git;a=commitdiff;h=667dffa787a87ef4ea43cc65957ce96077fdcd0a

staging: android: lowmemorykiller: Fix task_struct leak
https://www.codeaurora.org/gitweb/quic/la/?p=kernel/common.git;a=commitdiff;h=af0240f095a704f75f032bbcc01f670c65c163ba

Re: v3.4-rc2 out-of-memory problems (was Re: 3.4-rc1 sticks-and-crashs)

[Greg Kroah-Hartman]

On Mon, Apr 09, 2012 at 03:13:00PM -0700, Colin Cross wrote:
> On Mon, Apr 9, 2012 at 2:22 PM, David Rientjes <rientjes@google.com> wrote:
> > On Mon, 9 Apr 2012, Linus Torvalds wrote:
> >
> >> The real bug is actually that those notifiers are a f*cking joke, and
> >> the return value from the notifier is a mistake.
> >>
> >> So I personally think that the real problem is this code in
> >> profile_handoff_task:
> >>
> >> � � � � return (ret == NOTIFY_OK) ? 1 : 0;
> >>
> >> and ask yourself two questions:
> >>
> >> �- what the hell does NOTIFY_OK/NOTIFY_DONE mean?
> >> �- what happens if there are multiple notifiers that all (or some)
> >> return NOTIFY_OK?
> >>
> > NOTIFY_OK should never be a valid response for this notifier the way it's
> > currently implemented, it should be NOTIFY_STOP to stop iterating the call
> > chain to avoid a double free. �Right now it doesn't matter because only
> > oprofile is actually freeing the task_struct and lowmemorykiller should be
> > using NOTIFY_DONE.
> >
> > Then we have a completeness issue if multiple callbacks want to return
> > NOTIFY_STOP and an ordering issue if the oprofile callback is invoked
> > before lowmemorykiller.
> >
> >> I'll tell you what my answers are:
> >>
> >> �(a) NOTIFY_DONE is the "ok, everything is fine, you can free the
> >> task-struct". It's also what that handoff notifier thing returns if
> >> there are no notifiers registered at all.
> >>
> >> � � �So the fix to the Android lowmemorykiller is as simple as just
> >> changing NOTIFY_OK to NOTIFY_DONE, which will mean that the caller
> >> will properly free the task struct.
> >>
> >
> > I don't think so for Werner's config who also has CONFIG_OPROFILE=y, so
> > oprofile would return NOTIFY_OK and queue the task_struct for free, then
> > the second notifier callback to the lowmemorykiller would return
> > NOTIFY_DONE which would result in put_task_struct() doing free_task()
> > itself for a double free.
> >
> >> � � �The NOTIFY_OK/NOTIFY_DONE difference really does seem to be just
> >> "NOTIFY_OK means that I will free the task myself later". That's what
> >> the oprofile uses, and it frees the task.
> >>
> >> �(b) But the whole interface is a total f*cking mess. If *multiple*
> >> people return NOTIFY_OK, they're royally fucked. And the whole (and
> >> only) point of notifiers is that you can register multiple different
> >> ones independently.
> >>
> >> So quite frankly, the *real* bug is not in that android driver
> >> (although I'd say that we should just make it return NOTIFY_DONE and
> >> be done with it). The real bug is that the whole f*cking notifier is a
> >> mistake, and checking the error return was the biggest mistake of all.
> >>
> >
> > Right, we can't handoff the freeing of the task_struct to more than one
> > notifier. �It seems misdesigned from the beginning and what we really want
> > is to hijack task->usage for __put_task_struct(task) if we have such a
> > notifier callchain and require each one (currently just oprofile) to take
> > a reference on task->usage for NOTIFY_OK and then be responsible for
> > dropping the reference when it's done with it later instead of requiring
> > it to free the task_struct itself.
> >
> > That's _if_ we want to continue to have such an interface in the first
> > place where it's only really necessary right now for oprofile (and, hence,
> > wasn't implemented in an extendable way). �I'm thinking the
> > lowmemorykiller, as I eluded to, could be written in a way where we can
> > detect if a thread we've already killed has exited yet before killing
> > another one. �We can't just store a pointer to the task_struct of the
> > killed task since it could be reused for a fork later, but we could use
> > TIF_MEMDIE like the oom killer does.
> 
> This was a known issue in 2010, in the android tree the use of
> task_handoff_register was dropped one day after it was added and
> replaced with a new task_free_register hook.  I assume Greg dropped
> the fix during the android tree refresh in 3.0 because it depended on
> a change to kernel/fork.c.  The two relevant patches are (using
> codeaurora's gitweb becase we don't have one right now):
> 
> sched: Add a generic notifier when a task struct is about to be freed
> https://www.codeaurora.org/gitweb/quic/la/?p=kernel/common.git;a=commitdiff;h=667dffa787a87ef4ea43cc65957ce96077fdcd0a

Yes, I can't add a patch like that for this driver, that is why I
thought everyone was getting together to "properly" determine how to
solve this oom notifier problem.  Has that work stalled somwhere?

greg k-h

Re: v3.4-rc2 out-of-memory problems (was Re: 3.4-rc1 sticks-and-crashs)

[Linus Torvalds]

On Mon, Apr 9, 2012 at 3:13 PM, Colin Cross <ccross@google.com> wrote:
>
> sched: Add a generic notifier when a task struct is about to be freed
> https://www.codeaurora.org/gitweb/quic/la/?p=kernel/common.git;a=commitdiff;h=667dffa787a87ef4ea43cc65957ce96077fdcd0a

Oh, *HELL*NO*!

It's a fucking disaster in "Oh, one notifier was broken, SO LET'S ADD
ANOTHER RANDOM ONE TO FIX THAT".

The definition of insanity is doing the same thing over and over and
thinking you get a different result. Let's not do that kind of idiotic
thing.

Notifiers are evil crap. Let's make *fewer* of them, not add
yet-another-random-notifier-for-some-random-reason.

F*ck me, but how I hate those random notifiers. And I hate people who
add them willy nilly.

                             Linus

Re: v3.4-rc2 out-of-memory problems (was Re: 3.4-rc1 sticks-and-crashs)

[john stultz]

On Mon, Apr 9, 2012 at 3:21 PM, Greg Kroah-Hartman
<gregkh@linuxfoundation.org> wrote:
> On Mon, Apr 09, 2012 at 03:13:00PM -0700, Colin Cross wrote:
>> sched: Add a generic notifier when a task struct is about to be freed
>> https://www.codeaurora.org/gitweb/quic/la/?p=kernel/common.git;a=commitdiff;h=667dffa787a87ef4ea43cc65957ce96077fdcd0a
>
> Yes, I can't add a patch like that for this driver, that is why I
> thought everyone was getting together to "properly" determine how to
> solve this oom notifier problem. �Has that work stalled somwhere?

Anton Vorontsov  has been working on this (and just sent out some
related vmevent patches today). His hope is to use the vmevent  or mem
cgroup interface to notify a userland killer to get the same or
improved behavior as the in-kernel lowmemory killer.

thanks
-john

Re: v3.4-rc2 out-of-memory problems (was Re: 3.4-rc1 sticks-and-crashs)

[David Rientjes]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 3064 bytes --]

On Mon, 9 Apr 2012, Linus Torvalds wrote:

> > Right, we can't handoff the freeing of the task_struct to more than one
> > notifier.  It seems misdesigned from the beginning and what we really want
> > is to hijack task->usage for __put_task_struct(task) if we have such a
> > notifier callchain and require each one (currently just oprofile) to take
> > a reference on task->usage for NOTIFY_OK and then be responsible for
> > dropping the reference when it's done with it later instead of requiring
> > it to free the task_struct itself.
> 
> We could make notifier.c just "or" all the return values together, and
> then it's ok if *one* person returns NOTIFY_OK.
> 

You could that if you also turned the check for "ret == NOTIFY_OK" in 
profile_handoff_task() into "ret & NOTIFY_OK" in your patch, otherwise you 
get a double free from __put_task_struct() and oprofile.

> Of course, that's not how notifiers are documented to work, but quite
> frankly, notifiers with non-zero values that don't sat STOP are broken
> as-is anyway, so you might we well do a logical "or" of the return
> values and at least make things like this work.
> 

It works fine if the callbacks are correctly implemented, it's just that 
the task handoff in kernel/profile.c is broken because it assumes only one 
callback will return NOTIFY_OK, meaning it will eventually free, and its 
only checking the return value of the last notifier called to see if 
__put_task_struct() should immediately free.

In defense of notifiers, though, it works fine right now for memory 
hotplug.  The last issue I had with it was when slab lacked a callback 
when a node was onlined or offlined in 2.6.34 and then I added memory 
hotplug support for that allocator and it has since worked fine.  For 
things like MEM_GOING_OFFLINE, returning NOTIFY_BAD is great if the 
subsystem of interest can't allow the memory to go offline (in-use slab 
objects, for example).  In the memory hotplug usecase, we certainly don't 
want to stop at NOTIFY_OK because we need to notify every subsystem on the 
callchain.

> Oh well. So my suggestion right now would be something like the
> attached. It's still horribly broken, it actively breaks documented
> notifier behavior, but dammit, if the notifier people don't like
> 'or'ing return values together they should damn well return zero from
> the notifier that doesn't do anything. And returning an error will
> exit out, so..
> 

Instead of this and it's possible bad interactions with other notifiers 
during the -rc cycle, I think it would be better to

 (1)  fix the lowmemorykiller so it doesn't need to use these notifiers at 
      all, which isn't difficult, for 3.4, then

 (2a) change the task handoff to a refcount on task->usage after the final
      put_task_struct() using the notifier and then allow it to be freed 
      after everybody does a put_handoff_task_struct() for 3.5

	or

 (2b) remove the task handoff notifier callchain entirely and just tie it
      directly to oprofile since android won't be using it anymore after 
      (1).

Re: v3.4-rc2 out-of-memory problems (was Re: 3.4-rc1 sticks-and-crashs)

[David Rientjes]

On Mon, 9 Apr 2012, Colin Cross wrote:

> This was a known issue in 2010, in the android tree the use of
> task_handoff_register was dropped one day after it was added and
> replaced with a new task_free_register hook.

Why can't you just do this?  Are you concerned about the possibility of 
depleting all memory reserves?
---
 drivers/staging/android/lowmemorykiller.c |   47 ++++-------------------------
 1 file changed, 6 insertions(+), 41 deletions(-)

diff --git a/drivers/staging/android/lowmemorykiller.c b/drivers/staging/android/lowmemorykiller.c
--- a/drivers/staging/android/lowmemorykiller.c
+++ b/drivers/staging/android/lowmemorykiller.c
@@ -55,7 +55,6 @@ static int lowmem_minfree[6] = {
 };
 static int lowmem_minfree_size = 4;
 
-static struct task_struct *lowmem_deathpending;
 static unsigned long lowmem_deathpending_timeout;
 
 #define lowmem_print(level, x...)			\
@@ -64,24 +63,6 @@ static unsigned long lowmem_deathpending_timeout;
 			printk(x);			\
 	} while (0)
 
-static int
-task_notify_func(struct notifier_block *self, unsigned long val, void *data);
-
-static struct notifier_block task_nb = {
-	.notifier_call	= task_notify_func,
-};
-
-static int
-task_notify_func(struct notifier_block *self, unsigned long val, void *data)
-{
-	struct task_struct *task = data;
-
-	if (task == lowmem_deathpending)
-		lowmem_deathpending = NULL;
-
-	return NOTIFY_OK;
-}
-
 static int lowmem_shrink(struct shrinker *s, struct shrink_control *sc)
 {
 	struct task_struct *tsk;
@@ -97,19 +78,6 @@ static int lowmem_shrink(struct shrinker *s, struct shrink_control *sc)
 	int other_file = global_page_state(NR_FILE_PAGES) -
 						global_page_state(NR_SHMEM);
 
-	/*
-	 * If we already have a death outstanding, then
-	 * bail out right away; indicating to vmscan
-	 * that we have nothing further to offer on
-	 * this pass.
-	 *
-	 * Note: Currently you need CONFIG_PROFILING
-	 * for this to work correctly.
-	 */
-	if (lowmem_deathpending &&
-	    time_before_eq(jiffies, lowmem_deathpending_timeout))
-		return 0;
-
 	if (lowmem_adj_size < array_size)
 		array_size = lowmem_adj_size;
 	if (lowmem_minfree_size < array_size)
@@ -148,6 +116,11 @@ static int lowmem_shrink(struct shrinker *s, struct shrink_control *sc)
 		if (!p)
 			continue;
 
+		if (test_tsk_thread_flag(p, TIF_MEMDIE) &&
+		    time_before_eq(jiffies, lowmem_deathpending_timeout)) {
+			task_unlock(p);
+			return 0;
+		}
 		oom_score_adj = p->signal->oom_score_adj;
 		if (oom_score_adj < min_score_adj) {
 			task_unlock(p);
@@ -174,15 +147,9 @@ static int lowmem_shrink(struct shrinker *s, struct shrink_control *sc)
 		lowmem_print(1, "send sigkill to %d (%s), adj %d, size %d\n",
 			     selected->pid, selected->comm,
 			     selected_oom_score_adj, selected_tasksize);
-		/*
-		 * If CONFIG_PROFILING is off, then we don't want to stall
-		 * the killer by setting lowmem_deathpending.
-		 */
-#ifdef CONFIG_PROFILING
-		lowmem_deathpending = selected;
 		lowmem_deathpending_timeout = jiffies + HZ;
-#endif
 		send_sig(SIGKILL, selected, 0);
+		set_tsk_thread_flag(selected, TIF_MEMDIE);
 		rem -= selected_tasksize;
 	}
 	lowmem_print(4, "lowmem_shrink %lu, %x, return %d\n",
@@ -198,7 +165,6 @@ static struct shrinker lowmem_shrinker = {
 
 static int __init lowmem_init(void)
 {
-	task_handoff_register(&task_nb);
 	register_shrinker(&lowmem_shrinker);
 	return 0;
 }
@@ -206,7 +172,6 @@ static int __init lowmem_init(void)
 static void __exit lowmem_exit(void)
 {
 	unregister_shrinker(&lowmem_shrinker);
-	task_handoff_unregister(&task_nb);
 }
 
 module_param_named(cost, lowmem_shrinker.seeks, int, S_IRUGO | S_IWUSR);

Re: v3.4-rc2 out-of-memory problems (was Re: 3.4-rc1 sticks-and-crashs)

[Linus Torvalds]

On Mon, Apr 9, 2012 at 4:25 PM, David Rientjes <rientjes@google.com> wrote:
>
> You could that if you also turned the check for "ret == NOTIFY_OK" in
> profile_handoff_task() into "ret & NOTIFY_OK" in your patch, otherwise you
> get a double free from __put_task_struct() and oprofile.

Why? NOTIFY_DONE is zero.

I do agree that we *also* could do the "& NOTIFY_OK" and make it
clearer that we're oring bits together. And we could document the
stupid notifier interfaces to do this all, and just make the rules be
*sane* when you have multiple notifiers.

And sane rules would be either:

 - you always return an error return, and notifiers all return either
0 or a negative error number, and we stop on the first error and
return that.

 - you return a bitmask, and we or all bits together (and we can
certainly continue to have a "stop here" bit)

But the current notifier semantics are just insane. The whole "we
return the last return value" is crazy. It's by definition a random
number, since the whole point of notifiers is that there can be
multiple, and they aren't "ordered". So the whole "last return value"
is something I just look at and say: "Whoever designed that is a
f*cking moron".

(And if that happens to be some younger version of me, I am happy that
I got over it. But I'm pretty sure I have never touched that broken
notifier code in my life)

> It works fine if the callbacks are correctly implemented, it's just that
> the task handoff in kernel/profile.c is broken because it assumes only one
> callback will return NOTIFY_OK, meaning it will eventually free, and its
> only checking the return value of the last notifier called to see if
> __put_task_struct() should immediately free.

We can easily document it as "only oprofile is allowed to return
NOTIFY_OK, this notifier is a big mess, don't even *think* about
returning anything but NOTIFY_DONE".

> �(1) �fix the lowmemorykiller so it doesn't need to use these notifiers at
> � � �all, which isn't difficult, for 3.4, then

I do think that that makes sense. Fixing people to not use notifiers
is always a good idea. Why would anybody sane even care about the
process going away anyway? If some lowmemorykiller decides to kill off
a process that no longer exists, kill() should happily return ENOSRCH,
and we're all good

So it could just use a "pid", and test for existence with send_sig()
or lookup_pid() or something.

> �(2a) change the task handoff to a refcount on task->usage after the final
> � � �put_task_struct() using the notifier and then allow it to be freed
> � � �after everybody does a put_handoff_task_struct() for 3.5

The task handoff code runs too late right now. I guess we could easily
move it up, though.

At the same time, the *only* user of that stupid handoff thing is
oprofile, afaik, and if we use a refcount, why the hell doesn't
oprofile just use a refcount to begin with, instead of using that
notifier?: IOW, *both* users of the notifier seem to be just retarded.

So I'd rather just kill the stupid notifier entirely. In the meantime,
making lowmemorykiller just return zero instead just "fixes" it
(assuming we make the notifier semantics for multiple return codes
sane, which they clearly aren't).

Again, almost every notifier user has always been total crap. It's
just a stupid abstraction. "Something happened". "Oh, ok".

                    Linus

[patch] android, lowmemorykiller: remove task handoff notifier

[David Rientjes]

The task handoff notifier leaks task_struct since it never gets freed
after the callback returns NOTIFY_OK, which means it is responsible for
doing so.

It turns out the lowmemorykiller actually doesn't need this notifier at
all.  It's used to prevent unnecessary killing by waiting for a thread to
exit as a result of lowmem_shrink(), however, it's possible to do this in
the same way the kernel oom killer works by setting TIF_MEMDIE and avoid
killing if we're still waiting for it to exit.

The kernel oom killer will already automatically set TIF_MEMDIE for
threads that are attempting to allocate memory that have a fatal signal.
The thread selected by lowmem_shrink() will have such a signal after the
lowmemorykiller sends it a SIGKILL, so this won't result in an
unnecessary use of memory reserves for the thread to exit.

This has the added benefit that we don't have to rely on CONFIG_PROFILING
to prevent needlessly killing tasks.

Reported-by: werner <w.landgraf@ru.ru>
Cc: stable@vger.kernel.org
Signed-off-by: David Rientjes <rientjes@google.com>
---
 drivers/staging/android/lowmemorykiller.c |   48 +++++------------------------
 1 file changed, 7 insertions(+), 41 deletions(-)

diff --git a/drivers/staging/android/lowmemorykiller.c b/drivers/staging/android/lowmemorykiller.c
--- a/drivers/staging/android/lowmemorykiller.c
+++ b/drivers/staging/android/lowmemorykiller.c
@@ -55,7 +55,6 @@ static int lowmem_minfree[6] = {
 };
 static int lowmem_minfree_size = 4;
 
-static struct task_struct *lowmem_deathpending;
 static unsigned long lowmem_deathpending_timeout;
 
 #define lowmem_print(level, x...)			\
@@ -64,24 +63,6 @@ static unsigned long lowmem_deathpending_timeout;
 			printk(x);			\
 	} while (0)
 
-static int
-task_notify_func(struct notifier_block *self, unsigned long val, void *data);
-
-static struct notifier_block task_nb = {
-	.notifier_call	= task_notify_func,
-};
-
-static int
-task_notify_func(struct notifier_block *self, unsigned long val, void *data)
-{
-	struct task_struct *task = data;
-
-	if (task == lowmem_deathpending)
-		lowmem_deathpending = NULL;
-
-	return NOTIFY_OK;
-}
-
 static int lowmem_shrink(struct shrinker *s, struct shrink_control *sc)
 {
 	struct task_struct *tsk;
@@ -97,19 +78,6 @@ static int lowmem_shrink(struct shrinker *s, struct shrink_control *sc)
 	int other_file = global_page_state(NR_FILE_PAGES) -
 						global_page_state(NR_SHMEM);
 
-	/*
-	 * If we already have a death outstanding, then
-	 * bail out right away; indicating to vmscan
-	 * that we have nothing further to offer on
-	 * this pass.
-	 *
-	 * Note: Currently you need CONFIG_PROFILING
-	 * for this to work correctly.
-	 */
-	if (lowmem_deathpending &&
-	    time_before_eq(jiffies, lowmem_deathpending_timeout))
-		return 0;
-
 	if (lowmem_adj_size < array_size)
 		array_size = lowmem_adj_size;
 	if (lowmem_minfree_size < array_size)
@@ -148,6 +116,12 @@ static int lowmem_shrink(struct shrinker *s, struct shrink_control *sc)
 		if (!p)
 			continue;
 
+		if (test_tsk_thread_flag(p, TIF_MEMDIE) &&
+		    time_before_eq(jiffies, lowmem_deathpending_timeout)) {
+			task_unlock(p);
+			rcu_read_unlock();
+			return 0;
+		}
 		oom_score_adj = p->signal->oom_score_adj;
 		if (oom_score_adj < min_score_adj) {
 			task_unlock(p);
@@ -174,15 +148,9 @@ static int lowmem_shrink(struct shrinker *s, struct shrink_control *sc)
 		lowmem_print(1, "send sigkill to %d (%s), adj %d, size %d\n",
 			     selected->pid, selected->comm,
 			     selected_oom_score_adj, selected_tasksize);
-		/*
-		 * If CONFIG_PROFILING is off, then we don't want to stall
-		 * the killer by setting lowmem_deathpending.
-		 */
-#ifdef CONFIG_PROFILING
-		lowmem_deathpending = selected;
 		lowmem_deathpending_timeout = jiffies + HZ;
-#endif
 		send_sig(SIGKILL, selected, 0);
+		set_tsk_thread_flag(selected, TIF_MEMDIE);
 		rem -= selected_tasksize;
 	}
 	lowmem_print(4, "lowmem_shrink %lu, %x, return %d\n",
@@ -198,7 +166,6 @@ static struct shrinker lowmem_shrinker = {
 
 static int __init lowmem_init(void)
 {
-	task_handoff_register(&task_nb);
 	register_shrinker(&lowmem_shrinker);
 	return 0;
 }
@@ -206,7 +173,6 @@ static int __init lowmem_init(void)
 static void __exit lowmem_exit(void)
 {
 	unregister_shrinker(&lowmem_shrinker);
-	task_handoff_unregister(&task_nb);
 }
 
 module_param_named(cost, lowmem_shrinker.seeks, int, S_IRUGO | S_IWUSR);

Re: v3.4-rc2 out-of-memory problems (was Re: 3.4-rc1 sticks-and-crashs)

[David Rientjes]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 1747 bytes --]

On Mon, 9 Apr 2012, Linus Torvalds wrote:

> > You could that if you also turned the check for "ret == NOTIFY_OK" in
> > profile_handoff_task() into "ret & NOTIFY_OK" in your patch, otherwise you
> > get a double free from __put_task_struct() and oprofile.
> 
> Why? NOTIFY_DONE is zero.
> 

Oops, right.

> > �(1) �fix the lowmemorykiller so it doesn't need to use these notifiers at
> > � � �all, which isn't difficult, for 3.4, then
> 
> I do think that that makes sense. Fixing people to not use notifiers
> is always a good idea. Why would anybody sane even care about the
> process going away anyway? If some lowmemorykiller decides to kill off
> a process that no longer exists, kill() should happily return ENOSRCH,
> and we're all good
> 

It's apparently waiting for a killed thread to exit before selecting 
another victim or the one second timeout expires.  (And you only get to 
prevent needless kills if you have CONFIG_PROFILING, otherwise it doesn't 
care.)

> At the same time, the *only* user of that stupid handoff thing is
> oprofile, afaik, and if we use a refcount, why the hell doesn't
> oprofile just use a refcount to begin with, instead of using that
> notifier?: IOW, *both* users of the notifier seem to be just retarded.
> 

Agreed and since the current implementation relies on CONFIG_PROFILING I 
think it's safe to remove the notifier and add a hook only for oprofile so 
it can do free_task() when it wants to.  No refcounting required.

I've already proposed a patch that removes the notifier for 
lowmemorykiller with the added benefit that it doesn't rely on 
CONFIG_PROFILING at all.  If that's merged for 3.4, I'll remove the task 
handoff callchain entirely for 3.5 since oprofile is the only user.

Re: v3.4-rc2 out-of-memory problems (was Re: 3.4-rc1 sticks-and-crashs)

[Colin Cross]

On Mon, Apr 9, 2012 at 4:37 PM, David Rientjes <rientjes@google.com> wrote:
> On Mon, 9 Apr 2012, Colin Cross wrote:
>
>> This was a known issue in 2010, in the android tree the use of
>> task_handoff_register was dropped one day after it was added and
>> replaced with a new task_free_register hook.
>
> Why can't you just do this? �Are you concerned about the possibility of
> depleting all memory reserves?

The point of the lowmem_deathpending patch was to avoid a stutter
where the cpu would spend its time looping through the tasks due to
repeated calls to lowmem_shrink instead of processing the kill signal
to the selected thread.  With this patch, it will still loop through
tasks until it finds the one that was previously killed and then
abort.  It's possible that the improvements Anton made to the task
loop reduce the performance impact enough that this whole mess could
just be dropped (by reverting 1eda516, e5d7965, and 4755b72).

This may have also been impacted by another bug that is on my list of
things to look at: when asked the size of it's "cache", lowmemkiller
returns something on the order of all memory used by userspace, but
under some conditions will refuse to kill any of it due to the current
lowmem_minfree settings.  Due to the large size of the "cache", the
shrinker can call lowmem_shrink hundreds of times for a single
allocation, each time asking to reduce the size of the cache by 128
pages.  The original lowmem_deathpending patch may have been a
misguided "fix" for this bug.

> ---
> �drivers/staging/android/lowmemorykiller.c | � 47 ++++-------------------------
> �1 file changed, 6 insertions(+), 41 deletions(-)
>
> diff --git a/drivers/staging/android/lowmemorykiller.c b/drivers/staging/android/lowmemorykiller.c
> --- a/drivers/staging/android/lowmemorykiller.c
> +++ b/drivers/staging/android/lowmemorykiller.c
> @@ -55,7 +55,6 @@ static int lowmem_minfree[6] = {
> �};
> �static int lowmem_minfree_size = 4;
>
> -static struct task_struct *lowmem_deathpending;
> �static unsigned long lowmem_deathpending_timeout;
>
> �#define lowmem_print(level, x...) � � � � � � � � � � �\
> @@ -64,24 +63,6 @@ static unsigned long lowmem_deathpending_timeout;
> � � � � � � � � � � � �printk(x); � � � � � � � � � � �\
> � � � �} while (0)
>
> -static int
> -task_notify_func(struct notifier_block *self, unsigned long val, void *data);
> -
> -static struct notifier_block task_nb = {
> - � � � .notifier_call �= task_notify_func,
> -};
> -
> -static int
> -task_notify_func(struct notifier_block *self, unsigned long val, void *data)
> -{
> - � � � struct task_struct *task = data;
> -
> - � � � if (task == lowmem_deathpending)
> - � � � � � � � lowmem_deathpending = NULL;
> -
> - � � � return NOTIFY_OK;
> -}
> -
> �static int lowmem_shrink(struct shrinker *s, struct shrink_control *sc)
> �{
> � � � �struct task_struct *tsk;
> @@ -97,19 +78,6 @@ static int lowmem_shrink(struct shrinker *s, struct shrink_control *sc)
> � � � �int other_file = global_page_state(NR_FILE_PAGES) -
> � � � � � � � � � � � � � � � � � � � � � � � �global_page_state(NR_SHMEM);
>
> - � � � /*
> - � � � �* If we already have a death outstanding, then
> - � � � �* bail out right away; indicating to vmscan
> - � � � �* that we have nothing further to offer on
> - � � � �* this pass.
> - � � � �*
> - � � � �* Note: Currently you need CONFIG_PROFILING
> - � � � �* for this to work correctly.
> - � � � �*/
> - � � � if (lowmem_deathpending &&
> - � � � � � time_before_eq(jiffies, lowmem_deathpending_timeout))
> - � � � � � � � return 0;
> -
> � � � �if (lowmem_adj_size < array_size)
> � � � � � � � �array_size = lowmem_adj_size;
> � � � �if (lowmem_minfree_size < array_size)
> @@ -148,6 +116,11 @@ static int lowmem_shrink(struct shrinker *s, struct shrink_control *sc)
> � � � � � � � �if (!p)
> � � � � � � � � � � � �continue;
>
> + � � � � � � � if (test_tsk_thread_flag(p, TIF_MEMDIE) &&
> + � � � � � � � � � time_before_eq(jiffies, lowmem_deathpending_timeout)) {
> + � � � � � � � � � � � task_unlock(p);
> + � � � � � � � � � � � return 0;
> + � � � � � � � }
> � � � � � � � �oom_score_adj = p->signal->oom_score_adj;
> � � � � � � � �if (oom_score_adj < min_score_adj) {
> � � � � � � � � � � � �task_unlock(p);
> @@ -174,15 +147,9 @@ static int lowmem_shrink(struct shrinker *s, struct shrink_control *sc)
> � � � � � � � �lowmem_print(1, "send sigkill to %d (%s), adj %d, size %d\n",
> � � � � � � � � � � � � � � selected->pid, selected->comm,
> � � � � � � � � � � � � � � selected_oom_score_adj, selected_tasksize);
> - � � � � � � � /*
> - � � � � � � � �* If CONFIG_PROFILING is off, then we don't want to stall
> - � � � � � � � �* the killer by setting lowmem_deathpending.
> - � � � � � � � �*/
> -#ifdef CONFIG_PROFILING
> - � � � � � � � lowmem_deathpending = selected;
> � � � � � � � �lowmem_deathpending_timeout = jiffies + HZ;
> -#endif
> � � � � � � � �send_sig(SIGKILL, selected, 0);
> + � � � � � � � set_tsk_thread_flag(selected, TIF_MEMDIE);
> � � � � � � � �rem -= selected_tasksize;
> � � � �}
> � � � �lowmem_print(4, "lowmem_shrink %lu, %x, return %d\n",
> @@ -198,7 +165,6 @@ static struct shrinker lowmem_shrinker = {
>
> �static int __init lowmem_init(void)
> �{
> - � � � task_handoff_register(&task_nb);
> � � � �register_shrinker(&lowmem_shrinker);
> � � � �return 0;
> �}
> @@ -206,7 +172,6 @@ static int __init lowmem_init(void)
> �static void __exit lowmem_exit(void)
> �{
> � � � �unregister_shrinker(&lowmem_shrinker);
> - � � � task_handoff_unregister(&task_nb);
> �}
>
> �module_param_named(cost, lowmem_shrinker.seeks, int, S_IRUGO | S_IWUSR);

Re: v3.4-rc2 out-of-memory problems (was Re: 3.4-rc1 sticks-and-crashs)

[David Rientjes]

On Mon, 9 Apr 2012, Colin Cross wrote:

> The point of the lowmem_deathpending patch was to avoid a stutter
> where the cpu would spend its time looping through the tasks due to
> repeated calls to lowmem_shrink instead of processing the kill signal
> to the selected thread.

What did you do to avoid this without CONFIG_PROFILING?

> With this patch, it will still loop through
> tasks until it finds the one that was previously killed and then
> abort.  It's possible that the improvements Anton made to the task
> loop reduce the performance impact enough that this whole mess could
> just be dropped (by reverting 1eda516, e5d7965, and 4755b72).
> 

I don't understand how calling shrink_slab() from direct reclaim or using 
drop_caches manually taking slightly longer because it has to iterate the 
tasklist to the point of the killed thread will significantly stall the 
thread from exiting.

Much more likely is the killed thread cannot exit because you've killed it 
in a lowmem situation without giving it access to memory reserves so that 
it may exit quickly as my patch does.  That has a higher liklihood of 
stalling the exit than doing for_each_process().

Re: v3.4-rc2 out-of-memory problems (was Re: 3.4-rc1 sticks-and-crashs)

[Colin Cross]

On Mon, Apr 9, 2012 at 5:32 PM, David Rientjes <rientjes@google.com> wrote:
> On Mon, 9 Apr 2012, Colin Cross wrote:
>
>> The point of the lowmem_deathpending patch was to avoid a stutter
>> where the cpu would spend its time looping through the tasks due to
>> repeated calls to lowmem_shrink instead of processing the kill signal
>> to the selected thread.
>
> What did you do to avoid this without CONFIG_PROFILING?
>
>> With this patch, it will still loop through
>> tasks until it finds the one that was previously killed and then
>> abort. �It's possible that the improvements Anton made to the task
>> loop reduce the performance impact enough that this whole mess could
>> just be dropped (by reverting 1eda516, e5d7965, and 4755b72).
>>
>
> I don't understand how calling shrink_slab() from direct reclaim or using
> drop_caches manually taking slightly longer because it has to iterate the
> tasklist to the point of the killed thread will significantly stall the
> thread from exiting.

Before Anton's fix, iterating the tasklist involved taking every task
lock, which probably made it very expensive.  I tried a quick test
where I deliberately limited memory to the point that it was
triggering lowmemorykiller during boot, and it triggered about 5000
times taking on the order of 50ms total for all 5000 calls.  It was
about the same with your patch applied.

> Much more likely is the killed thread cannot exit because you've killed it
> in a lowmem situation without giving it access to memory reserves so that
> it may exit quickly as my patch does. �That has a higher liklihood of
> stalling the exit than doing for_each_process().

Re: [patch] android, lowmemorykiller: remove task handoff notifier

[Colin Cross]

On Mon, Apr 9, 2012 at 4:56 PM, David Rientjes <rientjes@google.com> wrote:
> The task handoff notifier leaks task_struct since it never gets freed
> after the callback returns NOTIFY_OK, which means it is responsible for
> doing so.
>
> It turns out the lowmemorykiller actually doesn't need this notifier at
> all. �It's used to prevent unnecessary killing by waiting for a thread to
> exit as a result of lowmem_shrink(), however, it's possible to do this in
> the same way the kernel oom killer works by setting TIF_MEMDIE and avoid
> killing if we're still waiting for it to exit.
>
> The kernel oom killer will already automatically set TIF_MEMDIE for
> threads that are attempting to allocate memory that have a fatal signal.
> The thread selected by lowmem_shrink() will have such a signal after the
> lowmemorykiller sends it a SIGKILL, so this won't result in an
> unnecessary use of memory reserves for the thread to exit.
>
> This has the added benefit that we don't have to rely on CONFIG_PROFILING
> to prevent needlessly killing tasks.
>
> Reported-by: werner <w.landgraf@ru.ru>
> Cc: stable@vger.kernel.org
> Signed-off-by: David Rientjes <rientjes@google.com>
> ---
> �drivers/staging/android/lowmemorykiller.c | � 48 +++++------------------------
> �1 file changed, 7 insertions(+), 41 deletions(-)
>

I did a quick test to measure the difference in time spent inside
lowmem_shrink with and without this patch, and they were about the
same.  So,
Acked-by: Colin Cross <ccross@android.com>

Re: v3.4-rc2 out-of-memory problems (was Re: 3.4-rc1 sticks-and-crashs)

[David Rientjes]

On Mon, 9 Apr 2012, Colin Cross wrote:

> Before Anton's fix, iterating the tasklist involved taking every task
> lock, which probably made it very expensive.

I'm not sure of the fix you're referring to, but it's not in 3.4-rc2 
because lowmem_shrink() still does find_lock_task_mm() for every user 
process on the system, which is necessary to safely do get_mm_rss().

Re: v3.4-rc2 out-of-memory problems (was Re: 3.4-rc1 sticks-and-crashs)

[Colin Cross]

On Mon, Apr 9, 2012 at 6:33 PM, David Rientjes <rientjes@google.com> wrote:
> On Mon, 9 Apr 2012, Colin Cross wrote:
>
>> Before Anton's fix, iterating the tasklist involved taking every task
>> lock, which probably made it very expensive.
>
> I'm not sure of the fix you're referring to, but it's not in 3.4-rc2
> because lowmem_shrink() still does find_lock_task_mm() for every user
> process on the system, which is necessary to safely do get_mm_rss().

I confused "staging: android/lowmemorykiller: Don't grab
tasklist_lock" and "staging: android/lowmemorykiller: Better mm
handling".  You're right, it still grabs the task lock.

Re: v3.4-rc2 out-of-memory problems (was Re: 3.4-rc1 sticks-and-crashs)

[Rik van Riel]

On 04/09/2012 09:52 PM, werner wrote:
> At least until now, and also tested hard by starting and stopping
> several memory-consuming operations, I'm happy to can inform that the
> computer didnt yet crash again, and that also slownessnesses what I
> observed under 3.3 (but without crashs) don't occur

That could be due to a few VM patches which I wrote, that
went in through -mm.

I am very interested in whether people do find a way to
break the VM with those patches, in ways that used to work
before.

If you find any, please let me know so I can fix them
before the 3.4 kernel comes out.

If everything you try works better than before, I'm not
going to complain about good news :)

-- 
All rights reversed

Re: v3.4-rc2 out-of-memory problems (was Re: 3.4-rc1 sticks-and-crashs)

[werner]

At least until now, and also tested hard by starting and 
stopping several memory-consuming operations,   I'm happy 
to can inform that the computer didnt yet crash again, and 
that also slownessnesses what I observed under 3.3 (but 
without crashs) don't occure and kmemleak (or a similar 
name) what sometimes started and I had to shut down under 
3.3 for recover normal speed, I dont see longer,  so that 
the the patch suggested by D.R. and L.T. cured the crashs 
(at least phenomenologically).    For be sure, however, 
it's still too early, only if the computer dont crash 
until tomorrow.

If there is found an agreement about something else what's 
senseful to test out, I can do that.

wl
---
Professional hosting for everyone - http://www.host.ru

Re: v3.4-rc2 out-of-memory problems (was Re: 3.4-rc1 sticks-and-crashs)

[werner]

Rik van Riel :
To avoid misunderstandings, that improvement is WITH / 
INCLUDING the patch suggested by D.R and L.T., namely: 
 put + free_task(task); after line 78 in subroutine 
lowmemorykiller.c   .  At least phenomenologically, it 
looks that it resolved the problem
I dont understanding so much about programming and about 
the problem (and "android" is neither running on my own 
computer nor on my coffee-machine, but I enable everything 
just because I don't know what hardware have the people 
which use my distro and compiled kernels) so that I don't 
understand all details of the discusion,  but if people 
come to an agreement what I could test further, one can 
mail it me and explain it me slowly, then I test it out.
wl

==========

On Mon, 09 Apr 2012 21:51:58 -0400
  Rik van Riel <riel@redhat.com> wrote:
> On 04/09/2012 09:52 PM, werner wrote:
>> At least until now, and also tested hard by starting and 
>>stopping
>> several memory-consuming operations, I'm happy to can 
>>inform that the
>> computer didnt yet crash again, and that also 
>>slownessnesses what I
>> observed under 3.3 (but without crashs) don't occur
> 
> That could be due to a few VM patches which I wrote, 
>that
> went in through -mm.
> 
> I am very interested in whether people do find a way to
> break the VM with those patches, in ways that used to 
>work
> before.
> 
> If you find any, please let me know so I can fix them
> before the 3.4 kernel comes out.
> 
> If everything you try works better than before, I'm not
> going to complain about good news :)
> 
> -- 
> All rights reversed
> 
> 

"werner" <w.landgraf@ru.ru>
---
Professional hosting for everyone - http://www.host.ru

Re: v3.4-rc2 out-of-memory problems (was Re: 3.4-rc1 sticks-and-crashs)

[werner]

After first I tested some hours the 1st,one-line patch by 
D.R., now is ready compiled and started to be tested his 
2nd patch, below.   I see he has it already comitted; it 
would have been better first wait to test it.

The loop suggested below, with this 2nd patch, gives 1560 
kB , compared with 1632 kb after the 1st patch, and 1432 
kB with kernel 3.3  .    On the other hand, 3.3 has 
clearly the same problem (even if not crashing, it's 
becoming often very slow, and then  there's running 
kmemleak, what I have to kill for return to the normal 
speed), but according this 'test' it would be good, so 
that it's questionable if this test is reliable.

As already reported, the 1st patch cured the problem at 
least subjectively.

To see if this 2nd patch is good, I have to wait now some 
hours and observe if the computer becomes slow or even 
crashs


wl


=================================================

On Mon, 9 Apr 2012 17:11:45 -0700 (PDT)
  David Rientjes <rientjes@google.com> wrote:
> On Mon, 9 Apr 2012, werner wrote:
> 
>> I continue now testing your first patch a few hours, if 
>>it's good or not.
>> Then, I can make another patch.  So you have still time 
>>to think and put all
>> together
>> what you want to be tested, and mail me that. Also 
>>explain me, if you want
>> other
>> patchs ADDITIONALLY or INSTEAD your first patch -- the 
>>best would be, to send
>> me
>> always accumulating patchs including everything together 
>>to be applied over
>> the
>> 'virgin' 3.4-rcX kernel.
>> For your information, I dont download the whole git, I 
>>download all 3.X.Y-rcZ
>> , and I
>> recompile everything again (patched), instead of 
>>compiling only the patched
>> subroutines.
>> 
> 
> Ok, when you want to test the latest patch, try this:
> 
> - revert back to the vanilla 3.4-rc2 kernel,
> 
> - boot and do this on the command line:
> 
> 	for i in $(seq 1 10000); do sleep 0 & done
> 	grep KernelStack /proc/meminfo
> 
> - record that number,
> 
> - apply the patch at https://lkml.org/lkml/2012/4/9/428,
> 
> - boot and do the same two command lines,
> 
> - compare the number with the previous number from the 
>first boot.
> 
> The number should be much lower after the patch is 
>applied.
> 
> Thanks!
> 
> 

"werner" <w.landgraf@ru.ru>
---
Professional hosting for everyone - http://www.host.ru

Re: v3.4-rc2 out-of-memory problems (was Re: 3.4-rc1 sticks-and-crashs)

[werner]

After first I tested some hours the 1st,one-line patch by 
D.R., now is ready compiled and started to be tested his 
2nd patch, below.   I see he has it already comitted; it 
would have been better first wait to test it.

The loop suggested below, with this 2nd patch, gives 1560 
kB , compared with 1632 kb after the 1st patch, and 1432 
kB with kernel 3.3  .    On the other hand, 3.3 has 
clearly the same problem (even if not crashing, it's 
becoming often very slow, and then  there's running 
kmemleak, what I have to kill for return to the normal 
speed), but according this 'test' it would be good, so 
that it's questionable if this test is reliable.

As already reported, the 1st patch cured the problem at 
least subjectively.

To see if this 2nd patch is good, I have to wait now some 
hours and observe if the computer becomes slow or even 
crashs


wl


=================================================

On Mon, 9 Apr 2012 17:11:45 -0700 (PDT)
  David Rientjes <rientjes@google.com> wrote:
> On Mon, 9 Apr 2012, werner wrote:
> 
>> I continue now testing your first patch a few hours, if 
>>it's good or not.
>> Then, I can make another patch.  So you have still time 
>>to think and put all
>> together
>> what you want to be tested, and mail me that. Also 
>>explain me, if you want
>> other
>> patchs ADDITIONALLY or INSTEAD your first patch -- the 
>>best would be, to send
>> me
>> always accumulating patchs including everything together 
>>to be applied over
>> the
>> 'virgin' 3.4-rcX kernel.
>> For your information, I dont download the whole git, I 
>>download all 3.X.Y-rcZ
>> , and I
>> recompile everything again (patched), instead of 
>>compiling only the patched
>> subroutines.
>> 
> 
> Ok, when you want to test the latest patch, try this:
> 
> - revert back to the vanilla 3.4-rc2 kernel,
> 
> - boot and do this on the command line:
> 
> 	for i in $(seq 1 10000); do sleep 0 & done
> 	grep KernelStack /proc/meminfo
> 
> - record that number,
> 
> - apply the patch at https://lkml.org/lkml/2012/4/9/428,
> 
> - boot and do the same two command lines,
> 
> - compare the number with the previous number from the 
>first boot.
> 
> The number should be much lower after the patch is 
>applied.
> 
> Thanks!
> 
> 

"werner" <w.landgraf@ru.ru>
---
Professional hosting for everyone - http://www.host.ru

Re: v3.4-rc2 out-of-memory problems (was Re: 3.4-rc1 sticks-and-crashs)

[werner]

That looks good;   until now no crash, no slowdown, no 
syslog message.  Thus I suppose that also the 2nd patch by 
D.R. (which was already comitted by him)  is OK and 
resolve the problem.

wl
---
Professional hosting for everyone - http://www.host.ru

Re: v3.4-rc2 out-of-memory problems (was Re: 3.4-rc1 sticks-and-crashs)

[werner]

Perhaps someone could check this

Ocasionally it still happens that the computer slows down 
very, during appr. 1 minute  (but it dont crash, nor is 
nothing visible in syslog).

I think this wasn't before approx. the 3.2 kernel, at 
least I didn't perceive it, but this happens at 3.3 and 
3.4-rc2-inclusive all patchs.

Below, ps alx   and   top   i printed in a file at an 
ocasion as this happened
I observed that then always kmemleak is almost at the 
beginning of top


wl



================= ps alx
F   UID   PID  PPID PRI  NI    VSZ   RSS WCHAN  STAT TTY 
       TIME COMMAND
4     0     1     0  20   0    824   320 poll_s Ss   ? 
         0:35 init [4]
1     0     2     0  20   0      0     0 kthrea S    ? 
         0:01 [kthreadd]
1     0     3     2  20   0      0     0 run_ks S    ? 
         6:12 [ksoftirqd/0]
1     0     6     2 -100  -      0     0 cpu_st S    ? 
         0:00 [migration/0]
1     0     7     2 -100  -      0     0 watchd S    ? 
         0:05 [watchdog/0]
1     0     8     2   0 -20      0     0 rescue S<   ? 
         0:00 [khelper]
5     0     9     2  20   0      0     0 devtmp S    ? 
         0:00 [kdevtmpfs]
1     0    10     2   0 -20      0     0 rescue S<   ? 
         0:00 [netns]
1     0  1086     2  20   0      0     0 bdi_sy S    ? 
         0:00 [sync_supers]
1     0  1088     2  20   0      0     0 bdi_fo S    ? 
         0:00 [bdi-default]
1     0  1089     2   0 -20      0     0 rescue S<   ? 
         0:00 [kintegrityd]
1     0  1090     2   0 -20      0     0 rescue S<   ? 
         0:00 [kblockd]
5     0  1108     2 -51   -      0     0 irq_wa S    ? 
         0:00 [irq/9-acpi]
1     0  1236     2   0 -20      0     0 rescue S<   ? 
         0:00 [tifm]
1     0  1268     2   0 -20      0     0 rescue S<   ? 
         0:00 [ata_sff]
1     0  1280     2  20   0      0     0 hub_th S    ? 
         0:00 [khubd]
1     0  1295     2   0 -20      0     0 rescue S<   ? 
         0:00 [md]
1     0  1420     2   0 -20      0     0 rescue S<   ? 
         0:00 [rpciod]
1     0  1450     2  20   0      0     0 watchd S    ? 
         0:00 [khungtaskd]
1     0  1456     2  20   0      0     0 kswapd S    ? 
         0:02 [kswapd0]
1     0  1457     2  25   5      0     0 ksm_sc SN   ? 
         0:00 [ksmd]
1     0  1532     2  39  19      0     0 khugep SN   ? 
         0:06 [khugepaged]
1     0  1533     2  20   0      0     0 fsnoti S    ? 
         0:00 [fsnotify_mark]
1     0  1572     2  20   0      0     0 ecrypt S    ? 
         0:00 [ecryptfs-kthrea]
1     0  1575     2   0 -20      0     0 rescue S<   ? 
         0:00 [nfsiod]
1     0  1580     2   0 -20      0     0 rescue S<   ? 
         0:00 [cifsiod]
1     0  1606     2  20   0      0     0 jfsIOW S    ? 
         0:00 [jfsIO]
1     0  1607     2  20   0      0     0 jfs_la S    ? 
         0:00 [jfsCommit]
1     0  1608     2  20   0      0     0 jfs_sy S    ? 
         0:00 [jfsSync]
1     0  1615     2   0 -20      0     0 rescue S<   ? 
         0:00 [xfsalloc]
1     0  1616     2   0 -20      0     0 rescue S<   ? 
         0:00 [xfs_mru_cache]
1     0  1617     2   0 -20      0     0 rescue S<   ? 
         0:00 [xfslogd]
1     0  1624     2   0 -20      0     0 rescue S<   ? 
         0:00 [ocfs2_wq]
1     0  1627     2   0 -20      0     0 rescue S<   ? 
         0:00 [user_dlm]
1     0  1633     2   0 -20      0     0 rescue S<   ? 
         0:00 [glock_workqueue]
1     0  1634     2   0 -20      0     0 rescue S<   ? 
         0:00 [delete_workqueu]
1     0  1638     2   0 -20      0     0 rescue S<   ? 
         0:00 [gfs_recovery]
1     0  1642     2   0 -20      0     0 rescue S<   ? 
         0:00 [crypto]
1     0  1780     2  20   0      0     0 worker S    ? 
         0:00 [kworker/u:2]
1     0  2408     2  20   0      0     0 scan_t S    ? 
         0:00 [cciss_scan]
1     0  2542     2   0 -20      0     0 rescue S<   ? 
         0:00 [iscsi_eh]
1     0  2551     2   0 -20      0     0 rescue S<   ? 
         0:00 [kmpath_rdacd]
1     0  2552     2   0 -20      0     0 rescue S<   ? 
         0:00 [fc_exch_workque]
1     0  2553     2   0 -20      0     0 rescue S<   ? 
         0:00 [fc_rport_eq]
1     0  2554     2   0 -20      0     0 fcoe_p S<   ? 
         0:00 [fcoethread/0]
1     0  2556     2   0 -20      0     0 rescue S<   ? 
         0:00 [fnic_event_wq]
1     0  2558     2   0 -20      0     0 bnx2fc S<   ? 
         0:00 [bnx2fc_l2_threa]
1     0  2559     2   0 -20      0     0 bnx2fc S<   ? 
         0:00 [bnx2fc_thread/0]
1     0  2636     2   0 -20      0     0 bnx2i_ S<   ? 
         0:00 [bnx2i_thread/0]
1     0  2675     2  20   0      0     0 scsi_e S    ? 
         0:00 [scsi_eh_2]
1     0  2678     2  20   0      0     0 scsi_e S    ? 
         0:25 [scsi_eh_3]
1     0  2688     2  20   0      0     0 scsi_e S    ? 
         0:00 [scsi_eh_4]
1     0  2691     2  20   0      0     0 scsi_e S    ? 
         0:00 [scsi_eh_5]
1     0  2756     2   0 -20      0     0 rescue S<   ? 
         0:00 [smflush]
1     0  2771     2  20   0      0     0 mtd_bl S    ? 
         0:00 [mtdblock0]
1     0  2784     2  20   0      0     0 mtd_bl S    ? 
         0:00 [mtdblock1]
1     0  2793     2  20   0      0     0 mtd_bl S    ? 
         0:00 [mtdblock2]
1     0  2863     2   0 -20      0     0 rescue S<   ? 
         0:00 [cnic_wq]
1     0  2874     2   0 -20      0     0 rescue S<   ? 
         0:00 [sfc_vfdi]
1     0  2875     2   0 -20      0     0 rescue S<   ? 
         0:00 [sfc_reset]
1     0  2890     2   0 -20      0     0 rescue S<   ? 
         0:00 [exec-osm]
1     0  2897     2   0 -20      0     0 rescue S<   ? 
         0:00 [block-osm]
1     0  2905     2   0 -20      0     0 rescue S<   ? 
         0:00 [firewire]
1     0  2982     2   0 -20      0     0 rescue S<   ? 
         0:00 [wusbd]
1     0  3045     2   0 -20      0     0 rescue S<   ? 
         0:00 [kpsmoused]
1     0  3171     2   0 -20      0     0 rescue S<   ? 
         0:00 [dm_bufio_cache]
1     0  3172     2   0 -20      0     0 rescue S<   ? 
         0:00 [kmpathd]
1     0  3173     2   0 -20      0     0 rescue S<   ? 
         0:00 [kmpath_handlerd]
1     0  3176     2   0 -20      0     0 rescue S<   ? 
         0:00 [edac-poller]
1     0  3188     2   0 -20      0     0 rescue S<   ? 
         0:00 [kvub300c]
1     0  3189     2   0 -20      0     0 rescue S<   ? 
         0:00 [kvub300p]
1     0  3190     2   0 -20      0     0 rescue S<   ? 
         0:00 [kvub300d]
1     0  3193     2   0 -20      0     0 rescue S<   ? 
         0:00 [kmemstick]
1     0  3260     2   0 -20      0     0 rescue S<   ? 
         0:00 [binder]
5     2  3416 11795  20   0  55952 12448 inet_c S    ? 
         0:00 /usr/local/apache2/bin/httpd -k restart
1     0  5534     2  30  10      0     0 -      RN   ? 
        90:25 [kmemleak]
1     0  5536     2   0 -20      0     0 rescue S<   ? 
         0:00 [kafs_vlupdated]
1     0  5537     2   0 -20      0     0 rescue S<   ? 
         0:00 [kafs_callbackd]
1     0  5538     2   0 -20      0     0 rescue S<   ? 
         0:00 [kafsd]
1     0  5541     2   0 -20      0     0 rescue S<   ? 
         0:00 [deferwq]
1     0  5546     2   0 -20      0     0 rescue S<   ? 
         0:00 [devfreq_wq]
1     0  5549     2  20   0      0     0 kjourn S    ? 
         0:54 [kjournald]
5     0  5577     1  16  -4   3124  1752 poll_s S<s  ? 
         0:04 /sbin/udevd --daemon
5     2  5950 11795  20   0  55908 12208 inet_c S    ? 
         0:07 /usr/local/apache2/bin/httpd -k restart
1     0  8011     2   0 -20      0     0 rescue S<   ? 
         0:00 [hd-audio0]
1     0  9620     2  20   0      0     0 bdi_wr S    ? 
         0:16 [flush-8:0]
1     0  9686     1  20   0   1928   664 poll_s Ss   ? 
         0:00 /usr/sbin/syslogd
5     0  9690     1  20   0   1880   428 syslog Ss   ? 
         0:00 /usr/sbin/klogd -c 3 -x
5     0  9904     1  20   0   1920   568 poll_s Ss   ? 
         0:00 /usr/sbin/inetd
1     0  9911     1  20   0   1876   560 poll_s Ss   ? 
         0:00 /usr/sbin/acpid
5    81  9922     1  20   0   2664   860 poll_s Ss   ? 
         0:00 /usr/bin/dbus-daemon --system
5    82  9927     1  20   0  15432  4544 poll_s Ssl  ? 
         0:01 /usr/sbin/hald --daemon=yes
0     0  9928  9927  20   0  11872  1256 poll_s Sl   ? 
         0:00 hald-runner
0     0  9950  9928  20   0   3800  1256 poll_s S    ? 
         0:02 hald-addon-input: Listening on 
/dev/input/event4 /dev/input/event0 /dev/input/event1 
/dev/input/event10 /dev/input/event2
0     0  9958  9928  20   0   3812  1076 poll_s S    ? 
         0:00 /usr/libexec/hald-addon-cpufreq
4    82  9959  9928  20   0   2572  1100 unix_s S    ? 
         0:00 hald-addon-acpi: listening on acpid socket 
/var/run/acpid.socket
0     0  9961  9928  20   0   3800  1076 poll_s S    ? 
         1:46 hald-addon-storage: polling /dev/hda (every 
2 sec)
0     0  9970  9928  20   0   3800  1076 poll_s S    ? 
         0:57 hald-addon-storage: polling /dev/sr0 (every 
2 sec)
5     0  9987     1  20   0   2536   932 poll_s Ss   ? 
         0:00 /usr/sbin/hcid -f /etc/bluetooth/hcid.conf
5     0  9991     1  20   0   1976   384 poll_s Ss   ? 
         0:00 /usr/sbin/sdpd
1     0 10003     1  20   0   2132   388 poll_s Ss   ? 
         0:00 /usr/bin/hidd --server
5     0 10008     2  10 -10      0     0 rfcomm S<   ? 
         0:00 [krfcommd]
4     0 10023     1  20   0   5832  2568 ep_pol Ss   ? 
         0:00 /usr/sbin/cupsd -C /etc/cups/cupsd.conf
1     0 10070     1  20   0   2096   712 hrtime S    ? 
         0:01 /usr/sbin/crond -l10
1     2 10072     1  20   0   2100   332 hrtime Ss   ? 
         0:00 /usr/sbin/atd -b 15 -l 1
1     0 10122     1  20   0   1864   304 hrtime Ss   ? 
         0:08 rkdet 1451
4     0 10131     1  20   0   3456  1960 wait   Ss   tty1 
      0:00 -bash
4     0 10132     1  20   0   3448  1952 wait   Ss   tty2 
      0:00 -bash
0     0 10133     1  20   0   1880   568 n_tty_ Ss+  tty3 
      0:00 /sbin/agetty 38400 tty3 linux
0     0 10134     1  20   0   1880   564 n_tty_ Ss+  tty4 
      0:00 /sbin/agetty 38400 tty4 linux
0     0 10135     1  20   0   1880   564 n_tty_ Ss+  tty5 
      0:00 /sbin/agetty 38400 tty5 linux
0     0 10136     1  20   0   1880   572 n_tty_ Ss+  tty6 
      0:00 /sbin/agetty 38400 tty6 linux
4     0 10137     1  20   0   3408   868 poll_s Ss   ? 
         0:00 /opt/kde/bin/kdm -nodaemon
4     0 10153 10137  20   0  54108 47384 poll_s Ss+  tty7 
     42:16 /usr/bin/X -br -nolisten tcp :0 vt7 -auth 
/var/run/xauth/A:0-EArqAj
1     0 10157 10137  20   0   3728  1196 wait   S    ? 
         0:00 -:0
4     0 10165 10157  20   0   3060  1500 wait   Ss   ? 
         0:00 /bin/sh /usr/bin/startkde
0     0 10183 10165  20   0  12404  1640 hrtime Sl   ? 
         8:54 /usr/bin/tpb
1     0 10219     1  20   0   1856    72 pipe_w S    ? 
         0:00 start_kdeinit --new-startup +kcminit_startup
1     0 10220     1  20   0  25160  7996 poll_s Ss   ? 
         0:01 kdeinit Running...
1     0 10223     1  20   0  24352  3216 poll_s S    ? 
         0:02 dcopserver [kdeinit] --nosid
5     0 10225 10220  20   0  29040  8412 poll_s S    ? 
         0:01 klauncher [kdeinit] --new-startup
1     0 10227     1  20   0  57396 32724 poll_s S    ? 
       256:52 kded [kdeinit] --new-startup
0     0 10234 10165  20   0   1860   272 hrtime S    ? 
         0:08 kwrapper ksmserver
1     0 10236     1  20   0  38976 22408 poll_s S    ? 
         0:00 ksmserver [kdeinit]
1     0 10237 10220  20   0  51880 27704 poll_s S    ? 
         0:42 kwin [kdeinit]
1     0 10239     1  20   0  48132 32020 poll_s S    ? 
         1:16 kdesktop [kdeinit]
1     0 10241     1  20   0  67968 44644 poll_s S    ? 
         1:25 kicker [kdeinit]
1     0 10248     1  20   0  38984 22168 poll_s S    ? 
         0:02 kaccess [kdeinit]
4     0 10256 10220  20   0 110796 43104 -      RLl  ? 
        31:15 artsd -F 9 -S 4096 -a alsa -b 8 -s 333 -m 
artsmessage -c drkonqi -l 3 -f
0     0 10260 10220  20   0  22420 10552 poll_s S    ? 
         0:00 nm-applet --sm-disable
1     0 10263     1  20   0  50312 25616 poll_s S    ? 
         0:10 knotify [kdeinit]
1     0 10271     1  20   0  26708  8832 poll_s S    ? 
         0:00 kalarmd --autostart
1     0 10275     1  20   0   3492   612 poll_s S    ? 
         0:00 dbus-launch --autolaunch 
9b9f86fbd8163bb39486a000475487ec --binary-syntax 
--close-stderr
1     0 10276     1  20   0   2520   748 poll_s Ss   ? 
         0:00 /usr/bin/dbus-daemon --fork --print-pid 5 
--print-address 7 --session
0     0 10278     1  20   0  31868 20800 poll_s S    ? 
         0:00 /usr/libexec/xfce4-notifyd
0     0 10280     1  20   0   4008  1788 poll_s S    ? 
         0:00 /usr/libexec/xfconfd
0     0 10286     1  20   0   5196  2824 poll_s S    ? 
         0:16 /usr/libexec/gconfd-2
5     2 10564 11795  20   0  56168 12740 inet_c S    ? 
         0:08 /usr/local/apache2/bin/httpd -k restart
5     0 10767     1  20   0  41376 12284 rt_sig Ssl  ? 
         0:00 named -d 5
0     0 11707     1  20   0   3072  1508 wait   S    ? 
         0:00 /bin/sh /usr/bin/mysqld_safe --user=root
4     0 11755 11707  20   0 103672 21484 poll_s Sl   ? 
         0:59 /usr/sbin/mysqld --basedir=/ --datadir=//var 
--user=root --pid-file=//var/werner.pid 
--log-error=//var/werner.err
5     0 11785     1  20   0   6904  2112 poll_s Ss   ? 
         0:12 sendmail: accepting connections
1     0 11786     1  20   0   6292  2768 poll_s S    ? 
         0:09 /usr/local/src/Unreal3.2/src/ircd
1     0 11795     1  20   0  53056  8876 poll_s Ss   ? 
         0:06 /usr/local/apache2/bin/httpd -k restart
5     2 11802 11795  20   0  55716 12324 inet_c S    ? 
         0:11 /usr/local/apache2/bin/httpd -k restart
5     0 11812     1  20   0   5900   980 poll_s Ss   ? 
         0:00 pure-ftpd (SERVER)
5     0 11825     1  20   0   3428   640 poll_s Ss   ? 
         0:00 rsync -v --daemon
1     0 11835     1  20   0   7608  5348 poll_s Ss   ? 
         0:05 /usr/local/bin/perl 
/usr/local/src/webmin/miniserv.pl 
/etc/webmin/miniserv.conf
1     0 11843     1  20   0   7216  5136 poll_s Ss   ? 
         0:06 /usr/local/bin/perl 
/usr/local/src/usermin/miniserv.pl 
/etc/usermin/miniserv.conf
5     2 17040 11795  20   0  55924 12480 inet_c S    ? 
         0:06 /usr/local/apache2/bin/httpd -k restart
1     0 20831     2  20   0      0     0 worker S    ? 
         0:00 [kworker/0:0]
1     0 21950 10220  20   0  26424  8332 poll_s S    ? 
         0:00 kio_file [kdeinit] file 
/tmp/ksocket-root/klauncherEGAF1a.
1     0 22463     2  20   0      0     0 worker S    ? 
         0:00 [kworker/0:2]
0     0 22802 10131  20   0   2624  1132 poll_s S+   tty1 
      0:05 top
4     0 24935 10220  20   0   6088  2676 poll_s S    ? 
         0:00 mrxvt
0     0 24936 24935  20   0   3396  1776 wait   Ss   pts/2 
     0:00 bash
0     0 24974 24936  20   0   2620  1196 poll_s S+   pts/2 
     0:02 top
1     0 25574     2  20   0      0     0 worker S    ? 
         0:00 [kworker/0:1]
0     0 26483 10132  20   0   2528   584 -      R+   tty2 
      0:00 ps alx
1     0 29868     2  20   0      0     0 worker S    ? 
         0:00 [kworker/u:0]
5     0 30236 10220  20   0  54416 38756 poll_s S    ? 
         0:04 konqueror [kdeinit] /
5     2 30420 11795  20   0  55664 12216 inet_c S    ? 
         0:05 /usr/local/apache2/bin/httpd -k restart
5     2 30590 11795  20   0  55672 12160 inet_c S    ? 
         0:04 /usr/local/apache2/bin/httpd -k restart
5     2 30596 11795  20   0  56228 12584 inet_c S    ? 
         0:05 /usr/local/apache2/bin/httpd -k restart
5     2 30597 11795  20   0  56404 12960 inet_c S    ? 
         0:05 /usr/local/apache2/bin/httpd -k restart
1     0 31140     1  20   0  41172 25692 poll_s S    ? 
         0:01 kio_uiserver [kdeinit]
5     2 31540 11795  20   0  55372 11884 inet_c S    ? 
         0:06 /usr/local/apache2/bin/httpd -k restart
4     0 31801 10220  20   0 591224 345664 -     Rl   ? 
       158:02 /usr/lib/firefox/firefox-bin

================= top

^[[H^[[J^[[0;10mtop - 16:20:01 up 2 days, 16:51,  5 users, 
 load average: 2.18, 1.54, 1.11^[[0;10m^[[39;49m^[[K
Tasks:^[[0;10m^[[39;49m^[[0;10m 168 
^[[0;10m^[[39;49mtotal,^[[0;10m^[[39;49m^[[0;10m   3 
^[[0;10m^[[39;49mrunning,^[[0;10m^[[39;49m^[[0;10m 165 
^[[0;10m^[[39;49msleeping,^[[0;10m^[[39;49m^[[0;10m   0 
^[[0;10m^[[39;49mstopped,^[[0;10m^[[39;49m^[[0;10m   0 
^[[0;10m^[[39;49mzombie^[[0;10m^[[39;49m^[[K
Cpu(s):^[[0;10m^[[39;49m^[[0;10m 
 9.7%^[[0;10m^[[39;49mus,^[[0;10m^[[39;49m^[[0;10m 
12.5%^[[0;10m^[[39;49msy,^[[0;10m^[[39;49m^[[0;10m 
 0.0%^[[0;10m^[[39;49mni,^[[0;10m^[[39;49m^[[0;10m 
76.7%^[[0;10m^[[39;49mid,^[[0;10m^[[39;49m^[[0;10m 
 1.0%^[[0;10m^[[39;49mwa,^[[0;10m^[[39;49m^[[0;10m 
 0.0%^[[0;10m^[[39;49mhi,^[[0;10m^[[39;49m^[[0;10m 
 0.1%^[[0;10m^[[39;49msi,^[[0;10m^[[39;49m^[[0;10m 
 0.0%^[[0;10m^[[39;49mst^[[0;10m^[[39;49m^[[K
Mem: ^[[0;10m^[[39;49m^[[0;10m  3088984k 
^[[0;10m^[[39;49mtotal,^[[0;10m^[[39;49m^[[0;10m  2861284k 
^[[0;10m^[[39;49mused,^[[0;10m^[[39;49m^[[0;10m   227700k 
^[[0;10m^[[39;49mfree,^[[0;10m^[[39;49m^[[0;10m   286872k 
^[[0;10m^[[39;49mbuffers^[[0;10m^[[39;49m^[[K
Swap:^[[0;10m^[[39;49m^[[0;10m        0k 
^[[0;10m^[[39;49mtotal,^[[0;10m^[[39;49m^[[0;10m        0k 
^[[0;10m^[[39;49mused,^[[0;10m^[[39;49m^[[0;10m        0k 
^[[0;10m^[[39;49mfree,^[[0;10m^[[39;49m^[[0;10m  1780732k 
^[[0;10m^[[39;49mcached^[[0;10m^[[39;49m^[[K
^[[6;1H
^[[7m  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM 
   TIME+  COMMAND 
                                                            
                                                            
                     ^[[0;10m^[[39;49m^[[K
^[[0;10m^[[0;10m31801 root      20   0  577m 337m  48m R 
66.9 11.2 158:27.70 firefox-bin 
                                                            
                                                            
                 ^[[0;10m^[[39;49m
^[[0;10m10227 root      20   0 57396  31m  27m S 12.7  1.1 
256:54.85 kded 
                                                            
                                                            
                        ^[[0;10m^[[39;49m
^[[0;10m^[[0;10m 5534 root      30  10     0    0    0 R 
 9.0  0.0  90:28.34 kmemleak 
                                                            
                                                            
                    ^[[0;10m^[[39;49m
^[[0;10m10256 root      20   0 94484  42m 7548 S  3.6  1.4 
 31:16.69 artsd 
                                                            
                                                            
                       ^[[0;10m^[[39;49m
^[[0;10m^[[0;10m26642 root      20   0  2620 1116  812 R 
 1.8  0.0   0:00.03 top 
                                                            
                                                            
                         ^[[0;10m^[[39;49m
^[[0;10m    1 root      20   0   824  320  280 S  0.0  0.0 
  0:35.88 init 
                                                            
                                                            
                        ^[[0;10m^[[39;49m
^[[0;10m    2 root      20   0     0    0    0 S  0.0  0.0 
  0:01.19 kthreadd 
                                                            
                                                            
                    ^[[0;10m^[[39;49m
^[[0;10m    3 root      20   0     0    0    0 S  0.0  0.0 
  6:12.80 ksoftirqd/0 
                                                            
                                                            
                 ^[[0;10m^[[39;49m
^[[0;10m    6 root      RT   0     0    0    0 S  0.0  0.0 
  0:00.00 migration/0 
                                                            
                                                            
                 ^[[0;10m^[[39;49m
^[[0;10m    7 root      RT   0     0    0    0 S  0.0  0.0 
  0:05.35 watchdog/0 
                                                            
                                                            
                  ^[[0;10m^[[39;49m
^[[0;10m    8 root       0 -20     0    0    0 S  0.0  0.0 
  0:00.00 khelper 
                                                            
                                                            
                     ^[[0;10m^[[39;49m
^[[0;10m    9 root      20   0     0    0    0 S  0.0  0.0 
  0:00.03 kdevtmpfs 
                                                            
                                                            
                   ^[[0;10m^[[39;49m
^[[0;10m   10 root       0 -20     0    0    0 S  0.0  0.0 
  0:00.00 netns 
                                                            
                                                            
                       ^[[0;10m^[[39;49m
^[[0;10m 1086 root      20   0     0    0    0 S  0.0  0.0 
  0:00.47 sync_supers 
                                                            
                                                            
                 ^[[0;10m^[[39;49m
^[[0;10m 1088 root      20   0     0    0    0 S  0.0  0.0 
  0:00.00 bdi-default 
                                                            
                                                            
                 ^[[0;10m^[[39;49m
^[[0;10m 1089 root       0 -20     0    0    0 S  0.0  0.0 
  0:00.00 kintegrityd 
                                                            
                                                            
                 ^[[0;10m^[[39;49m
^[[0;10m 1090 root       0 -20     0    0    0 S  0.0  0.0 
  0:00.00 kblockd 
                                                            
                                                            
                     ^[[0;10m^[[39;49m
^[[0;10m 1108 root     -51   0     0    0    0 S  0.0  0.0 
  0:00.00 irq/9-acpi 
                                                            
                                                            
                  ^[[0;10m^[[39;49m
^[[0;10m 1236 root       0 -20     0    0    0 S  0.0  0.0 
  0:00.00 tifm 
                                                            
                                                            
                        ^[[0;10m^[[39;49m
^[[0;10m 1268 root       0 -20     0    0    0 S  0.0  0.0 
  0:00.00 ata_sff 
                                                            
                                                            
                     ^[[0;10m^[[39;49m
^[[0;10m 1280 root      20   0     0    0    0 S  0.0  0.0 
  0:00.04 khubd 
                                                            
                                                            
                       ^[[0;10m^[[39;49m
^[[0;10m 1295 root       0 -20     0    0    0 S  0.0  0.0 
  0:00.00 md 
                                                            
                                                            
                          ^[[0;10m^[[39;49m
^[[0;10m 1420 root       0 -20     0    0    0 S  0.0  0.0 
  0:00.00 rpciod 
                                                            
                                                            
                      ^[[0;10m^[[39;49m
^[[0;10m 1450 root      20   0     0    0    0 S  0.0  0.0 
  0:00.03 khungtaskd 
                                                            
                                                            
                  ^[[0;10m^[[39;49m
^[[0;10m 1456 root      20   0     0    0    0 S  0.0  0.0 
  0:02.47 kswapd0 
                                                            
                                                            
                     ^[[0;10m^[[39;49m
^[[0;10m 1457 root      25   5     0    0    0 S  0.0  0.0 
  0:00.00 ksmd 
                                                            
                                                            
                        ^[[0;10m^[[39;49m
^[[0;10m 1532 root      39  19     0    0    0 S  0.0  0.0 
  0:06.17 khugepaged 
                                                            
                                                            
                  ^[[0;10m^[[39;49m
^[[0;10m 1533 root      20   0     0    0    0 S  0.0  0.0 
  0:00.00 fsnotify_mark 
                                                            
                                                            
               ^[[0;10m^[[39;49m
^[[0;10m 1572 root      20   0     0    0    0 S  0.0  0.0 
  0:00.00 ecryptfs-kthrea 
                                                            
                                                            
             ^[[0;10m^[[39;49m
^[[0;10m 1575 root       0 -20     0    0    0 S  0.0  0.0 
  0:00.00 nfsiod 
                                                            
                                                            
                      ^[[0;10m^[[39;49m
^[[0;10m 1580 root       0 -20     0    0    0 S  0.0  0.0 
  0:00.00 cifsiod 
                                                            
                                                            
                     ^[[0;10m^[[39;49m
^[[0;10m 1606 root      20   0     0    0    0 S  0.0  0.0 
  0:00.00 jfsIO 
                                                            
                                                            
                       ^[[0;10m^[[39;49m
^[[0;10m 1607 root      20   0     0    0    0 S  0.0  0.0 
  0:00.00 jfsCommit 
                                                            
                                                            
                   ^[[0;10m^[[39;49m
^[[0;10m 1608 root      20   0     0    0    0 S  0.0  0.0 
  0:00.00 jfsSync 
                                                            
                                                            
                     ^[[0;10m^[[39;49m
^[[0;10m 1615 root       0 -20     0    0    0 S  0.0  0.0 
  0:00.00 xfsalloc 
                                                            
                                                            
                    ^[[0;10m^[[39;49m
^[[0;10m 1616 root       0 -20     0    0    0 S  0.0  0.0 
  0:00.00 xfs_mru_cache 
                                                            
                                                            
               ^[[0;10m^[[39;49m
^[[0;10m 1617 root       0 -20     0    0    0 S  0.0  0.0 
  0:00.00 xfslogd 
                                                            
                                                            
                     ^[[0;10m^[[39;49m
^[[0;10m 1624 root       0 -20     0    0    0 S  0.0  0.0 
  0:00.00 ocfs2_wq 
                                                            
                                                            
                    ^[[0;10m^[[39;49m
^[[0;10m 1627 root       0 -20     0    0    0 S  0.0  0.0 
  0:00.00 user_dlm 
                                                            
                                                            
                    ^[[0;10m^[[39;49m
^[[0;10m 1633 root       0 -20     0    0    0 S  0.0  0.0 
  0:00.00 glock_workqueue 
                                                            
                                                            
             ^[[0;10m^[[39;49m
^[[0;10m 1634 root       0 -20     0    0    0 S  0.0  0.0 
  0:00.00 delete_workqueu 
                                                            
                                                            
             ^[[0;10m^[[39;49m
^[[0;10m 1638 root       0 -20     0    0    0 S  0.0  0.0 
  0:00.00 gfs_recovery 
                                                            
                                                            
                ^[[0;10m^[[39;49m
^[[0;10m 1642 root       0 -20     0    0    0 S  0.0  0.0 
  0:00.00 crypto 
                                                            
                                                            
                      ^[[0;10m^[[39;49m
^[[0;10m 1780 root      20   0     0    0    0 S  0.0  0.0 
  0:00.23 kworker/u:2 
                                                            
                                                            
                 ^[[0;10m^[[39;49m
^[[0;10m 2408 root      20   0     0    0    0 S  0.0  0.0 
  0:00.00 cciss_scan 
                                                            
                                                            
                  ^[[0;10m^[[39;49m
^[[0;10m 2542 root       0 -20     0    0    0 S  0.0  0.0 
  0:00.00 iscsi_eh 
                                                            
                                                            
                    ^[[0;10m^[[39;49m
^[[0;10m 2551 root       0 -20     0    0    0 S  0.0  0.0 
  0:00.00 kmpath_rdacd 
                                                            
                                                            
                ^[[0;10m^[[39;49m
^[[0;10m 2552 root       0 -20     0    0    0 S  0.0  0.0 
  0:00.00 fc_exch_workque 
                                                            
                                                            
             ^[[0;10m^[[39;49m
^[[0;10m 2553 root       0 -20     0    0    0 S  0.0  0.0 
  0:00.00 fc_rport_eq 
                                                            
                                                            
                 ^[[0;10m^[[39;49m
^[[0;10m 2554 root       0 -20     0    0    0 S  0.0  0.0 
  0:00.00 fcoethread/0 
                                                            
                                                            
                ^[[0;10m^[[39;49m
^[[0;10m 2556 root       0 -20     0    0    0 S  0.0  0.0 
  0:00.00 fnic_event_wq 
                                                            
                                                            
               ^[[0;10m^[[39;49m
^[[0;10m 2558 root       0 -20     0    0    0 S  0.0  0.0 
  0:00.00 bnx2fc_l2_threa 
                                                            
                                                            
             ^[[0;10m^[[39;49m
^[[0;10m 2559 root       0 -20     0    0    0 S  0.0  0.0 
  0:00.00 bnx2fc_thread/0 
                                                            
                                                            
             ^[[0;10m^[[39;49m
^[[0;10m 2636 root       0 -20     0    0    0 S  0.0  0.0 
  0:00.00 bnx2i_thread/0 
                                                            
                                                            
              ^[[0;10m^[[39;49m
^[[0;10m 2675 root      20   0     0    0    0 S  0.0  0.0 
  0:00.39 scsi_eh_2 
                                                            
                                                            
                   ^[[0;10m^[[39;49m
^[[0;10m 2678 root      20   0     0    0    0 S  0.0  0.0 
  0:25.28 scsi_eh_3 
                                                            
                                                            
                   ^[[0;10m^[[39;49m
^[[0;10m 2688 root      20   0     0    0    0 S  0.0  0.0 
  0:00.02 scsi_eh_4 
                                                            
                                                            
                   ^[[0;10m^[[39;49m
^[[0;10m 2691 root      20   0     0    0    0 S  0.0  0.0 
  0:00.33 scsi_eh_5 
                                                            
                                                            
                   ^[[0;10m^[[39;49m^[[6;1H^[[K^[[H^[[0;10mtop - 
16:20:04 up 2 days, 16:51,  5 users,  load average: 2.16, 
1.55, 1.11^[[0;10m^[[39;49m^[[K

Cpu(s):^[[0;10m^[[39;49m^[[0;10m 
77.0%^[[0;10m^[[39;49mus,^[[0;10m^[[39;49m^[[0;10m 
23.0%^[[0;10m^[[39;49msy,^[[0;10m^[[39;49m^[[0;10m 
 0.0%^[[0;10m^[[39;49mni,^[[0;10m^[[39;49m^[[0;10m 
 0.0%^[[0;10m^[[39;49mid,^[[0;10m^[[39;49m^[[0;10m 
 0.0%^[[0;10m^[[39;49mwa,^[[0;10m^[[39;49m^[[0;10m 
 0.0%^[[0;10m^[[39;49mhi,^[[0;10m^[[39;49m^[[0;10m 
 0.0%^[[0;10m^[[39;49msi,^[[0;10m^[[39;49m^[[0;10m 
 0.0%^[[0;10m^[[39;49mst^[[0;10m^[[39;49m^[[K
Mem: ^[[0;10m^[[39;49m^[[0;10m  3088984k 
^[[0;10m^[[39;49mtotal,^[[0;10m^[[39;49m^[[0;10m  2861540k 
^[[0;10m^[[39;49mused,^[[0;10m^[[39;49m^[[0;10m   227444k 
^[[0;10m^[[39;49mfree,^[[0;10m^[[39;49m^[[0;10m   286872k 
^[[0;10m^[[39;49mbuffers^[[0;10m^[[39;49m^[[K
Swap:^[[0;10m^[[39;49m^[[0;10m        0k 
^[[0;10m^[[39;49mtotal,^[[0;10m^[[39;49m^[[0;10m        0k 
^[[0;10m^[[39;49mused,^[[0;10m^[[39;49m^[[0;10m        0k 
^[[0;10m^[[39;49mfree,^[[0;10m^[[39;49m^[[0;10m  1780748k 
^[[0;10m^[[39;49mcached^[[0;10m^[[39;49m^[[K
^[[6;1H

^[[0;10m^[[0;10m31801 root      20   0  577m 337m  48m R 
75.1 11.2 158:29.99 firefox-bin 
                                                            
                                                            
                 ^[[0;10m^[[39;49m
^[[0;10m^[[0;10m 5534 root      30  10     0    0    0 R 
 9.5  0.0  90:28.63 kmemleak 
                                                            
                                                            
                    ^[[0;10m^[[39;49m
^[[0;10m10227 root      20   0 57396  31m  27m S  6.2  1.1 
256:55.04 kded 
                                                            
                                                            
                        ^[[0;10m^[[39;49m
^[[0;10m10256 root      20   0 94484  42m 7548 S  3.6  1.4 
 31:16.80 artsd 
                                                            
                                                            
                       ^[[0;10m^[[39;49m
^[[0;10m22802 root      20   0  2624 1132  828 S  1.0  0.0 
  0:05.91 top 
                                                            
                                                            
                         ^[[0;10m^[[39;49m
^[[0;10m^[[0;10m26642 root      20   0  2624 1132  828 R 
 1.0  0.0   0:00.06 top 
                                                            
                                                            
                         ^[[0;10m^[[39;49m
^[[0;10m24974 root      20   0  2620 1196  896 S  0.7  0.0 
  0:02.40 top 
                                                            
                                                            
                         ^[[0;10m^[[39;49m
^[[0;10m    3 root      20   0     0    0    0 S  0.3  0.0 
  6:12.81 ksoftirqd/0 
                                                            
                                                            
                 ^[[0;10m^[[39;49m
^[[0;10m24935 root      20   0  6088 2676 1848 S  0.3  0.1 
  0:00.63 mrxvt 
                                                            
                                                            
                       ^[[0;10m^[[39;49m
^[[0;10m    1 root      20   0   824  320  280 S  0.0  0.0 
  0:35.88 init 
                                                            
                                                            
                        ^[[0;10m^[[39;49m
^[[0;10m    2 root      20   0     0    0    0 S  0.0  0.0 
  0:01.19 kthreadd 
                                                            
                                                            
                    ^[[0;10m^[[39;49m
^[[0;10m    6 root      RT   0     0    0    0 S  0.0  0.0 
  0:00.00 migration/0 
                                                            
                                                            
                 ^[[0;10m^[[39;49m
^[[0;10m    7 root      RT   0     0    0    0 S  0.0  0.0 
  0:05.35 watchdog/0 
                                                            
                                                            
                  ^[[0;10m^[[39;49m
^[[0;10m    8 root       0 -20     0    0    0 S  0.0  0.0 
  0:00.00 khelper 
                                                            
                                                            
                     ^[[0;10m^[[39;49m
^[[0;10m    9 root      20   0     0    0    0 S  0.0  0.0 
  0:00.03 kdevtmpfs 
                                                            
                                                            
                   ^[[0;10m^[[39;49m
^[[0;10m   10 root       0 -20     0    0    0 S  0.0  0.0 
  0:00.00 netns 
                                                            
                                                            
                       ^[[0;10m^[[39;49m
^[[0;10m 1086 root      20   0     0    0    0 S  0.0  0.0 
  0:00.47 sync_supers 
                                                            
                                                            
                 ^[[0;10m^[[39;49m
^[[0;10m 1088 root      20   0     0    0    0 S  0.0  0.0 
  0:00.00 bdi-default 
                                                            
                                                            
                 ^[[0;10m^[[39;49m
^[[0;10m 1089 root       0 -20     0    0    0 S  0.0  0.0 
  0:00.00 kintegrityd 
                                                            
                                                            
                 ^[[0;10m^[[39;49m
^[[0;10m 1090 root       0 -20     0    0    0 S  0.0  0.0 
  0:00.00 kblockd 
                                                            
                                                            
                     ^[[0;10m^[[39;49m
^[[0;10m 1108 root     -51   0     0    0    0 S  0.0  0.0 
  0:00.00 irq/9-acpi 
                                                            
                                                            
                  ^[[0;10m^[[39;49m
^[[0;10m 1236 root       0 -20     0    0    0 S  0.0  0.0 
  0:00.00 tifm 
                                                            
                                                            
                        ^[[0;10m^[[39;49m
^[[0;10m 1268 root       0 -20     0    0    0 S  0.0  0.0 
  0:00.00 ata_sff 
                                                            
                                                            
                     ^[[0;10m^[[39;49m
^[[0;10m 1280 root      20   0     0    0    0 S  0.0  0.0 
  0:00.04 khubd 
                                                            
                                                            
                       ^[[0;10m^[[39;49m
^[[0;10m 1295 root       0 -20     0    0    0 S  0.0  0.0 
  0:00.00 md 
                                                            
                                                            
                          ^[[0;10m^[[39;49m
^[[0;10m 1420 root       0 -20     0    0    0 S  0.0  0.0 
  0:00.00 rpciod 
                                                            
                                                            
                      ^[[0;10m^[[39;49m
^[[0;10m 1450 root      20   0     0    0    0 S  0.0  0.0 
  0:00.03 khungtaskd 
                                                            
                                                            
                  ^[[0;10m^[[39;49m
^[[0;10m 1456 root      20   0     0    0    0 S  0.0  0.0 
  0:02.47 kswapd0 
                                                            
                                                            
                     ^[[0;10m^[[39;49m
^[[0;10m 1457 root      25   5     0    0    0 S  0.0  0.0 
  0:00.00 ksmd 
                                                            
                                                            
                        ^[[0;10m^[[39;49m
^[[0;10m 1532 root      39  19     0    0    0 S  0.0  0.0 
  0:06.17 khugepaged 
                                                            
                                                            
                  ^[[0;10m^[[39;49m
^[[0;10m 1533 root      20   0     0    0    0 S  0.0  0.0 
  0:00.00 fsnotify_mark 
                                                            
                                                            
               ^[[0;10m^[[39;49m
^[[0;10m 1572 root      20   0     0    0    0 S  0.0  0.0 
  0:00.00 ecryptfs-kthrea 
                                                            
                                                            
             ^[[0;10m^[[39;49m
^[[0;10m 1575 root       0 -20     0    0    0 S  0.0  0.0 
  0:00.00 nfsiod 
                                                            
                                                            
                      ^[[0;10m^[[39;49m
^[[0;10m 1580 root       0 -20     0    0    0 S  0.0  0.0 
  0:00.00 cifsiod 
                                                            
                                                            
                     ^[[0;10m^[[39;49m
^[[0;10m 1606 root      20   0     0    0    0 S  0.0  0.0 
  0:00.00 jfsIO 
                                                            
                                                            
                       ^[[0;10m^[[39;49m
^[[0;10m 1607 root      20   0     0    0    0 S  0.0  0.0 
  0:00.00 jfsCommit 
                                                            
                                                            
                   ^[[0;10m^[[39;49m
^[[0;10m 1608 root      20   0     0    0    0 S  0.0  0.0 
  0:00.00 jfsSync 
                                                            
                                                            
                     ^[[0;10m^[[39;49m
^[[0;10m 1615 root       0 -20     0    0    0 S  0.0  0.0 
  0:00.00 xfsalloc 
                                                            
                                                            
                    ^[[0;10m^[[39;49m
^[[0;10m 1616 root       0 -20     0    0    0 S  0.0  0.0 
  0:00.00 xfs_mru_cache 
                                                            
                                                            
               ^[[0;10m^[[39;49m
^[[0;10m 1617 root       0 -20     0    0    0 S  0.0  0.0 
  0:00.00 xfslogd 
                                                            
                                                            
                     ^[[0;10m^[[39;49m
^[[0;10m 1624 root       0 -20     0    0    0 S  0.0  0.0 
  0:00.00 ocfs2_wq 
                                                            
                                                            
                    ^[[0;10m^[[39;49m
^[[0;10m 1627 root       0 -20     0    0    0 S  0.0  0.0 
  0:00.00 user_dlm 
                                                            
                                                            
                    ^[[0;10m^[[39;49m
^[[0;10m 1633 root       0 -20     0    0    0 S  0.0  0.0 
  0:00.00 glock_workqueue 
                                                            
                                                            
             ^[[0;10m^[[39;49m
^[[0;10m 1634 root       0 -20     0    0    0 S  0.0  0.0 
  0:00.00 delete_workqueu 
                                                            
                                                            
             ^[[0;10m^[[39;49m
^[[0;10m 1638 root       0 -20     0    0    0 S  0.0  0.0 
  0:00.00 gfs_recovery 
                                                            
                                                            
                ^[[0;10m^[[39;49m
^[[0;10m 1642 root       0 -20     0    0    0 S  0.0  0.0 
  0:00.00 crypto 
                                                            
                                                            
                      ^[[0;10m^[[39;49m
^[[0;10m 1780 root      20   0     0    0    0 S  0.0  0.0 
  0:00.23 kworker/u:2 
                                                            
                                                            
                 ^[[0;10m^[[39;49m
^[[0;10m 2408 root      20   0     0    0    0 S  0.0  0.0 
  0:00.00 cciss_scan 
                                                            
                                                            
                  ^[[0;10m^[[39;49m
^[[0;10m 2542 root       0 -20     0    0    0 S  0.0  0.0 
  0:00.00 iscsi_eh 
                                                            
                                                            
                    ^[[0;10m^[[39;49m
^[[0;10m 2551 root       0 -20     0    0    0 S  0.0  0.0 
  0:00.00 kmpath_rdacd 
                                                            
                                                            
                ^[[0;10m^[[39;49m
^[[0;10m 2552 root       0 -20     0    0    0 S  0.0  0.0 
  0:00.00 fc_exch_workque 
                                                            
                                                            
             ^[[0;10m^[[39;49m
^[[0;10m 2553 root       0 -20     0    0    0 S  0.0  0.0 
  0:00.00 fc_rport_eq 
                                                            
                                                            
                 ^[[0;10m^[[39;49m
^[[0;10m 2554 root       0 -20     0    0    0 S  0.0  0.0 
  0:00.00 fcoethread/0 
                                                            
                                                            
                ^[[0;10m^[[39;49m
^[[0;10m 2556 root       0 -20     0    0    0 S  0.0  0.0 
  0:00.00 fnic_event_wq 
                                                            
                                                            
               ^[[0;10m^[[39;49m
^[[0;10m 2558 root       0 -20     0    0    0 S  0.0  0.0 
  0:00.00 bnx2fc_l2_threa 
                                                            
                                                            
             ^[[0;10m^[[39;49m
^[[0;10m 2559 root       0 -20     0    0    0 S  0.0  0.0 
  0:00.00 bnx2fc_thread/0 
                                                            
                                                            
             ^[[0;10m^[[39;49m
^[[0;10m 2636 root       0 -20     0    0    0 S  0.0  0.0 
  0:00.00 bnx2i_thread/0 
                                                            
                                                            
              ^[[0;10m^[[39;49m
^[[0;10m 2675 root      20   0     0    0    0 S  0.0  0.0 
  0:00.39 scsi_eh_2 
                                                            
                                                            
                   ^[[0;10m^[[39;49m^[[6;1H^[[K^[[H^[[0;10mtop - 
16:20:07 up 2 days, 16:51,  5 users,  load average: 2.15, 
1.55, 1.12^[[0;10m^[[39;49m^[[K

Cpu(s):^[[0;10m^[[39;49m^[[0;10m 
77.6%^[[0;10m^[[39;49mus,^[[0;10m^[[39;49m^[[0;10m 
22.4%^[[0;10m^[[39;49msy,^[[0;10m^[[39;49m^[[0;10m 
 0.0%^[[0;10m^[[39;49mni,^[[0;10m^[[39;49m^[[0;10m 
 0.0%^[[0;10m^[[39;49mid,^[[0;10m^[[39;49m^[[0;10m 
 0.0%^[[0;10m^[[39;49mwa,^[[0;10m^[[39;49m^[[0;10m 
 0.0%^[[0;10m^[[39;49mhi,^[[0;10m^[[39;49m^[[0;10m 
 0.0%^[[0;10m^[[39;49msi,^[[0;10m^[[39;49m^[[0;10m 
 0.0%^[[0;10m^[[39;49mst^[[0;10m^[[39;49m^[[K
Mem: ^[[0;10m^[[39;49m^[[0;10m  3088984k 
^[[0;10m^[[39;49mtotal,^[[0;10m^[[39;49m^[[0;10m  2861540k 
^[[0;10m^[[39;49mused,^[[0;10m^[[39;49m^[[0;10m   227444k 
^[[0;10m^[[39;49mfree,^[[0;10m^[[39;49m^[[0;10m   286872k 
^[[0;10m^[[39;49mbuffers^[[0;10m^[[39;49m^[[K
Swap:^[[0;10m^[[39;49m^[[0;10m        0k 
^[[0;10m^[[39;49mtotal,^[[0;10m^[[39;49m^[[0;10m        0k 
^[[0;10m^[[39;49mused,^[[0;10m^[[39;49m^[[0;10m        0k 
^[[0;10m^[[39;49mfree,^[[0;10m^[[39;49m^[[0;10m  1780768k 
^[[0;10m^[[39;49mcached^[[0;10m^[[39;49m^[[K
^[[6;1H

^[[0;10m^[[0;10m31801 root      20   0  577m 337m  48m R 
75.8 11.2 158:32.30 firefox-bin 
                                                            
                                                            
                 ^[[0;10m^[[39;49m
^[[0;10m^[[0;10m 5534 root      30  10     0    0    0 R 
 9.5  0.0  90:28.92 kmemleak 
                                                            
                                                            
                    ^[[0;10m^[[39;49m
^[[0;10m10227 root      20   0 57396  31m  27m S  5.9  1.1 
256:55.22 kded 
                                                            
                                                            
                        ^[[0;10m^[[39;49m
^[[0;10m10256 root      20   0 94484  42m 7548 S  3.3  1.4 
 31:16.90 artsd 
                                                            
                                                            
                       ^[[0;10m^[[39;49m
^[[0;10m24974 root      20   0  2620 1196  896 S  1.0  0.0 
  0:02.43 top 
                                                            
                                                            
                         ^[[0;10m^[[39;49m
^[[0;10m22802 root      20   0  2624 1132  828 S  0.7  0.0 
  0:05.93 top 
                                                            
                                                            
                         ^[[0;10m^[[39;49m
^[[0;10m24935 root      20   0  6088 2676 1848 S  0.7  0.1 
  0:00.65 mrxvt 
                                                            
                                                            
                       ^[[0;10m^[[39;49m
^[[0;10m^[[0;10m26642 root      20   0  2624 1132  828 R 
 0.7  0.0   0:00.08 top 
                                                            
                                                            
                         ^[[0;10m^[[39;49m
^[[0;10m 5549 root      20   0     0    0    0 S  0.3  0.0 
  0:54.85 kjournald 
                                                            
                                                            
                   ^[[0;10m^[[39;49m
^[[0;10m10183 root      20   0 12404 1640 1328 S  0.3  0.1 
  8:54.40 tpb 
                                                            
                                                            
                         ^[[0;10m^[[39;49m
^[[0;10m    1 root      20   0   824  320  280 S  0.0  0.0 
  0:35.88 init 
                                                            
                                                            
                        ^[[0;10m^[[39;49m
^[[0;10m    2 root      20   0     0    0    0 S  0.0  0.0 
  0:01.19 kthreadd 
                                                            
                                                            
                    ^[[0;10m^[[39;49m
^[[0;10m    3 root      20   0     0    0    0 S  0.0  0.0 
  6:12.81 ksoftirqd/0 
                                                            
                                                            
                 ^[[0;10m^[[39;49m
^[[0;10m    6 root      RT   0     0    0    0 S  0.0  0.0 
  0:00.00 migration/0 
                                                            
                                                            
                 ^[[0;10m^[[39;49m
^[[0;10m    7 root      RT   0     0    0    0 S  0.0  0.0 
  0:05.35 watchdog/0 
                                                            
                                                            
                  ^[[0;10m^[[39;49m
^[[0;10m    8 root       0 -20     0    0    0 S  0.0  0.0 
  0:00.00 khelper 
                                                            
                                                            
                     ^[[0;10m^[[39;49m
^[[0;10m    9 root      20   0     0    0    0 S  0.0  0.0 
  0:00.03 kdevtmpfs 
                                                            
                                                            
                   ^[[0;10m^[[39;49m
^[[0;10m   10 root       0 -20     0    0    0 S  0.0  0.0 
  0:00.00 netns 
                                                            
                                                            
                       ^[[0;10m^[[39;49m
^[[0;10m 1086 root      20   0     0    0    0 S  0.0  0.0 
  0:00.47 sync_supers 
                                                            
                                                            
                 ^[[0;10m^[[39;49m
^[[0;10m 1088 root      20   0     0    0    0 S  0.0  0.0 
  0:00.00 bdi-default 
                                                            
                                                            
                 ^[[0;10m^[[39;49m
^[[0;10m 1089 root       0 -20     0    0    0 S  0.0  0.0 
  0:00.00 kintegrityd 
                                                            
                                                            
                 ^[[0;10m^[[39;49m
^[[0;10m 1090 root       0 -20     0    0    0 S  0.0  0.0 
  0:00.00 kblockd 
                                                            
                                                            
                     ^[[0;10m^[[39;49m
^[[0;10m 1108 root     -51   0     0    0    0 S  0.0  0.0 
  0:00.00 irq/9-acpi 
                                                            
                                                            
                  ^[[0;10m^[[39;49m
^[[0;10m 1236 root       0 -20     0    0    0 S  0.0  0.0 
  0:00.00 tifm 
                                                            
                                                            
                        ^[[0;10m^[[39;49m
^[[0;10m 1268 root       0 -20     0    0    0 S  0.0  0.0 
  0:00.00 ata_sff 
                                                            
                                                            
                     ^[[0;10m^[[39;49m
^[[0;10m 1280 root      20   0     0    0    0 S  0.0  0.0 
  0:00.04 khubd 
                                                            
                                                            
                       ^[[0;10m^[[39;49m
^[[0;10m 1295 root       0 -20     0    0    0 S  0.0  0.0 
  0:00.00 md 
                                                            
                                                            
                          ^[[0;10m^[[39;49m
^[[0;10m 1420 root       0 -20     0    0    0 S  0.0  0.0 
  0:00.00 rpciod 
                                                            
                                                            
                      ^[[0;10m^[[39;49m
^[[0;10m 1450 root      20   0     0    0    0 S  0.0  0.0 
  0:00.03 khungtaskd 
                                                            
                                                            
                  ^[[0;10m^[[39;49m
^[[0;10m 1456 root      20   0     0    0    0 S  0.0  0.0 
  0:02.47 kswapd0 
                                                            
                                                            
                     ^[[0;10m^[[39;49m
^[[0;10m 1457 root      25   5     0    0    0 S  0.0  0.0 
  0:00.00 ksmd 
                                                            
                                                            
                        ^[[0;10m^[[39;49m
^[[0;10m 1532 root      39  19     0    0    0 S  0.0  0.0 
  0:06.17 khugepaged 
                                                            
                                                            
                  ^[[0;10m^[[39;49m
^[[0;10m 1533 root      20   0     0    0    0 S  0.0  0.0 
  0:00.00 fsnotify_mark 
                                                            
                                                            
               ^[[0;10m^[[39;49m
^[[0;10m 1572 root      20   0     0    0    0 S  0.0  0.0 
  0:00.00 ecryptfs-kthrea 
                                                            
                                                            
             ^[[0;10m^[[39;49m
^[[0;10m 1575 root       0 -20     0    0    0 S  0.0  0.0 
  0:00.00 nfsiod 
                                                            
                                                            
                      ^[[0;10m^[[39;49m
^[[0;10m 1580 root       0 -20     0    0    0 S  0.0  0.0 
  0:00.00 cifsiod 
                                                            
                                                            
                     ^[[0;10m^[[39;49m
^[[0;10m 1606 root      20   0     0    0    0 S  0.0  0.0 
  0:00.00 jfsIO 
                                                            
                                                            
                       ^[[0;10m^[[39;49m
^[[0;10m 1607 root      20   0     0    0    0 S  0.0  0.0 
  0:00.00 jfsCommit 
                                                            
                                                            
                   ^[[0;10m^[[39;49m
^[[0;10m 1608 root      20   0     0    0    0 S  0.0  0.0 
  0:00.00 jfsSync 
                                                            
                                                            
                     ^[[0;10m^[[39;49m
^[[0;10m 1615 root       0 -20     0    0    0 S  0.0  0.0 
  0:00.00 xfsalloc 
                                                            
                                                            
                    ^[[0;10m^[[39;49m
^[[0;10m 1616 root       0 -20     0    0    0 S  0.0  0.0 
  0:00.00 xfs_mru_cache 
                                                            
                                                            
               ^[[0;10m^[[39;49m
^[[0;10m 1617 root       0 -20     0    0    0 S  0.0  0.0 
  0:00.00 xfslogd 
                                                            
                                                            
                     ^[[0;10m^[[39;49m
^[[0;10m 1624 root       0 -20     0    0    0 S  0.0  0.0 
  0:00.00 ocfs2_wq 
                                                            
                                                            
                    ^[[0;10m^[[39;49m
^[[0;10m 1627 root       0 -20     0    0    0 S  0.0  0.0 
  0:00.00 user_dlm 
                                                            
                                                            
                    ^[[0;10m^[[39;49m
^[[0;10m 1633 root       0 -20     0    0    0 S  0.0  0.0 
  0:00.00 glock_workqueue 
                                                            
                                                            
             ^[[0;10m^[[39;49m
^[[0;10m 1634 root       0 -20     0    0    0 S  0.0  0.0 
  0:00.00 delete_workqueu 
                                                            
                                                            
             ^[[0;10m^[[39;49m
^[[0;10m 1638 root       0 -20     0    0    0 S  0.0  0.0 
  0:00.00 gfs_recovery 
                                                            
                                                            
                ^[[0;10m^[[39;49m
^[[0;10m 1642 root       0 -20     0    0    0 S  0.0  0.0 
  0:00.00 crypto 
                                                            
                                                            
                      ^[[0;10m^[[39;49m
^[[0;10m 1780 root      20   0     0    0    0 S  0.0  0.0 
  0:00.23 kworker/u:2 
                                                            
                                                            
                 ^[[0;10m^[[39;49m
^[[0;10m 2408 root      20   0     0    0    0 S  0.0  0.0 
  0:00.00 cciss_scan 
                                                            
                                                            
                  ^[[0;10m^[[39;49m
^[[0;10m 2542 root       0 -20     0    0    0 S  0.0  0.0 
  0:00.00 iscsi_eh 
                                                            
                                                            
                    ^[[0;10m^[[39;49m
^[[0;10m 2551 root       0 -20     0    0    0 S  0.0  0.0 
  0:00.00 kmpath_rdacd 
                                                            
                                                            
                ^[[0;10m^[[39;49m
^[[0;10m 2552 root       0 -20     0    0    0 S  0.0  0.0 
  0:00.00 fc_exch_workque 
                                                            
                                                            
             ^[[0;10m^[[39;49m
^[[0;10m 2553 root       0 -20     0    0    0 S  0.0  0.0 
  0:00.00 fc_rport_eq 
                                                            
                                                            
                 ^[[0;10m^[[39;49m
^[[0;10m 2554 root       0 -20     0    0    0 S  0.0  0.0 
  0:00.00 fcoethread/0 
                                                            
                                                            
                ^[[0;10m^[[39;49m
^[[0;10m 2556 root       0 -20     0    0    0 S  0.0  0.0 
  0:00.00 fnic_event_wq 
                                                            
                                                            
               ^[[0;10m^[[39;49m
^[[0;10m 2558 root       0 -20     0    0    0 S  0.0  0.0 
  0:00.00 bnx2fc_l2_threa 
                                                            
                                                            
             ^[[0;10m^[[39;49m
^[[0;10m 2559 root       0 -20     0    0    0 S  0.0  0.0 
  0:00.00 bnx2fc_thread/0 
                                                            
                                                            
             ^[[0;10m^[[39;49m^[[6;1H^[[K^[[H^[[0;10mtop - 
16:20:10 up 2 days, 16:52,  5 users,  load average: 2.15, 
1.55, 1.12^[[0;10m^[[39;49m^[[K

Cpu(s):^[[0;10m^[[39;49m^[[0;10m 
77.1%^[[0;10m^[[39;49mus,^[[0;10m^[[39;49m^[[0;10m 
22.5%^[[0;10m^[[39;49msy,^[[0;10m^[[39;49m^[[0;10m 
 0.0%^[[0;10m^[[39;49mni,^[[0;10m^[[39;49m^[[0;10m 
 0.0%^[[0;10m^[[39;49mid,^[[0;10m^[[39;49m^[[0;10m 
 0.0%^[[0;10m^[[39;49mwa,^[[0;10m^[[39;49m^[[0;10m 
 0.0%^[[0;10m^[[39;49mhi,^[[0;10m^[[39;49m^[[0;10m 
 0.3%^[[0;10m^[[39;49msi,^[[0;10m^[[39;49m^[[0;10m 
 0.0%^[[0;10m^[[39;49mst^[[0;10m^[[39;49m^[[K
Mem: ^[[0;10m^[[39;49m^[[0;10m  3088984k 
^[[0;10m^[[39;49mtotal,^[[0;10m^[[39;49m^[[0;10m  2861540k 
^[[0;10m^[[39;49mused,^[[0;10m^[[39;49m^[[0;10m   227444k 
^[[0;10m^[[39;49mfree,^[[0;10m^[[39;49m^[[0;10m   286872k 
^[[0;10m^[[39;49mbuffers^[[0;10m^[[39;49m^[[K
Swap:^[[0;10m^[[39;49m^[[0;10m        0k 
^[[0;10m^[[39;49mtotal,^[[0;10m^[[39;49m^[[0;10m        0k 
^[[0;10m^[[39;49mused,^[[0;10m^[[39;49m^[[0;10m        0k 
^[[0;10m^[[39;49mfree,^[[0;10m^[[39;49m^[[0;10m  1780780k 
^[[0;10m^[[39;49mcached^[[0;10m^[[39;49m^[[K
^[[6;1H

^[[0;10m^[[0;10m31801 root      20   0  577m 337m  48m R 
75.5 11.2 158:34.60 firefox-bin 
                                                            
                                                            
                 ^[[0;10m^[[39;49m
^[[0;10m^[[0;10m 5534 root      30  10     0    0    0 R 
 9.2  0.0  90:29.20 kmemleak 
                                                            
                                                            
                    ^[[0;10m^[[39;49m
^[[0;10m10227 root      20   0 57396  31m  27m S  6.2  1.1 
256:55.41 kded 
                                                            
                                                            
                        ^[[0;10m^[[39;49m
^[[0;10m10256 root      20   0 94484  42m 7548 S  3.6  1.4 
 31:17.01 artsd 
                                                            
                                                            
                       ^[[0;10m^[[39;49m
^[[0;10m22802 root      20   0  2624 1132  828 S  0.7  0.0 
  0:05.95 top 
                                                            
                                                            
                         ^[[0;10m^[[39;49m
^[[0;10m24974 root      20   0  2620 1196  896 S  0.7  0.0 
  0:02.45 top 
                                                            
                                                            
                         ^[[0;10m^[[39;49m
^[[0;10m^[[0;10m26642 root      20   0  2624 1132  828 R 
 0.7  0.0   0:00.10 top 
                                                            
                                                            
                         ^[[0;10m^[[39;49m
^[[0;10m    3 root      20   0     0    0    0 S  0.3  0.0 
  6:12.82 ksoftirqd/0 
                                                            
                                                            
                 ^[[0;10m^[[39;49m
^[[0;10m 9961 root      20   0  3800 1076  920 S  0.3  0.0 
  1:46.57 hald-addon-stor 
                                                            
                                                            
             ^[[0;10m^[[39;49m
^[[0;10m10183 root      20   0 12404 1640 1328 S  0.3  0.1 
  8:54.41 tpb 
                                                            
                                                            
                         ^[[0;10m^[[39;49m


^[[0;10m    6 root      RT   0     0    0    0 S  0.0  0.0 
  0:00.00 migration/0 
                                                            
                                                            
                 ^[[0;10m^[[39;49m
^[[0;10m    7 root      RT   0     0    0    0 S  0.0  0.0 
  0:05.35 watchdog/0 
                                                            
                                                            
                  ^[[0;10m^[[39;49m
^[[0;10m    8 root       0 -20     0    0    0 S  0.0  0.0 
  0:00.00 khelper 
                                                            
                                                            
                     ^[[0;10m^[[39;49m
^[[0;10m    9 root      20   0     0    0    0 S  0.0  0.0 
  0:00.03 kdevtmpfs 
                                                            
                                                            
                   ^[[0;10m^[[39;49m
^[[0;10m   10 root       0 -20     0    0    0 S  0.0  0.0 
  0:00.00 netns 
                                                            
                                                            
                       ^[[0;10m^[[39;49m
^[[0;10m 1086 root      20   0     0    0    0 S  0.0  0.0 
  0:00.47 sync_supers 
                                                            
                                                            
                 ^[[0;10m^[[39;49m
^[[0;10m 1088 root      20   0     0    0    0 S  0.0  0.0 
  0:00.00 bdi-default 
                                                            
                                                            
                 ^[[0;10m^[[39;49m
^[[0;10m 1089 root       0 -20     0    0    0 S  0.0  0.0 
  0:00.00 kintegrityd 
                                                            
                                                            
                 ^[[0;10m^[[39;49m
^[[0;10m 1090 root       0 -20     0    0    0 S  0.0  0.0 
  0:00.00 kblockd 
                                                            
                                                            
                     ^[[0;10m^[[39;49m
^[[0;10m 1108 root     -51   0     0    0    0 S  0.0  0.0 
  0:00.00 irq/9-acpi 
                                                            
                                                            
                  ^[[0;10m^[[39;49m
^[[0;10m 1236 root       0 -20     0    0    0 S  0.0  0.0 
  0:00.00 tifm 
                                                            
                                                            
                        ^[[0;10m^[[39;49m
^[[0;10m 1268 root       0 -20     0    0    0 S  0.0  0.0 
  0:00.00 ata_sff 
                                                            
                                                            
                     ^[[0;10m^[[39;49m
^[[0;10m 1280 root      20   0     0    0    0 S  0.0  0.0 
  0:00.04 khubd 
                                                            
                                                            
                       ^[[0;10m^[[39;49m
^[[0;10m 1295 root       0 -20     0    0    0 S  0.0  0.0 
  0:00.00 md 
                                                            
                                                            
                          ^[[0;10m^[[39;49m
^[[0;10m 1420 root       0 -20     0    0    0 S  0.0  0.0 
  0:00.00 rpciod 
                                                            
                                                            
                      ^[[0;10m^[[39;49m
^[[0;10m 1450 root      20   0     0    0    0 S  0.0  0.0 
  0:00.03 khungtaskd 
                                                            
                                                            
                  ^[[0;10m^[[39;49m
^[[0;10m 1456 root      20   0     0    0    0 S  0.0  0.0 
  0:02.47 kswapd0 
                                                            
                                                            
                     ^[[0;10m^[[39;49m
^[[0;10m 1457 root      25   5     0    0    0 S  0.0  0.0 
  0:00.00 ksmd 
                                                            
                                                            
                        ^[[0;10m^[[39;49m
^[[0;10m 1532 root      39  19     0    0    0 S  0.0  0.0 
  0:06.17 khugepaged 
                                                            
                                                            
                  ^[[0;10m^[[39;49m
^[[0;10m 1533 root      20   0     0    0    0 S  0.0  0.0 
  0:00.00 fsnotify_mark 
                                                            
                                                            
               ^[[0;10m^[[39;49m
^[[0;10m 1572 root      20   0     0    0    0 S  0.0  0.0 
  0:00.00 ecryptfs-kthrea 
                                                            
                                                            
             ^[[0;10m^[[39;49m
^[[0;10m 1575 root       0 -20     0    0    0 S  0.0  0.0 
  0:00.00 nfsiod 
                                                            
                                                            
                      ^[[0;10m^[[39;49m
^[[0;10m 1580 root       0 -20     0    0    0 S  0.0  0.0 
  0:00.00 cifsiod 
                                                            
                                                            
                     ^[[0;10m^[[39;49m
^[[0;10m 1606 root      20   0     0    0    0 S  0.0  0.0 
  0:00.00 jfsIO 
                                                            
                                                            
                       ^[[0;10m^[[39;49m
^[[0;10m 1607 root      20   0     0    0    0 S  0.0  0.0 
  0:00.00 jfsCommit 
                                                            
                                                            
                   ^[[0;10m^[[39;49m
^[[0;10m 1608 root      20   0     0    0    0 S  0.0  0.0 
  0:00.00 jfsSync 
                                                            
                                                            
                     ^[[0;10m^[[39;49m
^[[0;10m 1615 root       0 -20     0    0    0 S  0.0  0.0 
  0:00.00 xfsalloc 
                                                            
                                                            
                    ^[[0;10m^[[39;49m
^[[0;10m 1616 root       0 -20     0    0    0 S  0.0  0.0 
  0:00.00 xfs_mru_cache 
                                                            
                                                            
               ^[[0;10m^[[39;49m
^[[0;10m 1617 root       0 -20     0    0    0 S  0.0  0.0 
  0:00.00 xfslogd 
                                                            
                                                            
                     ^[[0;10m^[[39;49m
^[[0;10m 1624 root       0 -20     0    0    0 S  0.0  0.0 
  0:00.00 ocfs2_wq 
                                                            
                                                            
                    ^[[0;10m^[[39;49m
^[[0;10m 1627 root       0 -20     0    0    0 S  0.0  0.0 
  0:00.00 user_dlm 
                                                            
                                                            
                    ^[[0;10m^[[39;49m
^[[0;10m 1633 root       0 -20     0    0    0 S  0.0  0.0 
  0:00.00 glock_workqueue 
                                                            
                                                            
             ^[[0;10m^[[39;49m
^[[0;10m 1634 root       0 -20     0    0    0 S  0.0  0.0 
  0:00.00 delete_workqueu 
                                                            
                                                            
             ^[[0;10m^[[39;49m
^[[0;10m 1638 root       0 -20     0    0    0 S  0.0  0.0 
  0:00.00 gfs_recovery 
                                                            
                                                            
                ^[[0;10m^[[39;49m
^[[0;10m 1642 root       0 -20     0    0    0 S  0.0  0.0 
  0:00.00 crypto 
                                                            
                                                            
                      ^[[0;10m^[[39;49m
^[[0;10m 1780 root      20   0     0    0    0 S  0.0  0.0 
  0:00.23 kworker/u:2 
                                                            
                                                            
                 ^[[0;10m^[[39;49m
^[[0;10m 2408 root      20   0     0    0    0 S  0.0  0.0 
  0:00.00 cciss_scan 
                                                            
                                                            
                  ^[[0;10m^[[39;49m
^[[0;10m 2542 root       0 -20     0    0    0 S  0.0  0.0 
  0:00.00 iscsi_eh 
                                                            
                                                            
                    ^[[0;10m^[[39;49m
^[[0;10m 2551 root       0 -20     0    0    0 S  0.0  0.0 
  0:00.00 kmpath_rdacd 
                                                            
                                                            
                ^[[0;10m^[[39;49m
^[[0;10m 2552 root       0 -20     0    0    0 S  0.0  0.0 
  0:00.00 fc_exch_workque 
                                                            
                                                            
             ^[[0;10m^[[39;49m
^[[0;10m 2553 root       0 -20     0    0    0 S  0.0  0.0 
  0:00.00 fc_rport_eq 
                                                            
                                                            
                 ^[[0;10m^[[39;49m
^[[0;10m 2554 root       0 -20     0    0    0 S  0.0  0.0 
  0:00.00 fcoethread/0 
                                                            
                                                            
                ^[[0;10m^[[39;49m
^[[0;10m 2556 root       0 -20     0    0    0 S  0.0  0.0 
  0:00.00 fnic_event_wq 
                                                            
                                                            
               ^[[0;10m^[[39;49m
^[[0;10m 2558 root       0 -20     0    0    0 S  0.0  0.0 
  0:00.00 bnx2fc_l2_threa 
                                                            
                                                            
             ^[[0;10m^[[39;49m
^[[0;10m 2559 root       0 -20     0    0    0 S  0.0  0.0 
  0:00.00 bnx2fc_thread/0 
                                                            
                                                            
             ^[[0;10m^[[39;49m
^[[0;10m 2636 root       0 -20     0    0    0 S  0.0  0.0 
  0:00.00 bnx2i_thread/0 
                                                            
                                                            
              ^[[0;10m^[[39;49m^[[6;1H^[[K^[[H^[[0;10mtop - 
16:20:14 up 2 days, 16:52,  5 users,  load average: 2.14, 
1.56, 1.12^[[0;10m^[[39;49m^[[K

Cpu(s):^[[0;10m^[[39;49m^[[0;10m 
77.7%^[[0;10m^[[39;49mus,^[[0;10m^[[39;49m^[[0;10m 
22.3%^[[0;10m^[[39;49msy,^[[0;10m^[[39;49m^[[0;10m 
 0.0%^[[0;10m^[[39;49mni,^[[0;10m^[[39;49m^[[0;10m 
 0.0%^[[0;10m^[[39;49mid,^[[0;10m^[[39;49m^[[0;10m 
 0.0%^[[0;10m^[[39;49mwa,^[[0;10m^[[39;49m^[[0;10m 
 0.0%^[[0;10m^[[39;49mhi,^[[0;10m^[[39;49m^[[0;10m 
 0.0%^[[0;10m^[[39;49msi,^[[0;10m^[[39;49m^[[0;10m 
 0.0%^[[0;10m^[[39;49mst^[[0;10m^[[39;49m^[[K
Mem: ^[[0;10m^[[39;49m^[[0;10m  3088984k 
^[[0;10m^[[39;49mtotal,^[[0;10m^[[39;49m^[[0;10m  2861540k 
^[[0;10m^[[39;49mused,^[[0;10m^[[39;49m^[[0;10m   227444k 
^[[0;10m^[[39;49mfree,^[[0;10m^[[39;49m^[[0;10m   286876k 
^[[0;10m^[[39;49mbuffers^[[0;10m^[[39;49m^[[K
Swap:^[[0;10m^[[39;49m^[[0;10m        0k 
^[[0;10m^[[39;49mtotal,^[[0;10m^[[39;49m^[[0;10m        0k 
^[[0;10m^[[39;49mused,^[[0;10m^[[39;49m^[[0;10m        0k 
^[[0;10m^[[39;49mfree,^[[0;10m^[[39;49m^[[0;10m  1780796k 
^[[0;10m^[[39;49mcached^[[0;10m^[[39;49m^[[K
^[[6;1H

^[[0;10m^[[0;10m31801 root      20   0  577m 337m  48m R 
75.7 11.2 158:36.91 firefox-bin 
                                                            
                                                            
                 ^[[0;10m^[[39;49m
^[[0;10m^[[0;10m 5534 root      30  10     0    0    0 R 
 9.5  0.0  90:29.49 kmemleak 
                                                            
                                                            
                    ^[[0;10m^[[39;49m
^[[0;10m10227 root      20   0 57396  31m  27m S  6.2  1.1 
256:55.60 kded 
                                                            
                                                            
                        ^[[0;10m^[[39;49m
^[[0;10m10256 root      20   0 94484  42m 7548 S  3.6  1.4 
 31:17.12 artsd 
                                                            
                                                            
                       ^[[0;10m^[[39;49m
^[[0;10m22802 root      20   0  2624 1132  828 S  1.3  0.0 
  0:05.99 top 
                                                            
                                                            
                         ^[[0;10m^[[39;49m
^[[0;10m^[[0;10m26642 root      20   0  2624 1132  828 R 
 1.0  0.0   0:00.13 top 
                                                            
                                                            
                         ^[[0;10m^[[39;49m
^[[0;10m24974 root      20   0  2620 1196  896 S  0.7  0.0 
  0:02.47 top 
                                                            
                                                            
                         ^[[0;10m^[[39;49m
^[[0;10m24935 root      20   0  6088 2676 1848 S  0.3  0.1 
  0:00.66 mrxvt 
                                                            
                                                            
                       ^[[0;10m^[[39;49m
^[[0;10m    1 root      20   0   824  320  280 S  0.0  0.0 
  0:35.88 init 
                                                            
                                                            
                        ^[[0;10m^[[39;49m
^[[0;10m    2 root      20   0     0    0    0 S  0.0  0.0 
  0:01.19 kthreadd 
                                                            
                                                            
                    ^[[0;10m^[[39;49m
^[[0;10m    3 root      20   0     0    0    0 S  0.0  0.0 
  6:12.82 ksoftirqd/0 
                                                            
                                                            
                 ^[[0;10m^[[39;49m
^[[0;10m    6 root      RT   0     0    0    0 S  0.0  0.0 
  0:00.00 migration/0 
                                                            
                                                            
                 ^[[0;10m^[[39;49m
^[[0;10m    7 root      RT   0     0    0    0 S  0.0  0.0 
  0:05.35 watchdog/0 
                                                            
                                                            
                  ^[[0;10m^[[39;49m
^[[0;10m    8 root       0 -20     0    0    0 S  0.0  0.0 
  0:00.00 khelper 
                                                            
                                                            
                     ^[[0;10m^[[39;49m
^[[0;10m    9 root      20   0     0    0    0 S  0.0  0.0 
  0:00.03 kdevtmpfs 
                                                            
                                                            
                   ^[[0;10m^[[39;49m
^[[0;10m   10 root       0 -20     0    0    0 S  0.0  0.0 
  0:00.00 netns 
                                                            
                                                            
                       ^[[0;10m^[[39;49m
^[[0;10m 1086 root      20   0     0    0    0 S  0.0  0.0 
  0:00.47 sync_supers 
                                                            
                                                            
                 ^[[0;10m^[[39;49m
^[[0;10m 1088 root      20   0     0    0    0 S  0.0  0.0 
  0:00.00 bdi-default 
                                                            
                                                            
                 ^[[0;10m^[[39;49m
^[[0;10m 1089 root       0 -20     0    0    0 S  0.0  0.0 
  0:00.00 kintegrityd 
                                                            
                                                            
                 ^[[0;10m^[[39;49m
^[[0;10m 1090 root       0 -20     0    0    0 S  0.0  0.0 
  0:00.00 kblockd 
                                                            
                                                            
                     ^[[0;10m^[[39;49m
^[[0;10m 1108 root     -51   0     0    0    0 S  0.0  0.0 
  0:00.00 irq/9-acpi 
                                                            
                                                            
                  ^[[0;10m^[[39;49m
^[[0;10m 1236 root       0 -20     0    0    0 S  0.0  0.0 
  0:00.00 tifm 
                                                            
                                                            
                        ^[[0;10m^[[39;49m
^[[0;10m 1268 root       0 -20     0    0    0 S  0.0  0.0 
  0:00.00 ata_sff 
                                                            
                                                            
                     ^[[0;10m^[[39;49m
^[[0;10m 1280 root      20   0     0    0    0 S  0.0  0.0 
  0:00.04 khubd 
                                                            
                                                            
                       ^[[0;10m^[[39;49m
^[[0;10m 1295 root       0 -20     0    0    0 S  0.0  0.0 
  0:00.00 md 
                                                            
                                                            
                          ^[[0;10m^[[39;49m
^[[0;10m 1420 root       0 -20     0    0    0 S  0.0  0.0 
  0:00.00 rpciod 
                                                            
                                                            
                      ^[[0;10m^[[39;49m
^[[0;10m 1450 root      20   0     0    0    0 S  0.0  0.0 
  0:00.03 khungtaskd 
                                                            
                                                            
                  ^[[0;10m^[[39;49m
^[[0;10m 1456 root      20   0     0    0    0 S  0.0  0.0 
  0:02.47 kswapd0 
                                                            
                                                            
                     ^[[0;10m^[[39;49m
^[[0;10m 1457 root      25   5     0    0    0 S  0.0  0.0 
  0:00.00 ksmd 
                                                            
                                                            
                        ^[[0;10m^[[39;49m
^[[0;10m 1532 root      39  19     0    0    0 S  0.0  0.0 
  0:06.17 khugepaged 
                                                            
                                                            
                  ^[[0;10m^[[39;49m
^[[0;10m 1533 root      20   0     0    0    0 S  0.0  0.0 
  0:00.00 fsnotify_mark 
                                                            
                                                            
               ^[[0;10m^[[39;49m
^[[0;10m 1572 root      20   0     0    0    0 S  0.0  0.0 
  0:00.00 ecryptfs-kthrea 
                                                            
                                                            
             ^[[0;10m^[[39;49m
^[[0;10m 1575 root       0 -20     0    0    0 S  0.0  0.0 
  0:00.00 nfsiod 
                                                            
                                                            
                      ^[[0;10m^[[39;49m
^[[0;10m 1580 root       0 -20     0    0    0 S  0.0  0.0 
  0:00.00 cifsiod 
                                                            
                                                            
                     ^[[0;10m^[[39;49m
^[[0;10m 1606 root      20   0     0    0    0 S  0.0  0.0 
  0:00.00 jfsIO 
                                                            
                                                            
                       ^[[0;10m^[[39;49m
^[[0;10m 1607 root      20   0     0    0    0 S  0.0  0.0 
  0:00.00 jfsCommit 
                                                            
                                                            
                   ^[[0;10m^[[39;49m
^[[0;10m 1608 root      20   0     0    0    0 S  0.0  0.0 
  0:00.00 jfsSync 
                                                            
                                                            
                     ^[[0;10m^[[39;49m
^[[0;10m 1615 root       0 -20     0    0    0 S  0.0  0.0 
  0:00.00 xfsalloc 
                                                            
                                                            
                    ^[[0;10m^[[39;49m
^[[0;10m 1616 root       0 -20     0    0    0 S  0.0  0.0 
  0:00.00 xfs_mru_cache 
                                                            
                                                            
               ^[[0;10m^[[39;49m
^[[0;10m 1617 root       0 -20     0    0    0 S  0.0  0.0 
  0:00.00 xfslogd 
                                                            
                                                            
                     ^[[0;10m^[[39;49m
^[[0;10m 1624 root       0 -20     0    0    0 S  0.0  0.0 
  0:00.00 ocfs2_wq 
                                                            
                                                            
                    ^[[0;10m^[[39;49m
^[[0;10m 1627 root       0 -20     0    0    0 S  0.0  0.0 
  0:00.00 user_dlm 
                                                            
                                                            
                    ^[[0;10m^[[39;49m
^[[0;10m 1633 root       0 -20     0    0    0 S  0.0  0.0 
  0:00.00 glock_workqueue 
                                                            
                                                            
             ^[[0;10m^[[39;49m
^[[0;10m 1634 root       0 -20     0    0    0 S  0.0  0.0 
  0:00.00 delete_workqueu 
                                                            
                                                            
             ^[[0;10m^[[39;49m
^[[0;10m 1638 root       0 -20     0    0    0 S  0.0  0.0 
  0:00.00 gfs_recovery 
                                                            
                                                            
                ^[[0;10m^[[39;49m
^[[0;10m 1642 root       0 -20     0    0    0 S  0.0  0.0 
  0:00.00 crypto 
                                                            
                                                            
                      ^[[0;10m^[[39;49m
^[[0;10m 1780 root      20   0     0    0    0 S  0.0  0.0 
  0:00.23 kworker/u:2 
                                                            
                                                            
                 ^[[0;10m^[[39;49m
^[[0;10m 2408 root      20   0     0    0    0 S  0.0  0.0 
  0:00.00 cciss_scan 
                                                            
                                                            
                  ^[[0;10m^[[39;49m
^[[0;10m 2542 root       0 -20     0    0    0 S  0.0  0.0 
  0:00.00 iscsi_eh 
                                                            
                                                            
                    ^[[0;10m^[[39;49m
^[[0;10m 2551 root       0 -20     0    0    0 S  0.0  0.0 
  0:00.00 kmpath_rdacd 
                                                            
                                                            
                ^[[0;10m^[[39;49m
^[[0;10m 2552 root       0 -20     0    0    0 S  0.0  0.0 
  0:00.00 fc_exch_workque 
                                                            
                                                            
             ^[[0;10m^[[39;49m
^[[0;10m 2553 root       0 -20     0    0    0 S  0.0  0.0 
  0:00.00 fc_rport_eq 
                                                            
                                                            
                 ^[[0;10m^[[39;49m
^[[0;10m 2554 root       0 -20     0    0    0 S  0.0  0.0 
  0:00.00 fcoethread/0 
                                                            
                                                            
                ^[[0;10m^[[39;49m
^[[0;10m 2556 root       0 -20     0    0    0 S  0.0  0.0 
  0:00.00 fnic_event_wq 
                                                            
                                                            
               ^[[0;10m^[[39;49m
^[[0;10m 2558 root       0 -20     0    0    0 S  0.0  0.0 
  0:00.00 bnx2fc_l2_threa 
                                                            
                                                            
             ^[[0;10m^[[39;49m
^[[0;10m 2559 root       0 -20     0    0    0 S  0.0  0.0 
  0:00.00 bnx2fc_thread/0 
                                                            
                                                            
             ^[[0;10m^[[39;49m
^[[0;10m 2636 root       0 -20     0    0    0 S  0.0  0.0 
  0:00.00 bnx2i_thread/0 
                                                            
                                                            
              ^[[0;10m^[[39;49m
^[[0;10m 2675 root      20   0     0    0    0 S  0.0  0.0 
  0:00.39 scsi_eh_2 
                                                            
                                                            
                   ^[[0;10m^[[39;49m^[[6;1H^[[K^[[66;1H^[[?25h^[[?0c


---
Professional hosting for everyone - http://www.host.ru

Re: v3.4-rc2 out-of-memory problems (was Re: 3.4-rc1 sticks-and-crashs)

[Rik van Riel]

On 04/14/2012 03:38 PM, werner wrote:
> Perhaps someone could check this
>
> Ocasionally it still happens that the computer slows down very, during
> appr. 1 minute (but it dont crash, nor is nothing visible in syslog).
>
> I think this wasn't before approx. the 3.2 kernel, at least I didn't
> perceive it, but this happens at 3.3 and 3.4-rc2-inclusive all patchs.
>
> Below, ps alx and top i printed in a file at an ocasion as this happened
> I observed that then always kmemleak is almost at the beginning of top

I have a feeling you might enjoy things like the sysprof
tool, which can show you not only which process is using
CPU time, but also where it is spent.

-- 
All rights reversed

Re: v3.4-rc2 out-of-memory problems (was Re: 3.4-rc1 sticks-and-crashs)

[Srivatsa S. Bhat]

On 04/10/2012 05:25 AM, Linus Torvalds wrote:

> On Mon, Apr 9, 2012 at 4:25 PM, David Rientjes <rientjes@google.com> wrote:
>>
>> You could that if you also turned the check for "ret == NOTIFY_OK" in
>> profile_handoff_task() into "ret & NOTIFY_OK" in your patch, otherwise you
>> get a double free from __put_task_struct() and oprofile.
> 
> Why? NOTIFY_DONE is zero.
> 
> I do agree that we *also* could do the "& NOTIFY_OK" and make it
> clearer that we're oring bits together. And we could document the
> stupid notifier interfaces to do this all, and just make the rules be
> *sane* when you have multiple notifiers.
> 
> And sane rules would be either:
> 
>  - you always return an error return, and notifiers all return either
> 0 or a negative error number, and we stop on the first error and
> return that.
> 
>  - you return a bitmask, and we or all bits together (and we can
> certainly continue to have a "stop here" bit)
> 


Even I think 'or'ing the bits makes more sense than returning the last
return value.

CPU hotplug and suspend/resume are two of the things that I know of,
that use notifiers quite a bit. However, neither of them actually care
about the exact return value - if it is an error return, no matter which
one or for what reason, they do the same error handling; and it works
for them. IOW, if we change the documented behaviour of notifiers to
return 'or' of all return values, that would continue to work well
with these users.

Of course, there are other users like profile_handoff_task() that do
care about exactly what the return value was, but I guess we can
gradually adapt such users to the better, saner rules for the notifier
return values, as you proposed.

> But the current notifier semantics are just insane. The whole "we
> return the last return value" is crazy. It's by definition a random
> number, since the whole point of notifiers is that there can be
> multiple, and they aren't "ordered". So the whole "last return value"
> is something I just look at and say: "Whoever designed that is a
> f*cking moron".
> 

[...]

> 
> Again, almost every notifier user has always been total crap. It's
> just a stupid abstraction.



> "Something happened". "Oh, ok".
> 


Never saw such a concise and apt definition of notifiers before ;-)

However, unfortunately, what other better mechanism do we have, to
deal with things that affect stuff across multiple subsystems, like
some of the users mentioned above?  Hmm...

Regards,
Srivatsa S. Bhat

Re: v3.4-rc2 out-of-memory problems (was Re: 3.4-rc1 sticks-and-crashs)

[Linus Torvalds]

On Sat, Apr 14, 2012 at 12:38 PM, werner <w.landgraf@ru.ru> wrote:
>
> I observed that then always kmemleak is almost at the beginning of top

Just disable CONFIG_DEBUG_KMEMLEAK.

You do realize that it's a pure debugging aid - akin to
CONFIG_DEBUG_PAGEALLOC - that is *not* meant to be run on production
systems. Exactly because it is very expensive. It basically scans all
the kernel areas for pointers and tries to find leaks.

                 Linus