Gracefully killing kswapd, or any kernel thread

Gracefully killing kswapd, or any kernel thread

[Kristis Makris]

Hello,

I'm trying to kill a kernel thread gracefully, in particular kswapd,
without any success.

The goal is to start another kernel thread that contains updated kswapd
functionality, through a loadable module; no kernel recompilation.

I noticed that kernel threads block SIGKILL. Hence, on module load I'm
running:

task = find_task_by_name("kswapd");
if (task != NULL) {
    spin_lock_irq(&task->sigmask_lock);
    sigdelset(&task->blocked, SIGKILL);
    recalc_sigpending(task);
    spin_unlock_irq(&task->sigmask_lock);
    // Also tried issuing here a: kill_proc(task->pid, SIGKILL, 1);
}

Then from userspace I issue:

# ps aux |grep -i swap
root         4  0.0  0.0     0    0 ?        SW   18:36   0:00 [kswapd]
$ kill -9 4

After the kill is issued, kswapd is taking up 99.9% of CPU time and
remains at a runnable state:
# ps aux |grep -i swap
root         4  0.2  0.0     0    0 ?        RW   18:36   0:02 [kswapd]


Can anyone explain why this is happening ? I've tried this with linux
kernels 2.2.19 and 2.4.27 (with patch kdb-4.3). What is the proper way
of gracefully killing a kernel thread launched from the original kernel
image (not a module) in kernels < 2.6 (ie. without the new kernel thread
API that contains the stop_kthread call documented in
http://www.scs.ch/~frey/linux/kernelthreads.html)

I've also tried the same with kflushd, kupdate, and keventd in 2.2.19.
When I do issue a "kill -9" for them I see:

# ps aux
USER       PID %CPU %MEM   VSZ  RSS TTY      STAT START   TIME COMMAND
root         2  0.0  0.0     0    0 ?        SW   12:18   0:00 [kflushd]
root         3  1.5  0.0     0    0 ?        RW   12:18   0:16 [kupdate]
root         5  0.0  0.0     0    0 ?        SW   12:18   0:00 [keventd]

All 3 kernel threads remain in the process list. kupdate also appears to
be in a running state consuming 99.9% of the CPU when killed. What's so
special about kupdate and kswapd that makes them stay at a running
state, and why do all kernel threads seem unkillable?

Thanks,
Kristis

Re: Gracefully killing kswapd, or any kernel thread

[linux-os (Dick Johnson)]

On Wed, 7 Sep 2005, Kristis Makris wrote:

> Hello,
>
> I'm trying to kill a kernel thread gracefully, in particular kswapd,
> without any success.
>
> The goal is to start another kernel thread that contains updated kswapd
> functionality, through a loadable module; no kernel recompilation.
>
> I noticed that kernel threads block SIGKILL. Hence, on module load I'm
> running:
>
> task = find_task_by_name("kswapd");
> if (task != NULL) {
>    spin_lock_irq(&task->sigmask_lock);
>    sigdelset(&task->blocked, SIGKILL);
>    recalc_sigpending(task);
>    spin_unlock_irq(&task->sigmask_lock);
>    // Also tried issuing here a: kill_proc(task->pid, SIGKILL, 1);
> }
>
> Then from userspace I issue:
>
> # ps aux |grep -i swap
> root         4  0.0  0.0     0    0 ?        SW   18:36   0:00 [kswapd]
> $ kill -9 4
>
> After the kill is issued, kswapd is taking up 99.9% of CPU time and
> remains at a runnable state:
> # ps aux |grep -i swap
> root         4  0.2  0.0     0    0 ?        RW   18:36   0:02 [kswapd]
>
>
> Can anyone explain why this is happening ? I've tried this with linux
> kernels 2.2.19 and 2.4.27 (with patch kdb-4.3). What is the proper way
> of gracefully killing a kernel thread launched from the original kernel
> image (not a module) in kernels < 2.6 (ie. without the new kernel thread
> API that contains the stop_kthread call documented in
> http://www.scs.ch/~frey/linux/kernelthreads.html)
>
> I've also tried the same with kflushd, kupdate, and keventd in 2.2.19.
> When I do issue a "kill -9" for them I see:
>
> # ps aux
> USER       PID %CPU %MEM   VSZ  RSS TTY      STAT START   TIME COMMAND
> root         2  0.0  0.0     0    0 ?        SW   12:18   0:00 [kflushd]
> root         3  1.5  0.0     0    0 ?        RW   12:18   0:16 [kupdate]
> root         5  0.0  0.0     0    0 ?        SW   12:18   0:00 [keventd]
>
> All 3 kernel threads remain in the process list. kupdate also appears to
> be in a running state consuming 99.9% of the CPU when killed. What's so
> special about kupdate and kswapd that makes them stay at a running
> state, and why do all kernel threads seem unkillable?
>
> Thanks,
> Kristis

To kill a kernel thread, you need to make __it__ call exit(). It must be
CODED to do that! You can't do it externally although you can send
it a signal, after which it will spin forever....

// Main loop of the thread

     for(;;)
     {
         set_current_state(TASK_INTERRUPTIBLE);
         if(signal_pending(current))
             complete_and_exit(&info->quit, 0);  // <--- HERE!!!
         interruptible_sleep_on(&info->twait);
         do_work();
     }

// Start a thread

     info->pid = kernel_thread(local_thread, NULL, CLONE_FS|CLONE_FILES);



// Stop a thread

     if(info->pid)
     {
         (void)kill_proc(info->pid, SIGTERM, 1);
         wait_for_completion(&info->quit);   // MUST wait!
     }

With newer kernels, we use the allow_signal(SIGNAME) macro as one
of the first things executed to allow the kernel thread to respond
to a signal.

Cheers,
Dick Johnson
Penguin : Linux version 2.6.13 on an i686 machine (5589.51 BogoMips).
Warning : 98.36% of all statistics are fiction.
.
I apologize for the following. I tried to kill it with the above dot :

****************************************************************
The information transmitted in this message is confidential and may be privileged.  Any review, retransmission, dissemination, or other use of this information by persons or entities other than the intended recipient is prohibited.  If you are not the intended recipient, please notify Analogic Corporation immediately - by replying to this message or by sending an email to DeliveryErrors@analogic.com - and destroy all copies of this information, including any attachments, without reading or disclosing them.

Thank you.

Re: [ham] Re: Gracefully killing kswapd, or any kernel thread

[Kristis Makris]

> To kill a kernel thread, you need to make __it__ call exit(). It must be

There must be another way to do it. Perhaps one could have another
process effectively issue the contents of do_exit for the kswapd
task_struct ?

> CODED to do that! You can't do it externally although you can send

I'm clearly asking for the case where the thread wasn't coded to do
that.

> it a signal, after which it will spin forever....

kflushd and keventd don't seem to spin forever. I still haven't
determined what makes kswapd spin forever after it receives the signal.

Re: [ham] Re: Gracefully killing kswapd, or any kernel thread

[Kyle Moffett]

On Sep 7, 2005, at 17:07:12, Kristis Makris wrote:
>> To kill a kernel thread, you need to make __it__ call exit(). It  
>> must be
>
> There must be another way to do it. Perhaps one could have another
> process effectively issue the contents of do_exit for the kswapd
> task_struct ?

Umm, so then the kernel does what, exactly?  You have a process in some
indeterminate state, possibly holding semaphores, definitely pinning
memory/resources/etc, and you just stop it, turn it off, and expect
things to continue working?  This is similar in nature to that thread
a while ago about kernel error recovery and killing uninterruptible
user processes.  To extend this to kernel threads, unless the kernel
thread has been _specifically_ coded to be interruptible, it isn't, and
furthermore, *can't* be.

>> CODED to do that! You can't do it externally although you can send
>
> I'm clearly asking for the case where the thread wasn't coded to do
> that.

You can't.  This is flatly impossible.  Go see the thread a while back
about a hot-patch system call for several reasons why that is a bad
idea.  In particular, look at the post that discusses phone switches,
the one with the quote "'So why don't you just reboot the affected
switches?' [...] 'That assumes the switches had ever been booted in
the first place'".

>> it a signal, after which it will spin forever....
>
> kflushd and keventd don't seem to spin forever. I still haven't
> determined what makes kswapd spin forever after it receives the  
> signal.

Probably a while(1) loop that isn't intended to stop until the machine
physically powers off.  If you want to patch one specific kernel thread,
you might be able to do that, but you can't just expect to hot-patch
random parts of the kernel at runtime and have things work.

Cheers,
Kyle Moffett

-----BEGIN GEEK CODE BLOCK-----
Version: 3.12
GCM/CS/IT/U d- s++: a18 C++++>$ UB/L/X/*++++(+)>$ P+++(++++)>$ L++++(+ 
++) E
W++(+) N+++(++) o? K? w--- O? M++ V? PS+() PE+(-) Y+ PGP+++ t+(+++) 5  
X R?
tv-(--) b++++(++) DI+ D+ G e->++++$ h!*()>++$ r  !y?(-)
------END GEEK CODE BLOCK------

Re: [ham] Re: Gracefully killing kswapd, or any kernel thread

[linux-os (Dick Johnson)]

On Wed, 7 Sep 2005, Kristis Makris wrote:

>> To kill a kernel thread, you need to make __it__ call exit(). It must be
>
> There must be another way to do it. Perhaps one could have another
> process effectively issue the contents of do_exit for the kswapd
> task_struct ?
>
>> CODED to do that! You can't do it externally although you can send
>
> I'm clearly asking for the case where the thread wasn't coded to do
> that.
>

It was not clear. You just sent it a signal and expected it to go
away. No tasks, even user-mode tasks, just go away! They need to
execute the exit code in the context of the task that will
expected to eventually disappear. It's contents of "current"
that tells the exit code what to clean up.

This means that you could probably install a driver that patches
some code that the kernel thread executes. Nothing like
modifying code at run-time....

>> it a signal, after which it will spin forever....
>
> kflushd and keventd don't seem to spin forever. I still haven't
> determined what makes kswapd spin forever after it receives the signal.
>

It depends how the signal was masked and how the thread was written.
Notice in the example posted, the code was reset to TASK_INTERRUPTIBLE
each time the loop was executed. If this is not done, the task won't
get interrupted by a signal again.

>
>

Cheers,
Dick Johnson
Penguin : Linux version 2.6.13 on an i686 machine (5589.51 BogoMips).
Warning : 98.36% of all statistics are fiction.
.
I apologize for the following. I tried to kill it with the above dot :

****************************************************************
The information transmitted in this message is confidential and may be privileged.  Any review, retransmission, dissemination, or other use of this information by persons or entities other than the intended recipient is prohibited.  If you are not the intended recipient, please notify Analogic Corporation immediately - by replying to this message or by sending an email to DeliveryErrors@analogic.com - and destroy all copies of this information, including any attachments, without reading or disclosing them.

Thank you.

Re: [ham] Re: Gracefully killing kswapd, or any kernel thread

[Kristis Makris]

On Wed, 2005-09-07 at 18:36 -0400, linux-os (Dick Johnson) wrote:
> On Wed, 7 Sep 2005, Kristis Makris wrote:
> 
> >> To kill a kernel thread, you need to make __it__ call exit(). It must be

I was able to make it call do_exit(). However, even if I recompile a
kernel to have kswapd issued a do_exit(), I still see from ps:

root         4  0.0  0.0     0    0 ?        Z    06:20   0:00 [kswapd
<defunct>]

Why isn't the task_struct for it gone ?

Re: [ham] Re: Gracefully killing kswapd, or any kernel thread

[linux-os (Dick Johnson)]

On Fri, 9 Sep 2005, Kristis Makris wrote:

> On Wed, 2005-09-07 at 18:36 -0400, linux-os (Dick Johnson) wrote:
>> On Wed, 7 Sep 2005, Kristis Makris wrote:
>>
>>>> To kill a kernel thread, you need to make __it__ call exit(). It must be
>
> I was able to make it call do_exit(). However, even if I recompile a
> kernel to have kswapd issued a do_exit(), I still see from ps:
>
> root         4  0.0  0.0     0    0 ?        Z    06:20   0:00 [kswapd
> <defunct>]
>
> Why isn't the task_struct for it gone ?
>

Because it's now defunct <Z>, a zombie, waiting for somebody to
reap its return status. You are almost there, you need to issue
the kernel equivalent of waitpid() (sys_waitpid) to grab that
status and throw it away. That's what the code I showed you
did when it would shut down and remove a module that had
a kernel thread.

Cheers,
Dick Johnson
Penguin : Linux version 2.6.13 on an i686 machine (5589.53 BogoMips).
Warning : 98.36% of all statistics are fiction.
.
I apologize for the following. I tried to kill it with the above dot :

****************************************************************
The information transmitted in this message is confidential and may be privileged.  Any review, retransmission, dissemination, or other use of this information by persons or entities other than the intended recipient is prohibited.  If you are not the intended recipient, please notify Analogic Corporation immediately - by replying to this message or by sending an email to DeliveryErrors@analogic.com - and destroy all copies of this information, including any attachments, without reading or disclosing them.

Thank you.

Re: [ham] Re: Gracefully killing kswapd, or any kernel thread

[Kristis Makris]

On Fri, 2005-09-09 at 15:20 -0400, linux-os (Dick Johnson) wrote:

> Because it's now defunct <Z>, a zombie, waiting for somebody to
> reap its return status. You are almost there, you need to issue
> the kernel equivalent of waitpid() (sys_waitpid) to grab that
> status and throw it away. That's what the code I showed you
> did when it would shut down and remove a module that had
> a kernel thread.

It turns out I had to port reparent_to_init() from 2.4 to 2.2 to get it
going. issuing a sys_wait4 wasn't enough. Apparently there was no way
for the built-in kernel threads to exit at all in the first place. I
hope 2.6 at least issues a reparent_to_init in kernel_thread().