Re: [patch] sched: prevent bound kthreads from changing cpus_allowed

[Max Krasnyansky <maxk@qualcomm.com>]

Peter Zijlstra wrote:
> On Mon, 2008-06-09 at 13:59 -0700, Max Krasnyanskiy wrote:
>> David Rientjes wrote:
>>>>  2) Sometimes calls to kthread_bind are binding to any online cpu, such as in:
>>>>
>>>> 	drivers/infiniband/hw/ehca/ehca_irq.c:	kthread_bind(cct->task, any_online_cpu(cpu_online_map));
>>>>
>>>>     In such cases, the PF_THREAD_BOUND seems inappropriate.  The caller of
>>>>     kthread_bind() really doesn't seem to care where that thread is bound;
>>>>     they just want it on a CPU that is still online.
>>>>
>>> This particular case is simply moving the thread to any online cpu so that 
>>> it survives long enough for the subsequent kthread_stop() in 
>>> destroy_comp_task().  So I don't see a problem with this instance.
>>>
>>> A caller to kthread_bind() can always remove PF_THREAD_BOUND itself upon 
>>> return, but I haven't found any cases in the tree where that is currently 
>>> necessary.  And doing that would defeat the semantics of kthread_bind() 
>>> where these threads are supposed to be bound to a specific cpu and not 
>>> allowed to run on others.
>> Actually I have another use case here. Above example in particular may be ok 
>> but it does demonstrate the issue nicely. Which is that in some cases kthreads 
>> are bound to a CPU but do not have a strict "must run here" requirement and 
>> could be moved if needed.
>> For example I need an ability to move workqueue threads. Workqueue threads do 
>> kthread_bind().
> 
> Per cpu workqueues should stay on their cpu.
> 
> What you're really looking for is a more fine grained alternative to
> flush_workqueue().
Actually I had a discussion on that with Oleg Nesterov. If you remember my
original solution (ie centralized cpu_isolate_map) was to completely redirect
work onto other cpus. Then you pointed out that it's the flush_() that really
makes the box stuck. So I started thinking about redoing the flush. While
looking at the code I realized that if I only change the flush_() then queued
work can get stale so to speak. ie Machine does not get stuck but some work
submitted on the isolated cpus will sit there for a long time. Oleg pointed
out exact same thing. So the simplest solution that does not require any
surgery to the workqueue is to just move the threads to other cpus. I did not
want to get into too much detail on the workqueue stuff here. I'll start a
separate thread on this.
As I pointed out, there are a bunch of other kthreads like: kswapd, kacpid,
pdflush, khubd, etc, etc, that clearly do not need any pinning but still
violate cpuset constraints they inherit from kthreadd.

Max