Re: [PATCH RFC tip/core/rcu 11/11] rcu: move TREE_RCU from softirq to kthread

[Lai Jiangshan <laijs@cn.fujitsu.com>]

On 03/01/2011 07:51 AM, Paul E. McKenney wrote:
> On Mon, Feb 28, 2011 at 11:29:44AM +0800, Lai Jiangshan wrote:
>> On 02/26/2011 04:32 AM, Paul E. McKenney wrote:
>>>>> +/*
>>>>> + * Handle cases where the rcu_cpu_kthread() ends up on the wrong CPU.
>>>>> + * This can happen while the corresponding CPU is either coming online
>>>>> + * or going offline.  We cannot wait until the CPU is fully online
>>>>> + * before starting the kthread, because the various notifier functions
>>>>> + * can wait for RCU grace periods.  So we park rcu_cpu_kthread() until
>>>>> + * the corresponding CPU is online.
>>>>> + *
>>>>> + * Return 1 if the kthread needs to stop, 0 otherwise.
>>>>> + *
>>>>> + * Caller must disable bh.  This function can momentarily enable it.
>>>>> + */
>>>>> +static int rcu_cpu_kthread_should_stop(int cpu)
>>>>> +{
>>>>> +	while (cpu_is_offline(cpu) || smp_processor_id() != cpu) {
>>>>> +		if (kthread_should_stop())
>>>>> +			return 1;
>>>>> +		local_bh_enable();
>>>>> +		schedule_timeout_uninterruptible(1);
>>>>> +		if (smp_processor_id() != cpu)
>>>>> +			set_cpus_allowed_ptr(current, cpumask_of(cpu));
>>>>
>>>> The current task is PF_THREAD_BOUND,
>>>> Why do "set_cpus_allowed_ptr(current, cpumask_of(cpu));" ?
>>>
>>> Because I have seen CPU hotplug operations unbind PF_THREAD_BOUND threads.
>>> In addition, I end up having to spawn the kthread at CPU_UP_PREPARE time,
>>> at which point the thread must run unbound because its CPU isn't online
>>> yet.  I cannot invoke kthread_create() within the stop-machine handler
>>> (right?).  I cannot wait until CPU_ONLINE time because that results in
>>> hangs when other CPU notifiers wait for grace periods.
>>>
>>> Yes, I did find out about the hangs the hard way.  Why do you ask?  ;-)
>>
>> The current task is PF_THREAD_BOUND, "set_cpus_allowed_ptr(current, cpumask_of(cpu))"
>> will do nothing even it runs on the wrong CPU.
> 
> You lost me on this one.
> 
> Looking at set_cpus_allowed_ptr()...
> 
> The "again" loop won't happen because the task is already running.
> The CPU is online, so the cpumask_intersects() check won't kick
> us out.  We are working with the current task, so the check for
> PF_THREAD_BOUND, current, and cpumask_equal() won't kick us out.
> If the old and new cpumasks had been the same, we would not have called
> set_cpus_allowed_ptr() in the first place.  So we should get to
> the call to migrate_task().
> 
> What am I missing here?

You right. I forgot current tasks can change its cpumask even PF_THREAD_BOUND.