Re: CPU Hotplug broken -mm5 onwards

[Nick Piggin <nickpiggin@yahoo.com.au>]

Srivatsa Vaddagiri wrote:
> Hi,
> 	I found that I can't boot with CONFIG_HOTPLUG_CPU defined in both
> mm5 and mm6. Debugging this revealed it to be because exec path can now require 
> cpu hotplug sem (sched_migrate_task) and this has lead to a deadlock between 
> flush_workqueue and __call_usermodehelper. 
> 
> flush_workqueue takes cpu hotplug sem and blocks until workqueue is flushed.
> __call_usermodehelper, one of the queued work function, blocks because it
> also needs cpu hotplug sem during exec.  As of result of this, exec does not 
> progress and system does not boot.
> 
> I feel we can fix this by converting cpucontrol to a reader-writer semaphore or 
> big-reader-lock(?). One problem with reader-writer semaphore is there does not
> seem to be any down_write_interruptible, which is needed by cpu_down/up.
> 
> Comments?
> 

You are right, but it wasn't introduced in -mm or sched-domains
patches. However, one of Ingo's recent patches does balance on
exec for SMP, not just NUMA so it will make this more common.

So, Rusty has to fix it ;)

I think a rwsem might be a good idea anyway, because
sched_migrate_task can end up being called pretty often with
balance on exec and balance on clone. The semaphore could easily
place undue serialisation on that path.

> BTW, I think a cpu_is_offline check is needed in sched_migrate_task, since
> dest_cpu could have been downed by the time it has acquired the semaphore. 
> In which case, we could end up adding the task to dead cpu's runqueue?
> An alternate solution would be to put the same check in __migrate_task.
> 

Yes you are correct.

Can we arrange some of these checks to disappear when HOTPLUG_CPU
is not set? For example, make cpu_is_offline only valid to call for
CPUs that have been online sometime, and can evaluate to 0 if
HOTPLUG_CPU is not set?