Re: Fw: Re: [mm PATCH 4/6] RCU: (now) CPU hotplug

[Dipankar Sarma <dipankar@in.ibm.com>]

On Thu, Jan 25, 2007 at 02:36:45AM +0530, Dipankar Sarma wrote:
> On Wed, Jan 24, 2007 at 08:15:59AM -0800, Paul E. McKenney wrote:
> 
> It should be relatively easy. Setting the offlined cpu's flags
> to neutral state should do the trick in most cases.
> I will send out the patches tomorrow after reviewing the code
> some more.

Famous last words. 

It turns out that I have been bitten by the ugly cpu hotplug
locking mess while trying to get preemptible RCU code working
with CPU hotplug. The per-subsystem locking thing isn't really
user-friendly. Here is the dependency -

In cpu hotplug path (after CPU_LOCK_ACQUIRE) -

CPU_DOWN_PREPARE:sched domains -> detach_destroy_domains() ->
			synchronize_sched() -> sched_setaffinity()

sched_setaffinity() tries to acquire the scheduler cpu hotplug
mutex and deadlocks.

I see no easy way of getting around this - doing "cpu hotplug locked"
version of all those APIs would add lots of code which is bad.
We could try Gautham's idea of letting  each subsystem maintain
its own online cpu mask, but I bet implementing sched_setaffinity()
would not be very easy despite this.

What is the status on this now ? Is this a good example to
show why per-subsystem locks might be unmaintainable ? Can
we go back to a simple scalable refcount model
for CPU hotplug now ?

Thanks
Dipankar