Re: [PATCH 5/8] cpu: cpu-hotplug deadlock

[Peter Zijlstra <a.p.zijlstra@chello.nl>]

On Tue, 2008-04-29 at 20:45 +0400, Oleg Nesterov wrote:
> On 04/29, Peter Zijlstra wrote:
> >
> > The only thing that changed is that the mutex is not held; so what we
> > change is:
> > 
> >  LOCK
> > 
> >  ... do the full hotplug thing ...
> > 
> >  UNLOCK
> > 
> > into
> > 
> >  LOCK
> >   set state
> >  UNLOCK
> > 
> >  ... do the full hotplug thing ...
> > 
> >  LOCK
> >   unset state
> >  UNLOCK
> > 
> > So that the lock isn't held over the hotplug operation.
> 
> Well, yes I see, but... Ugh, I have a a blind spot here ;)
> 
> why this makes any difference from the semantics POV ? why it is bad
> to hold the mutex throughout the "full hotplug thing" ?

Darn, now you make me think ;-)

Ok, I think I have it; the crux of the matter is that we want
reader-in-writer recursion for the cpu hotplug lock.

So we want:

 cpu_hotplug.write_lock()
   A.lock()
     cpu_hotplug.read_lock()

When - as it was - the write lock is implemented as keeping the lock
internal lock (the lock guarding the lock state) locked over the entire
write section, and the read lock side is, LOCK; change state; UNLOCK,
the above will result in a deadlock like:

  C.lock
    A.lock
      C.lock

By making both the read and write side work like:

 LOCK
  change state
 UNLOCK

the internal lock will not deadlock.

So what I did was promote cpu_hotplug to a full lock that handled
read-in-read and read-in-write recursion and made cpu_hotplug.lock the
lock internal lock.

> > > (actually, since write-locks should be very rare, perhaps we don't need
> > >  2 wait_queues ?)
> > 
> > And just let them race the wakeup race, sure that might work. Gautham
> > even pointed out that it never happens because there is another
> > exclusive lock on the write path.
> > 
> > But you say you like that it doesn't depend on that anymore - me too ;-)
> 
> Yes. but let's suppose we have the single wait_queue, this doesn't make
> any difference from the correctness POV, no?
> 
> To clarify: I am not arguing! this makes sense, but I'm asking to be sure
> I didn't miss a subtle reason why do we "really" need 2 wait_queues.
> 
> Also. Let's suppose we have both read- and write- waiters, and cpu_hotplug_done()
> does wake_up(writer_queue). It is possible that another reader comes and does
> get_online_cpus() and increments .refcount first. After that, cpu_hotplug
> is "opened" for the read-lock, but other read-waiters continue to sleep, and
> the final put_online_cpus() wakes up write-waiters only. Yes, this all is
> correct, but not "symmetrical", and leads to the question "do we really need
> 2 wait_queues" again.

I don't think we do. It just didn't occur to me to pile read-waiters and
write-waiters on the same waitqueue.