Re: [PATCH 5/8] cpu: cpu-hotplug deadlock

Peter Zijlstra
2008-04-29T17:41:19Z

On Tue, 2008-04-29 at 20:45 +0400, Oleg Nesterov wrote:
> On 04/29, Peter Zijlstra wrote:
> >
> > The only thing that changed is that the mutex is not held; so what we
> > change is:
> >
> > LOCK
> >
> > ... do the full hotplug thing ...
> >
> > UNLOCK
> >
> > into
> >
> > LOCK
> > set state
> > UNLOCK
> >
> > ... do the full hotplug thing ...
> >
> > LOCK
> > unset state
> > UNLOCK
> >
> > So that the lock isn't held over the hotplug operation.
>
> Well, yes I see, but... Ugh, I have a a blind spot here ;)
>
> why this makes any difference from the semantics POV ? why it is bad
> to hold the mutex throughout the "full hotplug thing" ?

Darn, now you make me think ;-)

Ok, I think I have it; the crux of the matter is that we want
reader-in-writer recursion for the cpu hotplug lock.

So we want:

cpu_hotplug.write_lock()
A.lock()
cpu_hotplug.read_lock()

When - as it was - the write lock is implemented as keeping the lock
internal lock (the lock guarding the lock state) locked over the entire
write section, and the read lock side is, LOCK; change state; UNLOCK,
the above will result in a deadlock like:

C.lock
A.lock
C.lock

By making both the read and write side work like:

LOCK
change state
UNLOCK

the internal lock will not deadlock.

So what I did was promote cpu_hotplug to a full lock that handled
read-in-read and read-in-write recursion and made cpu_hotplug.lock the
lock internal lock.

> > > (actually, since write-locks should be very rare, perhaps we don't need
> > > 2 wait_queues ?)
> >
> > And just let them race the wakeup race, sure that might work. Gautham
> > even pointed out that it never happens because there is another
> > exclusive lock on the write path.
> >
> > But you say you like that it doesn't depend on that anymore - me too ;-)
>
> Yes. but let's suppose we have the single wait_queue, this doesn't make
> any difference from the correctness POV, no?
>
> To clarify: I am not arguing! this makes sense, but I'm asking to be sure
> I didn't miss a subtle reason why do we "really" need 2 wait_queues.
>
> Also. Let's suppose we have both read- and write- waiters, and cpu_hotplug_done()
> does wake_up(writer_queue). It is possible that another reader comes and does
> get_online_cpus() and increments .refcount first. After that, cpu_hotplug
> is "opened" for the read-lock, but other read-waiters continue to sleep, and
> the final put_online_cpus() wakes up write-waiters only. Yes, this all is
> correct, but not "symmetrical", and leads to the question "do we really need
> 2 wait_queues" again.

I don't think we do. It just didn't occur to me to pile read-waiters and
write-waiters on the same waitqueue.