Re: NPTL mutex and the scheduling priority

Sébastien Dugué
2006-06-14T13:14:52Z

On Tue, 2006-06-13 at 08:56 -0400, Jakub Jelinek wrote:
> On Tue, Jun 13, 2006 at 02:04:32PM +0200, S?bastien Dugu? wrote:
> > > Now not sure what do you mean by "use PI futexes all along in glibc",
> > > certainly you don't mean using them for normal mutexes, right?
> > > FUTEX_LOCK_PI has effects the normal futexes shouldn't have.
> > > The condvars can be also used with PP mutexes and using PI for the cv
> > > internal lock unconditionally wouldn't be the right thing either.
> >
> > I effectively meant using a PI futex for the cv __data.__futex but now
> > I realize it's a Really Bad Idea.
>
> For __data.__futex? That is not just a really bad idea, that's not possible
> at all. PI futex has a hardcoded format (owner tid plus 2 control bits in
> the MSB), while cv in __data.__futex needs to have an application controlled
> format (a counter).

Darn right, overlooked that!

>
> > To summarize (correct me if I'm wrong), we need a way in the broadcast
> > case to promote the cv __data.__futex type to the type of the external
> > mutex (PI, PP, normal) in the requeue path. Therefore we need the
> > ability to requeue waiters on a regular futex onto a PI futex.
>
> We need more things:
> Have FUTEX_WAKE/FUTEX_REQUEUE/FUTEX_WAKE_OP honor the scheduling policy/priorities
> rather than use FIFO (this is needed not just for proper behavior
> of pthread_mutex_unlock, but also for __data.__futex).

Well, this is what I tried to achieve by using plists.

>
> FUTEX_REQUEUE is used by pthread_cond_signal to requeue the __data.__futex
> onto __data.__lock.

You meant FUTEX_WAKE_OP, I guess. I could not find any place still
using FUTEX_REQUEUE in glibc 2.4.

> So it all depends on what futex type is used by the
> __data.__lock.

OK.

> A CV doesn't have a mutex associated to it at all times,
> when there is no contention on it, it really doesn't matter.

Right.

> Maybe
> it would be possible to use PI futex always (well, if FUTEX_LOCK_PI is
> supported, otherwise of course only use normal lock) for the internal lock
> though. All we want to ensure is that for the short time it is held
> (no blocking operation should be done while __data.__lock is held
> unless interrupted by signal) if the owning thread of __data.__lock is
> scheduled away (or interrupted by signal) it doesn't cause priority
> inversion. That even includes e.g. two threads with different priorities
> calling pthread_signal or pthread_broadcast concurrently (and in that
> case, we might not have an associated mutex at all).
> But, for PI __data.__lock, we need:
> 1) FUTEX_REQUEUE being able to requeue non-PI futex to PI futex

We're trying to look into that right now.

> 2) FUTEX_WAKE_OP alternative that allows the target futex be a PI futex:
> ATM NPTL uses FUTEX_OP_CLEAR_WAKE_IF_GT_ONE ((4 << 24) | 1)
> FUTEX_WAKE_OP operation, i.e. atomically
> FUTEX_WAKE (futex); int old = *lock; *lock = 0; if (old > 1) FUTEX_WAKE (lock);
> but with PI __data.__lock, we want instead atomically:
> FUTEX_WAKE (futex); /* This one is normal, non-PI */ FUTEX_UNLOCK_PI (lock);
> (or perhaps:
> FUTEX_WAKE (futex); int old = *lock;
> if (old & FUTEX_WAITERS)
> FUTEX_UNLOCK_PI (lock);
> else if (old == gettid ())
> *lock = 0;
> else
> FUTEX_UNLOCK_PI (lock);
> ).

Understood.

Thanks a lot for the info.

Sébastien.