Re: [Xenomai-core] Prio-inversion on cleanup?

[Jan Kiszka <jan.kiszka@domain.hid>]

[-- Attachment #1: Type: text/plain, Size: 5098 bytes --]

Philippe Gerum wrote:
> On Thu, 2006-06-29 at 12:48 +0200, Philippe Gerum wrote:
>> On Thu, 2006-06-29 at 12:34 +0200, Jan Kiszka wrote:
>>> Philippe Gerum wrote:
>>>> On Thu, 2006-06-29 at 11:24 +0200, Jan Kiszka wrote:
>>>>> Jan Kiszka wrote:
>>>>>> ...
>>>>>> The pthread is blocked on the irqbench device ioctl. On hitting ^C,
>>>>>> close() is invoked from the main thread for that device. The pthread is
>>>>>> woken up and obviously relaxed on some linux syscall (after being
>>>>>> interrupted twice by the periodic timer event of a "latency -p 300 -P
>>>>>> 50" instance). This passes the control over to the main thread while
>>>>>> keeping the pthread prio of 99. And this prio seems to survive for the
>>>>>> following 11 ms (full trace available on request).
>>>>>>
>>>>>> Any ideas what's going on?
>>>>>>
>>>>> Ok, I think I finally understood the issue. It seems to lie deep in the
>>>>> POSIX user-space lib, specifically its use of standard pthread services.
>>>>> Let me first clarify my scenario:
>>>>>
>>>>> A high-prio pthread of known and (theoretically) bounded workload shall
>>>>> be started and stopped while a low-prio thread is already running. The
>>>>> low-prio thread shall only be impacted by the real workload of the
>>>>> high-prio one, not by its creation/destruction - at least not
>>>>> significantly. To achieve this with the POSIX skin (actually this
>>>>> applies to preempt-rt in theory as well), I have to create the thread
>>>>> under SCHED_OTHER, raise its priority right before entering the
>>>>> workload, and lower it again before leaving the thread.
>>>>>
>>>>> But, unfortunately, __wrap_pthread_setschedparam() depends on some
>>>>> real_pthread functions to be called. One of them is
>>>>> __real_pthread_setschedparam, and this one issues a linux syscall for
>>>>> obvious reasons. When lowering the thread to SCHED_OTHER, this syscall
>>>>> is still issued under the original priority. And here we get bitten by
>>>>> the prio-inheritance feature of the nucleus which, in my case, lets
>>>>> significant parts of standard Linux execute under high priority,
>>>>> delaying my other real-time threads.
>>>>>
>>>>> Now I wonder how to resolve this, how to make pthread_setschedparam (a
>>>>> rather central RT-service) really real-time safe? I would say we need
>>>>> something like a lazy schedparam propagation to Linux which only takes
>>>>> place when the thread enters secondary mode intentionally or no other RT
>>>>> thread is ready. But I do not have a design for this at hand. Nasty.
>>>>>
>>>>> [My preferred way for every setup != CONFIG_PREEMPT_RT + CONFIG_XENOMAI
>>>>> would still be to switch this prio-inheritance off for the root thread.
>>>>> But this was nack'ed by Philippe several times before... ;)]
>>>>>
>>>> I nacked the proposal to _always_ switch it off. Some applications
>>>> deeply need this.
>>> I think to remember asking for a CONFIG switch here. Some applications
>>> actually benefit while others (I even think most) do not need it or even
>>> easily screw themselves up during init/cleanup. You know, my old
>>> concerns. :)
>>>
>> A dynamic switch is better there. You may want this behaviour to be
>> settable on a thread-by-thread basis.
>>
> 
> Actually, we could have both, dynamic and static switches, just like the
> interrupt shield.
> 

Hurray! (Otherwise, I would have kicked off a discussion about the
default setting ;-))

>>>>> Side note: the native skin does not seem to suffer from this effect as
>>>>> it only tracks the current prio at Xenomai level.
>>>>>
>>>> Switching off priority adjustment for the root thread before moving a
>>>> SCHED_FIFO shadow to SCHED_OTHER would prevent this side-effect. We'd
>>>> need to add a per-thread status bit to check whether we should run
>>>> xnpod_renice_root() or not for any given thread, and switch it on/off
>>>> from __wrap_pthread_setschedparam.
>>>>
>>> This doesn't sound bad and would probably help low-prio threads also in
>>> some other scenarios.
>>>
>> I'm currently implementing that at nucleus level.
>>
>>> Nevertheless, a syscall-less pthread_setschedparam would still be a good
>>> idea as well, this time having the caller in mind who wishes to change
>>> its priority without entering Linux.
>> Reading the comment Gilles put there, it's likely not possible to have a
>> syscall-less implementation on top of the NPTL. We need to give a chance
>> to the NPTL to track the priority update, otherwise,
>> pthread_getschedparam() is going to break.
>>

Yep, saw that. I would start thinking about what we gain by wrapping
that service as well (e.g. by mirroring the state in a per-__thread
variable). Would likely work as long as the thread stays in primary
mode, but we still need to propagate the setting when entering secondary
mode. Maybe some soft-IRQ to Linux can help here. It could apply the
scheduler change on Linux re-entry.

Well, I'm probably overseeing tons of pitfalls, or all this becomes
terribly complicated.

Jan


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 249 bytes --]