Re: [Xenomai] IO-APIC latencies

[Jan Kiszka <jan.kiszka@siemens.com>]

On 2012-09-17 20:37, Gilles Chanteperdrix wrote:
> On 09/17/2012 08:29 PM, Jan Kiszka wrote:
> 
>> On 2012-09-17 20:18, Gilles Chanteperdrix wrote:
>>> On 09/17/2012 08:15 PM, Jan Kiszka wrote:
>>>
>>>> On 2012-09-17 20:12, Jan Kiszka wrote:
>>>>> On 2012-09-17 20:08, Gilles Chanteperdrix wrote:
>>>>>> On 09/17/2012 08:05 PM, Jan Kiszka wrote:
>>>>>>
>>>>>>> On 2012-09-17 19:46, Gilles Chanteperdrix wrote:
>>>>>>>> ipipe_end is a nop when called from primary domain, yes, but this is not
>>>>>>>> very different from edge irqs. Also, fasteoi become a bit like MSI: in
>>>>>>>> the same way as we can not mask MSI from primary domain, we should not
>>>>>>>> mask IO-APIC fasteoi irqs, because the cost is too prohibitive. If we
>>>>>>>> can live with MSI without masking them in primary mode, I guess we can
>>>>>>>> do the same with fasteoi irqs.
>>>>>>>
>>>>>>> MSIs are edge triggered, fasteois are still level-based. They require
>>>>>>> masking at the point you defer them - what we do and what Linux may even
>>>>>>> extend beyond that. If you mask them by raising the task priority, you
>>>>>>> have to keep it raised until Linux finally handled the IRQ.
>>>>>>
>>>>>>
>>>>>> Yes.
>>>>>>
>>>>>>> Or you
>>>>>>> decide to mask it at IO-APIC level again.
>>>>>>
>>>>>>
>>>>>> We do not want that.
>>>>>>
>>>>>>> If you keep the TPR raised,
>>>>>>> you will block more than what Linux wants to block.
>>>>>>
>>>>>>
>>>>>> The point is that if the TPR keeps raised, it means that primary domain
>>>>>> has preempted Linux, so, we want it to keep that way. Otherwise the TPR
>>>>>> gets lowered when Linux has handled the interrupt.
>>>>>>
>>>>>> A week-end of testing made me sure of one thing: it works. I assure you.
>>>>>
>>>>> Probably, in the absence of IRQF_ONESHOT Linux interrupts. No longer if
>>>>> you face threaded IRQs - I assure you.
>>>>
>>>> Well, it may work (if mask/unmask callbacks work as native) but the
>>>> benefit is gone: masking at IO-APIC level will be done again. Given that
>>>> threaded IRQs become increasingly popular, it will also be hard to avoid
>>>> them in common setups.
>>>
>>>
>>> The thing is, if we no longer use the IO-APIC spinlock from primary
>>> domain, we may not have to turn it into an ipipe_spinlock, and may be
>>> able to preempt the IO-APIC masking.
>>
>> That might be true - but is the latency related to the lock or the
>> hardware access? In the latter case, you will still stall the CPU on it
>> and have to isolate the load on a non-RT CPU again.
>>
>> BTW, the task priority for the RT domain is a quite important parameter.
>> If you put it too low, Linux can run out of vectors. If you put it too
>> high, the same may happen to Xenomai - on bigger boxes.
> 
> 
> Yes, and there are only 16 levels. But Xenomai does not need to many levels.

How is telling you this? It's part of the system setup. And that may
lean toward RT or toward non-RT. This level should be adjusted according
to the current allocation of Linux and the RT domain for a particular
CPU, not hard-coded or compile-time defined.

> 
>>
>> On the other hand, it may be useful as a complete mask for Linux, ie.
>> when applied unconditionally on entry to the RT domain and released on
>> exit or fasteoi ending. Already played with that model? That would have
>> a value beyond IO-APICs.
> 
> 
> Yes, that is exactly what I did. We call this "PIC muting" on other
> architectures, I simply implemented it on x86 with the APIC TRR (that is
> the second curve in the plot posted at the beginning of this thread),
> but it does not solve the problem when the APIC timer interrupt happens
> while the I-pipe is busy acking an IO-APIC fasteoi. Which is why I
> implemented the second solution (third curve in the graph), which is
> exactly what you describe and corresponds the code I posted a few
> messages ago.

I would still refrain from directing non-RT IRQs to RT CPUs. But if
IO-APIC latency reductions comes as a byproduct of Linux IRQ muting,
then it might be worth the additional maintenance pain (shuffling
vectors around will cause pain, be assured).

Jan

-- 
Siemens AG, Corporate Technology, CT RTC ITP SDP-DE
Corporate Competence Center Embedded Linux