Re: [Adeos-main] latency results for ppc and x86

[Jan Kiszka <jan.kiszka@domain.hid>]

[-- Attachment #1: Type: text/plain, Size: 7125 bytes --]

Nicholas Mc Guire wrote:
>> Nicholas Mc Guire wrote:
>>>>>> well thats true for ADEOS/RTAI/RTLinux as well - we are also only
>>>>>> black-box testing the RT-kernel - there currently is absolutley NO
>>>>>> prof for worst-case timing in any of the flavours of RT-Linux.
>>>>>
>>>>> Nope, it isn't. There are neither sleeping not spinning lock nesting
>>>>> depths of that kind in Xenomai or Adeos/I-pipe (or older RT
>>>>> extensions,
>>>>> AFAIK) - ok, except for one spot in a driver we have scheduled for
>>>>> re-design already.
>>>
>>> that might be so - never the less there is no formal-proof that the
>>> worst
>>> case of ADEOS/I-pipe is X-microseconds, the latency/jitter numbers are
>>> based on black-box testing. In fact one problem is that there are not
>>> even
>>> code-coverage tools (or I just did not find them) that can provide
>>> coverage data for ADEOS - thus how can one guarantee worst-case ?
> 
>> The fact that tool support is "improvable" doesn't mean that such an
>> analysis is impossible. You may over-estimate, but you can derive
>> numbers for a given system (consisting of real-time core + RT
>> applications) based on a combined offline system analysis and runtime
>> measurements. But hardly anyone is doing this "for fun".
> 
> 
> with the current status I don't think a off-line analysis is resonable
> I don't think a model of ADEOS is resonably duable, alteast not a
> modleing that would lead to any usable results - I might be wrong - do
> you know of any such successfull approaches ? All testing is really
> inherently limited, from black-box testing you simply don't get any
> guarantees.

We are no longer black-box testing - thanks to our "KFT". I'm trying to
advertise this model heavily to users, but it still requires a bit too
much system knowledge. Still, modelling a system of I-pipe + Xenomai
remains an open challenge AFAIK.

> 
> <snip>
>>> its not a corener case demonstration, Ive been doing benchmarks on rt
>>> preempt now for quite some time, there is still an advantage if you run
>>> simple comparisons (jitter measurements) - but it is clearly going down,
>>> The problem I have with RT-preempt being 50us and ADEOS is 15us is
>>> simply that the sector that does need those numbers that RT-preempt will
>>> most likely
>>> never reach is generally interested in guaranteed times, and thats where
>>> it becomes tough to argue any of the hard-realtime extensions at this
>>> point - that is not saying RT-preempt can replace ADEOS/RTAI/RTLinux-gpl
>>> Im just saying that the numbers are no longer 2/3 orders of
>>> magnitude,which they were in 2.2.X/2.4.X and where arguing the use was
>>> simple.
> 
>> Granted, arguing becomes more hairy when you have to pull out low-level
>> system details like I posted (and not discussing individual issues of
>> certain patches). There are scenarios where I would recommend -rt as
>> well, but so far only few where RT extensions are fitting too.
> 
>>>
>>> Don't get me wrong Im not trying to argue away ADEOS/RTAI or I would
>>> have given up RTLinux/GPL quite some time ago - but I belive if these
>>> low-jitter/latency systems want to keep there acceptance in industry a
>>> key issue will be to improve the tools for verification/validation -
> 
>> Ack, and I'm sure they will emerge over the time. I don't expect this to
>> happen just because someone enjoys it (adding features is always
>> funnier), but because users will at some point really need them. It's a
>> process that will derive from the steadily growing professional user
>> base in both industry and academia.
> 
> let see - I hope you are right - I'm just starting into a FMEA/HAZOP for
> XtratuM "for fun" ;)

Will be interesting to hear/read about practical experiences.

> 
>>>
>>> <snip>
>>>
>>>  THAT is a problem in arguing for ADEOS/I-pipe - WHAT is the worst case
>>> now ? what is the cause of the worst case ? and can I really demonstrate
>>> by strong evidence that the worst case on this system is actually XXXX
>>> microseconds under arbitrary load and will not be higher in some strange
>>> corner cases ?
> 
>> Leaving the completely formal proof aside (that's something even
>> microkernels still cannot provide), you may go to the drawing board,
>> develop a model of your _specific_ system, derive worst-case
>> constellations, and trace the real system for those events (probably
>> also stimulating them) while measuring latencies. Then add some safety
>> margin ;), and you have worst-case numbers of a far higher quality then
>> by just experimenting with benchmarks. This process can become complex
>> (ie. costly), but it is doable.
> 
>> The point about co-scheduling approaches is here, that they already come
>> with a simpler base model (for the RT part), and they allow to "tune"
>> your system to simplify this model even further - without giving up an
>> integrated non-RT execution environment and its optimisations. We will
>> see the effect better on upcoming multi-core systems (not claiming that
>> Xenomai is already in /the/ perfect shape for them).
> 
> 
>> However, if you have suggestions on how to improve the current tool
>> situation, /me and likely others are all ears. And such improvements do
>> not have to be I-pipe/Xenomai-specific...
> 
> well one thing Im looking into for RTLinux is to extend things like
> kernel GCOV into RTLinux and KFI/KFT to RTLinux as this allows much
> better assessment. I guess that those extensions would equally be worth
> while
> for ADESO/I-pipe/Xenomai.
> 
> refs:
> 
>  KFT www.celinuxforum.org/CelfPubWiki/PatchArchive last one for 2.6.12
>  GCOV-Kernel part of LTP now (last one is for linux-2.6.16-gcov.patch.gz
> 

[Quick glance at GCOV patch] Hmm, the thrilling thing is typically
locking, but I don't see a single spinlock, just some semaphores that
cannot be called from arbitrary contexts anyway. Hmm. Did you already
played with it for some kernel?

Regarding KFT: we have such thing already. Partly derived from Ingo
Molnar's work, but with less impact during freeze, the function tracer
is in I-pipe since more than a year. It's heavily used (at least by the
core team) for application and kernel debugging, and for latency
spotting of course. Available for most I-pipe archs, even for the latest
x86_64-WiP. The funny thing is that even RTAI could make use of it - if
they only realised that it's in their patches.

Next to come (yeah, long announced) is LTTng support, i.e. patch and
front-end extensions for Xenomai. There is a working version lying
around somewhere in Canada, I just need to kick the guy again who did
that work for his thesis so that he roles out a release and we can start
discussing the patch integration. Good to be reminded...

So there is definitely not nothing - but surely still enough to do :).
If you see some potential in cooperating on front-ends (given that you
still seem to head for your own kernel-patch path), let us know. I guess
there should be common ground.

Jan


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 250 bytes --]