Re: RT patch acceptance

[Philippe Gerum <rpm@xenomai.org>]

Esben Nielsen wrote:
> On Tue, 24 May 2005, Karim Yaghmour wrote:
> 
> 
>>Esben Nielsen wrote:
>>
>>>I find that a bad approach:
>>>1) You don't have RT in userspace.
>>>2) You can't use Linux drivers for standeard hardware when you want it to
>>>be part of your deterministic RT application.
>>
>>Please have a look at RTAI/fusion. For the record, RTAI has been providing
>>hard-rt in standard Linux user-space for over 5 years now. With RTAI/Fusion
>>this gets even better as there isn't even a special API ...
>>
> 
> The tests I have read (I can't remember the links, but it was on lwn.net)
> states that the worst case latency is even worse than for a standeard 2.6
> kernel! 

You are likely talking about a benchmark conducted by Peter Laurich and 
published by Linuxdevices, I guess. In fact, the benchmark code was 
wrong, and has been amended by the author himself in a follow-up to the 
initial article:

http://www.linuxdevices.com/articles/AT3479098230.html

The figures obtained with RTAI's LXRT extension are now correct, and not 
surprisingly, LXRT is the best performer among the solutions aimed at 
providing RT support in user-space in this study, with 48 us worst-case 
scheduling latency under high load on a SBC based on a Celeron running 
at 650 MHz, and populated with 512 MB SDRAM. PREEMPT_RT was not tested 
by the benchmark, though.

> 
> If you gonna make usefull deterministic real-time in userspace you got to
> change stuff in kernel space and implement stuff like priority
> inheritance, priority ceiling or similar.  It can only turn up to be an
> ugly hack which will end up being as intruesive into the kernel as Ingo's
> approach. If you don't do anything like that you can not use _any_ Linux
> kernel resources from your RT processes even though you have reimplemented
> the pthread library to know about the "super RT" priorities.
> 

Indeed, this is why the RTAI project has an experimental branch called 
"fusion", distinct from the classic LXRT one, which aims at a better 
integration of the real-time services it provides into the common kernel 
framework. The idea is to allow RTAI applications to be alternatively 
controlled by the Linux kernel and the real-time nucleus, while keeping 
the RT priority scheme intact across seamless and automatic transitions 
between both.

If you think that PREEMPT_RT will be able to give you real-time 
guarantees that are as stringent as those already exhibited by 
additional nucleus/co-schedulers on any kind of hardware including the 
embedded ones, then you will likely conclude that we are currently 
chasing wild gooses (and that we owe you one of them for Xmas). If not, 
well, maybe considering a symbiotic approach between a fully preemptable 
Linux kernel providing regular services and a specialized co-scheduler 
providing extreme predictability should make some sense, given that both 
co-operate to control a single set of real-time tasks in user-space.

> But I give you: You will gain better interrupt latencies because
> interrupts are executed below the Linux proper. I.e. when the Linux
> kernel runs with interrupt disabled, they are really enabled in the RTAI
> subsystem.
>

IIRC, interrupt latency has never been the toughest problem with the 
vanilla Linux kernel with respect to predictability, but the scheduling 
latency still is. Hence Ingo's work, not to speak of previous efforts 
regarding preemptability, I guess.

> My estimate is that RTAI is good when you have a very small subsystem you
> need to run RT with very low latencies. Forinstance, controlling a fast
> device with limiting hardware resources to buffer events. 
> For large control systems I don't think it is the proper way to do it.
> There it is much better to run the control tasks as normal Linux
> user-space processes with RT-priority. I can see Ingo's kernel doing that,
> I can't see RTAI doing it except for very special situations where you
> don't make _any_ Linux system calls at all! You can't even use a
> normal Linux network device or character device from your RT application!
> 
> 

It happens that many if not most among the complex real-time 
applications have varying requirements with respect to determinism, 
depending on the task or execution context you consider from them. This 
is why the split application model, involving some kernel modules which 
implement the demanding RT stuff and some user-space programs connected 
to them, has been used for years.

What the RTAI project is trying to do now with its fusion branch, is to 
make compatible a larger spectrum of RT requirements without killing the 
design of your RT application as above. E.g. some tasks need to run on a 
10Khz period, whilst others can deal with ~100 us jitter under high load 
, just for the purpose of being able to call regular Linux services; but 
you want all of them running embodied in a single regular user-space 
process, and being able to use GDB for chasing the gremlins in there.

For us, this implies to make the most preemptable Linux kernel we can 
find and the fusion nucleus share the same semantics and co-operate, 
instead of blindly running side-by-side, so that RTAI eventually appears 
as a native support provided by the Linux kernel to the real-time 
application designers. Sometimes, this requires RTAI to impersonate some 
vanilla system calls such as nanosleep(), so that you really have 
micro-second level wakeups, with a little help of RTAI's integrated 
oneshot timer. This also requires a bunch of headaches, coffee, and 
machines going south, but that's nothing worth documenting in a README, 
I guess.

-- 

Philippe.