[Xenomai-help] Xenomai/rtnet vs. 2.6.21 realtime preempt patch

[Xenomai-help] Xenomai/rtnet vs. 2.6.21 realtime preempt patch

[M. Koehrer]

Hi everybody,

I have tried the latest 2.6.21 kernel + Ingo Molnar's realtime preempt patch
(see: http://rt.wiki.kernel.org/index.php/RT_PREEMPT_HOWTO)

And I am really amazed about the results.
On my Pentium 4D (dual core) system, 3.2 GHz, I ran the cyclictest
application (which is something similar to the "latency" application of Xenomai/RTAI)
and I get values below 15 microseconds on my PC (for a single task running with 200 microseconds).

Using kernel 2.6.20.4 + Xenomai 2.3.1 I get similar results for the latency application.

That means, that Ingo Molnar has done a real cool job in moving standard linux in the direction
of a real time OS.

However, one thing is not as fine as with Xenomai/rtnet:
I want to do real time UDP communication from a user space application.
Unfortunately, I was not  able to do that as fast as with the Xenomai/rtnet way.
My application does a kind of "ping-pong" with an embedded device that is connected as only
device to eth1 (Intel e1000 adapter).
Using Xenomai/rtnet I get request/response times at about 200 microseconds.

When I run an adapted application on the kernel with the preempt patch using the standard
linux UDP socket functions (sendto, recv), I get request/response times around 1 millisecond.
I have now the assumption that the TCP/IP stack that uses some internal tasks is not running with
real time priority. However, I have no idea how to move that code to real time priority or if it makes
sense to move it as it is probably not possible to handle only communication via a certain ethernet adapter 
with real time priority and all other stuff (house keeping via TCP, different Ethernet adapter) 
with standard priority.
Has anybody an idea about the internal behaviour of this or how to continue here?

Another approach could be to use a modified version of rtnet on base of the preempt patched kernel.
This could allow to use the preempt patch kernel as base for all threads, and the rtnet is used
for real time Ethernet communication...
Is something like this possible or this there a major error in the approach?

Currently, my application is working quite nice using Xenomai/rtnet however there are some
drawbacks like the issue with limited IRQs: Sharing of IRQs between Ethernet-Drivers of rtnet
and non-realtime drivers is not possible (at least I did not manage that...).
A smooth usage of real time features from a "standard" kernel could help here!

Thanks for all feedback on this comment!

Regards

Mathias

-- 
Mathias Koehrer
mathias_koehrer@domain.hid


50€ AMAZON-Einkaufsgutschein bei Bestellung von Arcor-DSL:
Viel oder wenig? Schnell oder langsam? Unbegrenzt surfen + telefonieren
ohne Zeit- und Volumenbegrenzung? DAS TOP ANGEBOT JETZT bei Arcor: günstig
und schnell mit DSL - das All-Inclusive-Paket für clevere Doppel-Sparer,
nur  39,85 €  inkl. DSL- und ISDN-Grundgebühr!
http://www.arcor.de/rd/emf-dsl-2

Re: [Xenomai-help] [RTnet-users] Xenomai/rtnet vs. 2.6.21 realtime preempt patch

[Jan Kiszka]

[-- Attachment #1: Type: text/plain, Size: 4301 bytes --]

M. Koehrer wrote:
> Hi everybody,
> 
> I have tried the latest 2.6.21 kernel + Ingo Molnar's realtime preempt patch
> (see: http://rt.wiki.kernel.org/index.php/RT_PREEMPT_HOWTO)
> 
> And I am really amazed about the results.
> On my Pentium 4D (dual core) system, 3.2 GHz, I ran the cyclictest
> application (which is something similar to the "latency" application of Xenomai/RTAI)
> and I get values below 15 microseconds on my PC (for a single task running with 200 microseconds).
> 
> Using kernel 2.6.20.4 + Xenomai 2.3.1 I get similar results for the latency application.
> 
> That means, that Ingo Molnar has done a real cool job in moving standard linux in the direction
> of a real time OS.

Yep, -rt is an amazing piece of work when considering the big chunk of
non-RT code it has to deal with through Linux. Still, there are open
design and implementation issues that the cyclictest does not tickle,
mainly futex determinism, the high locking complexity, and OOM-risks due
to preemptible RCU. Also, when your Linux box used to be tuned to
CONFIG_PREEMPT_NONE/VOLUNTARY for its non-RT load, your mileage may vary
with CONFIG_PREEMPT_RT.

If you want to dive deeper into testing, try e.g. irqbench (Wolfgang has
an -rt port via the rtdm-native branch of Xenomai) or benchmarks that
stresses locking (futexes) etc.

> 
> However, one thing is not as fine as with Xenomai/rtnet:
> I want to do real time UDP communication from a user space application.
> Unfortunately, I was not  able to do that as fast as with the Xenomai/rtnet way.
> My application does a kind of "ping-pong" with an embedded device that is connected as only
> device to eth1 (Intel e1000 adapter).
> Using Xenomai/rtnet I get request/response times at about 200 microseconds.

You are then using the plain Linux network stack which has quite a few
pitfalls /wrt latency and determinism.

> 
> When I run an adapted application on the kernel with the preempt patch using the standard
> linux UDP socket functions (sendto, recv), I get request/response times around 1 millisecond.
> I have now the assumption that the TCP/IP stack that uses some internal tasks is not running with
> real time priority. However, I have no idea how to move that code to real time priority or if it makes
> sense to move it as it is probably not possible to handle only communication via a certain ethernet adapter 
> with real time priority and all other stuff (house keeping via TCP, different Ethernet adapter) 
> with standard priority.
> Has anybody an idea about the internal behaviour of this or how to continue here?

Linux networking (namely: reception) goes through some softirq, and
those are threaded over -rt. You may want to play with the related
priority, but you then push quite a bunch of code up the prio ladder as
well - likely not what you want.

> 
> Another approach could be to use a modified version of rtnet on base of the preempt patched kernel.
> This could allow to use the preempt patch kernel as base for all threads, and the rtnet is used
> for real time Ethernet communication...
> Is something like this possible or this there a major error in the approach?

Nope, that's planned for the future. Thanks to Wolfgang's work on RTDM
for -rt, we already have a platform that is just lacking one feature for
RTnet (rtdm_tasks). We then need to extend RTnet's build system to
create RTnet for PREEMPT_RT. This will be at least a mid-term approach,
the long-term is something I'm still meditating on, but it will look
differently.

> 
> Currently, my application is working quite nice using Xenomai/rtnet however there are some
> drawbacks like the issue with limited IRQs: Sharing of IRQs between Ethernet-Drivers of rtnet
> and non-realtime drivers is not possible (at least I did not manage that...).
> A smooth usage of real time features from a "standard" kernel could help here!

Nope, not really. As I think to have explained earlier, IRQ sharing
between drivers that are designed for real-time and others that are not
will never work deterministically. That has nothing to do with the
design of your RTOS underneath. Actually, the same issue once came up
for -rt over some ARM board that did poor IRQ line sharing as well.

Jan


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 250 bytes --]

Re: [Xenomai-help] [RTnet-users] Xenomai/rtnet vs. 2.6.21 realtime preempt patch

[M. Koehrer]

Hi Jan,

> > I have tried the latest 2.6.21 kernel + Ingo Molnar's realtime preempt
> patch
> > (see: http://rt.wiki.kernel.org/index.php/RT_PREEMPT_HOWTO)
> > 
> > And I am really amazed about the results.
> > On my Pentium 4D (dual core) system, 3.2 GHz, I ran the cyclictest
> > application (which is something similar to the "latency" application of
> Xenomai/RTAI)
> > and I get values below 15 microseconds on my PC (for a single task running
> with 200 microseconds).
> > 
....
> > 
> > Currently, my application is working quite nice using Xenomai/rtnet
> however there are some
> > drawbacks like the issue with limited IRQs: Sharing of IRQs between
> Ethernet-Drivers of rtnet
> > and non-realtime drivers is not possible (at least I did not manage
> that...).
> > A smooth usage of real time features from a "standard" kernel could help
> here!
> 
> Nope, not really. As I think to have explained earlier, IRQ sharing
> between drivers that are designed for real-time and others that are not
> will never work deterministically. That has nothing to do with the
> design of your RTOS underneath. Actually, the same issue once came up
> for -rt over some ARM board that did poor IRQ line sharing as well.
> 
> Jan
I agree, the best is actually to have separate IRQs for real time and non real time drivers.
However, reality shows me, that this is very hard to get with standard PCs.
I can plug in one or two PCI boards to have an unique IRQ for them. However, if I want
to use more PCI boards IRQ sharing cannot be avoided.
As with the preempt patch, the duration of non-realtime IRQ routines seems to be fairly short.
>From this, I think it is at least an option to share IRQs between real time and non real time drivers
even if the IRQ routine of the non real time driver may lead to a delay of the real time IRQ routine.
It is a question of acceptable delays for the IRQs.

Regards

Mathias




-- 
Mathias Koehrer
mathias_koehrer@domain.hid


Viel oder wenig? Schnell oder langsam? Unbegrenzt surfen + telefonieren
ohne Zeit- und Volumenbegrenzung? DAS TOP ANGEBOT JETZT bei Arcor: günstig
und schnell mit DSL - das All-Inclusive-Paket für clevere Doppel-Sparer,
nur  39,85 €  inkl. DSL- und ISDN-Grundgebühr!
http://www.arcor.de/rd/emf-dsl-2

Re: [Xenomai-help] [RTnet-users] Xenomai/rtnet vs. 2.6.21 realtime preempt patch

[Jan Kiszka]

[-- Attachment #1: Type: text/plain, Size: 2988 bytes --]

M. Koehrer wrote:
> Hi Jan,
> 
>>> I have tried the latest 2.6.21 kernel + Ingo Molnar's realtime preempt
>> patch
>>> (see: http://rt.wiki.kernel.org/index.php/RT_PREEMPT_HOWTO)
>>>
>>> And I am really amazed about the results.
>>> On my Pentium 4D (dual core) system, 3.2 GHz, I ran the cyclictest
>>> application (which is something similar to the "latency" application of
>> Xenomai/RTAI)
>>> and I get values below 15 microseconds on my PC (for a single task running
>> with 200 microseconds).
> ....
>>> Currently, my application is working quite nice using Xenomai/rtnet
>> however there are some
>>> drawbacks like the issue with limited IRQs: Sharing of IRQs between
>> Ethernet-Drivers of rtnet
>>> and non-realtime drivers is not possible (at least I did not manage
>> that...).
>>> A smooth usage of real time features from a "standard" kernel could help
>> here!
>>
>> Nope, not really. As I think to have explained earlier, IRQ sharing
>> between drivers that are designed for real-time and others that are not
>> will never work deterministically. That has nothing to do with the
>> design of your RTOS underneath. Actually, the same issue once came up
>> for -rt over some ARM board that did poor IRQ line sharing as well.
>>
>> Jan
> I agree, the best is actually to have separate IRQs for real time and non real time drivers.
> However, reality shows me, that this is very hard to get with standard PCs.

MSI? -- Oh, damn... ;)

> I can plug in one or two PCI boards to have an unique IRQ for them. However, if I want
> to use more PCI boards IRQ sharing cannot be avoided.

Agreed. Theory is nice, but practice often rules.

> As with the preempt patch, the duration of non-realtime IRQ routines seems to be fairly short.

You got it: "seems". This approach might be fine for some applications,
but not for all.

> From this, I think it is at least an option to share IRQs between real time and non real time drivers
> even if the IRQ routine of the non real time driver may lead to a delay of the real time IRQ routine.
> It is a question of acceptable delays for the IRQs.

It is a question of acceptable delays in the actual (analysed and
measured) worst case vs. "probable" (more or less blindly measured)
worst case. If you know the driver that will sit on the same IRQ line
like eg. your RT NIC, you can decide on a case-by-case analysis for each
kernel release if its behaviour is acceptable for your scenario. But
then you could also go the clean way and establish a trampoline for that
non-RT driver (as I suggested a few years back).

I more and more think we should try to formalise the latter procedure
and help system developers to realise this by modified or additional
features of the API (RTDM, I-pipe). I just had a private thread with
some other guy from industry about a related issue. I will try to wrap
up my latest "shiny" ideas and come back with a proposal next week or so.

Jan


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 250 bytes --]