Differences of Roundtrip time, latency between Xenomai 2 and Xenomai 3

Differences of Roundtrip time, latency between Xenomai 2 and Xenomai 3

[danwe]

Hello,

I have made a comparison for Roundtrip time between Xenomai 2 / RTnet,
Xenomai 3 and Debian distribution. I would like to ask you some specific
questions as I am not sure.

1. I have read that Xenomai does work on Linux and the Interrupt Pipeline
does know which message is for real-time (Xenomai) and which not (Debian).
As I am copy Xenomai on a SD-Card am I just using real-time messages or is
it also possible to use non-real-time messages? Does Xenomai need Debian
distribution or can it work by itself?

2. I have seen some big difference in jitter between Xenomai 2 / RTnet and
Xenomai 3 without RTnet while testing the latency. Xenomai 3 does have a
much smaller jitter than Xenomai 2 / RTnet. Can you tell me where exactly
the difference come from?

3. I have also seen that the Roundtrip time for Xenomai 3 is twice the size
of Xenomai 2 / RTnet. So my question is: I thought that RTnet ist just for
synchronization and TDMA. Does RTnet also make it "more" real-time than
Xenomai 3 itself?

Kind regards

Daniel

Re: Differences of Roundtrip time, latency between Xenomai 2 and Xenomai 3

[Per Oberg]

----- Den 16 aug 2019, på kl 10:17, xenomai xenomai@xenomai.org skrev:

> Hello,

> I have made a comparison for Roundtrip time between Xenomai 2 / RTnet,
> Xenomai 3 and Debian distribution. I would like to ask you some specific
> questions as I am not sure.

> 1. I have read that Xenomai does work on Linux and the Interrupt Pipeline
> does know which message is for real-time (Xenomai) and which not (Debian).
> As I am copy Xenomai on a SD-Card am I just using real-time messages or is
> it also possible to use non-real-time messages? Does Xenomai need Debian
> distribution or can it work by itself?

You would probably need at least a small Linux distribution.
Xenomai does not now how to talk to all hardware there is, e.g. read write to disk or allocate memory etc. so someone else has to do that. And just to be clear, let's differentiate between a Linux kernel and a GNU/Linux OS. The programs you write needs libraries with common functions,  for example math functions or stdio or similar and these are not part of the Linux kernel. They are most often GNU (or from other sources). 

And also, the Xenomai kernel IS a Linux kernel with some addons to handle interrupts and some of the hardware. 

If you really want to slim down you would need

a) A Xenomai Linux kernel (would boot on its own, but may also need some help with loading firmware blobs for certain hardware)
b) A Xenomai compiled application. I guess if it's statically linked you could actually get away with running that instead of the regular init.

But in most cases you also need

c) A minimal Linux distribution, possibly just something to be able to load drivers, set the time etc. (See for example Yocto/Openembedded that can run busybox)
d) Some Xenomai runtime software like dynamically linked libraries for code you haven't statically linked and some software for controlling Xenomai drivers such as rtcan etc. 


> 2. I have seen some big difference in jitter between Xenomai 2 / RTnet and
> Xenomai 3 without RTnet while testing the latency. Xenomai 3 does have a
> much smaller jitter than Xenomai 2 / RTnet. Can you tell me where exactly
> the difference come from?

> 3. I have also seen that the Roundtrip time for Xenomai 3 is twice the size
> of Xenomai 2 / RTnet. So my question is: I thought that RTnet ist just for
> synchronization and TDMA. Does RTnet also make it "more" real-time than
> Xenomai 3 itself?

Lets see if I can clarify things a bit. Perhaps it will answer some of your questions indirectly. 

First, with regards to networking there are simply put two driver variants to choose from in Xenomai, the regular Kernel driver or the RTNet driver. 

The regular kernel driver is handled by the regular kernel which is allowed to process data when Xenomai thinks its fitting to do so. The RTNet driver is handled directly by Xenomai. The problem with handing over things to the regular kernel is that it might have other things that it thinks it needs to to while in charge of the CPU, so this should give a bit more random behavior. If we decide to wait for an answer/result we may be waiting a long time if the Linux kernel has other things to do so we may miss or deadline. 

Secondly, when programming for Xenomai you may choose a skin of your liking. When using the posix skin you may write posix UDP/TCP code for sending/reading a socket regardless of if its a RTNet driver or a regular driver. Xenomai should use the correct function automatically, but if it doesn't you may use macros or special functions with prefixes (__STD or __COBALT). 

In practice this means that sending a UDP message from Xenomai on a network card that has the RTNet driver loaded is completely handled by the real time part while sending on network card that has the regular Linux driver will need regular Linux kernel to step in.

> Kind regards

> Daniel

Best regards
Per Öberg 

Re: Differences of Roundtrip time, latency between Xenomai 2 and Xenomai 3

[Per Oberg]

----- Den 17 aug 2019, på kl 12:13, danwe daniel.wenninger92@gmail.com skrev:

> Hello Per,

Hello!
I'll re-add the list to my answer.

> thanks for your reply. It helped me a lot. But you did not really answer my
> question regarding the jitter:
> > 2. I have seen some big difference in jitter between Xenomai 2 / RTnet and
> > Xenomai 3 without RTnet while testing the latency. Xenomai 3 does have a
> > much smaller jitter than Xenomai 2 / RTnet. Can you tell me where exactly
> > the difference come from?

> So why exactly is the Jitter of Xenomai 2 bigger than Xenomai 3? Just because
> some intern improvements on Xenomai 3? Which improvements?

I don't know the answer to this, perhaps someone else does?

Your exact setup for comparing things matters a lot, as well as how you performed the test. For the high-jitter case I would make sure that I have no accidental dropouts to secondary mode (non realtime) etc. 

Another complicating factor is that there have been improvements in the regular Linux kernel too (e.g. PREEMT_RT) that improves the native real-time performance so kernel version and configuration matters a lot. A regular Linux kernel fully patched with the PREEMT_RT patches has quite good real time performance in itself. 

> And one more questioin: If Xenomai 3 without RTnet uses the normal Linux Kernel,
> why are my roundtrip times faster than on Debian system?

I cant answer this either exactly, but if you use Xenomai and regular network you are only using the driver part (kernel space) of the Linux kernel. When doing it using a regular Linux process you are in Linux user space and then your process is treated just as any other process. You could give your process higher priority and things should improve a bit, but to give a good answer one would need to know exactly how the test was setup and how the system with kernels are configured which is kind of out of scope for this answer.

B.t.w, if you send messages using Xenomai on a regular network card and use a blocking call you will most definitely end up in secondary mode and loose real time for that thread/process and everything related by inter process locks etc., but if it's only for the round trip test i guess it doesn't matter. 

> Kind regards

> Daniel Wenninger

> Am Fr., 16. Aug. 2019 um 14:01 Uhr schrieb Per Oberg < [ mailto:pero@wolfram.com
> | pero@wolfram.com ] >:

>> ----- Den 16 aug 2019, på kl 10:17, xenomai [ mailto:xenomai@xenomai.org |
>> xenomai@xenomai.org ] skrev:

>> > Hello,

>> > I have made a comparison for Roundtrip time between Xenomai 2 / RTnet,
>> > Xenomai 3 and Debian distribution. I would like to ask you some specific
>> > questions as I am not sure.

>> > 1. I have read that Xenomai does work on Linux and the Interrupt Pipeline
>> > does know which message is for real-time (Xenomai) and which not (Debian).
>> > As I am copy Xenomai on a SD-Card am I just using real-time messages or is
>> > it also possible to use non-real-time messages? Does Xenomai need Debian
>> > distribution or can it work by itself?

>> You would probably need at least a small Linux distribution.
>> Xenomai does not now how to talk to all hardware there is, e.g. read write to
>> disk or allocate memory etc. so someone else has to do that. And just to be
>> clear, let's differentiate between a Linux kernel and a GNU/Linux OS. The
>> programs you write needs libraries with common functions, for example math
>> functions or stdio or similar and these are not part of the Linux kernel. They
>> are most often GNU (or from other sources).

>> And also, the Xenomai kernel IS a Linux kernel with some addons to handle
>> interrupts and some of the hardware.

>> If you really want to slim down you would need

>> a) A Xenomai Linux kernel (would boot on its own, but may also need some help
>> with loading firmware blobs for certain hardware)
>> b) A Xenomai compiled application. I guess if it's statically linked you could
>> actually get away with running that instead of the regular init.

>> But in most cases you also need

>> c) A minimal Linux distribution, possibly just something to be able to load
>> drivers, set the time etc. (See for example Yocto/Openembedded that can run
>> busybox)
>> d) Some Xenomai runtime software like dynamically linked libraries for code you
>> haven't statically linked and some software for controlling Xenomai drivers
>> such as rtcan etc.

>> > 2. I have seen some big difference in jitter between Xenomai 2 / RTnet and
>> > Xenomai 3 without RTnet while testing the latency. Xenomai 3 does have a
>> > much smaller jitter than Xenomai 2 / RTnet. Can you tell me where exactly
>> > the difference come from?

>> > 3. I have also seen that the Roundtrip time for Xenomai 3 is twice the size
>> > of Xenomai 2 / RTnet. So my question is: I thought that RTnet ist just for
>> > synchronization and TDMA. Does RTnet also make it "more" real-time than
>> > Xenomai 3 itself?

>> Lets see if I can clarify things a bit. Perhaps it will answer some of your
>> questions indirectly.

>> First, with regards to networking there are simply put two driver variants to
>> choose from in Xenomai, the regular Kernel driver or the RTNet driver.

>> The regular kernel driver is handled by the regular kernel which is allowed to
>> process data when Xenomai thinks its fitting to do so. The RTNet driver is
>> handled directly by Xenomai. The problem with handing over things to the
>> regular kernel is that it might have other things that it thinks it needs to to
>> while in charge of the CPU, so this should give a bit more random behavior. If
>> we decide to wait for an answer/result we may be waiting a long time if the
>> Linux kernel has other things to do so we may miss or deadline.

>> Secondly, when programming for Xenomai you may choose a skin of your liking.
>> When using the posix skin you may write posix UDP/TCP code for sending/reading
>> a socket regardless of if its a RTNet driver or a regular driver. Xenomai
>> should use the correct function automatically, but if it doesn't you may use
>> macros or special functions with prefixes (__STD or __COBALT).

>> In practice this means that sending a UDP message from Xenomai on a network card
>> that has the RTNet driver loaded is completely handled by the real time part
>> while sending on network card that has the regular Linux driver will need
>> regular Linux kernel to step in.

>> > Kind regards

>> > Daniel

>> Best regards
>> Per Öberg

Per Öberg