[Xenomai-help] Xenomai vs. RTLinux

[Xenomai-help] Xenomai vs. RTLinux

[Jeff Webb]

Xenomai developers and users,

Our company is looking at the possibility of porting our hardware-in-the-loop (HWIL) simulations from RTLinuxFree to a new real-time operating system.  We are currently considering Xenomai and RTLinuxPro as possible options.  I am personally biased towards free software, but our customers do not necessarily share this bias.  I am attempting to put together a presentation comparing the advantages and disadvantages of the two systems  as they would be used in our HWIL simulations.  Any comments you have would be appreciated.

We are currently using RTLinuxFree-3.2-pre3 running on Fedora Core 1 (2.4.22 linux kernel) for our in-house HWIL simulations on rack-mounted x86 PC hardware.  We have been satisfied with the functionality of RTLinuxFree over the past few years, although we have been disappointed by the lack of maintenance and development that has occurred since the primary developers focused their efforts on developing RTLinuxPro.  Because there is no RTLinuxFree release that supports the 2.6 kernel, even more than two years after it's release, it looks like we will be forced to abandon this platform, or be stuck with the almost unsupported Fedora Core 1 for the foreseeable future.

Over the past week or so, I have installed Xenomai (2.1-rc3 and 2.1-rc4) and written a couple of test applications.  My experience was very good.  I will send an email with some feedback shortly.

I wrote some simple test programs to compare the scheduling jitter and interrupt latency for RTLinuxFree, Xenomai-kernel-space, and Xenomai-user-space applications.  My tests are not rigorous by any means, but it seems that the worst-case measurements are pretty similar.  Xenomai *may* be slightly better from the limited data I have taken.  Does anyone care to share their own observations?

The worst-case scheduling latency for the xenomai user-space application was only a few microseconds worse than the kernel-space version.  Is this what you would expect?

I did not do a very thorough test with my interrupt handler, but the results seemed similar to what I described above for the scheduler.  Is there a big penalty in terms of worst-case latency for user-space interrupt handlers that I may not have seen?  I'm just looking for any major problems that I might encounter if we port our real-time applications to user-space.

Is RTLinuxPro's PSDD (user-space) development environment similar to xenomai's user-space real-time threads?  Any known advantages/disadvantages to either system?

We currently use C and FORTRAN in our RTLinuxFree modules, and would like to be able to add C++ and FORTRAN as well.  Are there any problems you see with using these languages in xenomai user-space real-time programs?

Is there planned support for native AMD x86_64 support?  We have several of these machines on our desktops and would like to upgrade our HWIL machines as well.

We are also concerned about "future-proofing" as someone else put it the other day.  I hope that xenomai will not suffer the fate of RTLinuxFree in a few years.  Of course, I understand there are no guarantees.  Do you see any advantages to xenomai over RTLinuxPro in this respect?

Another issue is technical support and documentation.  The xenomai documentation seems good enough to me, although it sounds like RTLinuxPro is quite a bit more substantial.  It also looks like the developers on this list are very responsive and helpful, but I don't know how that compares to what we would get from FSMLabs.  Any input on these topics?

It appears that xenomai is released under the LGPL and can be used with proprietary applications.  Although our simulations are just for internal use, and I would always advocate releasing free software, I was wondering if xenomai is subject to the RTLinux patent, or if it uses a different process.

My first impression of xenomai is very positive.  I like the clean API, and the user-space real-time would make life *much* nicer for me.  I'm glad there is a such nice piece of free software available for real-time programming under GNU/Linux.  Thanks!

Thank you for taking the time to read these questions and comments.  Any comments or testimonies regarding xenomai or RTLinuxPro would be appreciated.

Thanks again,

Jeff Webb

RE: [Xenomai-help] Xenomai vs. RTLinux

[Christopher Stone]

> -----Original Message-----
> From: Jeff Webb [mailto:jeff.webb@domain.hid]
> Sent: March 8, 2006 5:48 PM
> To: xenomai@xenomai.org
> Subject: [Xenomai-help] Xenomai vs. RTLinux
> 
> Xenomai developers and users,
> 
> Our company is looking at the possibility of porting our hardware-in-the-
> loop (HWIL) simulations from RTLinuxFree to a new real-time operating
> system.  We are currently considering Xenomai and RTLinuxPro as possible
> options.  I am personally biased towards free software, but our customers
> do not necessarily share this bias.  I am attempting to put together a
> presentation comparing the advantages and disadvantages of the two systems
> as they would be used in our HWIL simulations.  Any comments you have
> would be appreciated.
> 
> We are currently using RTLinuxFree-3.2-pre3 running on Fedora Core 1
> (2.4.22 linux kernel) for our in-house HWIL simulations on rack-mounted
> x86 PC hardware.  We have been satisfied with the functionality of
> RTLinuxFree over the past few years, although we have been disappointed by
> the lack of maintenance and development that has occurred since the
> primary developers focused their efforts on developing RTLinuxPro.
> Because there is no RTLinuxFree release that supports the 2.6 kernel, even
> more than two years after it's release, it looks like we will be forced to
> abandon this platform, or be stuck with the almost unsupported Fedora Core
> 1 for the foreseeable future.
> 
> Over the past week or so, I have installed Xenomai (2.1-rc3 and 2.1-rc4)
> and written a couple of test applications.  My experience was very good.
> I will send an email with some feedback shortly.
> 
> I wrote some simple test programs to compare the scheduling jitter and
> interrupt latency for RTLinuxFree, Xenomai-kernel-space, and Xenomai-user-
> space applications.  My tests are not rigorous by any means, but it seems
> that the worst-case measurements are pretty similar.  Xenomai *may* be
> slightly better from the limited data I have taken.  Does anyone care to
> share their own observations?
> 
> The worst-case scheduling latency for the xenomai user-space application
> was only a few microseconds worse than the kernel-space version.  Is this
> what you would expect?
> 
> I did not do a very thorough test with my interrupt handler, but the
> results seemed similar to what I described above for the scheduler.  Is
> there a big penalty in terms of worst-case latency for user-space
> interrupt handlers that I may not have seen?  I'm just looking for any
> major problems that I might encounter if we port our real-time
> applications to user-space.
> 
> Is RTLinuxPro's PSDD (user-space) development environment similar to
> xenomai's user-space real-time threads?  Any known
> advantages/disadvantages to either system?
> 
> We currently use C and FORTRAN in our RTLinuxFree modules, and would like
> to be able to add C++ and FORTRAN as well.  Are there any problems you see
> with using these languages in xenomai user-space real-time programs?

[Christopher Stone] I am currently working on an Xenomai application (XLM -
http://www.openembedded.biz/content/view/36/27 ) that is completely written
in C++ and have not run in to any difficulties. There is a project called
xeno-- at https://gna.org/projects/xenomm to support C++ although it is not
very complete yet. 

> 
> Is there planned support for native AMD x86_64 support?  We have several
> of these machines on our desktops and would like to upgrade our HWIL
> machines as well.

[Christopher Stone] I definitely plan to add AMD x86_64 support for my XLM
project. I have a dual core AMD X2 chip I want to test XLM on. I don't have
specific dates when it will be available yet.

> 
> We are also concerned about "future-proofing" as someone else put it the
> other day.  I hope that xenomai will not suffer the fate of RTLinuxFree in
> a few years.  Of course, I understand there are no guarantees.  Do you see
> any advantages to xenomai over RTLinuxPro in this respect?

[Christopher Stone] I think the fact that RTLinuxPro is maintained by a
small company, with extremely limited resources when compared to the free
software community, is an almost debilitating disadvantage. How long can
FSMlabs really survive as a company in the face of the mounting competition
from better alternatives like Xenomai?  If FSMlabs goes bankrupt, you would
be left with nothing but uncertainty. With Xenomai, there is nothing to go
bankrupt, so the risk is nil. As long as the source code for Xenomai is
free, you will always be able to find an affordable independent consultant
to maintain it. This is one of the true ironies of the free software
movement. With free software you actually have less uncertainty, because you
are in control of the source code, not some guy who will do who knows what
with your future when his company begins to sink.

Another advantage of Xenomai is that there is no vendor lock-in. With
RTLinuxPro vendor lock in comes in two forms. You can only get your support
and consulting from one place, and the API is specific to RTLinuxPro. 

I guarantee you that the rates that FSMlabs charges for support and
consultancy are much higher than the rates you can get for Xenomai support
and consultancy from any of a growing number of companies. That is because
the Xenomai companies are small, have very little overhead, and have a
distinct advantage in the wages they pay. I would guess that FSMlabs
consultancy rate is twice my rate.

Vendor specific API's are like drugs, once you get hooked on them, it is
difficult to incur the cost of kicking the habit. With Xenomai you can use
any of a number of skins, but, would be wise to consider the POSIX skin. By
using the industry standard POSIX API your code remains portable and more
future proof. 

> 
> Another issue is technical support and documentation.  The xenomai
> documentation seems good enough to me, although it sounds like RTLinuxPro
> is quite a bit more substantial.  It also looks like the developers on
> this list are very responsive and helpful, but I don't know how that
> compares to what we would get from FSMLabs.  Any input on these topics?
> 
> It appears that xenomai is released under the LGPL and can be used with
> proprietary applications.  Although our simulations are just for internal
> use, and I would always advocate releasing free software, I was wondering
> if xenomai is subject to the RTLinux patent, or if it uses a different
> process.

[Christopher Stone] Xenomai uses a different process that was specifically
designed to circumvent the FSMlabs patent. I think that RTAI and Xenomai are
prime examples of how robust and "future proof" the free software community
is. RTAI was stung badly by the FSMlabs patent. However, instead of running
away, the community responded by developing a new process, and the community
is as strong as it ever was.

> 
> My first impression of xenomai is very positive.  I like the clean API,
> and the user-space real-time would make life *much* nicer for me.  I'm
> glad there is a such nice piece of free software available for real-time
> programming under GNU/Linux.  Thanks!
> 
> Thank you for taking the time to read these questions and comments.  Any
> comments or testimonies regarding xenomai or RTLinuxPro would be
> appreciated.
> 
> Thanks again,
> 
> Jeff Webb
> 
> _______________________________________________
> Xenomai-help mailing list
> Xenomai-help@domain.hid
> https://mail.gna.org/listinfo/xenomai-help
> 

Re: [Xenomai-help] Xenomai vs. RTLinux

[Jan Kiszka]

[-- Attachment #1: Type: text/plain, Size: 485 bytes --]

Christopher Stone wrote:
>> Is there planned support for native AMD x86_64 support?  We have several
>> of these machines on our desktops and would like to upgrade our HWIL
>> machines as well.
> 
> [Christopher Stone] I definitely plan to add AMD x86_64 support for my XLM
> project. I have a dual core AMD X2 chip I want to test XLM on. I don't have
> specific dates when it will be available yet.

Ha! Wasn't there somebody recently asking what to contribute? ;)

Jan


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 252 bytes --]

Re: [Xenomai-help] Xenomai vs. RTLinux

[Rodrigo Rosenfeld Rosas]

Christopher Stone escreveu:
> ...
> Vendor specific API's are like drugs, once you get hooked on them, it is
> difficult to incur the cost of kicking the habit. With Xenomai you can use
> any of a number of skins, but, would be wise to consider the POSIX skin. By
> using the industry standard POSIX API your code remains portable and more
> future proof. 
>   
Actually it depends. At the moment, as far as I know, RTDM drivers are 
not able to communicate to POSIX written programs in every way, like it 
does with programs written using the native API. So, if there is any 
plan to write real-time driver (and RTDM really makes this process much 
easier), so I would recommend using the native API instead. Unless 
Xenomai is going to support better intercommunication between different 
skins, if it is possible, or if someone will take the effort to maintain 
a POSIX version of RTDM...

Rodrigo.


		
_______________________________________________________
Yahoo! Acesso Grátis - Internet rápida e grátis. Instale o discador agora!
http://br.acesso.yahoo.com

Re: [Xenomai-help] Xenomai vs. RTLinux

[Gilles Chanteperdrix]

Rodrigo Rosenfeld Rosas wrote:
 > Christopher Stone escreveu:
 > > ...
 > > Vendor specific API's are like drugs, once you get hooked on them, it is
 > > difficult to incur the cost of kicking the habit. With Xenomai you can use
 > > any of a number of skins, but, would be wise to consider the POSIX skin. By
 > > using the industry standard POSIX API your code remains portable and more
 > > future proof. 
 > >   
 > Actually it depends. At the moment, as far as I know, RTDM drivers are 
 > not able to communicate to POSIX written programs in every way, like it 
 > does with programs written using the native API. 

Actually there is a major change around 2.1-rc3: most services of the
POSIX skins may be used by threads created by other skins. The exceptions to
this rule are services which involves the "POSIX skin specific" part of
Xenomai POSIX skin threads, namely only services involving signals and
cancellation.

Also note that in user-space, the RTDM and POSIX skins are fully integrated.

-- 


					    Gilles Chanteperdrix.

Re: [Xenomai-help] Xenomai vs. RTLinux

[Rodrigo Rosenfeld Rosas]

Em Quinta 09 Mar=E7o 2006 12:03, Gilles Chanteperdrix escreveu:

>Rodrigo Rosenfeld Rosas wrote:
> > Christopher Stone escreveu:
> > > ...
> > > Vendor specific API's are like drugs, once you get hooked on them, it
> > > is difficult to incur the cost of kicking the habit. With Xenomai you
> > > can use any of a number of skins, but, would be wise to consider the
> > > POSIX skin. By using the industry standard POSIX API your code remains
> > > portable and more future proof.
> >
> > Actually it depends. At the moment, as far as I know, RTDM drivers are
> > not able to communicate to POSIX written programs in every way, like it
> > does with programs written using the native API.
>
>Actually there is a major change around 2.1-rc3: most services of the
>POSIX skins may be used by threads created by other skins. The exceptions =
to
>this rule are services which involves the "POSIX skin specific" part of
>Xenomai POSIX skin threads, namely only services involving signals and
>cancellation.
>
>Also note that in user-space, the RTDM and POSIX skins are fully integrate=
d.

Thanks for that information. I wasn't aware of that.

Rodrigo.

		
_______________________________________________________
Yahoo! Acesso Grátis - Internet rápida e grátis. Instale o discador agora!
http://br.acesso.yahoo.com

Re: [Xenomai-help] Xenomai vs. RTLinux

[Jeff Webb]

Thanks to everyone for taking the time to share your thoughts and answer my questions.  I have added some comments below.


Jan Kiszka wrote:
>>means, but it seems that the worst-case measurements are pretty
>>similar.  Xenomai *may* be slightly better from the limited data I have
>>taken.  Does anyone care to share their own observations?
> 
> Would surprises me. From what I know of RTLinux/GPL (RTAI is similar
> here, BTW), the critical paths are involving less code than Xenomai.
> That's due to the different IRQ redirection layers, the nucleus
> abstraction of Xenomai, maybe also to some yet missing optimisations.


Thanks for your explanation and comparison.  If you are interested in my little tests, I put the source files and some output plots here: 

  http://www.judyandjeff.com/tech/xenotest

I ran the first test program (jitter2.c) under heavy disk activity and while playing the opengl tuxracer program.  You can see the results for RTLinux (rtlinuxfree1.png), xenomai-kernel (xenomai_kernel1.png), and xenomain-user (xenomai_user1.png) in the plots directory.  As you can see, the typical jitter of RTLinux task is quite a bit less, but the overall latency is higher, due to a fixed offset.  Perhaps this offset could be taken out by tweaking some RTLinux configuration.  I believe those big spikes (which occur in all three versions) are a result of switching video modes with those nasty proprietary NVIDIA drivers.  These spikes occur with the xenomai latency program as well.

I ran the second test program (irqtest.c) under the same conditions mentioned above, with a one-pulse-per-second interrupt source.  I would like to test with a higher-frequency interrupt, but that's all I have handy to apply externally at the moment.  The results can be seen in some o-scope captures (rtl_krn.png, xeno_krn.png, xeno_usr.png).  We don't see the big spikes here (likely due to the low interrupt frequency), but you can see the jitter.  The minimum interrupt latency seems pretty much the same for xenomai and rtlinux.  The average jitter seems less for RTLinux, but the worst case (for this run, anyway) is a little worse for RTLinux.


> Anyway, Xenomai is favouring flexibility and maintainability here over
> the last microsecond.


I agree with xenomai's approach.  The elegance of the design is worth a few clock cycles.  I just want to know the two systems are in the same ballpark.


>>The worst-case scheduling latency for the xenomai user-space application
>>was only a few microseconds worse than the kernel-space version.  Is
>>this what you would expect?
> 
> 
> Well, I guess it takes more information about your tests to comment on
> this. The hardware of the test system would be important, as well as the
> load scenario and the test duration.

I'm not really looking for hard numbers here, but more of a general answer.  Can you say in general, if there is any circumstance in which the user-space interrupt handler would perform much worse (percentage-wise) than a kernel-based handler?  Say an order of magnitude, factor of two, or something like that.  Can linux kernel interrupt handlers delay the handling of a real-time interrupt in user-space?

FYI, I'm running an AthlonXP system with that evil NVIDIA graphics card, along with a couple of synchronous serial cards and a counter/io board.  On board ethernet, etc.  Lots of devices on the PCI bus.  The real-time IRQ is shared with linux because of the way the PCI interrupts are assigned (and we have so many devices on the bus).
 

>>We currently use C and FORTRAN in our RTLinuxFree modules, and would
>>like to be able to add C++ and FORTRAN as well.  Are there any problems
>>you see with using these languages in xenomai user-space real-time
>>programs?
> 
> Based on the design of the userspace skins, there are no problems to
> expect. If minor compilation issues still pop up (I'm not aware of any),
> they will be easy to fix. There is even an ongoing effort to create a
> native skin class library for Xenomai: xeno-- (see xenomm at gna.org).


Very good.  That's what I concluded from reading the documentation, but it's nice to have that confirmed by someone with a good knowledge of the system.


> [Christopher Stone] I definitely plan to add AMD x86_64 support for my XLM
> project. I have a dual core AMD X2 chip I want to test XLM on. I don't have
> specific dates when it will be available yet.


It looks like several folks who have the expertise to contribute are interested.  That sounds very promising to me!


> [Christopher Stone] I think the fact that RTLinuxPro is maintained by a
> small company, with extremely limited resources...


Thanks for the arguments in favor of free software.  The support, stability, and longevity areas are where I am likely to face the most resistance.


> [Christopher Stone] I guarantee you that the rates that FSMlabs charges for support
> and consultancy are much higher than the rates you can get for Xenomai support
> and consultancy from any of a growing number of companies. 


No doubt.  I was just looking over the quotes they sent me...


> Vendor specific API's are like drugs, once you get hooked on them, it is
> difficult to incur the cost of kicking the habit. With Xenomai you can use
> any of a number of skins, but, would be wise to consider the POSIX skin. By
> using the industry standard POSIX API your code remains portable and more
> future proof. 


I am debating whether to use the native API or POSIX skins.  I have been using POSIX-type calls with RTLinux, but I really do prefer the simplicity and readability of the native API.  Some of the POSIX stuff seems a bit awkward and convoluted to me, but I suppose it is a standard.  Of course, the *_np calls aren't...  Does anyone else have an opinion on using native vs. POSIX?

Since my mind is on POSIX...  Is there a way to set up an RT_PIPE such that I could use fprintf from a real-time thread, or maybe even redirect the standard output so I could use printf from real-time?  This would greatly ease the porting of non-rt code to xenomai.  Maybe this idea is impossible or I'm very naive in my thinking.

> [Christopher Stone] Xenomai uses a different process that was specifically
> designed to circumvent the FSMlabs patent. I think that RTAI and Xenomai are
> prime examples of how robust and "future proof" the free software community
> is. RTAI was stung badly by the FSMlabs patent. However, instead of running
> away, the community responded by developing a new process, and the community
> is as strong as it ever was.

Good to hear.  Thanks for clearing that up.

-Jeff

Re: [Xenomai-help] Xenomai vs. RTLinux

[Gilles Chanteperdrix]

Jeff Webb wrote:
 > (snip)
 > Since my mind is on POSIX...  Is there a way to set up an RT_PIPE
 > such that I could use fprintf from a real-time thread, or maybe even
 > redirect the standard output so I could use printf from real-time?
 > This would greatly ease the porting of non-rt code to xenomai.  Maybe
 > this idea is impossible or I'm very naive in my thinking.

You can use a POSIX message queue for that purpose, the producer working
exclusively in primary mode, and the consumer oscillating between
primary mode when pending on the queue, and secondary mode when issuing
the calls to fprintf. It is a bit what the "satch" example does, except
that the producer runs in kernel-space.

-- 


					    Gilles Chanteperdrix.

[Xenomai-core] AMD x86_64 support

[Jeff Webb]

Christopher Stone wrote:
>> Is there planned support for native AMD x86_64 support?  We have several
>> of these machines on our desktops and would like to upgrade our HWIL
>> machines as well.
> 
> [Christopher Stone] I definitely plan to add AMD x86_64 support for my XLM
> project. I have a dual core AMD X2 chip I want to test XLM on. I don't have
> specific dates when it will be available yet.
> 

Thanks again for the helpful input you all gave me several weeks ago.  Since that time, I put together a presentation for my coworkers and customers on migrating our current hardware-in-the-loop simulation framework to a new RTOS.  We primarily discussed the advantages and disadvantages of Xenomai and RTLinuxPro.  In the end, everyone seemed to agree with my conclusion that Xenomai appears to be the best solution for us, with one qualification.  My boss is not comfortable with the decision until there is x86_64 support.

If no one is planning to add support for the x86_64 platform in the near future, it is possible that we may be able to provide some funding to help make this happen.  Could someone provide an estimate of the level of effort required (man-hours/length of time) to port Xenomai to this architecture?  Unfortunately, I believe our current contract may limit us to hiring U.S. citizens, which may prove to be a problem, judging by the traffic on this email list.  None of this is a certainty at this point, and I will investigate things further.  At this time, I just want to see if anyone would be interested in doing such work, if our customers are able to fund it.

Thanks,

Jeff Webb

Re: [Xenomai-help] Xenomai vs. RTLinux

[Jan Kiszka]

[-- Attachment #1: Type: text/plain, Size: 8215 bytes --]

Jeff Webb wrote:
> Xenomai developers and users,
> 
> Our company is looking at the possibility of porting our
> hardware-in-the-loop (HWIL) simulations from RTLinuxFree to a new
> real-time operating system.  We are currently considering Xenomai and
> RTLinuxPro as possible options.  I am personally biased towards free
> software, but our customers do not necessarily share this bias.  I am
> attempting to put together a presentation comparing the advantages and
> disadvantages of the two systems  as they would be used in our HWIL
> simulations.  Any comments you have would be appreciated.
> 
> We are currently using RTLinuxFree-3.2-pre3 running on Fedora Core 1
> (2.4.22 linux kernel) for our in-house HWIL simulations on rack-mounted
> x86 PC hardware.  We have been satisfied with the functionality of
> RTLinuxFree over the past few years, although we have been disappointed
> by the lack of maintenance and development that has occurred since the
> primary developers focused their efforts on developing RTLinuxPro. 
> Because there is no RTLinuxFree release that supports the 2.6 kernel,

Well, I'm not following this in every details, but there is something
now for RTLinux/GPL (the independent community RTLinux). Anyway, the
maintenance situation is not significantly better there - too few users,
too few developers.

> even more than two years after it's release, it looks like we will be
> forced to abandon this platform, or be stuck with the almost unsupported
> Fedora Core 1 for the foreseeable future.
> 
> Over the past week or so, I have installed Xenomai (2.1-rc3 and 2.1-rc4)
> and written a couple of test applications.  My experience was very
> good.  I will send an email with some feedback shortly.

You are welcome - oh, I see you already did!

> 
> I wrote some simple test programs to compare the scheduling jitter and
> interrupt latency for RTLinuxFree, Xenomai-kernel-space, and
> Xenomai-user-space applications.  My tests are not rigorous by any
> means, but it seems that the worst-case measurements are pretty
> similar.  Xenomai *may* be slightly better from the limited data I have
> taken.  Does anyone care to share their own observations?

Would surprises me. From what I know of RTLinux/GPL (RTAI is similar
here, BTW), the critical paths are involving less code than Xenomai.
That's due to the different IRQ redirection layers, the nucleus
abstraction of Xenomai, maybe also to some yet missing optimisations.
Anyway, Xenomai is favouring flexibility and maintainability here over
the last microsecond.

> 
> The worst-case scheduling latency for the xenomai user-space application
> was only a few microseconds worse than the kernel-space version.  Is
> this what you would expect?

Well, I guess it takes more information about your tests to comment on
this. The hardware of the test system would be important, as well as the
load scenario and the test duration.

> 
> I did not do a very thorough test with my interrupt handler, but the
> results seemed similar to what I described above for the scheduler.  Is
> there a big penalty in terms of worst-case latency for user-space
> interrupt handlers that I may not have seen?  I'm just looking for any
> major problems that I might encounter if we port our real-time
> applications to user-space.

A good impression of the major differences may give the latency test
tool of Xenomai when being run in its different modes: userspace task,
kernelspace task, timer handler (IRQ context). Specifically, when you
load your systems with various non-RT IRQs (disk, network, ...), the
worst case scenario of all those IRQ stubs hitting you between the RT
IRQ event and the scheduled RT task execution can make a difference of
some 10 us (depending on the architecture). You have to wait a bit to
let this happen. Analysing the worst case result with the Ipipe latency
tracer is a recommended to see if all possible IRQ actually already hit you.

BTW, smart people on this list already started discussing how to improve
this scenario even more: mask out non-RT IRQs when in RT context.

> 
> Is RTLinuxPro's PSDD (user-space) development environment similar to
> xenomai's user-space real-time threads?  Any known
> advantages/disadvantages to either system?

That's proprietary, err, stuff. I guess not many people from the
community ever had the "chance" to look at it. And I don't want to
compare anything here based on marketing brochures.

Only so much: Xenomai has a very advanced Linux integration, so advanced
that signals are delivered to your RT-threads (not in RT though),
invoking Linux services will let them run with raised priority (as far
as possible), and debugging works smoothly.

> 
> We currently use C and FORTRAN in our RTLinuxFree modules, and would
> like to be able to add C++ and FORTRAN as well.  Are there any problems
> you see with using these languages in xenomai user-space real-time
> programs?

Based on the design of the userspace skins, there are no problems to
expect. If minor compilation issues still pop up (I'm not aware of any),
they will be easy to fix. There is even an ongoing effort to create a
native skin class library for Xenomai: xeno-- (see xenomm at gna.org).

> 
> Is there planned support for native AMD x86_64 support?  We have several
> of these machines on our desktops and would like to upgrade our HWIL
> machines as well.

On my personal wish list, this is also of increasing importance. But I'm
not aware of currently ongoing efforts. There were some activity around
RTAI to work on Adeos for AMD64, what would also be the first step for a
Xenomai port, but that one is outdated (old Adeos design, not yet
Ipipe). Takes someone to push this...

> 
> We are also concerned about "future-proofing" as someone else put it the
> other day.  I hope that xenomai will not suffer the fate of RTLinuxFree
> in a few years.  Of course, I understand there are no guarantees.  Do
> you see any advantages to xenomai over RTLinuxPro in this respect?

No vendor-lock: Who knows what the future business concept of FSMLabs
will (have to) be? Think of TimeSys, they also already had to change
their product portfolio. Whatever projects will provide hard/soft RT for
Linux in the future, they all will definitely be open source - my strong
believe.

> 
> Another issue is technical support and documentation.  The xenomai
> documentation seems good enough to me, although it sounds like
> RTLinuxPro is quite a bit more substantial.  It also looks like the
> developers on this list are very responsive and helpful, but I don't
> know how that compares to what we would get from FSMLabs.  Any input on
> these topics?

Again, this can only be answered by people with knowledge about the
commercial product.

> 
> It appears that xenomai is released under the LGPL and can be used with
> proprietary applications.  Although our simulations are just for

Roughly: It's GPL for the kernel code and LGPL for userspace libs.

> internal use, and I would always advocate releasing free software, I was
> wondering if xenomai is subject to the RTLinux patent, or if it uses a
> different process.

Well, someone with good knowledge of RTLinux recently explained to me
that even FSMLabs are no longer actively using their own patent for
latest versions. It is simply irrelevant these days, specifically for
Xenomai with Adeos/Ipipe, which is cleanly implementing scientific
concepts pre-dating this patent.

> 
> My first impression of xenomai is very positive.  I like the clean API,
> and the user-space real-time would make life *much* nicer for me.  I'm
> glad there is a such nice piece of free software available for real-time
> programming under GNU/Linux.  Thanks!
> 
> Thank you for taking the time to read these questions and comments.  Any
> comments or testimonies regarding xenomai or RTLinuxPro would be
> appreciated.

Maybe other users with RTLinux migration experiences are listening and
can contribute more direct comparisons than I'm able to.

> 
> Thanks again,
> 
> Jeff Webb
> 

Jan


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 252 bytes --]

Re: [Xenomai-help] Xenomai vs. RTLinux

[Heikki Lindholm]

Jan Kiszka kirjoitti:
> Jeff Webb wrote:
> 
>>Xenomai developers and users,
>>
>>Our company is looking at the possibility of porting our
>>hardware-in-the-loop (HWIL) simulations from RTLinuxFree to a new
>>real-time operating system.  We are currently considering Xenomai and
>>RTLinuxPro as possible options.  I am personally biased towards free
>>software, but our customers do not necessarily share this bias.  I am
>>attempting to put together a presentation comparing the advantages and
>>disadvantages of the two systems  as they would be used in our HWIL
>>simulations.  Any comments you have would be appreciated.
>>
>>We are currently using RTLinuxFree-3.2-pre3 running on Fedora Core 1
>>(2.4.22 linux kernel) for our in-house HWIL simulations on rack-mounted
>>x86 PC hardware.  We have been satisfied with the functionality of
>>RTLinuxFree over the past few years, although we have been disappointed
>>by the lack of maintenance and development that has occurred since the
>>primary developers focused their efforts on developing RTLinuxPro. 
>>Because there is no RTLinuxFree release that supports the 2.6 kernel,
> 
> 
> Well, I'm not following this in every details, but there is something
> now for RTLinux/GPL (the independent community RTLinux). Anyway, the
> maintenance situation is not significantly better there - too few users,
> too few developers.

I did some investigation of these things last year and at least then 
only thing available
for linux 2.6 was a patch from fsmlabs, which (1) didn't work, (2) 
nobody maintained,
(3) nobody cared about, (4) was x86 only. The word from rtlinux/gpl folk 
was that they're probably going to use some more generic virtualizer- 
type-of-thing for 2.6 support instead of porting the existing rtlinux 
2.4 stuff over.


>>I wrote some simple test programs to compare the scheduling jitter and
>>interrupt latency for RTLinuxFree, Xenomai-kernel-space, and
>>Xenomai-user-space applications.  My tests are not rigorous by any
>>means, but it seems that the worst-case measurements are pretty
>>similar.  Xenomai *may* be slightly better from the limited data I have
>>taken.  Does anyone care to share their own observations?
> 
> 
> Would surprises me. From what I know of RTLinux/GPL (RTAI is similar
> here, BTW), the critical paths are involving less code than Xenomai.
> That's due to the different IRQ redirection layers, the nucleus
> abstraction of Xenomai, maybe also to some yet missing optimisations.
> Anyway, Xenomai is favouring flexibility and maintainability here over
> the last microsecond.

But if the target platform is something beginning with x86_64, few 
instructions here and there probably won't matter..

-- Heikki Lindholm

Re: [Xenomai-help] Xenomai vs. RTLinux

[Philippe Gerum]

Jeff Webb wrote:
>

<snip>

> The worst-case scheduling latency for the xenomai user-space application 
> was only a few microseconds worse than the kernel-space version.  Is 
> this what you would expect?
> 

Usually, yes. The gap may increase on low-end systems though, especially for those 
exhibiting unfriendly cache behaviour for instance. But the worst-case latency is 
still predictable, albeit higher.

> I did not do a very thorough test with my interrupt handler, but the 
> results seemed similar to what I described above for the scheduler.  Is 
> there a big penalty in terms of worst-case latency for user-space 
> interrupt handlers that I may not have seen?  I'm just looking for any 
> major problems that I might encounter if we port our real-time 
> applications to user-space.
> 
> Is RTLinuxPro's PSDD (user-space) development environment similar to 
> xenomai's user-space real-time threads?  Any known 
> advantages/disadvantages to either system?

My knowledge of PSDD being limited to the contents of the press releases, which 
are the services PSDD provides to applications, roughly?

> 
> We currently use C and FORTRAN in our RTLinuxFree modules, and would 
> like to be able to add C++ and FORTRAN as well.  Are there any problems 
> you see with using these languages in xenomai user-space real-time 
> programs?

None, provided your runtime environment can invoke callouts written in C in order 
to start Xenomai system calls. Those syscalls use the very same channel to 
re-enter the kernel than normal Linux system calls.

> 
> Is there planned support for native AMD x86_64 support?  We have several 
> of these machines on our desktops and would like to upgrade our HWIL 
> machines as well.

Not planned, yet.

> 
> We are also concerned about "future-proofing" as someone else put it the 
> other day.  I hope that xenomai will not suffer the fate of RTLinuxFree 
> in a few years.  Of course, I understand there are no guarantees.  Do 
> you see any advantages to xenomai over RTLinuxPro in this respect?
> 

I would rephrase your question this way: "what would prevent Xenomai from becoming 
a proprietary technology?". Short answer: critical pieces of the Xenomai codebase 
bear various GPL copyrights from different people or organizations having very 
little or no common interests, aside of keeping Xenomai free, that is. Another 
form of the famous "divide and conquer" approach.

> Another issue is technical support and documentation.  The xenomai 
> documentation seems good enough to me, although it sounds like 
> RTLinuxPro is quite a bit more substantial.  It also looks like the 
> developers on this list are very responsive and helpful, but I don't 
> know how that compares to what we would get from FSMLabs.  Any input on 
> these topics?
> 
> It appears that xenomai is released under the LGPL and can be used with 
> proprietary applications.  Although our simulations are just for 
> internal use, and I would always advocate releasing free software, I was 
> wondering if xenomai is subject to the RTLinux patent, or if it uses a 
> different process.

My answer must be awfully biased, but:
http://www.linuxdevices.com/news/NS7320885236.html

> 
> My first impression of xenomai is very positive.  I like the clean API, 
> and the user-space real-time would make life *much* nicer for me.  I'm 
> glad there is a such nice piece of free software available for real-time 
> programming under GNU/Linux.  Thanks!
> 

You are definitely welcome.

> Thank you for taking the time to read these questions and comments.  Any 
> comments or testimonies regarding xenomai or RTLinuxPro would be 
> appreciated.
> 
> Thanks again,
> 
> Jeff Webb
> 
> _______________________________________________
> Xenomai-help mailing list
> Xenomai-help@domain.hid
> https://mail.gna.org/listinfo/xenomai-help
> 


-- 

Philippe.

Re: [Xenomai-help] Xenomai vs. RTLinux

[Jeff Webb]

Philippe Gerum wrote:
>> The worst-case scheduling latency for the xenomai user-space 
>> application was only a few microseconds worse than the kernel-space 
>> version.  Is this what you would expect?
>>
> Usually, yes. The gap may increase on low-end systems though, especially 
> for those exhibiting unfriendly cache behaviour for instance. But the 
> worst-case latency is still predictable, albeit higher.


Thanks.  That answer was very helpful.


>> Is RTLinuxPro's PSDD (user-space) development environment similar to 
>> xenomai's user-space real-time threads?  Any known 
>> advantages/disadvantages to either system?
> 
> My knowledge of PSDD being limited to the contents of the press 
> releases, which are the services PSDD provides to applications, roughly?


I received some documentation describing PSDD from FSMLabs yesterday that answers most of my questions.  In case anyone is interested, here is a brief description of PSDD, based on how I understand it.  (Of course, my understanding could be wrong on some points.)  

PSDD is basically a user-space implementation of the RTCore API (most of it, at least), which is used in the standard RTLinuxPro.  The main program and real-time tasks can reside in the same source file and share the same memory space, but there is still a strict separation between real-time and non-real-time code.  The real-time threads can call reentrant functions in various libraries (such as sprintf), but cannot make linux system calls.  Memory allocation and any changes to the memory map must be done before the first the first real-time thread is started.  When the real-time threads are running, the process memory-map remains fixed.  A real-time allocator is provided that basically pre-allocates memory for malloc calls, so that library routines that use dynamic memory allocation will still work.  PSDD also requires that all of the RTCore API calls be prefixed by 'rtl_' (e.g. pthread_create -> rtl_pthread_create) to distinguish them from the user-space versions.  This see
ms to go against the practice of adhering to the standard POSIX API, which FSMLabs promotes so much.

>From what I understand, it appears that Xenomai's approach offers better user-space functionality.  The transparent migration of tasks from the Xenomai domain to the Linux domain is fantastic.  The ability to make linux system calls is very powerful, provided that one is careful about where this is done.  Also having to prefix POSIX calls with rtl_ is an ugly drawback for the PSDD side (especially since this doesn't match their kernel-based API).

I'm not quite clear on the memory allocation question.  Does Xenomai have the same restriction on allocating memory after a real-time task has been created?  I had been thinking that memory allocation could be done from the linux context, or by a real-time task in secondary mode.  (I thought I saw some documentation on this, but can't find it at the moment.  I could certainly be mistaken.)  If so, that's a big advantage for Xenomai as well.

Anyway, it appears to me that Xenomai user-space has some major advantages over PSDD.


>> We currently use C and FORTRAN in our RTLinuxFree modules, and would 
>> like to be able to add C++ and FORTRAN as well.  Are there any 
>> problems you see with using these languages in xenomai user-space 
>> real-time programs?
> 
> None, provided your runtime environment can invoke callouts written in C 
> in order to start Xenomai system calls. Those syscalls use the very same 
> channel to re-enter the kernel than normal Linux system calls.


I'm glad to hear this.  This is a great advantage of Xenomai from my perspective.


> I would rephrase your question this way: "what would prevent Xenomai 
> from becoming a proprietary technology?". Short answer: critical pieces 
> of the Xenomai codebase bear various GPL copyrights from different 
> people or organizations having very little or no common interests, aside 
> of keeping Xenomai free, that is. Another form of the famous "divide and 
> conquer" approach.


Thank you so much for making this point.  The shared copyright is a major factor in my mind.


> My answer must be awfully biased, but:
> http://www.linuxdevices.com/news/NS7320885236.html


Thanks for taking the time to answer my questions.  You were most helpful.

-Jeff

Re: [Xenomai-help] Xenomai vs. RTLinux

[Dmitry Adamushko]

[-- Attachment #1: Type: text/plain, Size: 1201 bytes --]

[skip-skip-skip]

I'm not quite clear on the memory allocation question.  Does Xenomai have
> the same restriction on allocating memory after a real-time task has been
> created?  I had been thinking that memory allocation could be done from the
> linux context, or by a real-time task in secondary mode.


As one may issue Linux system calls from real-time tasks (causing a switch
to the secondary domain), malloc() ( -> sbrk() system call) or any other
memory allocation mechanism like C++'s new() does not make a big difference.
The only additional advice though, is that all pages should be "locked" in
the physical memory so to prevent them from being occasionally swapped out.
mlockall() with MCL_FUTURE comes in handy to this goal.

Of course, the rule remains to be the same as you pointed out, - calling
malloc() or most of other linux system calls is not something one would like
to do (at least being in good memory) inside time-critical code fragments.


>
> -Jeff
>
> _______________________________________________
> Xenomai-help mailing list
> Xenomai-help@domain.hid
> https://mail.gna.org/listinfo/xenomai-help
>



--
Best regards,
Dmitry Adamushko

[-- Attachment #2: Type: text/html, Size: 1708 bytes --]