AW: [Xenomai-core] Fundamental Questions

AW: [Xenomai-core] Fundamental Questions

[Roderik_Wildenburg]

Dear Jan,
thank you for taking time to answer my questions and 
sorry for the delayed response, but I have been busy 
with some other work. 
Please find my follow-up questions inserted in the text.

> > 
> > 1.)
> > Essentially the question deals with the problem, how long a 
> Xenomai task in secondary mode can be delayed by normal Linux tasks. 
> > In detail : we plan to to have a lot of "near realtime" 
> ethernet communication from within Xenomai using the normal 
> Linux network stack (calling the normal socket API). The 
> question now is, how our network communication is influenced 
> by other Linux tasks performing also network communication, 
> let´s say an FTP transfer ?
> 
> Depending on the "normal" networking load, you will suffer 
> from more or less frequent (indeterministic) packet delays. 

Do you have an idea about the dimension weare talking about :
less than a millisecond, few milliseconds, seconds, or is the
delay complete indeterministic ?

> Xenomai will not improve this in any way. If your task in 
> secondary mode tries to send some data and requires to take a 
> networking lock currently held by another Linux task, it can 
> take a real long time until this request is completed. 



But at least, after a (linux-)systemcall (from what task ever) finished, 
Xenomai gets controll back before any other linux task, isn´t it ?
This means : between systemcalls a rescheduling back to Xenomai is performed 
or isn´t it ??
Sorry for the next stupid question, but what is a network lock. With what kind of 
action a task can lock the complete stack ? And how long could it block the stack ?
Could you give me an example for better understanding ?


> This 
> gets better with PREEMPT_RT but still remains non-RT because 
> the Linux networking stack is not designed for hard real-time.
> 


Next stupid question : what is PREEMPT_RT ? Is this kernel 2.6 or is it the 
Monta Vista approach for real time (making the kernel more preemtable) ?


> 
> If you communication can be soft-RT, you could indeed avoid 
> the separation - but you will then have to live with the side 
> effects. All you can do then is try to lower the number of 
> deadline misses by keeping the standard network traffic low 
> and managing the bandwidth of the participants (the Linux 
> network stack has some support for QoS, at least
> 2.6 I think).
> 
> BTW, as long as your network is not separated or you have no 
> control over the traffic arriving at your system, picking the 
> Linux stack in favour of RTnet (which is compatible with 
> non-RT networks) is indeed generally recommended. This way 
> you keep indeterministic load away from the real-time subsystem.
> 

Unfortunatelly we don´t want to limit non realtime traffic, we just 
want to make shure, that deterministic traffic has a higher priority 
than non RT traffic (like in other RTOS like vxWorks). 
Indeterministic traffic should get just the leftover bandwith.
What do you mean with : "Rtnet is compatible with non-RT networks" ? 
I thought RTnet uses a time slice mechanism and therefore could not be 
mixed with systems transmitting when ever they want. Do you refer to VNICs ?

 

> > 
> > I have created a scheduling scenario and I would ask you to 
> have a look on it and to tell me whether it is correct or 
> not. Thank you !
> > An corresponding question about this scheduling is : are there 
> > differences between a 2.4 and 2.6 Linux kernel ? (for our PowerPC 
> > plattform we intend to use the 2.4 kernel for performance reasons)
> > 
> > Scheduling scenario :
> > (I hope formating is not destroyed by email transfer)
> > 
> > Time moves downwards
> > 
> > v-Xenomai 
> >      v-Linux kernel
> >           v-Linux processes
> > 
> >           l1   Linux task1 running
> >      s1 < l1   Linux task1 makes systemcall
> >      s1        Linux task1 systemcall processed
> > -------------  Linux scheduling   
> >           l2   Linux task2 starts to run
> >      s2 < l2   Linux task2 makes systemcall
> >      s2        Linux task2 systemcall processed
> > +++++++++++++  Xenomai scheduling
> > x3             Xenomai task3 starts to run => primary mode
> > x3 > s3        Xenomai task3 makes systemcall => secondary mode
> >      s3        Xenomai task3 systemcall processed 
> > -------------  Linux scheduling => Xenomai task preemted
> 
> This preemption will only happen if the target Linux task has 
> a higher priority or the Xenomai task on secondary mode has 
> to block on some resource to be become free. As I sketched 
> above, this can actually happen in the network stack.


What do you mean with "higher priority" ? I thought Xenomai has
a higher priority than anything else in the linux system.
Could you give mean example about the resource (related to network
communication) s3 could wait for ?

> 
> >      s1        Linux task1 systemcall processed
> >      s1 > l1   Linux task1 systemcall ready => Linux task1 
> continues 
> >           l1   Linux task1 continues
> > -------------  Linux scheduling 
> >      s2        Linux task2 systemcall processed
> >      s2 > l2   Linux task2 systemcall ready => Linux task2 
> continues 
> >           l2   Linux task2 continues
> > -------------  Linux scheduling => Xenomai Task3 systemcall 
> continued
> >      s3        Xenomai Task3 systemcall continued
> > x3 < s3        Xenomai Task3 systemcall ready => Xenomai 
> Task contiunes
> > x3             but has been delayed by normal Linux tasks
> > 
> > 




> > 2.)
> > This question is not so crucial, but nevertheless interesting :
> > Who are the people behind Xenomai (I did not find anything 
> about this on the Xenomai webpage), what is their background 
> (University, Company)and who impels /sponsors the Xenomai 
> development (University, Company, Community)? 
> > I think I "identified" at least 2 core developers : 
> Philippe Gerum and Gilles Chanteperdrix, but I don´t know 
> very much more.
> 

> I can add my/our own profile here: working at the university 
> of Hannover (Real-Time Systems Group) with and on Xenomai to 
> use it on mobile service robots and to do research on 
> real-time security and industrial automation. Our institute 
> is currently developing a larger robotic software framework 
> on top of it. Primarily, we see Xenomai as a mature platform 
> for doing real-time with Linux, i.e. as a very useful tool.
> 
> > The background for this question is, to get a feeling, how 
> future-proof the xenomai development is. As far as I can see 
> open source projects often die or stall when the initiators 
> decide that "growing flowers" is a much more pleasant job ;-).
> > 
> 
> We had to decide last year which way to go for our future 
> platforms, also with potential industrial/commercial usage in 
> mind (e.g.
> proprietary real-time algorithms as user-space applications). 
> Based on our experience with the RTAI (> 4 years), my own 
> insight into RTAI/fusion (Xenomai predecessor), and the 
> status of PREEMPT_RT, we selected fusion (now Xenomai) as the 
> most matured and future-proven one.
> 
> Xenomai is enjoying an increasing acceptance in industry. Our 
> industry partners e.g. are either planning to start new with 
> Xenomai or port over from other RT-extensions - of course 
> encouraged by our own good experience. I talked to a lot of 
> smaller and real large companies over the last months, 
> specifically here in Germany, that started new 
> automation/embedded projects on top of Linux/Xenomai (as 
> usual, most of their activities remain non-public - 
> unfortunately). They are convinced of the future of this project.
> 
> Moreover, I believe that Xenomai has a very healthy 
> community. We have a quite active crowd of contributors and 
> "plain" users. There is definitely more than one person 
> knowing the internals of the system (as in many projects, 
> there is still one person knowing them best...). And the code 
> is also well documented (exceptions apart).
> 
> No one can safely predict what will happen to a project when 
> one of the main contributors stops working on it. But given 
> the number of "advanced" Xenomai users already at this 
> comparably early project stage, I'm convinced it would even 
> survive such an exceptional scenario, i.e.
> it has reached a critical mass. But I have no indications 
> that Philippe or Gilles will become gardeners. ;)
> 
> This said, the best thing one can do as a long-term user of 
> an open source project is to support its development actively by
> 
>  o promoting it (yes, it can already be that simple)  o 
> reporting problems  o fixing minor bugs and posting the fixes 
>  o contributing new features and enhancements (drivers, board support,
>    API extensions, ...)
>  o or directly funding the development of such enhancements
> 
> > 
> > Thank you for taking time to answer these questions, which 
> are interesting to others too, I think !?
> > 
> 
> Hope I was able to provide you some helpful details and arguments.
> 

Yes, you did ! Thank you very much !
Things you said about the interest of larger companies, the widespread
knowledgement and code maturity sounds quite good to me.

Form your mentioning of "industrial partners" I conclude you are working with 
or for companies also, not just for your universty.
So i would like to know whether you also offer external consultations ? 
Just in case, we want to make shure that things, we plan to do are possible 
with Xenomai.

Best Regards
Roderik

Re: AW: [Xenomai-core] Fundamental Questions

[Jan Kiszka]

[-- Attachment #1: Type: text/plain, Size: 9049 bytes --]

Roderik_Wildenburg@domain.hid wrote:
> Dear Jan,
> thank you for taking time to answer my questions and 
> sorry for the delayed response, but I have been busy 
> with some other work. 
> Please find my follow-up questions inserted in the text.
> 
>>> 1.)
>>> Essentially the question deals with the problem, how long a 
>> Xenomai task in secondary mode can be delayed by normal Linux tasks. 
>>> In detail : we plan to to have a lot of "near realtime" 
>> ethernet communication from within Xenomai using the normal 
>> Linux network stack (calling the normal socket API). The 
>> question now is, how our network communication is influenced 
>> by other Linux tasks performing also network communication, 
>> let´s say an FTP transfer ?
>>
>> Depending on the "normal" networking load, you will suffer 
>> from more or less frequent (indeterministic) packet delays. 
> 
> Do you have an idea about the dimension weare talking about :
> less than a millisecond, few milliseconds, seconds, or is the
> delay complete indeterministic ?

Normally you will only see millisecond delays or less. But in case of
overload (heavy data transfers, miss-configured or virus-contaminated
nodes etc.) you may face several hundred milliseconds or more - up to
packets drops. That depends on the infrastructure (network layout,
switches and their QoS features) and cannot be answered generally. Good
QoS-aware switches can help here.

> 
>> Xenomai will not improve this in any way. If your task in 
>> secondary mode tries to send some data and requires to take a 
>> networking lock currently held by another Linux task, it can 
>> take a real long time until this request is completed. 
> 
> 
> 
> But at least, after a (linux-)systemcall (from what task ever) finished, 
> Xenomai gets controll back before any other linux task, isn´t it ?
> This means : between systemcalls a rescheduling back to Xenomai is performed 
> or isn´t it ??

The Xenomai domain regains control as soon as a) the real-time task in
secondary mode can continue or b) some other real-time task becomes
runnable.

> Sorry for the next stupid question, but what is a network lock. With what kind of 
> action a task can lock the complete stack ? And how long could it block the stack ?
> Could you give me an example for better understanding ?

"Network lock" was an oversimplification. Actually, this is a complex
topic, and I had a wrong model in mind. To clarify:

The networking stack contains a lot of locks (NIC transmission access,
routing and ARP tables, input and output queues, ...), but they are
non-preemptible for the transmission and reception path on standard
Linux (as long as I do not oversee some corner case). So, contention is
now happening to the point where your RT-task issues a Linux syscall.
The kernel then has to preempt the interrupted Linux task (= interrupted
by Xenomai when the RT-task became runnable) to let your task run in
secondary mode. This normally happens within a few hundred microseconds
(CONFIG_PREEMPT enabled in the kernel), but there can be scenarios where
preemption is disabled for several hundred milliseconds or more - though
very rarely with CONFIG_PREEMPT, but possible. That can be ok for soft
RT if you are able to handle the exceptions.

Besides this, another critical issue are delays due to buffer
acquisitions. If there is no fitting buffer at hand on packet arrival or
transmission request, it can take a while to free the necessary
resources, specifically if your whole system just happen to run short on
memory (swapping is taking place, some application leaked memory, ...).

> 
> 
>> This 
>> gets better with PREEMPT_RT but still remains non-RT because 
>> the Linux networking stack is not designed for hard real-time.
>>
> 
> 
> Next stupid question : what is PREEMPT_RT ? Is this kernel 2.6 or is it the 
> Monta Vista approach for real time (making the kernel more preemtable) ?

Actually, neither of both. It is the ongoing community effort (which
includes MontaVista) to make the kernel natively preemptible, it is not
MontaVista's own project. Watch out for "-rt" or "real-time preempt" on
the Linux kernel mailing list, or look at
http://people.redhat.com/mingo/realtime-preempt (don't ask me for the
latest user-space pieces, i.e. glibc patches, they are either still
unpublished, well hidden, or non-existent yet).

> 
>> If you communication can be soft-RT, you could indeed avoid 
>> the separation - but you will then have to live with the side 
>> effects. All you can do then is try to lower the number of 
>> deadline misses by keeping the standard network traffic low 
>> and managing the bandwidth of the participants (the Linux 
>> network stack has some support for QoS, at least
>> 2.6 I think).
>>
>> BTW, as long as your network is not separated or you have no 
>> control over the traffic arriving at your system, picking the 
>> Linux stack in favour of RTnet (which is compatible with 
>> non-RT networks) is indeed generally recommended. This way 
>> you keep indeterministic load away from the real-time subsystem.
>>
> 
> Unfortunatelly we don´t want to limit non realtime traffic, we just 
> want to make shure, that deterministic traffic has a higher priority 
> than non RT traffic (like in other RTOS like vxWorks). 

RT != RT. Soft RT is feasible with improved stacks, hard RT requires
different approaches, e.g. buffer pre-allocation or traffic control for
non-RT data. Specifically, when you read about TCP and real-time, this
can only refer to soft RT applications, as TCP do not have hard RT
characteristics (undefined timeout and packet repetition behaviour).

> Indeterministic traffic should get just the leftover bandwith.
> What do you mean with : "Rtnet is compatible with non-RT networks" ? 

It "speaks" the same UDP/IP protocol, just removes some dynamics from
problematic parts (ARP, IP fragmentation).

> I thought RTnet uses a time slice mechanism and therefore could not be 
> mixed with systems transmitting when ever they want. Do you refer to VNICs ?

That's true, mixing up RTnet with non-RT stations in the same physical
network doesn't make sense, it destroys the determinism of other RT
threads on the RTnet node.


To sum your options up:

1) Go the "hard" way, and let RTnet control both RT and non-RT traffic
as well as manage the buffer resources.

2) Mix up RT and non-RT traffic for the sake of good non-RT performance
and fair soft-RT qualities with optimisations like QoS-switches.

The question is if you can handle rare deadline misses or if you need
hard guarantees.

> 
>>> I have created a scheduling scenario and I would ask you to 
>> have a look on it and to tell me whether it is correct or 
>> not. Thank you !
>>> An corresponding question about this scheduling is : are there 
>>> differences between a 2.4 and 2.6 Linux kernel ? (for our PowerPC 
>>> plattform we intend to use the 2.4 kernel for performance reasons)
>>>
>>> Scheduling scenario :
>>> (I hope formating is not destroyed by email transfer)
>>>
>>> Time moves downwards
>>>
>>> v-Xenomai 
>>>      v-Linux kernel
>>>           v-Linux processes
>>>
>>>           l1   Linux task1 running
>>>      s1 < l1   Linux task1 makes systemcall
>>>      s1        Linux task1 systemcall processed
>>> -------------  Linux scheduling   
>>>           l2   Linux task2 starts to run
>>>      s2 < l2   Linux task2 makes systemcall
>>>      s2        Linux task2 systemcall processed
>>> +++++++++++++  Xenomai scheduling
>>> x3             Xenomai task3 starts to run => primary mode
>>> x3 > s3        Xenomai task3 makes systemcall => secondary mode
>>>      s3        Xenomai task3 systemcall processed 
>>> -------------  Linux scheduling => Xenomai task preemted
>> This preemption will only happen if the target Linux task has 
>> a higher priority or the Xenomai task on secondary mode has 
>> to block on some resource to be become free. As I sketched 
>> above, this can actually happen in the network stack.
> 
> 
> What do you mean with "higher priority" ? I thought Xenomai has
> a higher priority than anything else in the linux system.

Either a Xenomai shadow thread in secondary mode or a standard task with
SCHED_FIFO/_RR set could become runnable. Xenomai will allow this switch
 - as long as no other Xenomai thread in primary mode is runnable, which
would then trigger switching back to the Xenomai domain.

> Could you give mean example about the resource (related to network
> communication) s3 could wait for ?

Yep, rescheduling due to lock contention is not likely for the
networking scenario (see above), only voluntary blocking, e.g. when
waiting on yet unavailable data.

As Linux IRQs can arrive and trigger interruptions while in secondary
mode, you may furthermore consider to use the IRQ-shield of Xenomai (see
config) to improve the predictability.

Jan


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 250 bytes --]

Re: AW: [Xenomai-core] Fundamental Questions

[Christopher Stone]

[-- Attachment #1: Type: text/plain, Size: 11029 bytes --]

In light of the desire for support below, would the Xenomai team consider listing companies capable of commercial support on their website, or create another mailing list for us to announce commercial offerings around Xenomai. I am prepared to contribute to Xenomai in order to receive this privilege.  
   
  Sorry for discussing commercial issues on your mailing list. If it makes it any better, we are a very small company, just trying to make a living, not a big corporate conglomerate. 
   
  As a final point, I believe Xenomai is very well positioned to become very popular and "future proof". I believe the next frontier for Linux is industrial grade Linux, or Linux on the factory floor and Xenomai will end up the technology of choice to make that happen. Contrary to many opinions I have heard, I beleve the rt-preempt work done by Ingo Molnar will enhance Xenomai and not replace it. I also think the break from RTAI was very smart as it has given you the flexibility to move Xenomai in the direction it needs to go. The recent releases have made Xenomai ready for the commercial world. So, kudos to the Xenomai team. You guys are proving to be great leaders with the right technology at the right time.
   
  Cheers,
      Chris Stone.

Roderik_Wildenburg@domain.hid wrote:
  Dear Jan,
thank you for taking time to answer my questions and 
sorry for the delayed response, but I have been busy 
with some other work. 
Please find my follow-up questions inserted in the text.

> > 
> > 1.)
> > Essentially the question deals with the problem, how long a 
> Xenomai task in secondary mode can be delayed by normal Linux tasks. 
> > In detail : we plan to to have a lot of "near realtime" 
> ethernet communication from within Xenomai using the normal 
> Linux network stack (calling the normal socket API). The 
> question now is, how our network communication is influenced 
> by other Linux tasks performing also network communication, 
> let´s say an FTP transfer ?
> 
> Depending on the "normal" networking load, you will suffer 
> from more or less frequent (indeterministic) packet delays. 

Do you have an idea about the dimension weare talking about :
less than a millisecond, few milliseconds, seconds, or is the
delay complete indeterministic ?

> Xenomai will not improve this in any way. If your task in 
> secondary mode tries to send some data and requires to take a 
> networking lock currently held by another Linux task, it can 
> take a real long time until this request is completed. 



But at least, after a (linux-)systemcall (from what task ever) finished, 
Xenomai gets controll back before any other linux task, isn´t it ?
This means : between systemcalls a rescheduling back to Xenomai is performed 
or isn´t it ??
Sorry for the next stupid question, but what is a network lock. With what kind of 
action a task can lock the complete stack ? And how long could it block the stack ?
Could you give me an example for better understanding ?


> This 
> gets better with PREEMPT_RT but still remains non-RT because 
> the Linux networking stack is not designed for hard real-time.
> 


Next stupid question : what is PREEMPT_RT ? Is this kernel 2.6 or is it the 
Monta Vista approach for real time (making the kernel more preemtable) ?


> 
> If you communication can be soft-RT, you could indeed avoid 
> the separation - but you will then have to live with the side 
> effects. All you can do then is try to lower the number of 
> deadline misses by keeping the standard network traffic low 
> and managing the bandwidth of the participants (the Linux 
> network stack has some support for QoS, at least
> 2.6 I think).
> 
> BTW, as long as your network is not separated or you have no 
> control over the traffic arriving at your system, picking the 
> Linux stack in favour of RTnet (which is compatible with 
> non-RT networks) is indeed generally recommended. This way 
> you keep indeterministic load away from the real-time subsystem.
> 

Unfortunatelly we don´t want to limit non realtime traffic, we just 
want to make shure, that deterministic traffic has a higher priority 
than non RT traffic (like in other RTOS like vxWorks). 
Indeterministic traffic should get just the leftover bandwith.
What do you mean with : "Rtnet is compatible with non-RT networks" ? 
I thought RTnet uses a time slice mechanism and therefore could not be 
mixed with systems transmitting when ever they want. Do you refer to VNICs ?



> > 
> > I have created a scheduling scenario and I would ask you to 
> have a look on it and to tell me whether it is correct or 
> not. Thank you !
> > An corresponding question about this scheduling is : are there 
> > differences between a 2.4 and 2.6 Linux kernel ? (for our PowerPC 
> > plattform we intend to use the 2.4 kernel for performance reasons)
> > 
> > Scheduling scenario :
> > (I hope formating is not destroyed by email transfer)
> > 
> > Time moves downwards
> > 
> > v-Xenomai 
> > v-Linux kernel
> > v-Linux processes
> > 
> > l1 Linux task1 running
> > s1 < l1 Linux task1 makes systemcall
> > s1 Linux task1 systemcall processed
> > ------------- Linux scheduling 
> > l2 Linux task2 starts to run
> > s2 < l2 Linux task2 makes systemcall
> > s2 Linux task2 systemcall processed
> > +++++++++++++ Xenomai scheduling
> > x3 Xenomai task3 starts to run => primary mode
> > x3 > s3 Xenomai task3 makes systemcall => secondary mode
> > s3 Xenomai task3 systemcall processed 
> > ------------- Linux scheduling => Xenomai task preemted
> 
> This preemption will only happen if the target Linux task has 
> a higher priority or the Xenomai task on secondary mode has 
> to block on some resource to be become free. As I sketched 
> above, this can actually happen in the network stack.


What do you mean with "higher priority" ? I thought Xenomai has
a higher priority than anything else in the linux system.
Could you give mean example about the resource (related to network
communication) s3 could wait for ?

> 
> > s1 Linux task1 systemcall processed
> > s1 > l1 Linux task1 systemcall ready => Linux task1 
> continues 
> > l1 Linux task1 continues
> > ------------- Linux scheduling 
> > s2 Linux task2 systemcall processed
> > s2 > l2 Linux task2 systemcall ready => Linux task2 
> continues 
> > l2 Linux task2 continues
> > ------------- Linux scheduling => Xenomai Task3 systemcall 
> continued
> > s3 Xenomai Task3 systemcall continued
> > x3 < s3 Xenomai Task3 systemcall ready => Xenomai 
> Task contiunes
> > x3 but has been delayed by normal Linux tasks
> > 
> > 




> > 2.)
> > This question is not so crucial, but nevertheless interesting :
> > Who are the people behind Xenomai (I did not find anything 
> about this on the Xenomai webpage), what is their background 
> (University, Company)and who impels /sponsors the Xenomai 
> development (University, Company, Community)? 
> > I think I "identified" at least 2 core developers : 
> Philippe Gerum and Gilles Chanteperdrix, but I don´t know 
> very much more.
> 

> I can add my/our own profile here: working at the university 
> of Hannover (Real-Time Systems Group) with and on Xenomai to 
> use it on mobile service robots and to do research on 
> real-time security and industrial automation. Our institute 
> is currently developing a larger robotic software framework 
> on top of it. Primarily, we see Xenomai as a mature platform 
> for doing real-time with Linux, i.e. as a very useful tool.
> 
> > The background for this question is, to get a feeling, how 
> future-proof the xenomai development is. As far as I can see 
> open source projects often die or stall when the initiators 
> decide that "growing flowers" is a much more pleasant job ;-).
> > 
> 
> We had to decide last year which way to go for our future 
> platforms, also with potential industrial/commercial usage in 
> mind (e.g.
> proprietary real-time algorithms as user-space applications). 
> Based on our experience with the RTAI (> 4 years), my own 
> insight into RTAI/fusion (Xenomai predecessor), and the 
> status of PREEMPT_RT, we selected fusion (now Xenomai) as the 
> most matured and future-proven one.
> 
> Xenomai is enjoying an increasing acceptance in industry. Our 
> industry partners e.g. are either planning to start new with 
> Xenomai or port over from other RT-extensions - of course 
> encouraged by our own good experience. I talked to a lot of 
> smaller and real large companies over the last months, 
> specifically here in Germany, that started new 
> automation/embedded projects on top of Linux/Xenomai (as 
> usual, most of their activities remain non-public - 
> unfortunately). They are convinced of the future of this project.
> 
> Moreover, I believe that Xenomai has a very healthy 
> community. We have a quite active crowd of contributors and 
> "plain" users. There is definitely more than one person 
> knowing the internals of the system (as in many projects, 
> there is still one person knowing them best...). And the code 
> is also well documented (exceptions apart).
> 
> No one can safely predict what will happen to a project when 
> one of the main contributors stops working on it. But given 
> the number of "advanced" Xenomai users already at this 
> comparably early project stage, I'm convinced it would even 
> survive such an exceptional scenario, i.e.
> it has reached a critical mass. But I have no indications 
> that Philippe or Gilles will become gardeners. ;)
> 
> This said, the best thing one can do as a long-term user of 
> an open source project is to support its development actively by
> 
> o promoting it (yes, it can already be that simple) o 
> reporting problems o fixing minor bugs and posting the fixes 
> o contributing new features and enhancements (drivers, board support,
> API extensions, ...)
> o or directly funding the development of such enhancements
> 
> > 
> > Thank you for taking time to answer these questions, which 
> are interesting to others too, I think !?
> > 
> 
> Hope I was able to provide you some helpful details and arguments.
> 

Yes, you did ! Thank you very much !
Things you said about the interest of larger companies, the widespread
knowledgement and code maturity sounds quite good to me.

Form your mentioning of "industrial partners" I conclude you are working with 
or for companies also, not just for your universty.
So i would like to know whether you also offer external consultations ? 
Just in case, we want to make shure that things, we plan to do are possible 
with Xenomai.

Best Regards
Roderik

_______________________________________________
Xenomai-core mailing list
Xenomai-core@domain.hid
https://mail.gna.org/listinfo/xenomai-core






Christopher Stone
Principal
Sombrio Systems Inc.
www.openembedded.biz
613-831-1892

[-- Attachment #2: Type: text/html, Size: 12488 bytes --]

Re: AW: [Xenomai-core] Fundamental Questions

[Jan Kiszka]

[-- Attachment #1: Type: text/plain, Size: 2343 bytes --]

Christopher Stone wrote:
> In light of the desire for support below, would the Xenomai team
> consider listing companies capable of commercial support on their
> website, or create another mailing list for us to announce commercial
> offerings around Xenomai. I am prepared to contribute to Xenomai in
> order to receive this privilege.

We had such a page on the old RTAI site, though not really populated I
think to remember. I personally have no concerns creating such a forum
for Xenomai as well. I think we could start with a link sub-page on
xenomai.org.

> 
> Sorry for discussing commercial issues on your mailing list. If it
> makes it any better, we are a very small company, just trying to make
> a living, not a big corporate conglomerate.

I'm convinced that this is not a question of big or small. There should
just be a good balance between taking and giving.

> 
> As a final point, I believe Xenomai is very well positioned to become
> very popular and "future proof". I believe the next frontier for
> Linux is industrial grade Linux, or Linux on the factory floor and
> Xenomai will end up the technology of choice to make that happen.
> Contrary to many opinions I have heard, I beleve the rt-preempt work
> done by Ingo Molnar will enhance Xenomai and not replace it. I also

Yep, that is one important point why Xenomai is future-proof in my eyes.
The day may come when significant parts of the PREEMPT_RT effort are
merged into mainline. But then Xenomai will still be able to offer
either staged degrees of hard-RT side by side with that approach, or
become a wrapping layer on top of the new deterministic and fast
infrastructure (PREEMPT_RT + RT-extended glibc). No one should expect
that this will happen soon, PREEMPT_RT has just started its tricky way
into the kernel.

> think the break from RTAI was very smart as it has given you the
> flexibility to move Xenomai in the direction it needs to go. The
> recent releases have made Xenomai ready for the commercial world. So,
> kudos to the Xenomai team. You guys are proving to be great leaders
> with the right technology at the right time.
> 

This nice compliment clearly has to be passed to our smart maintainer!

Jan


PS: Do you already have specific plans for potential contributions? I'm
just curious.


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 250 bytes --]

RE: AW: [Xenomai-core] Fundamental Questions

[Christopher Stone]

I do not have specific plans. 

I am working on something I am currently calling the Xen Loadable Module or
XLM. It is an Xenomai application that when loaded, turns the Linux kernel
into a Xen compatible hypervisor. For the rationale behind it see here:
http://www.openembedded.biz/content/view/36/27.

When it is ready, in 6 to 8 weeks, I am prepared to contribute it. It is a
significant work, but, is an Xenomai application, so I don't know if you
guys want it. It does fit with your goals with respect to industrial grade
Linux. 

I think that XLM actually illustrates a key point that people forget when
they compare rt-preempt to Xenomai. I believe that in the industrial grade
Linux world, the ability to support multiple OS's is key, especially in
light of the emerging dual core CPU's. Due to ADEOS, Xenomai has the
infrastructure to support doing things like running eCos on one core and
Linux on the other, or eCos and Linux, side by side, on the same CPU. XLM is
designed to make this easy. Rt-preempt has no such capability. 

I am not trying to discredit rt-preempt. It is a significant and useful
piece of work and contains some pretty smart coding. However, in my view,
rt-preempt is just part of the solution required for industrial grade linux.
Things like rt-preempt, Xenomai, and hopefully XLM will all be pieces of a
comprehensive industrial grade linux solution.

If XLM is not a suitable contribution to Xenomai, then, I can contribute
other ways such as other feature development or bug fixing. I would need
some direction from the leaders in order to contribute in that way.

Cheers,
   Chris.

----------------------------------------
Christopher Stone
Principal
Sombrio Systems Inc.
www.openembedded.biz
613-831-1892

-----Original Message-----
From: Jan Kiszka [mailto:jan.kiszka@domain.hid] 
Sent: March 8, 2006 1:18 PM
To: chris@domain.hid
Cc: xenomai@xenomai.org
Subject: Re: AW: [Xenomai-core] Fundamental Questions

Christopher Stone wrote:
> In light of the desire for support below, would the Xenomai team
> consider listing companies capable of commercial support on their
> website, or create another mailing list for us to announce commercial
> offerings around Xenomai. I am prepared to contribute to Xenomai in
> order to receive this privilege.

We had such a page on the old RTAI site, though not really populated I
think to remember. I personally have no concerns creating such a forum
for Xenomai as well. I think we could start with a link sub-page on
xenomai.org.

> 
> Sorry for discussing commercial issues on your mailing list. If it
> makes it any better, we are a very small company, just trying to make
> a living, not a big corporate conglomerate.

I'm convinced that this is not a question of big or small. There should
just be a good balance between taking and giving.

> 
> As a final point, I believe Xenomai is very well positioned to become
> very popular and "future proof". I believe the next frontier for
> Linux is industrial grade Linux, or Linux on the factory floor and
> Xenomai will end up the technology of choice to make that happen.
> Contrary to many opinions I have heard, I beleve the rt-preempt work
> done by Ingo Molnar will enhance Xenomai and not replace it. I also

Yep, that is one important point why Xenomai is future-proof in my eyes.
The day may come when significant parts of the PREEMPT_RT effort are
merged into mainline. But then Xenomai will still be able to offer
either staged degrees of hard-RT side by side with that approach, or
become a wrapping layer on top of the new deterministic and fast
infrastructure (PREEMPT_RT + RT-extended glibc). No one should expect
that this will happen soon, PREEMPT_RT has just started its tricky way
into the kernel.

> think the break from RTAI was very smart as it has given you the
> flexibility to move Xenomai in the direction it needs to go. The
> recent releases have made Xenomai ready for the commercial world. So,
> kudos to the Xenomai team. You guys are proving to be great leaders
> with the right technology at the right time.
> 

This nice compliment clearly has to be passed to our smart maintainer!

Jan


PS: Do you already have specific plans for potential contributions? I'm
just curious.

Re: AW: [Xenomai-core] Fundamental Questions

[Jan Kiszka]

[-- Attachment #1: Type: text/plain, Size: 2504 bytes --]

Christopher Stone wrote:
> I do not have specific plans. 
> 
> I am working on something I am currently calling the Xen Loadable Module or
> XLM. It is an Xenomai application that when loaded, turns the Linux kernel
> into a Xen compatible hypervisor. For the rationale behind it see here:
> http://www.openembedded.biz/content/view/36/27.
> 
> When it is ready, in 6 to 8 weeks, I am prepared to contribute it. It is a
> significant work, but, is an Xenomai application, so I don't know if you
> guys want it. It does fit with your goals with respect to industrial grade
> Linux. 
> 
> I think that XLM actually illustrates a key point that people forget when
> they compare rt-preempt to Xenomai. I believe that in the industrial grade
> Linux world, the ability to support multiple OS's is key, especially in
> light of the emerging dual core CPU's. Due to ADEOS, Xenomai has the
> infrastructure to support doing things like running eCos on one core and
> Linux on the other, or eCos and Linux, side by side, on the same CPU. XLM is
> designed to make this easy. Rt-preempt has no such capability. 

Sounds interesting, especially when considering future systems with
hardware support for virtualisation, thus removing the need to patch the
guest OS (there are still people with the desire to run Windows aside
the RTOS core for visualisation etc.). And if your approach have real
advantages over Xen, specifically on embedded systems or in combination
with hard real-time, this could become really great.

> 
> I am not trying to discredit rt-preempt. It is a significant and useful
> piece of work and contains some pretty smart coding. However, in my view,
> rt-preempt is just part of the solution required for industrial grade linux.
> Things like rt-preempt, Xenomai, and hopefully XLM will all be pieces of a
> comprehensive industrial grade linux solution.
> 
> If XLM is not a suitable contribution to Xenomai, then, I can contribute

Let's wait for some code first so that things like intrusiveness etc.
can be analysed.

> other ways such as other feature development or bug fixing. I would need
> some direction from the leaders in order to contribute in that way.

It's often best to pick a domain you are interested in on your own. This
can drive the overall development application-oriented in a very
constructive way. When you are using Xenomai for some projects, you may
happen to stumble over rough edges or lacking features.

Jan


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 250 bytes --]

Re: AW: [Xenomai-core] Fundamental Questions

[Herman Bruyninckx]

On Wed, 8 Mar 2006, Jan Kiszka wrote:

> Christopher Stone wrote:
>> I do not have specific plans.
>>
>> I am working on something I am currently calling the Xen Loadable Module or
>> XLM. It is an Xenomai application that when loaded, turns the Linux kernel
>> into a Xen compatible hypervisor. For the rationale behind it see here:
>> http://www.openembedded.biz/content/view/36/27.
>>
>> When it is ready, in 6 to 8 weeks, I am prepared to contribute it. It is a
>> significant work, but, is an Xenomai application, so I don't know if you
>> guys want it. It does fit with your goals with respect to industrial grade
>> Linux.
>>
>> I think that XLM actually illustrates a key point that people forget when
>> they compare rt-preempt to Xenomai. I believe that in the industrial grade
>> Linux world, the ability to support multiple OS's is key, especially in
>> light of the emerging dual core CPU's. Due to ADEOS, Xenomai has the
>> infrastructure to support doing things like running eCos on one core and
>> Linux on the other, or eCos and Linux, side by side, on the same CPU. XLM is
>> designed to make this easy. Rt-preempt has no such capability.
>
> Sounds interesting, especially when considering future systems with
> hardware support for virtualisation, thus removing the need to patch the
> guest OS (there are still people with the desire to run Windows aside
> the RTOS core for visualisation etc.). And if your approach have real
> advantages over Xen, specifically on embedded systems or in combination
> with hard real-time, this could become really great.
>
I have the same positive reaction towards this XLM suggestion!

In some of our projects with machine tool builders, this kind of
virtualization is high on their wish list, because it's a perfect way
(hopefully) to combine lots of legacy GUI code with the advantages of a
realtime Linux-based controller.

Herman

--
   K.U.Leuven, Mechanical Eng.,  Mechatronics & Robotics Research Group
     <http://people.mech.kuleuven.be/~bruyninc> Tel: +32 16 322480

Disclaimer: http://www.kuleuven.be/cwis/email_disclaimer.htm

Re: AW: [Xenomai-core] Fundamental Questions

[Philippe Gerum]

Christopher Stone wrote:
> In light of the desire for support below, would the Xenomai team 
> consider listing companies capable of commercial support on their 
> website, or create another mailing list for us to announce commercial 
> offerings around Xenomai. I am prepared to contribute to Xenomai in 
> order to receive this privilege.  
>  
> Sorry for discussing commercial issues on your mailing list. If it makes 
> it any better, we are a very small company, just trying to make a 
> living, not a big corporate conglomerate.
>  

We are about to start such listing on xenomai.org. Requests for inclusion should
be sent to Bruno who manages the website. He will likely ask for the graphical
elements needed to add the link (i.e. a logo of the proper size), and a short 
description of the services being advertised.

> As a final point, I believe Xenomai is very well positioned to become 
> very popular and "future proof". I believe the next frontier for Linux 
> is industrial grade Linux, or Linux on the factory floor and Xenomai 
> will end up the technology of choice to make that happen. Contrary to 
> many opinions I have heard, I beleve the rt-preempt work done by Ingo 
> Molnar will enhance Xenomai and not replace it.

My comments about this issue will likely be much longer than needed, but it's an
opportunity I'm going to take in order to clarify a few things, as an attempt to
explain why Xeno is Xeno, and which evolution process has led to the current
implementation.

Xenomai attempts to address four basic use cases:

- situations where revalidating the entire Linux stack, with the proper level of
confidence with respect to reliability and worst-case latency, regardless of the 
set of drivers or sub-systems one might use in a setup during the project 
lifetime, is not an option. A Xenomai-based application relying on the primary 
mode for getting determinism won't be impacted by some hidden piece of Linux 
kernel code which happens not to conform with the new locking semantics preempt_rt 
enforces. This advantage will last at least until all vanilla kernel code, and all 
vendor-supplied drivers one might want to use in her design, have been made 
conformant to such requirements. Of course, one could argue that "rogue" drivers 
could also mask off interrupts at hw level and thus impact Xenomai too, but the 
main difference there is about how complex it is to locate and fix the culprit in 
a short time, without adverse side-effects on the rest of the system. Debugging 
and fixing a tiny RTOS core is easier than chasing priority inversions in a 
full-blown GPOS kernel trying to cope with real-time issues.

- situations where real-time application duties tend to become more complex as the 
hardware improves, i.e. when applications need to cope at the same time with a 
broad spectrum of real-time constraints and non real-time duties, ranging from 
fast data acquisition to deterministic networking, and/or FFT crunching, while 
still being able to get a decent throughput for moving large amount of data over 
various storage/network devices without adverse effect on latencies.

- situations where the hardware the application needs to run on is fairly
limited. In such a case, a safe approach is to handle the most time-critical tasks
over a context which does not depend on the reserve of CPU power Linux has at hand
to complete its current unpreemptable duties, before turning its head to the
critical stuff. On the other hand, dealing with real-time principles at the core
level of a GPOS for all kernel activities - including those who have no real-time
constraints - is expensive in terms of CPU horsepower, and engineering complexity.

- situations where legacy code exists, that used to run over a non-POSIX API,
and/or over a non-Linux real-time environment.

So far, I've seen the following approaches implemented for addressing some of 
those issues:

1) pretending that anything that doesn't fall into the low microsecond-range
real-time constraint has no real-time constraint, enforcing a complete isolation
between the real-time core and Linux, so one has the only option to ask Linux its
"best effort" for dealing with "trivial" (i.e. "non ultra-low latency real-time")
duties. This just does not scale with the increasing complexity of applications,
and has the additional downside to lower the incentive to keep the regular Linux
programming model available to the real-time developer (i.e. "who cares for
user-space, GDB and/or Valgrind since we are only interested in coding data
acquisition loops!". This one is also known as the REMBO syndrom, as in "REal Men
only Bring an Oscilloscope").

2) pretending that burying the real-time principles deep enough inside the vanilla
Linux core will solve all problems for all real-time configurations, with high 
performances and stability, for every use, on every platform. Frankly, I have no 
clue whether the preempt_rt effort will eventually succeed or fail, the way it is 
currently advertised, that is (i.e. high reliability with ultra-low worst-case 
latencies, in the range of what the co-kernels/co-schedulers can do right now, 
regardless of the hardware in question).

This said, I have no doubt that this effort will succeed (and possibly get merged
eventually) to provide reliable real-time support for strictly defined
configurations. It's also obvious that it already brought some positive updates to
the vanilla kernel, by uncovering synchronization bugs and latency spots, because 
common issues are shared between SMP and real-time architectures. However, 
something looks like almost certain: making an RTOS from a GPOS like Linux will 
always carry the additional cost of dealing with the new set of real-time 
semantics and constraints when designing new drivers or sub-systems.
This raises the following question in turn: how and by whom, these pieces of 
software - which would need to be "real-time blessed" so that they don't break the 
whole kernel determinism - are going to be contributed?

The technical barrier on entry for contributing to a 15-years old kernel is
already fairly high, which de facto reduces the number of potential contributors;
not all patch reviewers on the LKML are interested in real-time issues, some
sub-system maintainers already expressed their reluctance to consider those issues
as fundamental Linux ones, and the cursor between throughput/fairness and
responsiveness might not always be set to favour the latter. Add to this the
requirement that each piece of contributed software does not mess up with the
real-time semantics imposed by preempt_rt (e.g. new locking scheme), and you will
get to a potential issue: either the barrier on entry for pushing code to the
kernel is kept as it is, and only a limited portion of the drivers making their
way to the vanilla kernel would be real-time conformant, hence making the task of
validating a configuration initially picked from kernel.org for an industrial
use, a nightmare. Or, raise the barrier so that real-time requirements are always
encompassed, but dry up the common contribution path doing so, then resort to only 
asking real-time Linux vendors for getting "certified" new kernels, drivers or 
updates (which those vendors might not consider as a downside though...).

3) providing foreign API emulation layers based on simple POSIX remapping in 
regular user-space, which don't fully conform to the actual dynamic behaviour of 
the original RTOS, when they ever provide real-time support at all. Those are 
often advertised as tools for making first order ports to Linux, before eventually 
going for the "big jump" to the native Linux API, which de facto implies that 
changing the API the application is written over, would represent any desirable 
value in the end. But, it may happen that switching APIs from e.g. VxWorks's WIND 
kernel to POSIX would be a deliberate waste of time and resources, depending on 
maintenance considerations, albeit moving to Linux still has a value. Legacy code 
has by essence sedimentation issues, with various engineers having made lots of 
assumptions over time, regarding the way the underlying API behaves. Therefore, 
switching APIs is not such a trivial operation, especially for large industrial 
applications which have been in the field for several years.

The way Xenomai has evolved since 2001 attempts to capture the essentials of the
above matters:

- first a generic real-time core has been devised, in order to exploit the 
numerous commonalities existing in the vast majority of traditional RTOSes. 
Properly emulating foreign APIs from traditional RTOSes could then be done 
efficiently and reliably, using simple building blocks, without reinventing the 
wheel API after API. One bug fixed in the core strengthens all APIs relying on it. 
This core includes a scheduling and synchronization system which is manageable in 
size and complexity, and immune to any Linux locking and preemptability issues, so 
it's cheaper to validate, and easier to fix without incurring collateral damages 
on other sub-systems. This is likely the only but important gift of the entire 
co-kernel legacy.

- step two was Adeos for replacing the old and unfortunately patented interrupt
interception mechanism, so that we could build over an arch-independent interface,
which in turn would deal with all the nitty-gritty details regarding interrupt and
event prioritization within the Linux kernel. The abstraction brought by Adeos 
also explains why Xenomai can be fairly easily ported to other architectures.

- then came the user-space support, with the fundamental choice of cooperating
with Linux on the domain boundaries (Xenomai / Linux). Cooperating with it,
seeking integration, neither fighting it nor seeking isolation unless absolutely 
required for keeping a high degree of determinism. For instance, properly dealing 
with Linux signals - at the core and API levels - is instrumental in enabling GDB 
for all real-time applications. Uniformizing the priority scales between the core 
Xenomai APIs and Linux by using the common SCHED_FIFO scale allows real-time 
threads to migrate between both domains, without losing the priority information. 
Keeping a common priority scheme between both domains makes the applications see 
an unified execution space for real-time threads, providing a gradual scale of 
real-time guarantees by means of migration between domains. Generally speaking, 
cooperating with Linux means that, each progress vanilla Linux makes toward 
deterministic response time, is immediately available to Xenomai-based 
applications would they need to rely on the secondary mode for accessing the rich 
set of services Linux provides. As soon as preempt_rt provides the same level of 
performances, stability and maintainability than Xeno, the time will have come to 
rebase the scheduler innards of the Xenomai nucleus over native POSIX services, 
for configurations that support preempt_rt, and solely focus on Xenomai's core 
value there: i.e. API abstraction.

- step four has been the integration and deep coupling with RTDM for supporting 
real-time device drivers in Xenomai-based systems. Aside of providing a 
well-thought framework, RTDM has the major advantage of normalizing the 
interactions between the application, the real-time core and the Linux kernel when 
dealing with hardware devices, around the use of a POSIXish interface in 
user-space. In other words, it prevents most changes occurring in the real-time 
core, and between this core and Linux, to impact both application and device 
driver sides.

To sum up, I definitely agree that preempt_rt and Xenomai could be a
perfect match when it comes to providing a broad spectrum of real-time services
with different levels of determinism on a single system with enough CPU power and
efficient hardware. And the integration between both will be pursued by the 
Xenomai project. But for the time being, we still need the Xenomai nucleus to 
guarantee ultra-low scheduling latencies for low maintenance costs, and Adeos to 
guarantee ultra-low interrupt latencies, at the very least.

This said, a lot of steps remain to be reached in order for Xenomai to take its 
part in building a complete, free as in free speech, industrial-grade environment 
for building real-time systems, regardless of the preempt_rt situation.

Aside of this, I might be awfully wrong since the very beginning too, which in 
such a case, and to refer to a precedent post on this list, would be the sign that 
time has come for me to consider growing flowers as a better occupation than IT. :o>

> I also think the break 
> from RTAI was very smart as it has given you the flexibility to move 
> Xenomai in the direction it needs to go.
> The recent releases have made 
> Xenomai ready for the commercial world. So, kudos to the Xenomai team. 
> You guys are proving to be great leaders with the right technology at 
> the right time.
>  
> Cheers,
>     Chris Stone.
> 

-- 

Philippe (the unsuspected gardener).