[Xenomai-help] RT OS Selection advice for parallel processing

[Xenomai-help] RT OS Selection advice for parallel processing

[Rudolf Meijering]

[-- Attachment #1: Type: text/plain, Size: 861 bytes --]

Hi,

We are in the process of developing a non-tracking safety Radar and would
really appreciate some guidance in selecting a real-time operating system
especially with the aim of offloading processing to a GPU. Can anyone share
their experience with real-time GPU offloading on either Xenomai or Realtime
Linux (PREEMPT_RT patch)? I have read reports on 900us delays with Nvidia
graphics drivers, has this been resolved?

Background:
The application basically requires the radar to output a disable
signal, within 100ms, when an object is detected. Radar data is received
every 1 ms and processed.

Platform: x86_64, most likely Intel i7 CPU
Total processing latency: 100ms
Interrupt frequency: 1KHz

>From what I can read both Xenomai and Realtime Linux should easily meet
these requirements, but what about RT GPU support?

Thanks a lot,
--
Rudolf Meijering

[-- Attachment #2: Type: text/html, Size: 1171 bytes --]

Re: [Xenomai-help] RT OS Selection advice for parallel processing

[Jan Kiszka]

On 2011-02-03 09:57, Rudolf Meijering wrote:
> Hi,
> 
> We are in the process of developing a non-tracking safety Radar and would
> really appreciate some guidance in selecting a real-time operating system
> especially with the aim of offloading processing to a GPU. Can anyone share
> their experience with real-time GPU offloading on either Xenomai or Realtime
> Linux (PREEMPT_RT patch)? I have read reports on 900us delays with Nvidia
> graphics drivers, has this been resolved?

The NVIDIA CUDA bits cause problems to any RT scenario when you trigger
DMA memory allocations/releases, thus mapping during runtime (wbinvd). I
can't tell you details, but my colleagues somehow solved this for our
non-RT GPU offloading tasks although we received no information which
APIs can trigger the problematic code paths (the binary blob is in the
loop so only NVIDIA can tell).

> 
> Background:
> The application basically requires the radar to output a disable
> signal, within 100ms, when an object is detected. Radar data is received
> every 1 ms and processed.
> 
> Platform: x86_64, most likely Intel i7 CPU
> Total processing latency: 100ms
> Interrupt frequency: 1KHz
> 
> From what I can read both Xenomai and Realtime Linux should easily meet
> these requirements, but what about RT GPU support?

Given the fact that the NVIDIA driver is neither designed nor properly
tested against PREEMPT-RT (you need extra hacks to make it work), I'm a
bit skeptical that you can expect any RT properties when trying to use
the GPU in time critical paths. No one (except maybe the vendor) knows
the timing of the GPU hardware and software bits under critical load
scenarios.

If "just" that 100 ms is the deadline and you measure over a long time
that your GPU tasks always complete much earlier, you may argue that
it's /probably/ safe. But if you need to certify this, well...

[ And, personally, I don't trust the NVIDIA driver a lot. But that might
be influenced by the fact that latest versions lock up my notebook
regularly after resume and that I filed another obvious locking issue of
the compilable driver stub around mmap_sem without any response from the
vendor. ]

Jan

-- 
Siemens AG, Corporate Technology, CT T DE IT 1
Corporate Competence Center Embedded Linux

Re: [Xenomai-help] RT OS Selection advice for parallel processing

[Gilles Chanteperdrix]

Jan Kiszka wrote:
> [ And, personally, I don't trust the NVIDIA driver a lot. But that might
> be influenced by the fact that latest versions lock up my notebook
> regularly after resume and that I filed another obvious locking issue of
> the compilable driver stub around mmap_sem without any response from the
> vendor. ]

Some colleague of mine is using the NVIDIA driver (through the opengl
mapping, not CUDA/VDPAU, whatever) on high-end NVIDIA boards in a non RT
situation. he definitely observed some huge latencies not due to
scheduling issues, when uploading textures to the GPU. We are talking
tens of milliseconds here.

-- 
					    Gilles.

Re: [Xenomai-help] RT OS Selection advice for parallel processing

[Jan Kiszka]

On 2011-02-03 12:04, Gilles Chanteperdrix wrote:
> Jan Kiszka wrote:
>> [ And, personally, I don't trust the NVIDIA driver a lot. But that might
>> be influenced by the fact that latest versions lock up my notebook
>> regularly after resume and that I filed another obvious locking issue of
>> the compilable driver stub around mmap_sem without any response from the
>> vendor. ]
> 
> Some colleague of mine is using the NVIDIA driver (through the opengl
> mapping, not CUDA/VDPAU, whatever) on high-end NVIDIA boards in a non RT
> situation. he definitely observed some huge latencies not due to
> scheduling issues, when uploading textures to the GPU. We are talking
> tens of milliseconds here.

Ugh, tens of milliseconds is heavy, more than I would expect from
wbinvd. OTOH, uploading textures may involve mapping the RAM that
contains them for the GPU. To exclude that as source, he could try
instrumenting CACHE_FLUSH() (in nv-linux.h).

Jan

-- 
Siemens AG, Corporate Technology, CT T DE IT 1
Corporate Competence Center Embedded Linux

Re: [Xenomai-help] RT OS Selection advice for parallel processing

[Gilles Chanteperdrix]

Jan Kiszka wrote:
> On 2011-02-03 12:04, Gilles Chanteperdrix wrote:
>> Jan Kiszka wrote:
>>> [ And, personally, I don't trust the NVIDIA driver a lot. But that might
>>> be influenced by the fact that latest versions lock up my notebook
>>> regularly after resume and that I filed another obvious locking issue of
>>> the compilable driver stub around mmap_sem without any response from the
>>> vendor. ]
>> Some colleague of mine is using the NVIDIA driver (through the opengl
>> mapping, not CUDA/VDPAU, whatever) on high-end NVIDIA boards in a non RT
>> situation. he definitely observed some huge latencies not due to
>> scheduling issues, when uploading textures to the GPU. We are talking
>> tens of milliseconds here.
> 
> Ugh, tens of milliseconds is heavy, more than I would expect from
> wbinvd. OTOH, uploading textures may involve mapping the RAM that
> contains them for the GPU. To exclude that as source, he could try
> instrumenting CACHE_FLUSH() (in nv-linux.h).

No, it really looks like a bug in the blob. Of course the 40ms is a spot
from time to time, usually, the upload is really fast.

-- 
					    Gilles.

Re: [Xenomai-help] RT OS Selection advice for parallel processing

[Jan Kiszka]

On 2011-02-03 12:27, Gilles Chanteperdrix wrote:
> Jan Kiszka wrote:
>> On 2011-02-03 12:04, Gilles Chanteperdrix wrote:
>>> Jan Kiszka wrote:
>>>> [ And, personally, I don't trust the NVIDIA driver a lot. But that might
>>>> be influenced by the fact that latest versions lock up my notebook
>>>> regularly after resume and that I filed another obvious locking issue of
>>>> the compilable driver stub around mmap_sem without any response from the
>>>> vendor. ]
>>> Some colleague of mine is using the NVIDIA driver (through the opengl
>>> mapping, not CUDA/VDPAU, whatever) on high-end NVIDIA boards in a non RT
>>> situation. he definitely observed some huge latencies not due to
>>> scheduling issues, when uploading textures to the GPU. We are talking
>>> tens of milliseconds here.
>>
>> Ugh, tens of milliseconds is heavy, more than I would expect from
>> wbinvd. OTOH, uploading textures may involve mapping the RAM that
>> contains them for the GPU. To exclude that as source, he could try
>> instrumenting CACHE_FLUSH() (in nv-linux.h).
> 
> No, it really looks like a bug in the blob. Of course the 40ms is a spot
> from time to time, usually, the upload is really fast.

Unless it's a stall on the PCI bus, the blob should leaves some traces
in ftrace.

Anyway, the key message is the same: you don't know for sure what the
blob is doing.

Jan

-- 
Siemens AG, Corporate Technology, CT T DE IT 1
Corporate Competence Center Embedded Linux

Re: [Xenomai-help] RT OS Selection advice for parallel processing

[Gilles Chanteperdrix]

Jan Kiszka wrote:
> On 2011-02-03 12:27, Gilles Chanteperdrix wrote:
>> Jan Kiszka wrote:
>>> On 2011-02-03 12:04, Gilles Chanteperdrix wrote:
>>>> Jan Kiszka wrote:
>>>>> [ And, personally, I don't trust the NVIDIA driver a lot. But that might
>>>>> be influenced by the fact that latest versions lock up my notebook
>>>>> regularly after resume and that I filed another obvious locking issue of
>>>>> the compilable driver stub around mmap_sem without any response from the
>>>>> vendor. ]
>>>> Some colleague of mine is using the NVIDIA driver (through the opengl
>>>> mapping, not CUDA/VDPAU, whatever) on high-end NVIDIA boards in a non RT
>>>> situation. he definitely observed some huge latencies not due to
>>>> scheduling issues, when uploading textures to the GPU. We are talking
>>>> tens of milliseconds here.
>>> Ugh, tens of milliseconds is heavy, more than I would expect from
>>> wbinvd. OTOH, uploading textures may involve mapping the RAM that
>>> contains them for the GPU. To exclude that as source, he could try
>>> instrumenting CACHE_FLUSH() (in nv-linux.h).
>> No, it really looks like a bug in the blob. Of course the 40ms is a spot
>> from time to time, usually, the upload is really fast.
> 
> Unless it's a stall on the PCI bus, the blob should leaves some traces
> in ftrace.

I am talking about the kernel-space blob. ftrance does just show that
the ioctl has been submitted.



-- 
					    Gilles.

Re: [Xenomai-help] RT OS Selection advice for parallel processing

[Jan Kiszka]

On 2011-02-03 12:49, Gilles Chanteperdrix wrote:
> Jan Kiszka wrote:
>> On 2011-02-03 12:27, Gilles Chanteperdrix wrote:
>>> Jan Kiszka wrote:
>>>> On 2011-02-03 12:04, Gilles Chanteperdrix wrote:
>>>>> Jan Kiszka wrote:
>>>>>> [ And, personally, I don't trust the NVIDIA driver a lot. But that might
>>>>>> be influenced by the fact that latest versions lock up my notebook
>>>>>> regularly after resume and that I filed another obvious locking issue of
>>>>>> the compilable driver stub around mmap_sem without any response from the
>>>>>> vendor. ]
>>>>> Some colleague of mine is using the NVIDIA driver (through the opengl
>>>>> mapping, not CUDA/VDPAU, whatever) on high-end NVIDIA boards in a non RT
>>>>> situation. he definitely observed some huge latencies not due to
>>>>> scheduling issues, when uploading textures to the GPU. We are talking
>>>>> tens of milliseconds here.
>>>> Ugh, tens of milliseconds is heavy, more than I would expect from
>>>> wbinvd. OTOH, uploading textures may involve mapping the RAM that
>>>> contains them for the GPU. To exclude that as source, he could try
>>>> instrumenting CACHE_FLUSH() (in nv-linux.h).
>>> No, it really looks like a bug in the blob. Of course the 40ms is a spot
>>> from time to time, usually, the upload is really fast.
>>
>> Unless it's a stall on the PCI bus, the blob should leaves some traces
>> in ftrace.
> 
> I am talking about the kernel-space blob. ftrance does just show that
> the ioctl has been submitted.

Function tracing takes you at least into the blob wrapper, both for
requests issued to the blob and for its callbacks to access Linux
services. It looks like NVIDIA as concentrating both arch and kernel
specific abstractions there, not in the blob (but that's just an
impression).

Jan

-- 
Siemens AG, Corporate Technology, CT T DE IT 1
Corporate Competence Center Embedded Linux

Re: [Xenomai-help] RT OS Selection advice for parallel processing

[Gilles Chanteperdrix]

Jan Kiszka wrote:
> On 2011-02-03 12:49, Gilles Chanteperdrix wrote:
>> Jan Kiszka wrote:
>>> On 2011-02-03 12:27, Gilles Chanteperdrix wrote:
>>>> Jan Kiszka wrote:
>>>>> On 2011-02-03 12:04, Gilles Chanteperdrix wrote:
>>>>>> Jan Kiszka wrote:
>>>>>>> [ And, personally, I don't trust the NVIDIA driver a lot. But that might
>>>>>>> be influenced by the fact that latest versions lock up my notebook
>>>>>>> regularly after resume and that I filed another obvious locking issue of
>>>>>>> the compilable driver stub around mmap_sem without any response from the
>>>>>>> vendor. ]
>>>>>> Some colleague of mine is using the NVIDIA driver (through the opengl
>>>>>> mapping, not CUDA/VDPAU, whatever) on high-end NVIDIA boards in a non RT
>>>>>> situation. he definitely observed some huge latencies not due to
>>>>>> scheduling issues, when uploading textures to the GPU. We are talking
>>>>>> tens of milliseconds here.
>>>>> Ugh, tens of milliseconds is heavy, more than I would expect from
>>>>> wbinvd. OTOH, uploading textures may involve mapping the RAM that
>>>>> contains them for the GPU. To exclude that as source, he could try
>>>>> instrumenting CACHE_FLUSH() (in nv-linux.h).
>>>> No, it really looks like a bug in the blob. Of course the 40ms is a spot
>>>> from time to time, usually, the upload is really fast.
>>> Unless it's a stall on the PCI bus, the blob should leaves some traces
>>> in ftrace.
>> I am talking about the kernel-space blob. ftrance does just show that
>> the ioctl has been submitted.
> 
> Function tracing takes you at least into the blob wrapper, both for
> requests issued to the blob and for its callbacks to access Linux
> services. It looks like NVIDIA as concentrating both arch and kernel
> specific abstractions there, not in the blob (but that's just an
> impression).

Sorry, I misunderstood ftrace, I thought strace. No, we did not try ftrace.

-- 
					    Gilles.

Re: [Xenomai-help] RT OS Selection advice for parallel processing

[Rudolf Meijering]

[-- Attachment #1: Type: text/plain, Size: 2914 bytes --]

Thanks a lot for everyone's input! It seems like for our application, GPU
off-loading is not the way to go. We have redesigned our system for all
processing to take place on the Intel i7. Total maximum processing latency
stays 100ms, and interrupt frequency goes up to 1.5 KHz.

I have done some research on the pro's and con's of PREEMPT_RT vs Xenomai.
As far as I can determine some of the benefits include:
1. Programming with Xenomai native API's are more elegant than POSIX
real-time API’s.
2. Xenomai has possibly less overhead than a full preemptive kernel. The
user interface does not require RT capabilities.

Are there any other substantial benefits worth considering? Which would you
use?

Regards,

On Thu, Feb 3, 2011 at 3:21 PM, Gilles Chanteperdrix <
gilles.chanteperdrix@xenomai.org> wrote:

> Jan Kiszka wrote:
> > On 2011-02-03 12:49, Gilles Chanteperdrix wrote:
> >> Jan Kiszka wrote:
> >>> On 2011-02-03 12:27, Gilles Chanteperdrix wrote:
> >>>> Jan Kiszka wrote:
> >>>>> On 2011-02-03 12:04, Gilles Chanteperdrix wrote:
> >>>>>> Jan Kiszka wrote:
> >>>>>>> [ And, personally, I don't trust the NVIDIA driver a lot. But that
> might
> >>>>>>> be influenced by the fact that latest versions lock up my notebook
> >>>>>>> regularly after resume and that I filed another obvious locking
> issue of
> >>>>>>> the compilable driver stub around mmap_sem without any response
> from the
> >>>>>>> vendor. ]
> >>>>>> Some colleague of mine is using the NVIDIA driver (through the
> opengl
> >>>>>> mapping, not CUDA/VDPAU, whatever) on high-end NVIDIA boards in a
> non RT
> >>>>>> situation. he definitely observed some huge latencies not due to
> >>>>>> scheduling issues, when uploading textures to the GPU. We are
> talking
> >>>>>> tens of milliseconds here.
> >>>>> Ugh, tens of milliseconds is heavy, more than I would expect from
> >>>>> wbinvd. OTOH, uploading textures may involve mapping the RAM that
> >>>>> contains them for the GPU. To exclude that as source, he could try
> >>>>> instrumenting CACHE_FLUSH() (in nv-linux.h).
> >>>> No, it really looks like a bug in the blob. Of course the 40ms is a
> spot
> >>>> from time to time, usually, the upload is really fast.
> >>> Unless it's a stall on the PCI bus, the blob should leaves some traces
> >>> in ftrace.
> >> I am talking about the kernel-space blob. ftrance does just show that
> >> the ioctl has been submitted.
> >
> > Function tracing takes you at least into the blob wrapper, both for
> > requests issued to the blob and for its callbacks to access Linux
> > services. It looks like NVIDIA as concentrating both arch and kernel
> > specific abstractions there, not in the blob (but that's just an
> > impression).
>
> Sorry, I misunderstood ftrace, I thought strace. No, we did not try ftrace.
>
> --
>                                             Gilles.
>

[-- Attachment #2: Type: text/html, Size: 3804 bytes --]

Re: [Xenomai-help] RT OS Selection advice for parallel processing

[Gilles Chanteperdrix]

Rudolf Meijering wrote:
> Thanks a lot for everyone's input! It seems like for our application, GPU
> off-loading is not the way to go. We have redesigned our system for all
> processing to take place on the Intel i7. Total maximum processing latency
> stays 100ms, and interrupt frequency goes up to 1.5 KHz.
> 
> I have done some research on the pro's and con's of PREEMPT_RT vs Xenomai.
> As far as I can determine some of the benefits include:
> 1. Programming with Xenomai native API's are more elegant than POSIX
> real-time API’s.
> 2. Xenomai has possibly less overhead than a full preemptive kernel. The
> user interface does not require RT capabilities.
> 
> Are there any other substantial benefits worth considering? Which would you
> use?

Another factor, is that with Xenomai, the application needs a clear
separation between real-time and non real-time tasks. They do not use
the same services, they need to interact through special means. This is
a double edged sword:
- on one edge, with this clear separation, you have no doubt about which
code will be deterministic and which will not be;
- on the other edge, you end up using two APIS for interacting with two
different schedulers, which complicates things a bit.

-- 
                                                                Gilles.

Re: [Xenomai-help] RT OS Selection advice for parallel processing

[Richard Cochran]

On Fri, Feb 04, 2011 at 03:33:04PM +0200, Rudolf Meijering wrote:
> I have done some research on the pro's and con's of PREEMPT_RT vs Xenomai.
> As far as I can determine some of the benefits include:
> 1. Programming with Xenomai native API's are more elegant than POSIX
> real-time API’s.
> 2. Xenomai has possibly less overhead than a full preemptive kernel. The
> user interface does not require RT capabilities.
> 
> Are there any other substantial benefits worth considering? Which would you
> use?

In my view, the most important advantage of Xenomai is superior real
time performance. To convince yourself about this, try the following
simple experiment.

1. Compile both RT_PREEMPT and Xenomai for your target hardware.

2. Run cyclictest under various loads and priorities. (For Xenomai, be
   sure to use the version of cyclictest included with the Xenomai libraries.)

For the Freescale P2020 PowerPC platform, I recently tested Xenomai
2.4 (kernel 2.6.30) and the latest RT_PREEMPT (kernel 2.6.33). Xenomai
has maximal latencies around 10 usec, and RT_PREEMPT has around 40
usec, but with occasional spikes to over 400 usec.

Obviously, there are bugs or unfixed latencies in RT_PREEMPT on that
platform. One can always roll up the sleeves and go fixing these
problems, but my point is that Xenomai offers better performance out
of the box.

In general, I would choose a Linux flavor for an application like
this:

1. Does plain old Linux offer good enough real time performance for my
   needs? If yes, then use Linux.

2. Does RT_PREEMPT do it? If yes, use it, because it is closer to
   mainline Linux than Xenomai.

3. Does Xenomai fill the bill? If yes, use it.

4. Out of choices, you can't use Linux after all.

For my needs, I have landed at choice #3 more than once. My experience
with Xenomai has been extremely positive, and after meeting the main
developers (Philippe, Gilles, and Jan) at the first ever Xenomai users
meeting (at the 11th RTLW, 2009, in Dresden) I have tremendous
confidence in them. Also, it was eye opening to see how Xenomai has
been used in several demanding commerical projects.

Another important consideration is which kernel version to use. Both
Xenomai and RT_PREEMPT are only offered on selected kernel
versions. It might be that some new Linux kernel feature is not
present in your RT kernel version, so you have to choose carefully.

Finally, for safety critical applications, it might be hard to get a
Linux based system certified. I have never attempted this, but I
imagine that an analytical correctness "proof" would be infeasible for
a Linux based system.

HTH,

Richard