Good performance (hard realtime ??) on 2.6.16 patched with patch-2.6.16-rt29 from Ingo Molnar

Good performance (hard realtime ??) on 2.6.16 patched with patch-2.6.16-rt29 from Ingo Molnar

[kernel]

Hello, 

I have tested 2.6.16 patched with 2.6.16-rt29 on two ordinary boxes (Dell 
PIII, 780 Mhz, IDE disks) connected back to back with ethernet cross at 
100Mbit, one broadcasting small udp messages @50Hz, the other receiving 
those messages. Distro Mandriva 2006.0, used .config from them. 

Conditions:
* Sending and receiving process @RT priority, FIFO, prio 99
* Sending box is almost idle, receiving box heavily loaded (cpu / mem).
* Watchdog/0 non RT (this setting is critical!)
* IRQ from network card on RT prio, rx thread from softirq-net-rx on RT prio 
(FIFO)
* posix_cpu_timer on RT
* kernel compiled without preemption debugging, complete realtime, 
scheduler@100Hz
* receiving process memory locked MCL_CURRENT|MCL_FUTURE
* Heavily loaded (continuously compiling kernel, with 100 jobs in parallel) 

I see following figures:
max latency 512 microseconds on receiving box, during 24 hours of heavily 
testing. 

** this looks like hard realtime, am I correct, or is this coincidence ? 

Thanks
Carl. 

Re: Good performance (hard realtime ??) on 2.6.16 patched with patch-2.6.16-rt29 from Ingo Molnar

[Felix Oxley]

On 12 Jun 2006, at 10:50, kernel@wolff-online.nl wrote:

> Hello,
> I have tested 2.6.16 patched with 2.6.16-rt29 on two ordinary boxes  
> (Dell PIII, 780 Mhz, IDE disks) connected back to back with  
> ethernet cross at 100Mbit, one broadcasting small udp messages  
> @50Hz, the other receiving those messages. Distro Mandriva 2006.0,  
> used .config from them.
> Conditions:
> * Sending and receiving process @RT priority, FIFO, prio 99
> * Sending box is almost idle, receiving box heavily loaded (cpu /  
> mem).
> * Watchdog/0 non RT (this setting is critical!)
> * IRQ from network card on RT prio, rx thread from softirq-net-rx  
> on RT prio (FIFO)
> * posix_cpu_timer on RT
> * kernel compiled without preemption debugging, complete realtime,  
> scheduler@100Hz
> * receiving process memory locked MCL_CURRENT|MCL_FUTURE
> * Heavily loaded (continuously compiling kernel, with 100 jobs in  
> parallel)
> I see following figures:
> max latency 512 microseconds on receiving box, during 24 hours of  
> heavily testing.
> ** this looks like hard realtime, am I correct, or is this  
> coincidence ?
> Thanks
> Carl.
>

Could you explain to me, in nice easy steps, how you got the  
measurement?
Specifically what triggered the 'start' of your time slice?
BTW, where you using hi resolution timers?
You say the max latency was 512usecs. What were the mean and mode?

(Regarding Hard Real Time, my understanding is that that depends on a  
_guarantee_ that the system will always be able to produce the  
'result' within the required interval. Ingo's -rt patches may give  
exceedingly good responsiveness but they offer no guarantees, so they  
cannot be considered Hard Real Time)

//felix

Re: Good performance (hard realtime ??) on 2.6.16 patched with patch-2.6.16-rt29 from Ingo Molnar

[Carl, Wolff]

Felix Oxley writes: 

> Could you explain to me, in nice easy steps, how you got the  measurement?
1) for details see: http://tapas.affenbande.org/?page_id=6. 

2) Sending box and sending process
* periodically sends udp message using broadcast adress 192.168.2.255
* this process at RT priority FIFO (using chrt tool)
* put IRQ of network card to RT 99 FIFO prio
* box is idle (to create sort of 'stable') reference timing 

3) receiving box
* blocking read on udp socket
* put to RT FIFO 99 (chrt tool)
* process cannot be swapped out (mlockall call)
* while [1] do make -j100 && make clean; done;
* put softiqr-net-rx to RT FIFO 99 (using chrt)
* watchdog/0 not RT (i'm not sure this is mandatory)
* put IRQ of network card to RT FIFO prio 99
* Time measurement the cpu clock using gettimeofday calls 

> Specifically what triggered the 'start' of your time slice?
on sending box: usleep with 20000 interval (20 millisecond)
on receiving box: blocking read on udp socket 

> BTW, where you using hi resolution timers?
No, only usleep (maye it uses highres timers in the libs/kernel) 

> You say the max latency was 512usecs. What were the mean and mode?
Not measured... 

> (Regarding Hard Real Time, my understanding is that that depends on a  
> _guarantee_ that the system will always be able to produce the  'result' 
> within the required interval. Ingo's -rt patches may give  exceedingly 
> good responsiveness but they offer no guarantees, so they  cannot be 
> considered Hard Real Time)
Correct. In my case the repsonsiveness is acceptable in a real time control 
system where sporadic deadline misses are acceptable. 

Thanks
Carl. 

Re: Good performance (hard realtime ??) on 2.6.16 patched with patch-2.6.16-rt29 from Ingo Molnar

[Lee Revell]

On Mon, 2006-06-12 at 11:12 +0100, Felix Oxley wrote:
> (Regarding Hard Real Time, my understanding is that that depends on a
> _guarantee_ that the system will always be able to produce the  
> 'result' within the required interval. Ingo's -rt patches may give  
> exceedingly good responsiveness but they offer no guarantees, so they
> cannot be considered Hard Real Time) 

The -rt kernel is capable of hard realtime, modulo any bugs, but no one
has yet done an analysis of the few non-preemptible code paths to
determine what guarantees it can make.

Lee

Re: Good performance (hard realtime ??) on 2.6.16 patched with patch-2.6.16-rt29 from Ingo Molnar

[Esben Nielsen]

On Mon, 12 Jun 2006, Lee Revell wrote:

> On Mon, 2006-06-12 at 11:12 +0100, Felix Oxley wrote:
>> (Regarding Hard Real Time, my understanding is that that depends on a
>> _guarantee_ that the system will always be able to produce the
>> 'result' within the required interval. Ingo's -rt patches may give
>> exceedingly good responsiveness but they offer no guarantees, so they
>> cannot be considered Hard Real Time)
>
> The -rt kernel is capable of hard realtime, modulo any bugs, but no one
> has yet done an analysis of the few non-preemptible code paths to
> determine what guarantees it can make.
>

For me a "hard real time system" is one where a missing a deadline is a 
bug just as well as having a stray pointer is a bug. Therefore any code 
running non-preemptible and not being bound is a bug. In fact you have to 
consider all code running at equal and higher priority than your RT 
application - and do remember that PI can move tasks up in priority as well!

One may ask: What is bounded? The plist is an examble where 
operations are "bounded" mathematically speaking, but in practise one 
will never come even close to that bound. It is therefore very unlikely 
that you hit the worst case behaviour in a lab test before deploying your 
system - and that is very unfortunate :-(
I have, in another place, made a real priority heap, which operates in 
O(log N), N being the number of elements. N can in  principle be very big 
- but it is limited to the amount of elements in the system (which was 
static for the given application) and that is certainly limited to the 
amount of memory of the system. And log 1G is still only 9 compared to 
log 100 which is 2. Thus _in practice_ something O(log N) is can be 
considered bounded and is more testabl, i.e. you can come closer to the 
worst case behaviour in a lab test.

Which OSes, by the way, makes these mathematically guarentees? The hard 
realtime OS I work with at work certainly does _not_ give such. Yes, they 
do in the sales material, and the core of the OS does have a certification 
for safety critical systems. But I can still find exambles of unbounded 
code. The mutexes, forinstance, does not use plists and thus adding a 
waiter on a mutex is not a bounded operation, _theoretically_ speaking. A 
lot of the drivers and platform specific code are really bad. But the 
worst is that it doesn't implement priority inheritance correctly: It 
leaves a thread boosted until it releases the last mutex held. Therefore, 
something you considered low priority, non-RT, code can suddenly run at 
high priority!

But then again: That is an _embedded_ OS. We should in principle have 
control over _all_ the code and therefore should be able to avoid the
almost theoretical pitfalls of unbounded code.
Linux is supposed to be a general perpose OS. I.e. not priviliged users 
can run whatever code they want with normal priority (SCHED_OTHER) and the 
system should still be hard realtime! Therefore Linux has to be implemented
even without the theoretical unbounded code.

When I compare the state of preempt-realtime and the RTOS used at work, I 
would say preempt-realtime does pretty well. Especially the better 
priority inheritance implementation and the MMU support makes it far 
better at shielding the RT application from the arbitrary non-RT code. 
With the RTOS you have to be sure that every piece of code running on the 
system behaves correctly. With preempt-realtime Linux you "only" have to 
thrust the kernel and your RT application itself.

Esben


> Lee
>
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/
>