[Xenomai-help] Encouraging Xenomai test results

[Xenomai-help] Encouraging Xenomai test results

[Martin Shepherd]

Just in case it's of interest to anybody, here are some results of
testing a custom RTDM device driver and Xenomai on a computer with a
Celeron 430 (1.8GHz) processor. As discussed in other threads, I have
to turn off X86_PM_TIMER and HPET_TIMER to get Xenomai working
properly on this system. So the test was performed with these timers
turned off in the kernel configuration.

My custom RTDM device driver works with PCI-DIO-72/96/120 digital I/O
cards from Acces I/O. This family of cards include from three to five
8255-5 parallel peripheral interface chips, providing from 72 to 120
bits of digital I/O, along with optional interrupt generation. I have
the 96-bit version. Using this, I connected a wire from an I/O pin
that my program configured to generate an interrupt on rising edges,
to a pin that I configured as a digital output. This allowed the test
program to trigger an interrupt by writing to said digital output bit,
and then measure how much time elapsed before the driver notified it
of the resulting interrupt. My test program ran under Xenomai in user
mode, and used one ioctl to write to the output pin, and another to
wait for the driver to indicate that it had received an interrupt from
the card. It read the time before asserting the output pin, and read
the time again, after being notified of the resulting interrupt. It
then used the difference between these two times as an estimate of the
roundtrip latency. This was done in a loop, rate-limited by sleeping
for 1ms after each measurement. This thus generated interrupts at
1KHz. After every 10000 measurements, the program printed out the min
and max latencies that it had seen so far.

In order to sample not only the interrupt latency, but also task
switching latency, I also ran a second realtime Xenomai task at
slightly lower priority. This executed an iterative double precision
recurrence algorithm (with no function calls) continually for about 3
seconds at a time, pausing for 1.2s between each run to give some time
to Linux itself.

After an hour of running the program (ie. 3600000 measurements), the
range of roundtrip latencies that the program measured, were from 9us
to 29us. This seems pretty good to me.

While performing these measurements, I tried running various Linux
loads, such as dd if=/dev/zero of=/dev/null, lots of "find" commands
to hammer the disk, etc, but nothing that I did in Linux seemed to
have any effect on the Xenomai timings.

One thing that I don't understand is why whenever I ran my Xenomai
test program, the Linux clock ran slow, losing about half an hour
during the hour-long test. My realtime floating-point load task was
designed to preempt the Linux kernel for about 3 seconds, then go to
sleep for about 1.2 seconds, to give Adeos time to replay pending
interrupts to the Linux kernel. Although the realtime interrupt
strobing task was still running during the 1.2s pauses in the load
task, it only ran for 30us every 1ms.  So I am wondering why so many
timer interrupts appear not to have been played back by Adeos? I
didn't seen any evidence of non-timer interrupts getting lost. Is this
perhaps a side effect of turning off X86_PM_TIMER and HPET_TIMER, and
leaving both Linux and Xenomai sharing the sole remaining timer?

Martin

Re: [Xenomai-help] Encouraging Xenomai test results

[Gilles Chanteperdrix]

Martin Shepherd wrote:
> One thing that I don't understand is why whenever I ran my Xenomai
> test program, the Linux clock ran slow, losing about half an hour
> during the hour-long test. My realtime floating-point load task was
> designed to preempt the Linux kernel for about 3 seconds, then go to
> sleep for about 1.2 seconds, to give Adeos time to replay pending
> interrupts to the Linux kernel. Although the realtime interrupt
> strobing task was still running during the 1.2s pauses in the load
> task, it only ran for 30us every 1ms.  So I am wondering why so many
> timer interrupts appear not to have been played back by Adeos? I
> didn't seen any evidence of non-timer interrupts getting lost. Is this
> perhaps a side effect of turning off X86_PM_TIMER and HPET_TIMER, and
> leaving both Linux and Xenomai sharing the sole remaining timer?

Adeos no longer plays back the missed interrupts. It only plays back one
interrupt and let Linux' lost tick compensation algorithms do the rest.
Running real-time tasks for 3 seconds without leaving Linux run is a bad
idea (it is even surprising that it works at all). For Linux to work
correctly, we usually advise people not to let real-time task run for a
period longer than Linux tick period.

-- 
					    Gilles.

Re: [Xenomai-help] Encouraging Xenomai test results

[Martin Shepherd]

On Thu, 14 May 2009, Gilles Chanteperdrix wrote:
> Adeos no longer plays back the missed interrupts. It only plays back one
> interrupt and let Linux' lost tick compensation algorithms do the rest.

Is there a way to get ipipe to report if it has had to drop any
interrupts for a given IRQ number?

> Running real-time tasks for 3 seconds without leaving Linux run is a bad
> idea (it is even surprising that it works at all).

Understood, although I still think that this was a useful exercise,
even if no sane application would lock out linux for so long, because
it showed me how the system will react if a bug or a hardware problem
causes my code to operate seriously out of spec. For safety reasons,
this is useful to know.

Anyway, I have now re-run the test program, this time configured to
only lock out Linux for 3 milliseconds at a time, separated by gaps of
1.5 milliseconds. As expected, this time the Linux clock, with its 4
millisecond period, appeared to be unaffected by the test. To be
precise, the Linux clock on this computer generally drifts at about
-1ms per minute (measured using a couple of tier-1 NTP servers), and
it continued to drift at this rate while I was running the test
program. The min and max roundtrip latencies of the new test run were
unchanged from what I reported earlier.

> For Linux to work correctly, we usually advise people not to let
> real-time task run for a period longer than Linux tick period.

That's a useful bit of information.

Thanks for the help,

Martin