[Xenomai-help] Massive improvement when Xenomai patched

[Xenomai-help] Massive improvement when Xenomai patched

[Mike McTernan]

Hi,

I've patched Blackfin uClinux 2008R1.5 with Adeos and Xenomai 2.4.6.1
and started benchmarking our system.  At this point the apps are
unmodified and I only change between a Xenomai enabled kernel and an
unmodified kernel.  This is verified by checking the boot messages and
the process list under /proc/xenomai/stats.

The critical app creates a thread to service a level sensitive interrupt
in user-space via a device driver.  The driver is a Linux char device,
the guts being:

    static irqreturn_t fpgairq_handler(int irq, void *p)
    {
        /* Disable this interrupt */
        disable_irq_nosync(irq);

        /* Enable any blocked task to run */
        fpgairq_waitcond = 1;
        wake_up_interruptible(&fpgairq_waitq);

        return IRQ_HANDLED;
    }

    static ssize_t fpgairq_read(struct file *filp, char *buf, size_t
size, loff_t *offp)
    {
        const int irq = gpio_to_irq(PIN_IRQ);

        ....

        /* Clear the wait condition */
        fpgairq_waitcond = 0;

        /* Enable the interrupt source */
        enable_irq(irq);

        /* Now wait for an interrupt */
        wait_event_interruptible(fpgairq_waitq, fpgairq_waitcond != 0);

        /* The interrupt handler will have disabled the interrupt again
*/

        return size;
    }
    
The user-space code sets it's pthread_setschedparam() to SCHED_FIFO with
a high priority.  It then loops, reading from the device to wait for the
next interrupt and then writes via a different device to acknowledge the
external interrupt source.  Measuring the duration for which the line is
asserted gives the time between the interrupt being asserted and it
being acknowledged.

I made measurements over ~2000 sweeps on a scope with the unpatched and
patched kernel, both when otherwise idle and then when under load.  To
create load I execute the following line on a telnet session:

  $ while true; do ls -l /mnt; done
  
The results:

  Configuration                    Duration (uS)
                           Min       Mean      Max
  Vanilla                  9.9       16.8     30.8
  Xenomai Kernel          19.0       27.5     50.4
  Vanilla + load          10.9    11017.4  19619.7
  Xenomai Kernel + load   59.4      215.5   1849.8

Clearly Xenomai appears to add some overhead to the interrupt handling
and acknowledgement.  That's to be expected as the apps are executing in
the secondary and the drivers are not RTDM, and the overhead is very
small in anycase.

But I don't understand why under load the scheduling of the non-Xenomai
task is so much improved.  The worst case is more than x10 better, the
average case x50.  It's incredible!

Is this to be expected?

Regards,

Mike

Re: [Xenomai-help] Massive improvement when Xenomai patched

[Philippe Gerum]

Mike McTernan wrote:
> Hi,
> 
> I've patched Blackfin uClinux 2008R1.5 with Adeos and Xenomai 2.4.6.1
> and started benchmarking our system.  At this point the apps are
> unmodified and I only change between a Xenomai enabled kernel and an
> unmodified kernel.  This is verified by checking the boot messages and
> the process list under /proc/xenomai/stats.
>

<snip>

> I made measurements over ~2000 sweeps on a scope with the unpatched and
> patched kernel, both when otherwise idle and then when under load.  To
> create load I execute the following line on a telnet session:
> 
>   $ while true; do ls -l /mnt; done
>   
> The results:
> 
>   Configuration                    Duration (uS)
>                            Min       Mean      Max
>   Vanilla                  9.9       16.8     30.8
>   Xenomai Kernel          19.0       27.5     50.4
>   Vanilla + load          10.9    11017.4  19619.7
>   Xenomai Kernel + load   59.4      215.5   1849.8
> 
> Clearly Xenomai appears to add some overhead to the interrupt handling
> and acknowledgement.  That's to be expected as the apps are executing in
> the secondary and the drivers are not RTDM, and the overhead is very
> small in anycase.
> 

As a matter of fact, on the Blackfin with I-pipe series <= 1.8, the IRQs are
threaded, which causes such overhead. This is no more the case since 1.9-00.

> But I don't understand why under load the scheduling of the non-Xenomai
> task is so much improved.  The worst case is more than x10 better, the
> average case x50.  It's incredible!
>

Beyond that, it's a clear violation of Murphy's law.

> Is this to be expected?
> 

I suspect the IRQ latency figures look better because the I-pipe lowers the
current interrupt priority much earlier than the original code does. Moving the
interrupt handlers over threads reflecting the interrupt priorities used to be
the way the I-pipe did this on Blackfin at the expense of more overhead.
Additionally, the I-pipe for 2.6.22/blackfin makes the kernel tick on TMR0
instead of the core timer (the latter is then dedicated to Xenomai), which may
have some impact depending on the priority value set for CONFIG_IRQ_TMR0.

Once your driver is RTDM-based and the IRQ hooked via rtdm_irq_request(), you
should see the worst-case latency drop below 60 us on a bf537, likely less than
45 us on a bf561. The extra bonus will then likely be due to the changes in
arch/blackfin/lib/ins.S. These PIO helpers are responsible for huge latency
spots in the vanilla implementation, particularly under significant network
load, and the I-pipe makes them preemptible by real-time IRQs, even if Linux
IRQs are still masked.

-- 
Philippe.