Using patch-2.6.33.7.2-rt30 increases latency and CPU usage?

Using patch-2.6.33.7.2-rt30 increases latency and CPU usage?

[Grant Edwards]

I've been doing some interrupt latency testing using a board running a
400MHz ARM9 (an Atmel at19sam9g20).  I'm measuring interrupt latency
of an ISR attached to external interrupt pin IRQ0.

I built three kernels from scratch and ran the same test on all three:

 2.6.30          
 2.6.33.7        
 2.6.33.7-rt30  

All kernels are built using the standard AT91 patches from
http://maxim.org.za/at91_26.html.  The 2.6.30 kernel also has a set of
patches distributed by Atmel.

The 2.6.33.7-rt30 kernel is built from the same sources as the 2.6.33.7
kernel with the addition of the 2.6.33.6-rt30 patch.

I don't have enough data to say much about worst-case latency, but
after looking a few thousand samples I can say that...

 * 2.6.30 and 2.6.33.7 are pretty much the same: there is as much
   variation between test runs as there is between kernels.

 * Typical latency with RT patch is 3X the latency without it.

 * CPU idle time during test is 10% with RT patch and 30% without.

IOW: the real-time patch for 2.6.33.7 makes both the typical interrupt
latency and the CPU usage significantly worse.

Typical latency without the RT patch is 5-15us.

Typical latency with is 15-50us (I've never seen latency below 15us
with the RT patch).

Is this the expected behavior?

I don't have enough data to state conclusively what the worst-case
latencies are, but it looks like in all cases they're 200-250us.

In all cases the kernel was configured to be the most preemptible it
could be.  In the 2.6.33.7-rt30 kernel I don't have any of the tracers
enabled.

The kernel configurations are as identical as I could make them.  In
the 2.6.33.7-rt30 case, I started with the .config file used to build
the other 2.6.33.7 kernel and ran old-config making sure to select the
"fully preemptible" option when it asked.

Why would a "vanilla" 2.6.33.7 AT91 kernel perform so much better than
one with the RT patch?

-- 
Grant Edwards               grant.b.edwards        Yow! RHAPSODY in Glue!
                                  at
                               gmail.com

Re: Using patch-2.6.33.7.2-rt30 increases latency and CPU usage?

[Grant Edwards]

On 2012-05-09, Grant Edwards <grant.b.edwards@gmail.com> wrote:

> IOW: the real-time patch for 2.6.33.7 makes both the typical interrupt
> latency and the CPU usage significantly worse.
>
> Typical latency without the RT patch is 5-15us.
>
> Typical latency with is 15-50us (I've never seen latency below 15us
> with the RT patch).
>
> Is this the expected behavior?

I've been loaned a clue by somebody on the OSADL mailing list: the RT
patches are for improving _user_space_ reponse, and may do so at the
expense of both CPU usage and interrupt latency.

As a result, I'm better off without the RT patch if what I care about
is interrupt latency.

Am right I correct in the following conclusions?

  * In 2.6.33.7, ISRs are really ISRs

  * In 2.6.33.7-rt30, ISRs are kernel threads.

  
-- 
Grant Edwards               grant.b.edwards        Yow! Don't hit me!!  I'm in
                                  at               the Twilight Zone!!!
                              gmail.com            

Re: Using patch-2.6.33.7.2-rt30 increases latency and CPU usage?

[Joachim Achtzehnter]

Grant Edwards wrote:

> I've been loaned a clue by somebody on the OSADL mailing list: the RT
> patches are for improving _user_space_ reponse, and may do so at the
> expense of both CPU usage and interrupt latency.

The RT patches improve *worst case* latency. They are primarily intended 
for applications that must *always* meet their deadlines, not merely 
most of the time. In return you tend to get increased average latency as 
well as reduced throughput.

> As a result, I'm better off without the RT patch if what I care
> about is interrupt latency.

Yes, if you only care about *typical* interrupt latency but don't mind 
the occasional long delay.

Joachim

-- 
joachima@netacquire.com http://www.netacquire.com

Re: Using patch-2.6.33.7.2-rt30 increases latency and CPU usage?

[Grant Edwards]

On 2012-05-09, Joachim Achtzehnter <joachima@netacquire.com> wrote:
> Grant Edwards wrote:
>
>> I've been loaned a clue by somebody on the OSADL mailing list: the RT
>> patches are for improving _user_space_ reponse, and may do so at the
>> expense of both CPU usage and interrupt latency.
>
> The RT patches improve *worst case* latency. They are primarily
> intended for applications that must *always* meet their deadlines,
> not merely most of the time. In return you tend to get increased
> average latency as well as reduced throughput.

Good point.

I eventually figured out that my increased interrupt latency is due to
my ISR being run in a kernel thread instead of as a real ISR.  Adding
the IRQF_NODELAY flag gets me the same average latency I had without
the RT patch.  [It looks like that flag has a different name in 2.6.39
and later?]

>> As a result, I'm better off without the RT patch if what I care
>> about is interrupt latency.
>
> Yes, if you only care about *typical* interrupt latency but don't
> mind the occasional long delay.

Unfortunately, the requirements are a bit fuzzy -- I've got an ISR
deadline of about 20us that I'm trying to meet [I wouldn't mind a
little chat with the person who designed _that_ requirement into the
hardware].  What I don't know is how hard that deadline is.  With the
RT patch (and without IRQF_NODELAY), I miss the deadline most of the
time (I'd guess about 80% of the time).

Without RT or with RT and IRQF_NODELAY, it looks like I meet the
deadline maybe 98% of the time (under test conditions).  What I don't
know is if once the deadline is missed it matters weather it's missed
by 50us or by 250us [or if 98% is going to be anywhere close to
acceptible, for that matter].

-- 
Grant

Re: Using patch-2.6.33.7.2-rt30 increases latency and CPU usage?

[Remy Bohmer]

Hi,

2012/5/10 Grant Edwards <grant.b.edwards@gmail.com>:
> On 2012-05-09, Joachim Achtzehnter <joachima@netacquire.com> wrote:
>> Grant Edwards wrote:
> Unfortunately, the requirements are a bit fuzzy -- I've got an ISR
> deadline of about 20us that I'm trying to meet [I wouldn't mind a
> little chat with the person who designed _that_ requirement into the
> hardware].  What I don't know is how hard that deadline is.  With the
> RT patch (and without IRQF_NODELAY), I miss the deadline most of the
> time (I'd guess about 80% of the time).
>
> Without RT or with RT and IRQF_NODELAY, it looks like I meet the
> deadline maybe 98% of the time (under test conditions).  What I don't
> know is if once the deadline is missed it matters weather it's missed
> by 50us or by 250us [or if 98% is going to be anywhere close to
> acceptible, for that matter].

Given my experience with these cores and the RT patch: This 20usec is
too close. There are some places in the code that have interrupt
disable section of about 30usec (also process ctx-switch cache flush
have high impact)
So, I expect you will not get this very robust.
If the 20usec requirement is hard, you can also look at using an FIQ.

Kind regards,

Remy
--
To unsubscribe from this list: send the line "unsubscribe linux-rt-users" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Using patch-2.6.33.7.2-rt30 increases latency and CPU usage?

[Grant Edwards]

On Thu, May 10, 2012 at 11:46:26AM +0200, Remy Bohmer wrote:
> 2012/5/10 Grant Edwards <grant.b.edwards@gmail.com>:
>
>> Unfortunately, the requirements are a bit fuzzy -- I've got an ISR
>> deadline of about 20us that I'm trying to meet [I wouldn't mind a
>> little chat with the person who designed _that_ requirement into the
>> hardware].  What I don't know is how hard that deadline is.  With the
>> RT patch (and without IRQF_NODELAY), I miss the deadline most of the
>> time (I'd guess about 80% of the time).
> 
> Given my experience with these cores and the RT patch: This 20usec is
> too close.

That's pretty much what I've decided.

> There are some places in the code that have interrupt disable section
> of about 30usec (also process ctx-switch cache flush have high
> impact)
>
> So, I expect you will not get this very robust.
>
> If the 20usec requirement is hard, you can also look at using an FIQ.

Does the normal AT91 kernel not use FIQs at all?  If so, I might be
able to dedicate the FIQ to this function and have it happen without
the Linux kernel knowing about it (or affecting it).  Any
communication between it and a normal user-task would have to be
handled carefully...

-- 
Grant
--
To unsubscribe from this list: send the line "unsubscribe linux-rt-users" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Using patch-2.6.33.7.2-rt30 increases latency and CPU usage?

[Remy Bohmer]

Hi,

2012/5/10 Grant Edwards <grant.b.edwards@gmail.com>:
>> If the 20usec requirement is hard, you can also look at using an FIQ.
>
> Does the normal AT91 kernel not use FIQs at all?  If so, I might be
> able to dedicate the FIQ to this function and have it happen without
> the Linux kernel knowing about it (or affecting it).  Any
> communication between it and a normal user-task would have to be
> handled carefully...

No, the kernel does not use the FIQ at all.
It is free for you to use as you want.
Communication with the FIQ handler can be done using a queue based on
atomic instructions.

Kind regards,

Remy
--
To unsubscribe from this list: send the line "unsubscribe linux-rt-users" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Using patch-2.6.33.7.2-rt30 increases latency and CPU usage?

[Grant Edwards]

On 2012-05-11, Remy Bohmer <linux@bohmer.net> wrote:
> Hi,
>
> 2012/5/10 Grant Edwards <grant.b.edwards@gmail.com>:
>>> If the 20usec requirement is hard, you can also look at using an FIQ.
>>
>> Does the normal AT91 kernel not use FIQs at all? ?If so, I might be
>> able to dedicate the FIQ to this function and have it happen without
>> the Linux kernel knowing about it (or affecting it). ?Any
>> communication between it and a normal user-task would have to be
>> handled carefully...
>
> No, the kernel does not use the FIQ at all. It is free for you to use
> as you want. Communication with the FIQ handler can be done using a
> queue based on atomic instructions.

Thanks!  I was about to start looking into that...

-- 
Grant Edwards               grant.b.edwards        Yow! I just forgot my whole
                                  at               philosophy of life!!!
                              gmail.com            

Re: Using patch-2.6.33.7.2-rt30 increases latency and CPU usage?

[Tim Sander]

Hi
> > No, the kernel does not use the FIQ at all. It is free for you to use
> > as you want. Communication with the FIQ handler can be done using a
> > queue based on atomic instructions.
Well i think there is some FIQ infrastructure in the kernel and even an FIQ 
switch. But the only thing it does is reserving the infrastructure. Not more 
not less. I have been toying around with the FIQ interrupt. On our HW (i.mx35) 
we still have about 30µs interrupt jitter i would like to reduce. But the 
Memory for the FIQ was to small and i didn't manage to jump to some c handler 
code within the driver. Does anybody know has some examples how to do that.

Further since the FIQ is an own mode i thought the only thing one could use is 
memory accesses? But when working within the TCM or FIQ memory the jitter was 
very good (it was about 5ns if i can trust my memory).

Best regards
Tim
--
To unsubscribe from this list: send the line "unsubscribe linux-rt-users" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Using patch-2.6.33.7.2-rt30 increases latency and CPU usage?

[Steven Rostedt]

On Wed, 2012-05-09 at 23:18 +0000, Grant Edwards wrote:

> I eventually figured out that my increased interrupt latency is due to
> my ISR being run in a kernel thread instead of as a real ISR.  Adding
> the IRQF_NODELAY flag gets me the same average latency I had without
> the RT patch.  [It looks like that flag has a different name in 2.6.39
> and later?]

Note, adding a IRQF_NODELAY is very dangerous. To keep the kernel fully
preemptible (well mostly), spin-locks have been converted into mutexes.
If the irq handler has one of these spinlocks that have been converted,
it may schedule in the interrupt handler and crash the kernel. You can
not schedule in interrupt context, which is why the RT kernel makes the
interrupt handlers into threads.

Also note, you may get better response if you up the interrupt thread's
priority. Up it to 98, and you should see much better responses.

Now I don't think you will make the 20us under load, but you won't do
that in the non-RT patch either.

-- Steve

Re: Using patch-2.6.33.7.2-rt30 increases latency and CPU usage?

[Grant Edwards]

On Tue, May 15, 2012 at 01:33:12PM -0400, Steven Rostedt wrote:
> On Wed, 2012-05-09 at 23:18 +0000, Grant Edwards wrote:
> 
> > I eventually figured out that my increased interrupt latency is due to
> > my ISR being run in a kernel thread instead of as a real ISR.  Adding
> > the IRQF_NODELAY flag gets me the same average latency I had without
> > the RT patch.  [It looks like that flag has a different name in 2.6.39
> > and later?]
> 
> Note, adding a IRQF_NODELAY is very dangerous. To keep the kernel
> fully preemptible (well mostly), spin-locks have been converted into
> mutexes. If the irq handler has one of these spinlocks that have been
> converted, it may schedule in the interrupt handler and crash the
> kernel. You can not schedule in interrupt context, which is why the
> RT kernel makes the interrupt handlers into threads.

Noted.  Since the IRQF_NODELAY approach still misses the deadline 2-3%
of the time, that approach is probably off the list anyway.  Right now
we're looking at using either FIQ interrupt or Xenomai.

> Also note, you may get better response if you up the interrupt
> thread's priority. Up it to 98, and you should see much better
> responses.

Thanks for the tip.  I'll keep that in mind for some other projects
that would need latency times in the 100-200us range, which should be
doable.

> Now I don't think you will make the 20us under load, but you won't do
> that in the non-RT patch either.

Right.  With the non-RT patch, the average response is about the same,
but the worst-case numbers are 2X-3X.

-- 
Grant

Re: Using patch-2.6.33.7.2-rt30 increases latency and CPU usage?

[Steven Rostedt]

On Tue, 2012-05-15 at 17:58 -0500, Grant Edwards wrote:

> Thanks for the tip.  I'll keep that in mind for some other projects
> that would need latency times in the 100-200us range, which should be
> doable.

If it isn't doable, definitely report it. For modern intel hardware, we
are expecting all response times under 100us. Otherwise we consider it a
bug. Except for the cases where the hardware itself is the cause (SMIs
and other nasties).

-- Steve