[RFC] ARM: Let GIC to route IRQs to any allowed CPUs

[RFC] ARM: Let GIC to route IRQs to any allowed CPUs

[TAO HU]

[RFC] ARM: Let GIC to route IRQs to any allowed CPUs

[Russell King - ARM Linux]

On Wed, Jan 12, 2011 at 04:18:51PM +0800, TAO HU wrote:
> The current logic will always route IRQs to the first allowed CPU
> Following two commands have the same result (all IRQs to CPU0)
> cmd1: $ echo 3 > proc/irq/xxx/smp_affinity
> cmd2: $ echo 1 > proc/irq/xxx/smp_affinity
> While below command will route all IRQs to CPU1
> cmd3: $ echo 2 > proc/irq/xxx/smp_affinity
> 
> With this patch, cmd1 will route IRQs to both CPU0 and CPU1
> Meanwhile, cmd2 and cmd3 will work as before

There's a reason we don't do this...

We don't want to deliver one IRQ to both CPU0 and CPU1 at the same time,
which is what will happen with this patch.  The GIC hardware doesn't
do IRQ balancing between CPUs.

The result will be CPUs fighting over servicing such interrupts.

[RFC] ARM: Let GIC to route IRQs to any allowed CPUs

[TAO HU]

Hi, Russell King

Thanks.

For "The result will be CPUs fighting over servicing such interrupts."
What's the consequence? The performance downgrade?

Does it imply a design flaw in GIC or in ARM SMP?

On Wed, Jan 12, 2011 at 4:51 PM, Russell King - ARM Linux
<linux@arm.linux.org.uk> wrote:
> On Wed, Jan 12, 2011 at 04:18:51PM +0800, TAO HU wrote:
>> The current logic will always route IRQs to the first allowed CPU
>> Following two commands have the same result (all IRQs to CPU0)
>> cmd1: $ echo 3 > proc/irq/xxx/smp_affinity
>> cmd2: $ echo 1 > proc/irq/xxx/smp_affinity
>> While below command will route all IRQs to CPU1
>> cmd3: $ echo 2 > proc/irq/xxx/smp_affinity
>>
>> With this patch, cmd1 will route IRQs to both CPU0 and CPU1
>> Meanwhile, cmd2 and cmd3 will work as before
>
> There's a reason we don't do this...
>
> We don't want to deliver one IRQ to both CPU0 and CPU1 at the same time,
> which is what will happen with this patch. ?The GIC hardware doesn't
> do IRQ balancing between CPUs.
>
> The result will be CPUs fighting over servicing such interrupts.
>



-- 
Best Regards
Hu Tao

[RFC] ARM: Let GIC to route IRQs to any allowed CPUs

[Santosh Shilimkar]

Hu Tao,
(Please avoid top posting)
> -----Original Message-----
> From: linux-arm-kernel-bounces at lists.infradead.org [mailto:linux-
> arm-kernel-bounces at lists.infradead.org] On Behalf Of TAO HU
> Sent: Wednesday, January 12, 2011 2:35 PM
> To: Russell King - ARM Linux
> Cc: TAO HU; linux-arm-kernel at lists.infradead.org
> Subject: Re: [RFC] ARM: Let GIC to route IRQs to any allowed CPUs
>
> Hi, Russell King
>
> Thanks.
>
> For "The result will be CPUs fighting over servicing such
> interrupts."
> What's the consequence? The performance downgrade?
>
Few problems I can see here.
1. Spurious interrupts will be seen by CPU who looses the IRQ race
2. When interrupts are targeted on two CPU's, and if one of the CPU is
in low power mode, the ON CPU will not see the interrupt till the     OFF
CPU becomes ON again.
3. Unnecessary wakeups on both CPUs which can screw up power savings.
4. prone to aggravate race conditions in the existing drivers.

Regards,
Santosh

[RFC] ARM: Let GIC to route IRQs to any allowed CPUs

[Russell King - ARM Linux]

On Wed, Jan 12, 2011 at 05:04:43PM +0800, TAO HU wrote:
> Hi, Russell King
> 
> Thanks.
> 
> For "The result will be CPUs fighting over servicing such interrupts."
> What's the consequence? The performance downgrade?

It means you'll have the same interrupt trying to be serviced on two
CPUs at the same time.  One will succeed and service it.  The other
will waste time just finding out that there's nothing for it to do,
competing with resources and wasting time that it could otherwise have
spent executing userspace or whatever.

> Does it imply a design flaw in GIC or in ARM SMP?

No.  The hardware is doing what you're asking it to.  Some OSes may want
this behaviour so it would be wrong for the hardware to deny it.

There's quite a bit of history behind interrupt balancing across CPU
cores, and it's not a simple issue to get to grips with.

I believe the x86 kernel uses software to balance interrupts across the
cores using an algorithm in the kernel.  This tries to distribute the
interrupts between the cores.  When this was tried on ARM, although it
moved interrupts around the cores, it was no better than having all
interrupts routed to one core.

On x86 they now do away with their kernel algorithm, and instead run a
boot-time utility (irqbalance) which does a one-time distribution of
interrupts across the cores.  The idea is that it is more important to
keep an interrupt handler running on the same core than it is to
constantly switch it between the cores.

If a handler keeps switching between the cores, you have to migrate cache
lines between the cores, which adds to the cache coherency traffic, and
on x86 results in a reduction in performance.

So, with all that in mind, when I was sorting out the initial SMP merging,
I tried out various algorithms for automatic interrupt distribution, and
never got any of them to work satisfactorily.  In light of discussions with
x86 folk, particularly Arjan van de Ven, I decided not to merge any of them
and leave it as a matter for userspace policy to control how interrupts are
distributed to the cores - just like the majority of x86 platforms now do.

AFAIK, there's nothing stopping anyone running 'irqbalance' (the x86
utility) on ARM - it should just be accessing procfs files.  See
http://irqbalance.org/ for more information on the program, and on the
issues surrounding IRQ distribution in SMP systems.