Re: [PATCH] irq-gic: select all CPU's selected in interrupt affinity settings - Russell King

From: Russell King - ARM Linux admin <linux@armlinux.org.uk>
To: Leonid Movshovich <event.riga@gmail.com>
Cc: Robin Murphy <robin.murphy@arm.com>,
	linux-arm-kernel@lists.infradead.org
Subject: Re: [PATCH] irq-gic: select all CPU's selected in interrupt affinity settings
Date: Wed, 20 Nov 2019 10:50:17 +0000	[thread overview]
Message-ID: <20191120105017.GN25745@shell.armlinux.org.uk> (raw)
In-Reply-To: <CAPaFbasJGxV=mEGHAOWdt31vapCfV_RyD1Ptz6aa_WTp6htPKg@mail.gmail.com>

On Wed, Nov 20, 2019 at 10:44:39AM +0000, Leonid Movshovich wrote:
> On Wed, 20 Nov 2019 at 01:15, Robin Murphy <robin.murphy@arm.com> wrote:
> >
> > On 2019-11-20 12:24 am, Leonid Movshovich wrote:
> > > On Tue, 19 Nov 2019 at 23:36, Russell King - ARM Linux admin
> > > <linux@armlinux.org.uk> wrote:
> > >>
> > >> On Tue, Nov 19, 2019 at 11:12:26PM +0000, event wrote:
> > >>> So far only a CPU selected with top affinity bit was selected. This
> > >>> resulted in all interrupts
> > >>> being processed by CPU0 by default despite "FF" default affinity
> > >>> setting for all interrupts
> > >>
> > >> Have you checked whether this causes _ALL_ CPUs in the mask to be
> > >> delivered a single interrupt, thereby causing _ALL_ CPUs to be
> > >> slowed down and hit the same locks at the same time.
> > >>
> > >
> > > Yes, I've checked this. No, interrupt is delivered to only one CPU.
> > > Also ARM GIC architecture specification specifically states in chapter
> > > 3.1.1 that hardware interrupts are delivered to a single CPU in
> > > multiprocessor system ("1-N model").
> >
> > But see also section 3.2.3 - just because only one CPU actually runs the
> > given ISR doesn't necessarily guarantee that the others *weren't*
> > interrupted. I'd also hesitate to make any assumptions that all GIC
> > implementations behave exactly the same way.
> >
> > Robin.
> 
> Yes, that's right, however:
> 1. They are only interrupted for a split-second, since interrupt is
> immediately ACKed in gic_handle_irq

Even that is detrimental - consider cpuidle where a CPU is placed in
a low power state waiting for an interrupt, and it keeps getting woken
for interrupts that it isn't able to handle.  The effect will be to
stop the CPU hitting the lower power states, which would be a regression
over how the kernel behaves today.

> 2. More important that smp_affinity in procfs is defined to allow user
> to configure multiple CPU's to handle interrupts (see
> Documentation/IRQ-affinity.txt) which is effectively prohibited in
> current implementation. I mean, when user sets it to FF, she expects
> all CPUs to process interrupts, not CPU0 only

The reason we've ended up with that on ARM is precisely because it
wasted CPU resources, and my attempts at writing code to distribute
the interrupt between CPU cores did not have a successful outcome.
So, the best thing that could be done was to route interrupts to the
first core, and run irqbalance to distribute the interrupts in a
sensible, cache friendly way between CPU cores.

And no, the current implementation is *NOT* prohibited.  You can't
prohibit something that hardware hasn't been able to provide.

-- 
RMK's Patch system: https://www.armlinux.org.uk/developer/patches/
FTTC broadband for 0.8mile line in suburbia: sync at 12.1Mbps down 622kbps up
According to speedtest.net: 11.9Mbps down 500kbps up

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel