Re: [PATCH] irq-gic: select all CPU's selected in interrupt affinity settings

[Russell King - ARM Linux admin <linux@armlinux.org.uk>]

On Wed, Nov 20, 2019 at 05:37:38PM +0000, Leonid Movshovich wrote:
> On Wed, 20 Nov 2019 at 17:18, Russell King - ARM Linux admin
> <linux@armlinux.org.uk> wrote:
> >
> > On Wed, Nov 20, 2019 at 04:45:59PM +0000, Leonid Movshovich wrote:
> > > On Wed, 20 Nov 2019 at 15:39, Marc Zyngier <maz@kernel.org> wrote:
> > > >
> > > > On 2019-11-20 15:28, Leonid Movshovich wrote:
> > > > > On Wed, 20 Nov 2019 at 15:04, Marc Zyngier <maz@kernel.org> wrote:
> > > > >>
> > > > >> On 2019-11-20 01:15, Robin Murphy wrote:
> > > > >> > On 2019-11-20 12:24 am, Leonid Movshovich wrote:
> > > > >> >> On Tue, 19 Nov 2019 at 23:36, Russell King - ARM Linux admin
> > > > >> >> <linux@armlinux.org.uk> wrote:
> > > > >> >>>
> > > > >> >>> On Tue, Nov 19, 2019 at 11:12:26PM +0000, event wrote:
> > > > >> >>>> So far only a CPU selected with top affinity bit was selected.
> > > > >> >>>> This
> > > > >> >>>> resulted in all interrupts
> > > > >> >>>> being processed by CPU0 by default despite "FF" default
> > > > >> affinity
> > > > >> >>>> setting for all interrupts
> > > > >> >>>
> > > > >> >>> Have you checked whether this causes _ALL_ CPUs in the mask to
> > > > >> be
> > > > >> >>> delivered a single interrupt, thereby causing _ALL_ CPUs to be
> > > > >> >>> slowed down and hit the same locks at the same time.
> > > > >> >>>
> > > > >> >> Yes, I've checked this. No, interrupt is delivered to only one
> > > > >> CPU.
> > > > >> >> Also ARM GIC architecture specification specifically states in
> > > > >> >> chapter
> > > > >> >> 3.1.1 that hardware interrupts are delivered to a single CPU in
> > > > >> >> multiprocessor system ("1-N model").
> > > > >> >
> > > > >> > But see also section 3.2.3 - just because only one CPU actually
> > > > >> runs
> > > > >> > the given ISR doesn't necessarily guarantee that the others
> > > > >> *weren't*
> > > > >> > interrupted. I'd also hesitate to make any assumptions that all
> > > > >> GIC
> > > > >> > implementations behave exactly the same way.
> > > > >>
> > > > >> What happens is that *all* CPUs are being sent the interrupt, and
> > > > >> there
> > > > >> is some logic in the GIC that ensures that only one sees it (the
> > > > >> first
> > > > >> one to read the IAR register). All the other see a spurious (1023)
> > > > >> interrupt, and have wasted some precious cycles in doing so.
> > > > >
> > > > > Cycles are only precious when system is under high load. Under high
> > > > > load, to achieve fair spread of interrupts between CPUs one would
> > > > > need
> > > > > a userspace app (irqbalance) to sit there and constantly rebalance
> > > > > smp_affinity based on /proc/interrupts. Hard to believe such an
> > > > > approach wastes less cycles.
> > > >
> > > > You'd be surprised. As always when looking at these things, do come up
> > > > with actual figures with a wide range of workloads that show benefits
> > > > for the approach you're suggesting.
> > > >
> > > > Also, if your system isn't under high load, why would you even care
> > > > about this kind of distribution?
> > >
> > > Coming back to my network example, under moderate load, without
> > > distribution, you'd get one CPU struggling to process all the traffic,
> > > while others sitting idle.
> >
> > And you think that receiving TCP packet 1 on CPU0, TCP packet 2 on
> > CPU1, TCP packet 2 on CPU2 etc will help?
> >
> > I guess you're not aware of network features such as GRO which
> > combine consecutive packets.  Forcing each packet onto a different
> > CPU will bounce the cache lines associated with managing the state
> > between different CPUs => negative performance impact.
> 
> I guess, you're not aware that TCP is not the only protocol in the
> internet. And that GRO is not a "network feature" but rather a NIC
> feature. And, that not all NICs support it.

You have an answer to everything.  Pointless continuing this, sorry.

-- 
RMK's Patch system: https://www.armlinux.org.uk/developer/patches/
FTTC broadband for 0.8mile line in suburbia: sync at 12.1Mbps down 622kbps up
According to speedtest.net: 11.9Mbps down 500kbps up

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel