Re: [PATCH] irq-gic: select all CPU's selected in interrupt affinity settings

[Russell King - ARM Linux admin <linux@armlinux.org.uk>]

On Wed, Nov 20, 2019 at 04:45:59PM +0000, Leonid Movshovich wrote:
> On Wed, 20 Nov 2019 at 15:39, Marc Zyngier <maz@kernel.org> wrote:
> >
> > On 2019-11-20 15:28, Leonid Movshovich wrote:
> > > On Wed, 20 Nov 2019 at 15:04, Marc Zyngier <maz@kernel.org> wrote:
> > >>
> > >> On 2019-11-20 01:15, Robin Murphy wrote:
> > >> > On 2019-11-20 12:24 am, Leonid Movshovich wrote:
> > >> >> On Tue, 19 Nov 2019 at 23:36, Russell King - ARM Linux admin
> > >> >> <linux@armlinux.org.uk> wrote:
> > >> >>>
> > >> >>> On Tue, Nov 19, 2019 at 11:12:26PM +0000, event wrote:
> > >> >>>> So far only a CPU selected with top affinity bit was selected.
> > >> >>>> This
> > >> >>>> resulted in all interrupts
> > >> >>>> being processed by CPU0 by default despite "FF" default
> > >> affinity
> > >> >>>> setting for all interrupts
> > >> >>>
> > >> >>> Have you checked whether this causes _ALL_ CPUs in the mask to
> > >> be
> > >> >>> delivered a single interrupt, thereby causing _ALL_ CPUs to be
> > >> >>> slowed down and hit the same locks at the same time.
> > >> >>>
> > >> >> Yes, I've checked this. No, interrupt is delivered to only one
> > >> CPU.
> > >> >> Also ARM GIC architecture specification specifically states in
> > >> >> chapter
> > >> >> 3.1.1 that hardware interrupts are delivered to a single CPU in
> > >> >> multiprocessor system ("1-N model").
> > >> >
> > >> > But see also section 3.2.3 - just because only one CPU actually
> > >> runs
> > >> > the given ISR doesn't necessarily guarantee that the others
> > >> *weren't*
> > >> > interrupted. I'd also hesitate to make any assumptions that all
> > >> GIC
> > >> > implementations behave exactly the same way.
> > >>
> > >> What happens is that *all* CPUs are being sent the interrupt, and
> > >> there
> > >> is some logic in the GIC that ensures that only one sees it (the
> > >> first
> > >> one to read the IAR register). All the other see a spurious (1023)
> > >> interrupt, and have wasted some precious cycles in doing so.
> > >
> > > Cycles are only precious when system is under high load. Under high
> > > load, to achieve fair spread of interrupts between CPUs one would
> > > need
> > > a userspace app (irqbalance) to sit there and constantly rebalance
> > > smp_affinity based on /proc/interrupts. Hard to believe such an
> > > approach wastes less cycles.
> >
> > You'd be surprised. As always when looking at these things, do come up
> > with actual figures with a wide range of workloads that show benefits
> > for the approach you're suggesting.
> >
> > Also, if your system isn't under high load, why would you even care
> > about this kind of distribution?
> 
> Coming back to my network example, under moderate load, without
> distribution, you'd get one CPU struggling to process all the traffic,
> while others sitting idle.

And you think that receiving TCP packet 1 on CPU0, TCP packet 2 on
CPU1, TCP packet 2 on CPU2 etc will help?

I guess you're not aware of network features such as GRO which
combine consecutive packets.  Forcing each packet onto a different
CPU will bounce the cache lines associated with managing the state
between different CPUs => negative performance impact.

Userspace doesn't see individual packets.

-- 
RMK's Patch system: https://www.armlinux.org.uk/developer/patches/
FTTC broadband for 0.8mile line in suburbia: sync at 12.1Mbps down 622kbps up
According to speedtest.net: 11.9Mbps down 500kbps up

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel