Re: [PATCH] x86/x2apic: introduce a mixed physical/cluster mode

["Roger Pau Monné" <roger.pau@citrix.com>]

On Mon, Oct 30, 2023 at 05:34:48PM +0100, Jan Beulich wrote:
> On 30.10.2023 17:10, Roger Pau Monné wrote:
> > On Mon, Oct 30, 2023 at 03:32:56PM +0100, Jan Beulich wrote:
> >> On 24.10.2023 15:51, Roger Pau Monne wrote:
> >>> The current implementation of x2APIC requires to either use Cluster Logical or
> >>> Physical mode for all interrupts.  However the selection of Physical vs Logical
> >>> is not done at APIC setup, an APIC can be addressed both in Physical or Logical
> >>> destination modes concurrently.
> >>>
> >>> Introduce a new x2APIC mode called mixed, which uses Logical Cluster mode for
> >>> IPIs, and Physical mode for external interrupts, thus attempting to use the
> >>> best method for each interrupt type.
> >>>
> >>> Using Physical mode for external interrupts allows more vectors to be used, and
> >>> interrupt balancing to be more accurate.
> >>>
> >>> Using Logical Cluster mode for IPIs allows less accesses to the ICR register
> >>> when sending those, as multiple CPUs can be targeted with a single ICR register
> >>> write.
> >>>
> >>> A simple test calling flush_tlb_all() 10000 times in a tight loop on a 96 CPU
> >>> box gives the following average figures:
> >>>
> >>> Physical mode: 26617931ns
> >>> Mixed mode:    23865337ns
> >>>
> >>> So ~10% improvement versus plain Physical mode.
> >>
> >> Nice.
> >>
> >>>  Note that Xen uses Cluster
> >>> mode by default, and hence is already using the fastest way for IPI delivery at
> >>> the cost of reducing the amount of vectors available system-wide.
> >>>
> >>> Make the newly introduced mode the default one.
> >>>
> >>> Suggested-by: Andrew Cooper <andrew.cooper3@citrix.com>
> >>> Signed-off-by: Roger Pau Monné <roger.pau@citrix.com>
> >>> ---
> >>> Do we want to keep a full Logical Cluster mode available?  I don't see a reason
> >>> to target external interrupts in Logical Cluster mode, but maybe there's
> >>> something I'm missing.
> >>>
> >>> It's not clear to me whether the ACPI FADT flags are meant to apply only to
> >>> external interrupt delivery mode, or also to IPI delivery.  If
> >>> ACPI_FADT_APIC_PHYSICAL is only meant to apply to external interrupts and not
> >>> IPIs then we could possibly get rid of physical mode IPI delivery.
> >>>
> >>> Still need to put this under XenServer extensive testing, but wanted to get
> >>> some feedback before in case approach greatly changes.
> >>
> >> Looks quite okay, just a couple of minor remarks:
> > 
> > Thanks.
> > 
> > Do we still want to keep the pure cluster mode?
> 
> I think we should keep it for a full release cycle or two.

OK, will make the requested changes and send v2 then.

Thanks, Roger.