Re: [PATCH] x86/apic: reduce cache line misses in __x2apic_send_IPI_mask()

[Peter Zijlstra <peterz@infradead.org>]

On Thu, Oct 07, 2021 at 03:13:46PM -0700, Eric Dumazet wrote:
> It seems wasteful to use tiny per-cpu variables and block hard irqs.
> 
> Quick and very dirty patch :
> 
> diff --git a/arch/x86/kernel/apic/x2apic_cluster.c
> b/arch/x86/kernel/apic/x2apic_cluster.c
> index e696e22d0531976f7cba72ed17443592eac72c13..c5076d40d4ea7bc9ffb06728531d91777a32cef4
> 100644
> --- a/arch/x86/kernel/apic/x2apic_cluster.c
> +++ b/arch/x86/kernel/apic/x2apic_cluster.c
> @@ -44,15 +44,18 @@ static void
>  __x2apic_send_IPI_mask(const struct cpumask *mask, int vector, int apic_dest)
>  {
>         unsigned int cpu, clustercpu;
> -       struct cpumask *tmpmsk;
> +       cpumask_var_t tmpmsk;
> +#ifdef CONFIG_CPUMASK_OFFSTACK
>         unsigned long flags;
> +#endif
>         u32 dest;
> 
>         /* x2apic MSRs are special and need a special fence: */
>         weak_wrmsr_fence();
> +#ifdef CONFIG_CPUMASK_OFFSTACK
>         local_irq_save(flags);
> -
>         tmpmsk = this_cpu_cpumask_var_ptr(ipi_mask);
> +#endif
>         cpumask_copy(tmpmsk, mask);
>         /* If IPI should not be sent to self, clear current CPU */
>         if (apic_dest != APIC_DEST_ALLINC)
> @@ -74,7 +77,9 @@ __x2apic_send_IPI_mask(const struct cpumask *mask,
> int vector, int apic_dest)
>                 cpumask_andnot(tmpmsk, tmpmsk, &cmsk->mask);
>         }
> 
> +#ifdef CONFIG_CPUMASK_OFFSTACK
>         local_irq_restore(flags);
> +#endif
>  }

I'm really conflicted about this. On the one hand, yes absolutely. On
the other hand, urgh, code ugly :-)