Re: [RFC patch] Use IPI_shortcut for lapic timer broadcast

["Pallipadi, Venkatesh" <venkatesh.pallipadi@intel.com>]

On Sun, 2009-06-28 at 23:47 -0700, Luming Yu wrote:
> Hello,
> 
> We need to use IPI shortcut to send lapic timer broadcast
> to avoid the latency of sending IPI one bye one on systems with many
> logical processors when NO_HZ is disabled.
> Without this patch,I have seen upstream kernel with RHEL 5 kernel
> config boot hang .
> 
> The patch also changes physflat_send_IPI_all to IPI shortcut mode.
> 
> Please review, and apply.
> 
> **The patch is enclosed in text attachment*
> **Using web client to send the patch* *
> **below is for review, please apply attached  patch*/
> 
> Thanks,
> Luming
> 
> 
> Signed-off-by: Yu Luming <luming.yu@intel.com>
> 
>  apic.c         |    4 +++-
>  apic_flat_64.c |    7 ++++++-
>  2 files changed, 9 insertions(+), 2 deletions(-)
> 
> --- linux-2.6.30-rc6/arch/x86/kernel/apic/apic.c.0	2009-06-28
> 20:22:55.000000000 -0600
> +++ linux-2.6.30-rc6/arch/x86/kernel/apic/apic.c	2009-06-29
> 00:21:44.000000000 -0600
> @@ -419,7 +419,9 @@
>  static void lapic_timer_broadcast(const struct cpumask *mask)
>  {
>  #ifdef CONFIG_SMP
> -	apic->send_IPI_mask(mask, LOCAL_TIMER_VECTOR);
> +	if (cpus_empty(*mask))
> +		return;
> +	apic->send_IPI_all(LOCAL_TIMER_VECTOR);
>  #endif
>  }
> 

Hmm.. this change looks wrong. This will cause unnecessary timer
broadcasts when nohz is actually enabled. I mean, when nohz is enabled,
we want to send this interrupt broadcast to one or some subset of CPUs
and this change sends the interrupt to all CPUs.

On a 16 logical CPUs with nohz enabled, powertop without this patch I
see
Wakeups-from-idle per second : 11.6     interval: 15.0s
and with this patch I see
Wakeups-from-idle per second : 24.7     interval: 15.0s

Also, cat /proc/interrupts | grep LOC shows this problem where all CPUs
seem to get roughly same number of local APIC interrupts over a period
of time with this patch.

We should only do the allbutself when nohz is disabled with a big
unlikely() around that code.

Thanks,
Venki

> --- linux-2.6.30-rc6/arch/x86/kernel/apic/apic_flat_64.c.0	2009-06-29
> 00:13:26.000000000 -0600
> +++ linux-2.6.30-rc6/arch/x86/kernel/apic/apic_flat_64.c	2009-06-29
> 00:11:23.000000000 -0600
> @@ -274,7 +274,12 @@
> 
>  static void physflat_send_IPI_all(int vector)
>  {
> -	physflat_send_IPI_mask(cpu_online_mask, vector);
> +	if (vector == NMI_VECTOR) {
> +		physflat_send_IPI_mask(cpu_online_mask, vector);
> +	} else {
> +		__default_send_IPI_shortcut(APIC_DEST_ALLINC,
> +					    vector, apic->dest_logical);
> +	}
>  }
> 
>  static unsigned int physflat_cpu_mask_to_apicid(const struct cpumask *cpumask)