[patch for 2.6.30 2/2] arch/x86/kernel/cpu/cpufreq/acpi-cpufreq.c: avoid cross-CPU interrupts

[patch for 2.6.30 2/2] arch/x86/kernel/cpu/cpufreq/acpi-cpufreq.c: avoid cross-CPU interrupts

[akpm]

From: Andrew Morton <akpm@linux-foundation.org>

In drv_read(), check to see whether we can run the rdmsr() on the current
CPU.  If so, do that.  So smp_call_function_single() can avoid the IPI.

Arguably, cpumask_any() should do this.

Cc: Rusty Russell <rusty@rustcorp.com.au>
Cc: Ingo Molnar <mingo@elte.hu>
Cc: Venkatesh Pallipadi <venkatesh.pallipadi@intel.com>
Cc: Len Brown <len.brown@intel.com>
Cc: Zhao Yakui <yakui.zhao@intel.com>
Cc: Dave Jones <davej@redhat.com>
Cc: Thomas Gleixner <tglx@linutronix.de>
Tested-by: Mike Galbraith <efault@gmx.de>
Cc: "Zhang, Yanmin" <yanmin_zhang@linux.intel.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
---

 arch/x86/kernel/cpu/cpufreq/acpi-cpufreq.c |   17 +++++++++++++++--
 1 file changed, 15 insertions(+), 2 deletions(-)

diff -puN arch/x86/kernel/cpu/cpufreq/acpi-cpufreq.c~arch-x86-kernel-cpu-cpufreq-acpi-cpufreqc-avoid-cross-cpu-interrupts arch/x86/kernel/cpu/cpufreq/acpi-cpufreq.c
--- a/arch/x86/kernel/cpu/cpufreq/acpi-cpufreq.c~arch-x86-kernel-cpu-cpufreq-acpi-cpufreqc-avoid-cross-cpu-interrupts
+++ a/arch/x86/kernel/cpu/cpufreq/acpi-cpufreq.c
@@ -197,9 +197,22 @@ static void do_drv_write(void *_cmd)
 
 static void drv_read(struct drv_cmd *cmd)
 {
-	cmd->val = 0;
+	int target_cpu;		/* The CPU on which to perform thr rdmsr() */
+	int this_cpu;
+
+	/*
+	 * If the current CPU is in cmd->mask then run the rdmsr() on this
+	 * CPU to avoid the cross-cpu interrupt.
+	 */
+	this_cpu = get_cpu();
+	if (cpu_isset(this_cpu, *(cmd->mask)))
+		target_cpu = this_cpu;
+	else
+		target_cpu = cpumask_any(cmd->mask);
 
-	smp_call_function_single(cpumask_any(cmd->mask), do_drv_read, cmd, 1);
+	cmd->val = 0;
+	smp_call_function_single(target_cpu, do_drv_read, cmd, 1);
+	put_cpu();
 }
 
 static void drv_write(struct drv_cmd *cmd)
_

Re: [patch for 2.6.30 2/2] arch/x86/kernel/cpu/cpufreq/acpi-cpufreq.c: avoid cross-CPU interrupts

[Jaswinder Singh Rajput]

On Fri, 2009-04-10 at 23:17 -0700, akpm@linux-foundation.org wrote:
> From: Andrew Morton <akpm@linux-foundation.org>
> 
> In drv_read(), check to see whether we can run the rdmsr() on the current
> CPU.  If so, do that.  So smp_call_function_single() can avoid the IPI.
> 
> Arguably, cpumask_any() should do this.
> 
> Cc: Rusty Russell <rusty@rustcorp.com.au>
> Cc: Ingo Molnar <mingo@elte.hu>
> Cc: Venkatesh Pallipadi <venkatesh.pallipadi@intel.com>
> Cc: Len Brown <len.brown@intel.com>
> Cc: Zhao Yakui <yakui.zhao@intel.com>
> Cc: Dave Jones <davej@redhat.com>
> Cc: Thomas Gleixner <tglx@linutronix.de>
> Tested-by: Mike Galbraith <efault@gmx.de>
> Cc: "Zhang, Yanmin" <yanmin_zhang@linux.intel.com>
> Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
> ---
> 
>  arch/x86/kernel/cpu/cpufreq/acpi-cpufreq.c |   17 +++++++++++++++--
>  1 file changed, 15 insertions(+), 2 deletions(-)
> 
> diff -puN arch/x86/kernel/cpu/cpufreq/acpi-cpufreq.c~arch-x86-kernel-cpu-cpufreq-acpi-cpufreqc-avoid-cross-cpu-interrupts arch/x86/kernel/cpu/cpufreq/acpi-cpufreq.c
> --- a/arch/x86/kernel/cpu/cpufreq/acpi-cpufreq.c~arch-x86-kernel-cpu-cpufreq-acpi-cpufreqc-avoid-cross-cpu-interrupts
> +++ a/arch/x86/kernel/cpu/cpufreq/acpi-cpufreq.c
> @@ -197,9 +197,22 @@ static void do_drv_write(void *_cmd)
>  
>  static void drv_read(struct drv_cmd *cmd)
>  {
> -	cmd->val = 0;
> +	int target_cpu;		/* The CPU on which to perform thr rdmsr() */

+ 	int target_cpu; 	/* The CPU on which to perform the rdmsr() */

--
JSR

Re: [patch for 2.6.30 2/2] arch/x86/kernel/cpu/cpufreq/acpi-cpufreq.c: avoid cross-CPU interrupts

[Dave Jones]

On Fri, Apr 10, 2009 at 11:17:18PM -0700, Andrew Morton wrote:
 > From: Andrew Morton <akpm@linux-foundation.org>
 > 
 > In drv_read(), check to see whether we can run the rdmsr() on the current
 > CPU.  If so, do that.  So smp_call_function_single() can avoid the IPI.

Wouldn't it be a better to make smp_call_function_single do this check
itself, so all callers benefit from this optimisation?

*looks*

Wait, won't this already be caught by this code in smp_call_function_single() ?

286         this_cpu = get_cpu();
...
291         if (cpu == this_cpu) {
292                 local_irq_save(flags);
293                 func(info);
294                 local_irq_restore(flags);
295         } else {



	Dave 

Re: [patch for 2.6.30 2/2] arch/x86/kernel/cpu/cpufreq/acpi-cpufreq.c: avoid cross-CPU interrupts

[Andrew Morton]

On Sat, 11 Apr 2009 20:06:05 -0400 Dave Jones <davej@redhat.com> wrote:

> On Fri, Apr 10, 2009 at 11:17:18PM -0700, Andrew Morton wrote:
>  > From: Andrew Morton <akpm@linux-foundation.org>
>  > 
>  > In drv_read(), check to see whether we can run the rdmsr() on the current
>  > CPU.  If so, do that.  So smp_call_function_single() can avoid the IPI.
> 
> Wouldn't it be a better to make smp_call_function_single do this check
> itself, so all callers benefit from this optimisation?
> 
> *looks*
> 
> Wait, won't this already be caught by this code in smp_call_function_single() ?
> 
> 286         this_cpu = get_cpu();
> ...
> 291         if (cpu == this_cpu) {
> 292                 local_irq_save(flags);
> 293                 func(info);
> 294                 local_irq_restore(flags);
> 295         } else {
> 
> 
> 

The problem is that the caller (acpi-cpufreq) is doing

	cpu = cpumask_any(mask);
	smp_call_function_single(cpu);

and cpumask_any(mask) does cpumask_first(mask).  Which might be a
different CPU, even though this thread of control is running on a CPU
which is present in `mask'.

- We could fix this by making cpumask_any(mask) return this-cpu if
  this-cpu is present `mask'.

- We could fix this by changing smp_call_function_single() to take a
  mask, rather than a particular CPU.  Then of course it preferentially
  chooses this-cpu if possible.

  Or write a new smp_call_function_any(mask, ...);

I suspect that changing cpumask_any() to preferentially return this-cpu
will always give us the behaviour that we prefer, but I haven't looked
into it.

Re: [patch for 2.6.30 2/2] arch/x86/kernel/cpu/cpufreq/acpi-cpufreq.c: avoid cross-CPU interrupts

[Rusty Russell]

On Sun, 12 Apr 2009 10:16:44 am Andrew Morton wrote:
> I suspect that changing cpumask_any() to preferentially return this-cpu
> will always give us the behaviour that we prefer, but I haven't looked
> into it.

How's this?

Subject: cpumask: cpumask_closest()

Impact: new function

Andrew points out that acpi-cpufreq uses cpumask_any, when it really
would prefer to use the same CPU if possible (to avoid an IPI).  In 
general, this seems a good idea to offer.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
CC: Andrew Morton <akpm@linux-foundation.org>

diff --git a/include/linux/cpumask.h b/include/linux/cpumask.h
--- a/include/linux/cpumask.h
+++ b/include/linux/cpumask.h
@@ -931,6 +931,8 @@ static inline void cpumask_copy(struct c
  */
 #define cpumask_of(cpu) (get_cpu_mask(cpu))
 
+unsigned int cpumask_closest(const struct cpumask *mask);
+
 /**
  * cpumask_scnprintf - print a cpumask into a string as comma-separated hex
  * @buf: the buffer to sprintf into
diff --git a/lib/cpumask.c b/lib/cpumask.c
--- a/lib/cpumask.c
+++ b/lib/cpumask.c
@@ -170,3 +170,26 @@ void __init free_bootmem_cpumask_var(cpu
 	free_bootmem((unsigned long)mask, cpumask_size());
 }
 #endif
+
+/**
+ * cpumask_closest - return the closest cpu in mask.
+ * @mask: the cpus to choose from.
+ *
+ * Returns >= nr_cpu_ids if no bits are set in @mask.
+ */
+unsigned int cpumask_closest(const struct cpumask *mask)
+{
+	unsigned int cpu = raw_smp_processor_id();
+
+	/* Try for same CPU. */
+	if (cpumask_test_cpu(cpu, mask))
+		return cpu;
+
+	/* Try for same node. */
+	cpu = cpumask_any_and(cpumask_of_node(cpu), mask);
+	if (cpu <= nr_cpu_ids)
+		return cpu;
+
+	/* Anything will do. */
+	return cpumask_any(mask);
+}

Re: [patch for 2.6.30 2/2] arch/x86/kernel/cpu/cpufreq/acpi-cpufreq.c: avoid cross-CPU interrupts

[Andrew Morton]

On Tue, 14 Apr 2009 18:21:36 +0930 Rusty Russell <rusty@rustcorp.com.au> wrote:

> On Sun, 12 Apr 2009 10:16:44 am Andrew Morton wrote:
> > I suspect that changing cpumask_any() to preferentially return this-cpu
> > will always give us the behaviour that we prefer, but I haven't looked
> > into it.
> 
> How's this?
> 
> Subject: cpumask: cpumask_closest()
> 
> Impact: new function
> 
> Andrew points out that acpi-cpufreq uses cpumask_any, when it really
> would prefer to use the same CPU if possible (to avoid an IPI).  In 
> general, this seems a good idea to offer.
> 
> Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
> CC: Andrew Morton <akpm@linux-foundation.org>
> 
> diff --git a/include/linux/cpumask.h b/include/linux/cpumask.h
> --- a/include/linux/cpumask.h
> +++ b/include/linux/cpumask.h
> @@ -931,6 +931,8 @@ static inline void cpumask_copy(struct c
>   */
>  #define cpumask_of(cpu) (get_cpu_mask(cpu))
>  
> +unsigned int cpumask_closest(const struct cpumask *mask);
> +
>  /**
>   * cpumask_scnprintf - print a cpumask into a string as comma-separated hex
>   * @buf: the buffer to sprintf into
> diff --git a/lib/cpumask.c b/lib/cpumask.c
> --- a/lib/cpumask.c
> +++ b/lib/cpumask.c
> @@ -170,3 +170,26 @@ void __init free_bootmem_cpumask_var(cpu
>  	free_bootmem((unsigned long)mask, cpumask_size());
>  }
>  #endif
> +
> +/**
> + * cpumask_closest - return the closest cpu in mask.
> + * @mask: the cpus to choose from.
> + *
> + * Returns >= nr_cpu_ids if no bits are set in @mask.
> + */
> +unsigned int cpumask_closest(const struct cpumask *mask)
> +{
> +	unsigned int cpu = raw_smp_processor_id();
> +
> +	/* Try for same CPU. */
> +	if (cpumask_test_cpu(cpu, mask))
> +		return cpu;
> +
> +	/* Try for same node. */
> +	cpu = cpumask_any_and(cpumask_of_node(cpu), mask);
> +	if (cpu <= nr_cpu_ids)
> +		return cpu;
> +
> +	/* Anything will do. */
> +	return cpumask_any(mask);
> +}

Should it be exported?

It looks all racy against hotplug.  What are the caller's
responsibilities here?

<greps a bit>

any_online_cpu() could use cpumask_closest(), against (*mask & cpu_online_map).

I think all cpumask_any() call sites can be migrated to
cpumask_closest() with, at worst, no benefit.

Re: [patch for 2.6.30 2/2] arch/x86/kernel/cpu/cpufreq/acpi-cpufreq.c: avoid cross-CPU interrupts

[Rusty Russell]

On Wed, 15 Apr 2009 02:48:17 am Andrew Morton wrote:
> On Tue, 14 Apr 2009 18:21:36 +0930 Rusty Russell <rusty@rustcorp.com.au> wrote:
> > Subject: cpumask: cpumask_closest()
..
> Should it be exported?

Ah yes.

> It looks all racy against hotplug.  What are the caller's
> responsibilities here?

Kind of independent.  There's no implied internal reference to online_mask.

> any_online_cpu() could use cpumask_closest(), against (*mask & cpu_online_map).

Note that I've been killing any_online_cpu().  It passes a cpumask on stack, 
and cpumask_any(cpu_online_mask) / cpumask_any_and(mask, cpu_online_mask) work
just as well.

> I think all cpumask_any() call sites can be migrated to
> cpumask_closest() with, at worst, no benefit.

OK, here's the updated patch.

Rusty.

Subject: cpumask: cpumask_closest() and cpumask_closest_and()

Impact: new functions

Andrew points out that acpi-cpufreq uses cpumask_any, when it really
would prefer to use the same CPU if possible (to avoid an IPI).  In
general, this seems a good idea to offer.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
CC: Andrew Morton <akpm@linux-foundation.org>
---
 include/linux/cpumask.h |    4 +++
 lib/cpumask.c           |   54 ++++++++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 58 insertions(+)

diff --git a/include/linux/cpumask.h b/include/linux/cpumask.h
--- a/include/linux/cpumask.h
+++ b/include/linux/cpumask.h
@@ -931,6 +931,10 @@ static inline void cpumask_copy(struct c
  */
 #define cpumask_of(cpu) (get_cpu_mask(cpu))
 
+unsigned int cpumask_closest(const struct cpumask *mask);
+unsigned int cpumask_closest_and(const struct cpumask *mask1,
+				 const struct cpumask *mask2);
+
 /**
  * cpumask_scnprintf - print a cpumask into a string as comma-separated hex
  * @buf: the buffer to sprintf into
diff --git a/lib/cpumask.c b/lib/cpumask.c
--- a/lib/cpumask.c
+++ b/lib/cpumask.c
@@ -170,3 +170,57 @@ void __init free_bootmem_cpumask_var(cpu
 	free_bootmem((unsigned long)mask, cpumask_size());
 }
 #endif
+
+/**
+ * cpumask_closest - return the closest cpu in mask.
+ * @mask: the cpus to choose from.
+ *
+ * Returns >= nr_cpu_ids if no bits are set in @mask.
+ */
+unsigned int cpumask_closest(const struct cpumask *mask)
+{
+	unsigned int cpu = raw_smp_processor_id();
+
+	/* Try for same CPU. */
+	if (cpumask_test_cpu(cpu, mask))
+		return cpu;
+
+	/* Try for same node. */
+	cpu = cpumask_any_and(cpumask_of_node(cpu), mask);
+	if (cpu <= nr_cpu_ids)
+		return cpu;
+
+	/* Anything will do. */
+	return cpumask_any(mask);
+}
+EXPORT_SYMBOL(cpumask_closest);
+
+/**
+ * cpumask_closest_and - return the closest cpu in both masks.
+ * @mask1: one set of cpus to choose from.
+ * @mask2: the other set of cpus to choose from.
+ *
+ * The same as cpumask_closest(@mask1 & @mask2).
+ * Returns >= nr_cpu_ids if no bits are set in both..
+ */
+unsigned int cpumask_closest_and(const struct cpumask *mask1,
+				 const struct cpumask *mask2)
+{
+	unsigned int cpu = raw_smp_processor_id();
+	const struct cpumask *nodemask;
+
+	/* Try for same CPU. */
+	if (cpumask_test_cpu(cpu, mask1) && cpumask_test_cpu(cpu, mask2))
+		return cpu;
+
+	/* Try for same node. */
+	nodemask = cpumask_of_node(cpu);
+	for_each_cpu_and(cpu, nodemask, mask1) {
+		if (cpumask_test_cpu(cpu, mask2))
+			return cpu;
+	}
+
+	/* Anything will do. */
+	return cpumask_any_and(mask1, mask2);
+}
+EXPORT_SYMBOL(cpumask_closest_and);

Re: [patch for 2.6.30 2/2] arch/x86/kernel/cpu/cpufreq/acpi-cpufreq.c: avoid cross-CPU interrupts

[Len Brown]

> From: Andrew Morton <akpm@linux-foundation.org>
> 
> In drv_read(), check to see whether we can run the rdmsr() on the current
> CPU.  If so, do that.  So smp_call_function_single() can avoid the IPI.
> 
> Arguably, cpumask_any() should do this.
> 
> Cc: Rusty Russell <rusty@rustcorp.com.au>
> Cc: Ingo Molnar <mingo@elte.hu>
> Cc: Venkatesh Pallipadi <venkatesh.pallipadi@intel.com>
> Cc: Len Brown <len.brown@intel.com>
> Cc: Zhao Yakui <yakui.zhao@intel.com>
> Cc: Dave Jones <davej@redhat.com>
> Cc: Thomas Gleixner <tglx@linutronix.de>
> Tested-by: Mike Galbraith <efault@gmx.de>
> Cc: "Zhang, Yanmin" <yanmin_zhang@linux.intel.com>
> Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
> 
> ---
> 
>  arch/x86/kernel/cpu/cpufreq/acpi-cpufreq.c |   17 +++++++++++++++--
>  1 file changed, 15 insertions(+), 2 deletions(-)
> 
> diff -puN arch/x86/kernel/cpu/cpufreq/acpi-cpufreq.c~arch-x86-kernel-cpu-cpufreq-acpi-cpufreqc-avoid-cross-cpu-interrupts arch/x86/kernel/cpu/cpufreq/acpi-cpufreq.c
> --- a/arch/x86/kernel/cpu/cpufreq/acpi-cpufreq.c~arch-x86-kernel-cpu-cpufreq-acpi-cpufreqc-avoid-cross-cpu-interrupts
> +++ a/arch/x86/kernel/cpu/cpufreq/acpi-cpufreq.c
> @@ -197,9 +197,22 @@ static void do_drv_write(void *_cmd)
>  
>  static void drv_read(struct drv_cmd *cmd)
>  {
> -	cmd->val = 0;
> +	int target_cpu;		/* The CPU on which to perform thr rdmsr() */
> +	int this_cpu;
> +
> +	/*
> +	 * If the current CPU is in cmd->mask then run the rdmsr() on this
> +	 * CPU to avoid the cross-cpu interrupt.
> +	 */
> +	this_cpu = get_cpu();
> +	if (cpu_isset(this_cpu, *(cmd->mask)))
> +		target_cpu = this_cpu;
> +	else
> +		target_cpu = cpumask_any(cmd->mask);
>  
> -	smp_call_function_single(cpumask_any(cmd->mask), do_drv_read, cmd, 1);
> +	cmd->val = 0;
> +	smp_call_function_single(target_cpu, do_drv_read, cmd, 1);
> +	put_cpu();
>  }
>  
>  static void drv_write(struct drv_cmd *cmd)
> _

Rather than this patch I would expect we would want to either:

A. as we went to the trouble to detect the local case
   in drv_read, why call smp_call_function at all for that case?

or

B. optimize smp_call_function_single to beneift all users
   instead of just this customer.

thanks,
-Len Brown, Intel Open Source Technology Center

Re: [patch for 2.6.30 2/2] arch/x86/kernel/cpu/cpufreq/acpi-cpufreq.c: avoid cross-CPU interrupts

[Andrew Morton]

On Sun, 19 Apr 2009 22:57:24 -0400 (EDT) Len Brown <lenb@kernel.org> wrote:

> 
> 
> > +++ a/arch/x86/kernel/cpu/cpufreq/acpi-cpufreq.c
> > @@ -197,9 +197,22 @@ static void do_drv_write(void *_cmd)
> >  
> >  static void drv_read(struct drv_cmd *cmd)
> >  {
> > -	cmd->val = 0;
> > +	int target_cpu;		/* The CPU on which to perform thr rdmsr() */
> > +	int this_cpu;
> > +
> > +	/*
> > +	 * If the current CPU is in cmd->mask then run the rdmsr() on this
> > +	 * CPU to avoid the cross-cpu interrupt.
> > +	 */
> > +	this_cpu = get_cpu();
> > +	if (cpu_isset(this_cpu, *(cmd->mask)))
> > +		target_cpu = this_cpu;
> > +	else
> > +		target_cpu = cpumask_any(cmd->mask);
> >  
> > -	smp_call_function_single(cpumask_any(cmd->mask), do_drv_read, cmd, 1);
> > +	cmd->val = 0;
> > +	smp_call_function_single(target_cpu, do_drv_read, cmd, 1);
> > +	put_cpu();
> >  }
> >  
> >  static void drv_write(struct drv_cmd *cmd)
> > _
> 
> Rather than this patch I would expect we would want to either:
> 
> A. as we went to the trouble to detect the local case
>    in drv_read, why call smp_call_function at all for that case?

Sure, that would work.

I felt it was a little cleaner to always delegate the call to
smp_call_function() rather than open-coding smp_call_function()'s
internal implementation details at this site.  We'd need to do:

	local_irq_disable();		/* Because this is what smp_call_function_single() does */
	do_drv_read(...);
	local_irq_enable();

> or
> 
> B. optimize smp_call_function_single to beneift all users
>    instead of just this customer.

Yep.  That would be a new smp_call_function_any() which takes a cpumask
rather than a single CPU number.  I think Rusty was cooking something
up..

Re: [patch for 2.6.30 2/2] arch/x86/kernel/cpu/cpufreq/acpi-cpufreq.c: avoid cross-CPU interrupts

[Len Brown]

> Andrew points out that acpi-cpufreq uses cpumask_any, when it really
> would prefer to use the same CPU if possible (to avoid an IPI).  In
> general, this seems a good idea to offer.

I like this idea.

I'll drop andrew's patch "acpi-cpufreq.c: avoid cross-CPU interrupts"
b/c that is not an emergency and this is more clever.

thanks,
-Len Brown, Intel Open Source Technology Center.