Re: [PATCH UPDATED 04/16] x86: Use local variable to cache smp_processor_id() in setup_local_APIC()

Re: [PATCH UPDATED 04/16] x86: Use local variable to cache smp_processor_id() in setup_local_APIC()

[Cyrill Gorcunov]

On Tue, Nov 30, 2010 at 05:17:19PM +0100, Tejun Heo wrote:
> Subject: x86: Use local variable to cache smp_processor_id() in setup_local_APIC()
> 
> This is a trivial clean up.
> 
> * Move initialization of @cpu inside preemption disabled region as
>   suggested by Cyrill Gorcunov.
> 
> Signed-off-by: Tejun Heo <tj@kernel.org>
> Cc: Yinghai Lu <yinghai@kernel.org>
> Cc: Cyrill Gorcunov <gorcunov@gmail.com>
> ---
> Updated as suggested.  Git tree updated too.
> 
> Thanks.
> 

Thanks Tejun, but please don't modify the snippet below
(ie leave it with smp_processor_id). It happens when preempt
gets enabled again but cpu cached with preempt disabled.
So we should not mess this I believe. ok?

Other than that
Acked-by: Cyrill Gorcunov <gorcunov@gmail.com>

Thanks!

> @@ -1368,7 +1367,7 @@ void __cpuinit setup_local_APIC(void)
> 
>  #ifdef CONFIG_X86_MCE_INTEL
>  	/* Recheck CMCI information after local APIC is up on CPU #0 */
> -	if (smp_processor_id() == 0)
> +	if (cpu == 0)
>  		cmci_recheck();
>  #endif
>  }
> 
  Cyrill

[PATCHSET] x86: unify x86_32 and 64 NUMA init paths, take#2

[Tejun Heo]

Hello,

This is the second take of unify-x86_32-and-64-NUMA-init-paths
patchset.  Changes from the last take[L] are,

* As suggested by Ingo, 0004 from the previous series is further split
  into 0004-0011.

* Comment noting the ugliness of x86_32 logical apicid mapping
  difference added.

This patchset started as an attempt to fix percpu setup problem on
x86_32 NUMA configurations reported by Eric Dumazet.  On x86_32, NUMA
configuration is initialized during SMP bringup which happens way
later than percpu setup.  As percpu setup code doesn't have access to
NUMA configuration, it gets set up as if the system is UMA.

The NUMA init paths differ subtly yet significantly between x86_32 and
64 making it quite difficult to follow.  Custom apic configurations on
x86_32 worsens the problem.

There is no fundamental reason why x86_32 can't be initialized in the
same steps as x86_64.  They just were written differently and just
weren't unified during x86_32/64 code merge.  This patchset unifies
them and in the process fixes percpu setup problem on 32bit NUMA.

This patchset contains the following sixteen patches.

 0001-x86-Kill-unused-static-boot_cpu_logical_apicid-in-sm.patch
 0002-x86-Rename-x86_32-MAX_APICID-to-MAX_LOCAL_APIC.patch
 0003-x86-Make-default_send_IPI_mask_sequence-allbutself_l.patch
 0004-x86-Use-local-variable-to-cache-smp_processor_id-in-.patch
 0005-x86-Replace-cpu_2_logical_apicid-with-early-percpu-v.patch
 0006-x86-Always-use-x86_cpu_to_logical_apicid-for-cpu-log.patch
 0007-x86-Remove-custom-apic-cpu_to_logical_apicid-impleme.patch
 0008-x86-Use-apic-cpu_to_logical_apicid-to-find-out-logic.patch
 0009-x86-Implement-cpu_to_logical_apicid-for-bigsmp_32.patch
 0010-x86-Implement-cpu_to_logical_apicid-for-summit_32.patch
 0011-x86-Implement-cpu_to_logical_apicid-for-numaq_32.patch
 0012-x86-Replace-apic-apicid_to_node-with-numa_cpu_node.patch
 0013-x86-Unify-cpu-apicid-NUMA-node-mapping-between-32-an.patch
 0014-x86-Unify-CPU-NUMA-node-mapping-between-32-and-64bit.patch
 0015-x86-Unify-node_to_cpumask_map-handling-between-32-an.patch
 0016-x86-Unify-NUMA-initialization-between-32-and-64bit.patch

0001-0004 are small preparation patches.

0005-0011 change the role of apic->cpu_to_logical_apicid().  It's now
called once for each cpu early during boot and the output is recorded.
apic works fine but I couldn't test other apic configurations.

I couldn't figure out how to determine logical_apicid early for numaq
and it still has the stub cpu_to_logical_apicid() implementation which
would result in suboptimal percpu configuration.  NUMA configuration
is still updated during SMP bringup, so the situation is the same as
before the series, but if someone knows how to convert this one,
please go ahead.

0005 replaces apic->apicid_to_node() with ->numa_cpu_node() as some
need physical apicid while others logical and both can be easily found
from cpu.

0006-0009 unify different parts of NUMA initialization/handling step
by step.  Afterwards, x86_32 and 64 behave mostly the same regarding
NUMA initialization making the whole thing much easier to understand
and percpu setup works correctly on x86_32 too.

This patchset is on top of the current mainline
(19650e8580987c0ffabc2fe2cbc16b944789df8b) and also avaialbe in the
following git branch,

 git://git.kernel.org/pub/scm/linux/kernel/git/tj/misc.git x86_32-numa

and contains the following changes.

 arch/x86/Kconfig                      |    2 
 arch/x86/include/asm/apic.h           |   22 ++-
 arch/x86/include/asm/apicdef.h        |    1 
 arch/x86/include/asm/ipi.h            |    8 -
 arch/x86/include/asm/mpspec.h         |    3 
 arch/x86/include/asm/numa.h           |   61 ++++++++++
 arch/x86/include/asm/numa_32.h        |    7 -
 arch/x86/include/asm/numa_64.h        |   16 --
 arch/x86/include/asm/smp.h            |    3 
 arch/x86/include/asm/topology.h       |   17 --
 arch/x86/kernel/acpi/boot.c           |    8 -
 arch/x86/kernel/apic/apic.c           |   52 ++++++--
 arch/x86/kernel/apic/apic_flat_64.c   |    4 
 arch/x86/kernel/apic/apic_noop.c      |    4 
 arch/x86/kernel/apic/bigsmp_32.c      |   33 ++---
 arch/x86/kernel/apic/es7000_32.c      |   33 ++---
 arch/x86/kernel/apic/ipi.c            |   12 -
 arch/x86/kernel/apic/numaq_32.c       |   29 +++-
 arch/x86/kernel/apic/probe_32.c       |    2 
 arch/x86/kernel/apic/summit_32.c      |   46 ++-----
 arch/x86/kernel/apic/x2apic_cluster.c |    2 
 arch/x86/kernel/apic/x2apic_phys.c    |    2 
 arch/x86/kernel/apic/x2apic_uv_x.c    |    2 
 arch/x86/kernel/cpu/amd.c             |   18 +-
 arch/x86/kernel/cpu/common.c          |    2 
 arch/x86/kernel/cpu/intel.c           |    5 
 arch/x86/kernel/setup.c               |    2 
 arch/x86/kernel/setup_percpu.c        |   11 +
 arch/x86/kernel/smpboot.c             |   68 -----------
 arch/x86/mm/k8topology_64.c           |    2 
 arch/x86/mm/numa.c                    |  179 +++++++++++++++++++++++++++++
 arch/x86/mm/numa_32.c                 |    7 +
 arch/x86/mm/numa_64.c                 |  205 ----------------------------------
 arch/x86/mm/srat_32.c                 |    6 
 arch/x86/mm/srat_64.c                 |   10 -
 35 files changed, 433 insertions(+), 451 deletions(-)

Thanks.

--
tejun

[L] http://lkml.org/lkml/2010/11/11/96

[PATCH 04/16] x86: Use local variable to cache smp_processor_id() in setup_local_APIC()

[Tejun Heo]

This is a trivial clean up.

Signed-off-by: Tejun Heo <tj@kernel.org>
Cc: Yinghai Lu <yinghai@kernel.org>
---
 arch/x86/kernel/apic/apic.c |   13 ++++++-------
 1 files changed, 6 insertions(+), 7 deletions(-)

diff --git a/arch/x86/kernel/apic/apic.c b/arch/x86/kernel/apic/apic.c
index 3f838d5..2710f07 100644
--- a/arch/x86/kernel/apic/apic.c
+++ b/arch/x86/kernel/apic/apic.c
@@ -1201,6 +1201,7 @@ static void __cpuinit lapic_setup_esr(void)
  */
 void __cpuinit setup_local_APIC(void)
 {
+	int cpu = smp_processor_id();
 	unsigned int value, queued;
 	int i, j, acked = 0;
 	unsigned long long tsc = 0, ntsc;
@@ -1342,21 +1343,19 @@ void __cpuinit setup_local_APIC(void)
 	 * TODO: set up through-local-APIC from through-I/O-APIC? --macro
 	 */
 	value = apic_read(APIC_LVT0) & APIC_LVT_MASKED;
-	if (!smp_processor_id() && (pic_mode || !value)) {
+	if (!cpu && (pic_mode || !value)) {
 		value = APIC_DM_EXTINT;
-		apic_printk(APIC_VERBOSE, "enabled ExtINT on CPU#%d\n",
-				smp_processor_id());
+		apic_printk(APIC_VERBOSE, "enabled ExtINT on CPU#%d\n", cpu);
 	} else {
 		value = APIC_DM_EXTINT | APIC_LVT_MASKED;
-		apic_printk(APIC_VERBOSE, "masked ExtINT on CPU#%d\n",
-				smp_processor_id());
+		apic_printk(APIC_VERBOSE, "masked ExtINT on CPU#%d\n", cpu);
 	}
 	apic_write(APIC_LVT0, value);
 
 	/*
 	 * only the BP should see the LINT1 NMI signal, obviously.
 	 */
-	if (!smp_processor_id())
+	if (!cpu)
 		value = APIC_DM_NMI;
 	else
 		value = APIC_DM_NMI | APIC_LVT_MASKED;
@@ -1368,7 +1367,7 @@ void __cpuinit setup_local_APIC(void)
 
 #ifdef CONFIG_X86_MCE_INTEL
 	/* Recheck CMCI information after local APIC is up on CPU #0 */
-	if (smp_processor_id() == 0)
+	if (cpu == 0)
 		cmci_recheck();
 #endif
 }
-- 
1.7.1

[PATCH UPDATED 04/16] x86: Use local variable to cache smp_processor_id() in setup_local_APIC()

[Tejun Heo]

Subject: x86: Use local variable to cache smp_processor_id() in setup_local_APIC()

This is a trivial clean up.

* Move initialization of @cpu inside preemption disabled region as
  suggested by Cyrill Gorcunov.

Signed-off-by: Tejun Heo <tj@kernel.org>
Cc: Yinghai Lu <yinghai@kernel.org>
Cc: Cyrill Gorcunov <gorcunov@gmail.com>
---
Updated as suggested.  Git tree updated too.

Thanks.

 arch/x86/kernel/apic/apic.c |   15 +++++++--------
 1 file changed, 7 insertions(+), 8 deletions(-)

Index: work/arch/x86/kernel/apic/apic.c
===================================================================
--- work.orig/arch/x86/kernel/apic/apic.c
+++ work/arch/x86/kernel/apic/apic.c
@@ -1202,7 +1202,7 @@ static void __cpuinit lapic_setup_esr(vo
 void __cpuinit setup_local_APIC(void)
 {
 	unsigned int value, queued;
-	int i, j, acked = 0;
+	int cpu, i, j, acked = 0;
 	unsigned long long tsc = 0, ntsc;
 	long long max_loops = cpu_khz;

@@ -1226,6 +1226,7 @@ void __cpuinit setup_local_APIC(void)
 	perf_events_lapic_init();

 	preempt_disable();
+	cpu = smp_processor_id();

 	/*
 	 * Double-check whether this APIC is really registered.
@@ -1342,21 +1343,19 @@ void __cpuinit setup_local_APIC(void)
 	 * TODO: set up through-local-APIC from through-I/O-APIC? --macro
 	 */
 	value = apic_read(APIC_LVT0) & APIC_LVT_MASKED;
-	if (!smp_processor_id() && (pic_mode || !value)) {
+	if (!cpu && (pic_mode || !value)) {
 		value = APIC_DM_EXTINT;
-		apic_printk(APIC_VERBOSE, "enabled ExtINT on CPU#%d\n",
-				smp_processor_id());
+		apic_printk(APIC_VERBOSE, "enabled ExtINT on CPU#%d\n", cpu);
 	} else {
 		value = APIC_DM_EXTINT | APIC_LVT_MASKED;
-		apic_printk(APIC_VERBOSE, "masked ExtINT on CPU#%d\n",
-				smp_processor_id());
+		apic_printk(APIC_VERBOSE, "masked ExtINT on CPU#%d\n", cpu);
 	}
 	apic_write(APIC_LVT0, value);

 	/*
 	 * only the BP should see the LINT1 NMI signal, obviously.
 	 */
-	if (!smp_processor_id())
+	if (!cpu)
 		value = APIC_DM_NMI;
 	else
 		value = APIC_DM_NMI | APIC_LVT_MASKED;
@@ -1368,7 +1367,7 @@ void __cpuinit setup_local_APIC(void)

 #ifdef CONFIG_X86_MCE_INTEL
 	/* Recheck CMCI information after local APIC is up on CPU #0 */
-	if (smp_processor_id() == 0)
+	if (cpu == 0)
 		cmci_recheck();
 #endif
 }