[PATCH v3 0/4] SMP support for CNS3xxx + some SMP SCU cleanups

[PATCH v3 0/4] SMP support for CNS3xxx + some SMP SCU cleanups

[Anton Vorontsov]

Hello Russell,

These patches were posted before, i.e.

1/3 ARM: Factor out secondary CPU bootup code from VExpress platforms
 http://lists.infradead.org/pipermail/linux-arm-kernel/2010-June/018410.html
2/3 ARM: RealView: Switch to generic SCU routines
 http://lists.infradead.org/pipermail/linux-arm-kernel/2010-June/018411.html
3/3 ARM: cns3xxx: Add support for SMP
 http://lists.infradead.org/pipermail/linux-arm-kernel/2010-June/018412.html

In this updated series I split patch 1/3 into two, i.e. now it is:

1/4 ARM: SCU: Add generic routines for secondary CPU bootup
and
3/4 ARM: VExpress: Switch to generic SCU routines

This was done so that new feature addition doesn't depend on a
cleanup patch.

I tested 4/4 on a RealView PBX platform, so it should be pretty
much OK. 3/4 was only compile-tested, but it should be OK as
generic SCU code was mostly copied from VExpress platforms.

In the end, I consider patches 1/4 and 2/4 as 100% safe, so I'd
like them to go into 2.6.36 via your patch tracking system. What
do you think?

Thanks,

-- 
Anton Vorontsov
email: cbouatmailru at gmail.com
irc://irc.freenode.net/bd2

[PATCH 1/4] ARM: SCU: Add generic routines for secondary CPU bootup

[Anton Vorontsov]

For CNS3xxx we want to reuse the original ARM approach of booting
secondary CPUs. This patch turns VExpress' code into "SCU library",
so that now platform code can call the generic routines.

Note that this patch doesn't convert VExpress platform to the
generic routines. Plus, there are also mach-{omap2,ux500,realview}
platforms that might benefit from this change, but we'll convert
them via separate patches.

Signed-off-by: Anton Vorontsov <avorontsov@mvista.com>
---
 arch/arm/include/asm/smp_scu.h |    9 +++
 arch/arm/kernel/Makefile       |    2 +-
 arch/arm/kernel/smp_scu.c      |  136 ++++++++++++++++++++++++++++++++++++++++
 arch/arm/kernel/smp_scu_head.S |   37 +++++++++++
 4 files changed, 183 insertions(+), 1 deletions(-)
 create mode 100644 arch/arm/kernel/smp_scu_head.S

diff --git a/arch/arm/include/asm/smp_scu.h b/arch/arm/include/asm/smp_scu.h
index 2376835..271b252 100644
--- a/arch/arm/include/asm/smp_scu.h
+++ b/arch/arm/include/asm/smp_scu.h
@@ -1,7 +1,16 @@
 #ifndef __ASMARM_ARCH_SCU_H
 #define __ASMARM_ARCH_SCU_H
 
+#include <linux/init.h>
+
 unsigned int scu_get_core_count(void __iomem *);
 void scu_enable(void __iomem *);
 
+extern volatile int __cpuinitdata scu_pen_release;
+void scu_secondary_startup(void);
+void __cpuinit scu_secondary_init(unsigned int cpu);
+int __cpuinit scu_boot_secondary(unsigned int cpu, struct task_struct *idle);
+void __init scu_init_cpus(void __iomem *scu_base);
+void __init scu_prepare_cpus(void __iomem *scu_base, unsigned int max_cpus);
+
 #endif
diff --git a/arch/arm/kernel/Makefile b/arch/arm/kernel/Makefile
index 26d302c..2fc7cb1 100644
--- a/arch/arm/kernel/Makefile
+++ b/arch/arm/kernel/Makefile
@@ -28,7 +28,7 @@ obj-$(CONFIG_ARTHUR)		+= arthur.o
 obj-$(CONFIG_ISA_DMA)		+= dma-isa.o
 obj-$(CONFIG_PCI)		+= bios32.o isa.o
 obj-$(CONFIG_SMP)		+= smp.o
-obj-$(CONFIG_HAVE_ARM_SCU)	+= smp_scu.o
+obj-$(CONFIG_HAVE_ARM_SCU)	+= smp_scu.o smp_scu_head.o
 obj-$(CONFIG_HAVE_ARM_TWD)	+= smp_twd.o
 obj-$(CONFIG_DYNAMIC_FTRACE)	+= ftrace.o
 obj-$(CONFIG_KEXEC)		+= machine_kexec.o relocate_kernel.o
diff --git a/arch/arm/kernel/smp_scu.c b/arch/arm/kernel/smp_scu.c
index 9ab4149..b18414c 100644
--- a/arch/arm/kernel/smp_scu.c
+++ b/arch/arm/kernel/smp_scu.c
@@ -8,10 +8,15 @@
  * it under the terms of the GNU General Public License version 2 as
  * published by the Free Software Foundation.
  */
+#include <linux/compiler.h>
+#include <linux/kernel.h>
 #include <linux/init.h>
 #include <linux/io.h>
+#include <linux/jiffies.h>
+#include <linux/delay.h>
 
 #include <asm/smp_scu.h>
+#include <asm/localtimer.h>
 #include <asm/cacheflush.h>
 
 #define SCU_CTRL		0x00
@@ -50,3 +55,134 @@ void __init scu_enable(void __iomem *scu_base)
 	 */
 	flush_cache_all();
 }
+
+/*
+ * control for which core is the next to come out of the secondary
+ * boot "holding pen"
+ */
+volatile int __cpuinitdata scu_pen_release = -1;
+
+static DEFINE_SPINLOCK(boot_lock);
+
+void __cpuinit scu_secondary_init(unsigned int cpu)
+{
+	/*
+	 * let the primary processor know we're out of the
+	 * pen, then head off into the C entry point
+	 */
+	scu_pen_release = -1;
+	smp_wmb();
+
+	/*
+	 * Synchronise with the boot thread.
+	 */
+	spin_lock(&boot_lock);
+	spin_unlock(&boot_lock);
+}
+
+int __cpuinit scu_boot_secondary(unsigned int cpu, struct task_struct *idle)
+{
+	unsigned long timeout;
+
+	/*
+	 * Set synchronisation state between this boot processor
+	 * and the secondary one
+	 */
+	spin_lock(&boot_lock);
+
+	/*
+	 * This is really belt and braces; we hold unintended secondary
+	 * CPUs in the holding pen until we're ready for them.  However,
+	 * since we haven't sent them a soft interrupt, they shouldn't
+	 * be there.
+	 */
+	scu_pen_release = cpu;
+	__cpuc_flush_dcache_area((void *)&scu_pen_release,
+				 sizeof(scu_pen_release));
+	outer_clean_range(__pa(&scu_pen_release), __pa(&scu_pen_release + 1));
+
+	/*
+	 * Send the secondary CPU a soft interrupt, thereby causing
+	 * the boot monitor to read the system wide flags register,
+	 * and branch to the address found there.
+	 */
+	smp_cross_call(cpumask_of(cpu));
+
+	timeout = jiffies + (1 * HZ);
+	while (time_before(jiffies, timeout)) {
+		smp_rmb();
+		if (scu_pen_release == -1)
+			break;
+
+		udelay(10);
+	}
+
+	/*
+	 * now the secondary core is starting up let it run its
+	 * calibrations, then wait for it to finish
+	 */
+	spin_unlock(&boot_lock);
+
+	return scu_pen_release != -1 ? -ENOSYS : 0;
+}
+
+/*
+ * Initialise the CPU possible map early - this describes the CPUs
+ * which may be present or become present in the system.
+ */
+void __init scu_init_cpus(void __iomem *scu_base)
+{
+	unsigned int ncores = scu_base ? scu_get_core_count(scu_base) : 1;
+	unsigned int i;
+
+	/* sanity check */
+	if (ncores == 0) {
+		pr_err("%s: strange CM count of 0? Default to 1\n", __func__);
+		ncores = 1;
+	}
+
+	if (ncores > NR_CPUS) {
+		pr_warn("%s: no. of cores (%d) greater than configured "
+		       "maximum of %d - clipping\n", __func__, ncores, NR_CPUS);
+		ncores = NR_CPUS;
+	}
+
+	for (i = 0; i < ncores; i++)
+		set_cpu_possible(i, true);
+}
+
+void __init scu_prepare_cpus(void __iomem *scu_base, unsigned int max_cpus)
+{
+	unsigned int ncores = num_possible_cpus();
+	unsigned int cpu = smp_processor_id();
+	int i;
+
+	smp_store_cpu_info(cpu);
+
+	/*
+	 * are we trying to boot more cores than exist?
+	 */
+	if (max_cpus > ncores)
+		max_cpus = ncores;
+
+	/*
+	 * Initialise the present map, which describes the set of CPUs
+	 * actually populated at the present time.
+	 */
+	for (i = 0; i < max_cpus; i++)
+		set_cpu_present(i, true);
+
+	/*
+	 * Initialise the SCU if there are more than one CPU and let
+	 * them know where to start.
+	 */
+	if (max_cpus > 1) {
+		/*
+		 * Enable the local timer or broadcast device for the
+		 * boot CPU, but only if we have more than one CPU.
+		 */
+		percpu_timer_setup();
+
+		scu_enable(scu_base);
+	}
+}
diff --git a/arch/arm/kernel/smp_scu_head.S b/arch/arm/kernel/smp_scu_head.S
new file mode 100644
index 0000000..200010b
--- /dev/null
+++ b/arch/arm/kernel/smp_scu_head.S
@@ -0,0 +1,37 @@
+/*
+ *  Copyright (c) 2003 ARM Limited
+ *  All Rights Reserved
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 as
+ * published by the Free Software Foundation.
+ */
+#include <linux/linkage.h>
+#include <linux/init.h>
+
+	__INIT
+
+/*
+ * SCU specific entry point for secondary CPUs.  This provides
+ * a "holding pen" into which all secondary cores are held until we're
+ * ready for them to initialise.
+ */
+ENTRY(scu_secondary_startup)
+	mrc	p15, 0, r0, c0, c0, 5
+	and	r0, r0, #15
+	adr	r4, 1f
+	ldmia	r4, {r5, r6}
+	sub	r4, r4, r5
+	add	r6, r6, r4
+pen:	ldr	r7, [r6]
+	cmp	r7, r0
+	bne	pen
+
+	/*
+	 * we've been released from the holding pen: secondary_stack
+	 * should now contain the SVC stack for this core
+	 */
+	b	secondary_startup
+
+1:	.long	.
+	.long	scu_pen_release
-- 
1.7.0.5

[PATCH 2/4] ARM: cns3xxx: Add support for SMP

[Anton Vorontsov]

Thanks to the previous refactorings, nothing fancy needs to be
done, just use generic SCU routines.

Signed-off-by: Anton Vorontsov <avorontsov@mvista.com>
---
 arch/arm/Kconfig                         |    2 +-
 arch/arm/mach-cns3xxx/Kconfig            |    1 +
 arch/arm/mach-cns3xxx/Makefile           |    1 +
 arch/arm/mach-cns3xxx/include/mach/smp.h |   31 +++++++++++++
 arch/arm/mach-cns3xxx/platsmp.c          |   69 ++++++++++++++++++++++++++++++
 5 files changed, 103 insertions(+), 1 deletions(-)
 create mode 100644 arch/arm/mach-cns3xxx/include/mach/smp.h
 create mode 100644 arch/arm/mach-cns3xxx/platsmp.c

diff --git a/arch/arm/Kconfig b/arch/arm/Kconfig
index 4b54156..7bc76de 100644
--- a/arch/arm/Kconfig
+++ b/arch/arm/Kconfig
@@ -1101,7 +1101,7 @@ config SMP
 	bool "Symmetric Multi-Processing (EXPERIMENTAL)"
 	depends on EXPERIMENTAL && (REALVIEW_EB_ARM11MP || REALVIEW_EB_A9MP ||\
 		 MACH_REALVIEW_PB11MP || MACH_REALVIEW_PBX || ARCH_OMAP4 ||\
-		 ARCH_U8500 || ARCH_VEXPRESS_CA9X4)
+		 ARCH_U8500 || ARCH_VEXPRESS_CA9X4 || ARCH_CNS3XXX)
 	depends on GENERIC_CLOCKEVENTS
 	select USE_GENERIC_SMP_HELPERS
 	select HAVE_ARM_SCU if (ARCH_REALVIEW || ARCH_OMAP4 || ARCH_U8500 || ARCH_VEXPRESS_CA9X4)
diff --git a/arch/arm/mach-cns3xxx/Kconfig b/arch/arm/mach-cns3xxx/Kconfig
index 9ebfcc4..0ba82aa 100644
--- a/arch/arm/mach-cns3xxx/Kconfig
+++ b/arch/arm/mach-cns3xxx/Kconfig
@@ -3,6 +3,7 @@ menu "CNS3XXX platform type"
 
 config MACH_CNS3420VB
 	bool "Support for CNS3420 Validation Board"
+	select HAVE_ARM_SCU if SMP
 	help
 	  Include support for the Cavium Networks CNS3420 MPCore Platform
 	  Baseboard.
diff --git a/arch/arm/mach-cns3xxx/Makefile b/arch/arm/mach-cns3xxx/Makefile
index 11033f1..ba4ee7b 100644
--- a/arch/arm/mach-cns3xxx/Makefile
+++ b/arch/arm/mach-cns3xxx/Makefile
@@ -1,3 +1,4 @@
 obj-$(CONFIG_ARCH_CNS3XXX)		+= core.o pm.o devices.o
+obj-$(CONFIG_SMP)			+= platsmp.o
 obj-$(CONFIG_PCI)			+= pcie.o
 obj-$(CONFIG_MACH_CNS3420VB)		+= cns3420vb.o
diff --git a/arch/arm/mach-cns3xxx/include/mach/smp.h b/arch/arm/mach-cns3xxx/include/mach/smp.h
new file mode 100644
index 0000000..afb2fef
--- /dev/null
+++ b/arch/arm/mach-cns3xxx/include/mach/smp.h
@@ -0,0 +1,31 @@
+/*
+ * Copyright 2002 ARM Ltd.
+ * Copyright 2008 Cavium Networks
+ *
+ * This file is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License, Version 2, as
+ * published by the Free Software Foundation.
+ */
+
+#ifndef __MACH_SMP_H
+#define __MACH_SMP_H
+
+#include <asm/hardware/gic.h>
+
+#define hard_smp_processor_id()				\
+	({						\
+		unsigned int cpunum;			\
+		__asm__("mrc p15, 0, %0, c0, c0, 5"	\
+			: "=r" (cpunum));		\
+		cpunum &= 0x0F;				\
+	})
+
+/*
+ * We use IRQ1 as the IPI
+ */
+static inline void smp_cross_call(const struct cpumask *mask)
+{
+	gic_raise_softirq(mask, 1);
+}
+
+#endif /* __MACH_SMP_H */
diff --git a/arch/arm/mach-cns3xxx/platsmp.c b/arch/arm/mach-cns3xxx/platsmp.c
new file mode 100644
index 0000000..fc1595c
--- /dev/null
+++ b/arch/arm/mach-cns3xxx/platsmp.c
@@ -0,0 +1,69 @@
+/*
+ * Copyright 2002 ARM Ltd
+ * Copyright 2008 Cavium Networks
+ * Copyright 2010 MontaVista Software, LLC.
+ *
+ * This file is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License, Version 2, as
+ * published by the Free Software Foundation.
+ */
+
+#include <linux/compiler.h>
+#include <linux/kernel.h>
+#include <linux/init.h>
+#include <linux/smp.h>
+#include <linux/io.h>
+#include <asm/unified.h>
+#include <asm/smp_scu.h>
+#include <mach/cns3xxx.h>
+#include "core.h"
+
+static void __iomem *scu_base_addr(void)
+{
+	return (void __iomem *)CNS3XXX_TC11MP_SCU_BASE_VIRT;
+}
+
+void __cpuinit platform_secondary_init(unsigned int cpu)
+{
+	trace_hardirqs_off();
+
+	/*
+	 * if any interrupts are already enabled for the primary
+	 * core (e.g. timer irq), then they will not have been enabled
+	 * for us: do so
+	 */
+	gic_cpu_init(0, gic_cpu_base_addr);
+
+	scu_secondary_init(cpu);
+}
+
+int __cpuinit boot_secondary(unsigned int cpu, struct task_struct *idle)
+{
+	return scu_boot_secondary(cpu, idle);
+}
+
+void __init smp_init_cpus(void)
+{
+	scu_init_cpus(scu_base_addr());
+}
+
+static void __init poke_milo(void)
+{
+	if (num_present_cpus() <= 1)
+		return;
+	/*
+	 * Write the address of secondary startup into the system-wide flags
+	 * register. The BootMonitor waits for this register to become
+	 * non-zero.
+	 */
+	__raw_writel(BSYM(virt_to_phys(scu_secondary_startup)),
+		     (void __iomem *)(0xFFF07000 + 0x0600));
+
+	mb();
+}
+
+void __init smp_prepare_cpus(unsigned int max_cpus)
+{
+	scu_prepare_cpus(scu_base_addr(), max_cpus);
+	poke_milo();
+}
-- 
1.7.0.5

[PATCH 3/4] ARM: VExpress: Switch to generic SCU routines

[Anton Vorontsov]

This shrinks platform-specific code quite a bit.

Signed-off-by: Anton Vorontsov <avorontsov@mvista.com>
---
 arch/arm/mach-vexpress/Makefile  |    2 +-
 arch/arm/mach-vexpress/headsmp.S |   39 ----------
 arch/arm/mach-vexpress/platsmp.c |  150 ++++----------------------------------
 3 files changed, 15 insertions(+), 176 deletions(-)
 delete mode 100644 arch/arm/mach-vexpress/headsmp.S

diff --git a/arch/arm/mach-vexpress/Makefile b/arch/arm/mach-vexpress/Makefile
index 1b71b77..6533d11 100644
--- a/arch/arm/mach-vexpress/Makefile
+++ b/arch/arm/mach-vexpress/Makefile
@@ -4,5 +4,5 @@
 
 obj-y					:= v2m.o
 obj-$(CONFIG_ARCH_VEXPRESS_CA9X4)	+= ct-ca9x4.o
-obj-$(CONFIG_SMP)			+= platsmp.o headsmp.o
+obj-$(CONFIG_SMP)			+= platsmp.o
 obj-$(CONFIG_LOCAL_TIMERS)		+= localtimer.o
diff --git a/arch/arm/mach-vexpress/headsmp.S b/arch/arm/mach-vexpress/headsmp.S
deleted file mode 100644
index 8a78ff6..0000000
--- a/arch/arm/mach-vexpress/headsmp.S
+++ /dev/null
@@ -1,39 +0,0 @@
-/*
- *  linux/arch/arm/mach-vexpress/headsmp.S
- *
- *  Copyright (c) 2003 ARM Limited
- *  All Rights Reserved
- *
- * This program is free software; you can redistribute it and/or modify
- * it under the terms of the GNU General Public License version 2 as
- * published by the Free Software Foundation.
- */
-#include <linux/linkage.h>
-#include <linux/init.h>
-
-	__INIT
-
-/*
- * Versatile Express specific entry point for secondary CPUs.  This
- * provides a "holding pen" into which all secondary cores are held
- * until we're ready for them to initialise.
- */
-ENTRY(vexpress_secondary_startup)
-	mrc	p15, 0, r0, c0, c0, 5
-	and	r0, r0, #15
-	adr	r4, 1f
-	ldmia	r4, {r5, r6}
-	sub	r4, r4, r5
-	add	r6, r6, r4
-pen:	ldr	r7, [r6]
-	cmp	r7, r0
-	bne	pen
-
-	/*
-	 * we've been released from the holding pen: secondary_stack
-	 * should now contain the SVC stack for this core
-	 */
-	b	secondary_startup
-
-1:	.long	.
-	.long	pen_release
diff --git a/arch/arm/mach-vexpress/platsmp.c b/arch/arm/mach-vexpress/platsmp.c
index 6709706..2d4146f 100644
--- a/arch/arm/mach-vexpress/platsmp.c
+++ b/arch/arm/mach-vexpress/platsmp.c
@@ -9,15 +9,9 @@
  * published by the Free Software Foundation.
  */
 #include <linux/init.h>
-#include <linux/errno.h>
-#include <linux/delay.h>
-#include <linux/device.h>
-#include <linux/jiffies.h>
 #include <linux/smp.h>
 #include <linux/io.h>
 
-#include <asm/cacheflush.h>
-#include <asm/localtimer.h>
 #include <asm/smp_scu.h>
 #include <asm/unified.h>
 
@@ -27,21 +21,11 @@
 
 #include "core.h"
 
-extern void vexpress_secondary_startup(void);
-
-/*
- * control for which core is the next to come out of the secondary
- * boot "holding pen"
- */
-volatile int __cpuinitdata pen_release = -1;
-
 static void __iomem *scu_base_addr(void)
 {
 	return MMIO_P2V(A9_MPCORE_SCU);
 }
 
-static DEFINE_SPINLOCK(boot_lock);
-
 void __cpuinit platform_secondary_init(unsigned int cpu)
 {
 	trace_hardirqs_off();
@@ -53,138 +37,32 @@ void __cpuinit platform_secondary_init(unsigned int cpu)
 	 */
 	gic_cpu_init(0, gic_cpu_base_addr);
 
-	/*
-	 * let the primary processor know we're out of the
-	 * pen, then head off into the C entry point
-	 */
-	pen_release = -1;
-	smp_wmb();
-
-	/*
-	 * Synchronise with the boot thread.
-	 */
-	spin_lock(&boot_lock);
-	spin_unlock(&boot_lock);
+	scu_secondary_init(cpu);
 }
 
 int __cpuinit boot_secondary(unsigned int cpu, struct task_struct *idle)
 {
-	unsigned long timeout;
-
-	/*
-	 * Set synchronisation state between this boot processor
-	 * and the secondary one
-	 */
-	spin_lock(&boot_lock);
-
-	/*
-	 * This is really belt and braces; we hold unintended secondary
-	 * CPUs in the holding pen until we're ready for them.  However,
-	 * since we haven't sent them a soft interrupt, they shouldn't
-	 * be there.
-	 */
-	pen_release = cpu;
-	__cpuc_flush_dcache_area((void *)&pen_release, sizeof(pen_release));
-	outer_clean_range(__pa(&pen_release), __pa(&pen_release + 1));
-
-	/*
-	 * Send the secondary CPU a soft interrupt, thereby causing
-	 * the boot monitor to read the system wide flags register,
-	 * and branch to the address found there.
-	 */
-	smp_cross_call(cpumask_of(cpu));
-
-	timeout = jiffies + (1 * HZ);
-	while (time_before(jiffies, timeout)) {
-		smp_rmb();
-		if (pen_release == -1)
-			break;
-
-		udelay(10);
-	}
-
-	/*
-	 * now the secondary core is starting up let it run its
-	 * calibrations, then wait for it to finish
-	 */
-	spin_unlock(&boot_lock);
-
-	return pen_release != -1 ? -ENOSYS : 0;
+	return scu_boot_secondary(cpu, idle);
 }
 
-/*
- * Initialise the CPU possible map early - this describes the CPUs
- * which may be present or become present in the system.
- */
 void __init smp_init_cpus(void)
 {
-	void __iomem *scu_base = scu_base_addr();
-	unsigned int i, ncores;
-
-	ncores = scu_base ? scu_get_core_count(scu_base) : 1;
-
-	/* sanity check */
-	if (ncores == 0) {
-		printk(KERN_ERR
-		       "vexpress: strange CM count of 0? Default to 1\n");
-
-		ncores = 1;
-	}
+	scu_init_cpus(scu_base_addr());
+}
 
-	if (ncores > NR_CPUS) {
-		printk(KERN_WARNING
-		       "vexpress: no. of cores (%d) greater than configured "
-		       "maximum of %d - clipping\n",
-		       ncores, NR_CPUS);
-		ncores = NR_CPUS;
-	}
+/* If there are more than one CPU let them know where to start. */
+static void __init smp_point_cpus(void)
+{
+	if (num_present_cpus() <= 1)
+		return;
 
-	for (i = 0; i < ncores; i++)
-		set_cpu_possible(i, true);
+	writel(~0, MMIO_P2V(V2M_SYS_FLAGSCLR));
+	writel(BSYM(virt_to_phys(scu_secondary_startup)),
+	       MMIO_P2V(V2M_SYS_FLAGSSET));
 }
 
 void __init smp_prepare_cpus(unsigned int max_cpus)
 {
-	unsigned int ncores = num_possible_cpus();
-	unsigned int cpu = smp_processor_id();
-	int i;
-
-	smp_store_cpu_info(cpu);
-
-	/*
-	 * are we trying to boot more cores than exist?
-	 */
-	if (max_cpus > ncores)
-		max_cpus = ncores;
-
-	/*
-	 * Initialise the present map, which describes the set of CPUs
-	 * actually populated at the present time.
-	 */
-	for (i = 0; i < max_cpus; i++)
-		set_cpu_present(i, true);
-
-	/*
-	 * Initialise the SCU if there are more than one CPU and let
-	 * them know where to start.
-	 */
-	if (max_cpus > 1) {
-		/*
-		 * Enable the local timer or broadcast device for the
-		 * boot CPU, but only if we have more than one CPU.
-		 */
-		percpu_timer_setup();
-
-		scu_enable(scu_base_addr());
-
-		/*
-		 * Write the address of secondary startup into the
-		 * system-wide flags register. The boot monitor waits
-		 * until it receives a soft interrupt, and then the
-		 * secondary CPU branches to this address.
-		 */
-		writel(~0, MMIO_P2V(V2M_SYS_FLAGSCLR));
-		writel(BSYM(virt_to_phys(vexpress_secondary_startup)),
-			MMIO_P2V(V2M_SYS_FLAGSSET));
-	}
+	scu_prepare_cpus(scu_base_addr(), max_cpus);
+	smp_point_cpus();
 }
-- 
1.7.0.5

[PATCH 4/4] ARM: RealView: Switch to generic SCU routines

[Anton Vorontsov]

This shrinks platform-specific code quite a bit.

Signed-off-by: Anton Vorontsov <avorontsov@mvista.com>
---
 arch/arm/mach-realview/Makefile  |    2 +-
 arch/arm/mach-realview/headsmp.S |   39 ---------
 arch/arm/mach-realview/hotplug.c |    5 +-
 arch/arm/mach-realview/platsmp.c |  161 +++-----------------------------------
 4 files changed, 14 insertions(+), 193 deletions(-)
 delete mode 100644 arch/arm/mach-realview/headsmp.S

diff --git a/arch/arm/mach-realview/Makefile b/arch/arm/mach-realview/Makefile
index a01b76b..d456163 100644
--- a/arch/arm/mach-realview/Makefile
+++ b/arch/arm/mach-realview/Makefile
@@ -8,6 +8,6 @@ obj-$(CONFIG_MACH_REALVIEW_PB11MP)	+= realview_pb11mp.o
 obj-$(CONFIG_MACH_REALVIEW_PB1176)	+= realview_pb1176.o
 obj-$(CONFIG_MACH_REALVIEW_PBA8)	+= realview_pba8.o
 obj-$(CONFIG_MACH_REALVIEW_PBX)		+= realview_pbx.o
-obj-$(CONFIG_SMP)			+= platsmp.o headsmp.o
+obj-$(CONFIG_SMP)			+= platsmp.o
 obj-$(CONFIG_HOTPLUG_CPU)		+= hotplug.o
 obj-$(CONFIG_LOCAL_TIMERS)		+= localtimer.o
diff --git a/arch/arm/mach-realview/headsmp.S b/arch/arm/mach-realview/headsmp.S
deleted file mode 100644
index 4075473..0000000
--- a/arch/arm/mach-realview/headsmp.S
+++ /dev/null
@@ -1,39 +0,0 @@
-/*
- *  linux/arch/arm/mach-realview/headsmp.S
- *
- *  Copyright (c) 2003 ARM Limited
- *  All Rights Reserved
- *
- * This program is free software; you can redistribute it and/or modify
- * it under the terms of the GNU General Public License version 2 as
- * published by the Free Software Foundation.
- */
-#include <linux/linkage.h>
-#include <linux/init.h>
-
-	__INIT
-
-/*
- * Realview specific entry point for secondary CPUs.  This provides
- * a "holding pen" into which all secondary cores are held until we're
- * ready for them to initialise.
- */
-ENTRY(realview_secondary_startup)
-	mrc	p15, 0, r0, c0, c0, 5
-	and	r0, r0, #15
-	adr	r4, 1f
-	ldmia	r4, {r5, r6}
-	sub	r4, r4, r5
-	add	r6, r6, r4
-pen:	ldr	r7, [r6]
-	cmp	r7, r0
-	bne	pen
-
-	/*
-	 * we've been released from the holding pen: secondary_stack
-	 * should now contain the SVC stack for this core
-	 */
-	b	secondary_startup
-
-1:	.long	.
-	.long	pen_release
diff --git a/arch/arm/mach-realview/hotplug.c b/arch/arm/mach-realview/hotplug.c
index f95521a..cf2c1ea 100644
--- a/arch/arm/mach-realview/hotplug.c
+++ b/arch/arm/mach-realview/hotplug.c
@@ -13,10 +13,9 @@
 #include <linux/smp.h>
 #include <linux/completion.h>
 
+#include <asm/smp_scu.h>
 #include <asm/cacheflush.h>
 
-extern volatile int pen_release;
-
 static DECLARE_COMPLETION(cpu_killed);
 
 static inline void cpu_enter_lowpower(void)
@@ -72,7 +71,7 @@ static inline void platform_do_lowpower(unsigned int cpu)
 		    :
 		    : "memory", "cc");
 
-		if (pen_release == cpu) {
+		if (scu_pen_release == cpu) {
 			/*
 			 * OK, proper wakeup, we're done
 			 */
diff --git a/arch/arm/mach-realview/platsmp.c b/arch/arm/mach-realview/platsmp.c
index 0092658..3c1000b 100644
--- a/arch/arm/mach-realview/platsmp.c
+++ b/arch/arm/mach-realview/platsmp.c
@@ -9,17 +9,11 @@
  * published by the Free Software Foundation.
  */
 #include <linux/init.h>
-#include <linux/errno.h>
-#include <linux/delay.h>
-#include <linux/device.h>
-#include <linux/jiffies.h>
 #include <linux/smp.h>
 #include <linux/io.h>
 
-#include <asm/cacheflush.h>
 #include <mach/hardware.h>
 #include <asm/mach-types.h>
-#include <asm/localtimer.h>
 #include <asm/unified.h>
 
 #include <mach/board-eb.h>
@@ -29,14 +23,6 @@
 
 #include "core.h"
 
-extern void realview_secondary_startup(void);
-
-/*
- * control for which core is the next to come out of the secondary
- * boot "holding pen"
- */
-volatile int __cpuinitdata pen_release = -1;
-
 static void __iomem *scu_base_addr(void)
 {
 	if (machine_is_realview_eb_mp())
@@ -50,16 +36,6 @@ static void __iomem *scu_base_addr(void)
 		return (void __iomem *)0;
 }
 
-static inline unsigned int get_core_count(void)
-{
-	void __iomem *scu_base = scu_base_addr();
-	if (scu_base)
-		return scu_get_core_count(scu_base);
-	return 1;
-}
-
-static DEFINE_SPINLOCK(boot_lock);
-
 void __cpuinit platform_secondary_init(unsigned int cpu)
 {
 	trace_hardirqs_off();
@@ -71,151 +47,36 @@ void __cpuinit platform_secondary_init(unsigned int cpu)
 	 */
 	gic_cpu_init(0, gic_cpu_base_addr);
 
-	/*
-	 * let the primary processor know we're out of the
-	 * pen, then head off into the C entry point
-	 */
-	pen_release = -1;
-	smp_wmb();
-
-	/*
-	 * Synchronise with the boot thread.
-	 */
-	spin_lock(&boot_lock);
-	spin_unlock(&boot_lock);
+	scu_secondary_init(cpu);
 }
 
 int __cpuinit boot_secondary(unsigned int cpu, struct task_struct *idle)
 {
-	unsigned long timeout;
-
-	/*
-	 * set synchronisation state between this boot processor
-	 * and the secondary one
-	 */
-	spin_lock(&boot_lock);
-
-	/*
-	 * The secondary processor is waiting to be released from
-	 * the holding pen - release it, then wait for it to flag
-	 * that it has been released by resetting pen_release.
-	 *
-	 * Note that "pen_release" is the hardware CPU ID, whereas
-	 * "cpu" is Linux's internal ID.
-	 */
-	pen_release = cpu;
-	flush_cache_all();
-
-	/*
-	 * XXX
-	 *
-	 * This is a later addition to the booting protocol: the
-	 * bootMonitor now puts secondary cores into WFI, so
-	 * poke_milo() no longer gets the cores moving; we need
-	 * to send a soft interrupt to wake the secondary core.
-	 * Use smp_cross_call() for this, since there's little
-	 * point duplicating the code here
-	 */
-	smp_cross_call(cpumask_of(cpu));
-
-	timeout = jiffies + (1 * HZ);
-	while (time_before(jiffies, timeout)) {
-		smp_rmb();
-		if (pen_release == -1)
-			break;
-
-		udelay(10);
-	}
-
-	/*
-	 * now the secondary core is starting up let it run its
-	 * calibrations, then wait for it to finish
-	 */
-	spin_unlock(&boot_lock);
+	return scu_boot_secondary(cpu, idle);
+}
 
-	return pen_release != -1 ? -ENOSYS : 0;
+void __init smp_init_cpus(void)
+{
+	scu_init_cpus(scu_base_addr());
 }
 
 static void __init poke_milo(void)
 {
-	/* nobody is to be released from the pen yet */
-	pen_release = -1;
-
+	if (num_present_cpus() <= 1)
+		return;
 	/*
 	 * Write the address of secondary startup into the system-wide flags
 	 * register. The BootMonitor waits for this register to become
 	 * non-zero.
 	 */
-	__raw_writel(BSYM(virt_to_phys(realview_secondary_startup)),
+	__raw_writel(BSYM(virt_to_phys(scu_secondary_startup)),
 		     __io_address(REALVIEW_SYS_FLAGSSET));
 
 	mb();
 }
 
-/*
- * Initialise the CPU possible map early - this describes the CPUs
- * which may be present or become present in the system.
- */
-void __init smp_init_cpus(void)
-{
-	unsigned int i, ncores = get_core_count();
-
-	for (i = 0; i < ncores; i++)
-		set_cpu_possible(i, true);
-}
-
 void __init smp_prepare_cpus(unsigned int max_cpus)
 {
-	unsigned int ncores = get_core_count();
-	unsigned int cpu = smp_processor_id();
-	int i;
-
-	/* sanity check */
-	if (ncores == 0) {
-		printk(KERN_ERR
-		       "Realview: strange CM count of 0? Default to 1\n");
-
-		ncores = 1;
-	}
-
-	if (ncores > NR_CPUS) {
-		printk(KERN_WARNING
-		       "Realview: no. of cores (%d) greater than configured "
-		       "maximum of %d - clipping\n",
-		       ncores, NR_CPUS);
-		ncores = NR_CPUS;
-	}
-
-	smp_store_cpu_info(cpu);
-
-	/*
-	 * are we trying to boot more cores than exist?
-	 */
-	if (max_cpus > ncores)
-		max_cpus = ncores;
-
-	/*
-	 * Initialise the present map, which describes the set of CPUs
-	 * actually populated at the present time.
-	 */
-	for (i = 0; i < max_cpus; i++)
-		set_cpu_present(i, true);
-
-	/*
-	 * Initialise the SCU if there are more than one CPU and let
-	 * them know where to start. Note that, on modern versions of
-	 * MILO, the "poke" doesn't actually do anything until each
-	 * individual core is sent a soft interrupt to get it out of
-	 * WFI
-	 */
-	if (max_cpus > 1) {
-		/*
-		 * Enable the local timer or broadcast device for the
-		 * boot CPU, but only if we have more than one CPU.
-		 */
-		percpu_timer_setup();
-
-		scu_enable(scu_base_addr());
-		poke_milo();
-	}
+	scu_prepare_cpus(scu_base_addr(), max_cpus);
+	poke_milo();
 }
-- 
1.7.0.5

[PATCH 1/4] ARM: SCU: Add generic routines for secondary CPU bootup

[Russell King - ARM Linux]

On Tue, Jul 27, 2010 at 09:11:40PM +0400, Anton Vorontsov wrote:
> For CNS3xxx we want to reuse the original ARM approach of booting
> secondary CPUs. This patch turns VExpress' code into "SCU library",
> so that now platform code can call the generic routines.

I'm not sure this should become a "SCU library" - the name seems wrong.
When you consider that the SCU is the 'snoop control unit', which has
no relation to how secondary CPUs are booted, it makes little sense.

The secondary CPU boot protocol has much more to do with the boot
loader than with the hardware.

I also suspect that we can get rid of the 'holding pen' stuff and just
rely on the IPI being sent to the intended target CPU.  I don't think
anyone has really updated this since the initial ARM SMP implementation.

[PATCH 1/4] ARM: SCU: Add generic routines for secondary CPU bootup

[Anton Vorontsov]

On Sat, Jul 31, 2010 at 04:41:34PM +0100, Russell King - ARM Linux wrote:
> On Tue, Jul 27, 2010 at 09:11:40PM +0400, Anton Vorontsov wrote:
> > For CNS3xxx we want to reuse the original ARM approach of booting
> > secondary CPUs. This patch turns VExpress' code into "SCU library",
> > so that now platform code can call the generic routines.
> 
> I'm not sure this should become a "SCU library" - the name seems wrong.

Well, how would you name it? Would you also move these routines
into a separate file (is there any preference for the file name)?

> When you consider that the SCU is the 'snoop control unit', which has
> no relation to how secondary CPUs are booted, it makes little sense.
> 
> The secondary CPU boot protocol has much more to do with the boot
> loader than with the hardware.

Yup.

> I also suspect that we can get rid of the 'holding pen' stuff and just
> rely on the IPI being sent to the intended target CPU.  I don't think
> anyone has really updated this since the initial ARM SMP implementation.

Sure, but I belive that this patch set is a step into the right
direction, basically it just gets rid of code duplication, and
then we can improve the bootup code for all interested machines.

With these patches, machines don't* bother with the holding pen,
it just becomes an implementation detail of the SCU/secondary
CPU bootup code.

So it makes sense to consolidate the code anyway (which exactly
what these patches do), right?

* Except realview, it also uses scu_pen_release for hotplug
  handling.

-- 
Anton Vorontsov
email: cbouatmailru at gmail.com
irc://irc.freenode.net/bd2