[PATCH v3 0/7] x86: BSP or CPU0 online/offline

[PATCH v3 0/7] x86: BSP or CPU0 online/offline

[Fenghua Yu]

From: Fenghua Yu <fenghua.yu@intel.com>

BSP or CPU0 has been the last obstacle to CPU hotplug on x86. This patch set
implements BSP online and offline and removes this obstacle to CPU hotplug.

RAS needs the feature. If socket0 needs to be hotplugged for any reason (any
thread on socket0 is bad, shared cache issue, uncore issue, etc), CPU0 is
required to be offline or hot replaced to keep the system run.

v3: Register a pm notifier to check if CPU0 is online before hibernate/suspend.
Small wording changes in document and print info.

v2: Add locking changes between cpu hotplug and hibernate/suspend. Change PIC
irq bound to CPU0 detection.

Fenghua Yu (7):
  x86/topology.c: Support functions for BSP online/offline
  x86/common.c: Init BSP data during BSP online
  x86/mtrr/main.c: Ask the first online CPU to save mtrr
  x86/smpboot.c: Don't offline BSP if any irq can not be migrated out
    of it
  Documentations/cpu-hotplug.tx, kernel-parameters.txt: Add x86 CPU0
    online/offline feature
  x86/i387.c: Thread xstate is initialized only on BSP once
  x86/power/cpu.c: Don't hibernate/suspend if CPU0 is offline

 Documentation/cpu-hotplug.txt       |   19 +++++++++++++++
 Documentation/kernel-parameters.txt |   13 ++++++++++
 arch/x86/include/asm/processor.h    |    1 +
 arch/x86/kernel/cpu/common.c        |   13 ++++++++--
 arch/x86/kernel/cpu/mtrr/main.c     |    9 +++++-
 arch/x86/kernel/i387.c              |    9 ++++++-
 arch/x86/kernel/smpboot.c           |   43 ++++++++++++++++++++++++++++-----
 arch/x86/kernel/topology.c          |   24 +++++++++++++-----
 arch/x86/power/cpu.c                |   44 +++++++++++++++++++++++++++++++++++
 9 files changed, 155 insertions(+), 20 deletions(-)

[PATCH v3 1/7] x86/topology.c: Support functions for BSP online/offline

[Fenghua Yu]

From: Fenghua Yu <fenghua.yu@intel.com>

By default, BSP can't be hotpluggable because bsp_hotpluggable is 0. Kernel
parameter bsp_hotplug can enable BSP hotplug feature.

Signed-off-by: Fenghua Yu <fenghua.yu@intel.com>
---
 arch/x86/kernel/topology.c |   24 +++++++++++++++++-------
 1 files changed, 17 insertions(+), 7 deletions(-)

diff --git a/arch/x86/kernel/topology.c b/arch/x86/kernel/topology.c
index 76ee977..a3fc939 100644
--- a/arch/x86/kernel/topology.c
+++ b/arch/x86/kernel/topology.c
@@ -35,18 +35,28 @@
 static DEFINE_PER_CPU(struct x86_cpu, cpu_devices);
 
 #ifdef CONFIG_HOTPLUG_CPU
+
+static int bsp_hotpluggable;
+
+static int __init enable_bsp_hotplug(char *str)
+{
+	bsp_hotpluggable = 1;
+	return 1;
+}
+
+__setup("bsp_hotplug", enable_bsp_hotplug);
+
 int __ref arch_register_cpu(int num)
 {
 	/*
-	 * CPU0 cannot be offlined due to several
-	 * restrictions and assumptions in kernel. This basically
-	 * doesn't add a control file, one cannot attempt to offline
-	 * BSP.
+	 * Resume from suspend/hibernate depends on BSP. PIC interrupts depend
+	 * on BSP.
 	 *
-	 * Also certain PCI quirks require not to enable hotplug control
-	 * for all CPU's.
+	 * If the BSP depencies are under control, one can tell kernel to
+	 * enable BSP hotplug. This basically adds a control file and
+	 * one can attempt to offline BSP.
 	 */
-	if (num)
+	if (num || bsp_hotpluggable)
 		per_cpu(cpu_devices, num).cpu.hotpluggable = 1;
 
 	return register_cpu(&per_cpu(cpu_devices, num).cpu, num);
-- 
1.6.0.3

[PATCH v3 2/7] x86/common.c: Init BSP data during BSP online

[Fenghua Yu]

From: Fenghua Yu <fenghua.yu@intel.com>

During BSP online, enable x2apic, set_numa_node, add numa mask, and enable
sep cpu for BSP.

Signed-off-by: Fenghua Yu <fenghua.yu@intel.com>
---
 arch/x86/include/asm/processor.h |    1 +
 arch/x86/kernel/cpu/common.c     |   13 ++++++++++---
 2 files changed, 11 insertions(+), 3 deletions(-)

diff --git a/arch/x86/include/asm/processor.h b/arch/x86/include/asm/processor.h
index b650435..4057161 100644
--- a/arch/x86/include/asm/processor.h
+++ b/arch/x86/include/asm/processor.h
@@ -162,6 +162,7 @@ extern struct pt_regs *idle_regs(struct pt_regs *);
 
 extern void early_cpu_init(void);
 extern void identify_boot_cpu(void);
+extern void identify_boot_cpu_online(void);
 extern void identify_secondary_cpu(struct cpuinfo_x86 *);
 extern void print_cpu_info(struct cpuinfo_x86 *);
 extern void init_scattered_cpuid_features(struct cpuinfo_x86 *c);
diff --git a/arch/x86/kernel/cpu/common.c b/arch/x86/kernel/cpu/common.c
index aa003b1..faa3bda 100644
--- a/arch/x86/kernel/cpu/common.c
+++ b/arch/x86/kernel/cpu/common.c
@@ -916,6 +916,14 @@ void __init identify_boot_cpu(void)
 #endif
 }
 
+void __cpuinit identify_boot_cpu_online(void)
+{
+	numa_add_cpu(smp_processor_id());
+#ifdef CONFIG_X86_32
+	enable_sep_cpu();
+#endif
+}
+
 void __cpuinit identify_secondary_cpu(struct cpuinfo_x86 *c)
 {
 	BUG_ON(c == &boot_cpu_data);
@@ -1163,7 +1171,7 @@ void __cpuinit cpu_init(void)
 	oist = &per_cpu(orig_ist, cpu);
 
 #ifdef CONFIG_NUMA
-	if (cpu != 0 && percpu_read(numa_node) == 0 &&
+	if (percpu_read(numa_node) == 0 &&
 	    early_cpu_to_node(cpu) != NUMA_NO_NODE)
 		set_numa_node(early_cpu_to_node(cpu));
 #endif
@@ -1195,8 +1203,7 @@ void __cpuinit cpu_init(void)
 	barrier();
 
 	x86_configure_nx();
-	if (cpu != 0)
-		enable_x2apic();
+	enable_x2apic();
 
 	/*
 	 * set up and load the per-CPU TSS
-- 
1.6.0.3

[PATCH v3 3/7] x86/mtrr/main.c: Ask the first online CPU to save mtrr

[Fenghua Yu]

From: Fenghua Yu <fenghua.yu@intel.com>

Ask the first online CPU to save mtrr instead of asking BSP. BSP could be
offline when mtrr_save_state() is called.

Signed-off-by: Fenghua Yu <fenghua.yu@intel.com>
---
 arch/x86/kernel/cpu/mtrr/main.c |    9 +++++++--
 1 files changed, 7 insertions(+), 2 deletions(-)

diff --git a/arch/x86/kernel/cpu/mtrr/main.c b/arch/x86/kernel/cpu/mtrr/main.c
index 6b96110..e4c1a41 100644
--- a/arch/x86/kernel/cpu/mtrr/main.c
+++ b/arch/x86/kernel/cpu/mtrr/main.c
@@ -695,11 +695,16 @@ void mtrr_ap_init(void)
 }
 
 /**
- * Save current fixed-range MTRR state of the BSP
+ * Save current fixed-range MTRR state of the first cpu in cpu_online_mask.
  */
 void mtrr_save_state(void)
 {
-	smp_call_function_single(0, mtrr_save_fixed_ranges, NULL, 1);
+	int first_cpu;
+
+	get_online_cpus();
+	first_cpu = cpumask_first(cpu_online_mask);
+	smp_call_function_single(first_cpu, mtrr_save_fixed_ranges, NULL, 1);
+	put_online_cpus();
 }
 
 void set_mtrr_aps_delayed_init(void)
-- 
1.6.0.3

[PATCH v3 4/7] x86/smpboot.c: Don't offline BSP if any irq can not be migrated out of it

[Fenghua Yu]

From: Fenghua Yu <fenghua.yu@intel.com>

Don't offline BSP if any irq can not be migrated out of it.

Call identify_boot_cpu_online() for BSP in smp_callin() and continue to online
BSP in native_cpu_up().

Signed-off-by: Fenghua Yu <fenghua.yu@intel.com>
---
 arch/x86/kernel/smpboot.c |   43 ++++++++++++++++++++++++++++++++++++-------
 1 files changed, 36 insertions(+), 7 deletions(-)

diff --git a/arch/x86/kernel/smpboot.c b/arch/x86/kernel/smpboot.c
index 9f548cb..83838c5 100644
--- a/arch/x86/kernel/smpboot.c
+++ b/arch/x86/kernel/smpboot.c
@@ -136,8 +136,8 @@ EXPORT_PER_CPU_SYMBOL(cpu_info);
 atomic_t init_deasserted;
 
 /*
- * Report back to the Boot Processor.
- * Running on AP.
+ * Report back to the Boot Processor during boot time or to the caller processor
+ * during CPU online.
  */
 static void __cpuinit smp_callin(void)
 {
@@ -224,6 +224,13 @@ static void __cpuinit smp_callin(void)
 	smp_store_cpu_info(cpuid);
 
 	/*
+	 * This function won't run on the BSP during boot time. It run
+	 * on BSP only when BSP is offlined and onlined again.
+	 */
+	if (cpuid == 0)
+		identify_boot_cpu_online();
+
+	/*
 	 * This must be done before setting cpu_online_mask
 	 * or calling notify_cpu_starting.
 	 */
@@ -839,7 +846,7 @@ int __cpuinit native_cpu_up(unsigned int cpu)
 
 	pr_debug("++++++++++++++++++++=_---CPU UP  %u\n", cpu);
 
-	if (apicid == BAD_APICID || apicid == boot_cpu_physical_apicid ||
+	if (apicid == BAD_APICID ||
 	    !physid_isset(apicid, phys_cpu_present_map)) {
 		printk(KERN_ERR "%s: bad cpu %d\n", __func__, cpu);
 		return -EINVAL;
@@ -1283,12 +1290,34 @@ int native_cpu_disable(void)
 	 * Perhaps use cpufreq to drop frequency, but that could go
 	 * into generic code.
 	 *
-	 * We won't take down the boot processor on i386 due to some
+	 * We won't take down the boot processor on i386 if some
 	 * interrupts only being able to be serviced by the BSP.
-	 * Especially so if we're not using an IOAPIC	-zwane
+	 * Especially so if we're not using an IOAPIC
+	 * -original comment from zwane, changed by fyu
 	 */
-	if (cpu == 0)
-		return -EBUSY;
+	if (cpu == 0) {
+		int irq;
+		struct irq_desc *desc;
+		struct irq_data *data;
+		struct irq_chip *chip;
+
+		for_each_irq_desc(irq, desc) {
+			raw_spin_lock(&desc->lock);
+			if (!irq_has_action(irq)) {
+				raw_spin_unlock(&desc->lock);
+				continue;
+			}
+
+			data = irq_desc_get_irq_data(desc);
+			chip = irq_data_get_irq_chip(data);
+			if (!chip->irq_set_affinity) {
+				pr_debug("irq%d can't move out of BSP\n", irq);
+				raw_spin_unlock(&desc->lock);
+				return -EBUSY;
+			}
+			raw_spin_unlock(&desc->lock);
+		}
+	}
 
 	clear_local_APIC();
 
-- 
1.6.0.3

[PATCH v3 5/7] Documentations/cpu-hotplug.tx, kernel-parameters.txt: Add x86 CPU0 online/offline feature

[Fenghua Yu]

From: Fenghua Yu <fenghua.yu@intel.com>

Add x86 CPU0 online/offline feature in the two documentations.

By default BSP is not pluggable. Kernel parameter bsp_hotplug enables BSP
online/offline feature.

The documentations point out two BSP dependencies. First, resume from hibernate
or suspend always starts from BSP. So hibernate and suspend are prevented if BSP
is offline. Another dependency is PIC interrupts always go to BSP.

It's said that some machines may depend on CPU0 to poweroff/reboot. But I
haven't seen such dependency on a few tested machines.

Please let me know if you see any CPU0 dependencies on your machine.

Signed-off-by: Fenghua Yu <fenghua.yu@intel.com>
---
---
 Documentation/cpu-hotplug.txt       |   19 +++++++++++++++++++
 Documentation/kernel-parameters.txt |   13 +++++++++++++
 2 files changed, 32 insertions(+), 0 deletions(-)

diff --git a/Documentation/cpu-hotplug.txt b/Documentation/cpu-hotplug.txt
index a20bfd4..93af1a0 100644
--- a/Documentation/cpu-hotplug.txt
+++ b/Documentation/cpu-hotplug.txt
@@ -207,6 +207,25 @@ by making it not-removable.
 
 In such cases you will also notice that the online file is missing under cpu0.
 
+Q: Is CPU0 removable on X86?
+A: Yes. If given kernel option bsp_hotplug, CPU0 is removable.
+
+But some features depend on BSP. Known dependencies are:
+1. Resume from hibernate/suspend depends on BSP. Hibernate/suspend will fail if
+BSP is offline and you need to online BSP before hibernate/suspend can continue.
+2. PIC interrupts also depend on BSP. BSP can't be removed if a PIC interrupt
+is detected.
+
+It's said poweroff/reboot may depend on BSP on some machines although I haven't
+seen any poweroff/reboot failure so far after BSP is offline on a few tested
+machines.
+
+Please let me know if you know or see any other dependencies of BSP.
+
+If the dependencies are under your control, you can turn on bsp_hotplug.
+
+--Fenghua Yu
+
 Q: How do i find out if a particular CPU is not removable?
 A: Depending on the implementation, some architectures may show this by the
 absence of the "online" file. This is done if it can be determined ahead of
diff --git a/Documentation/kernel-parameters.txt b/Documentation/kernel-parameters.txt
index a0c5c5f..55582db 100644
--- a/Documentation/kernel-parameters.txt
+++ b/Documentation/kernel-parameters.txt
@@ -1842,6 +1842,19 @@ bytes respectively. Such letter suffixes can also be entirely omitted.
 
 	nox2apic	[X86-64,APIC] Do not enable x2APIC mode.
 
+	bsp_hotplug	[X86] BSP (aka CPU0) is hotpluggable.
+			Some features depend on BSP. Known dependencies are:
+			1. Resume from suspend/hibernate depends on BSP.
+			Suspend/hibernate will fail if BSP is offline and you
+			need to online BSP before suspend/hibernate.
+			2. PIC interrupts also depend on BSP. BSP can't be
+			removed if a PIC interrupt is detected.
+			It's said poweroff/reboot may depend on BSP on some
+			machines although I haven't seen such issues so far
+			after BSP is offline on a few tested machines.
+			If the dependencies are under your control, you can
+			turn on bsp_hotplug.
+
 	nptcg=		[IA-64] Override max number of concurrent global TLB
 			purges which is reported from either PAL_VM_SUMMARY or
 			SAL PALO.
-- 
1.6.0.3

[PATCH v3 6/7] x86/i387.c: Thread xstate is initialized only on BSP once

[Fenghua Yu]

From: Fenghua Yu <fenghua.yu@intel.com>

init_thread_xstate() is only called on BSP once to avoid to override
xstate_size.

Signed-off-by: Fenghua Yu <fenghua.yu@intel.com>
---
 arch/x86/kernel/i387.c |    9 ++++++++-
 1 files changed, 8 insertions(+), 1 deletions(-)

diff --git a/arch/x86/kernel/i387.c b/arch/x86/kernel/i387.c
index 739d859..f721a61 100644
--- a/arch/x86/kernel/i387.c
+++ b/arch/x86/kernel/i387.c
@@ -93,6 +93,7 @@ void __cpuinit fpu_init(void)
 {
 	unsigned long cr0;
 	unsigned long cr4_mask = 0;
+	static int once = 1;
 
 	if (cpu_has_fxsr)
 		cr4_mask |= X86_CR4_OSFXSR;
@@ -107,8 +108,14 @@ void __cpuinit fpu_init(void)
 		cr0 |= X86_CR0_EM;
 	write_cr0(cr0);
 
-	if (!smp_processor_id())
+	/*
+	 * init_thread_xstate is only called on BSP once to avoid to override
+	 * xstate_size.
+	 */
+	if (!smp_processor_id() && once) {
+		once = 0;
 		init_thread_xstate();
+	}
 
 	mxcsr_feature_mask_init();
 	/* clean state in init */
-- 
1.6.0.3

[PATCH v3 7/7] x86/power/cpu.c: Don't hibernate/suspend if CPU0 is offline

[Fenghua Yu]

From: Fenghua Yu <fenghua.yu@intel.com>

Because x86 BIOS requires CPU0 to resume from sleep, suspend or hibernate
can't be executed if CPU0 is detected offline.

Signed-off-by: Fenghua Yu <fenghua.yu@intel.com>
---
 arch/x86/power/cpu.c |   44 ++++++++++++++++++++++++++++++++++++++++++++
 1 files changed, 44 insertions(+), 0 deletions(-)

diff --git a/arch/x86/power/cpu.c b/arch/x86/power/cpu.c
index f10c0af..a4ec084 100644
--- a/arch/x86/power/cpu.c
+++ b/arch/x86/power/cpu.c
@@ -236,3 +236,47 @@ void restore_processor_state(void)
 #ifdef CONFIG_X86_32
 EXPORT_SYMBOL(restore_processor_state);
 #endif
+
+/*
+ * When bsp_check() is called in hibernate and suspend, cpu hotplug
+ * is disabled already. So it's unnessary to handle race condition between
+ * cpumask query and cpu hotplug.
+ */
+static int bsp_check(void)
+{
+	if (cpumask_first(cpu_online_mask) != 0) {
+		printk(KERN_WARNING "CPU0 is offline.\n");
+		return -ENODEV;
+	}
+
+	return 0;
+}
+
+static int bsp_pm_callback(struct notifier_block *nb, unsigned long action,
+			   void *ptr)
+{
+	int ret = 0;
+
+	switch (action) {
+	case PM_SUSPEND_PREPARE:
+	case PM_HIBERNATION_PREPARE:
+		ret = bsp_check();
+		break;
+	default:
+		break;
+	}
+	return notifier_from_errno(ret);
+}
+
+static int __init bsp_pm_check_init(void)
+{
+	/*
+	 * Set this bsp_pm_callback as lower priority than
+	 * cpu_hotplug_pm_callback. So cpu_hotplug_pm_callback will be called
+	 * earlier to disable cpu hotplug before bsp online check.
+	 */
+	pm_notifier(bsp_pm_callback, -INT_MAX);
+	return 0;
+}
+
+core_initcall(bsp_pm_check_init);
-- 
1.6.0.3

Re: [PATCH v3 1/7] x86/topology.c: Support functions for BSP online/offline

[Konrad Rzeszutek Wilk]

On Wed, Nov 09, 2011 at 04:34:11PM -0800, Fenghua Yu wrote:
> From: Fenghua Yu <fenghua.yu@intel.com>
> 
> By default, BSP can't be hotpluggable because bsp_hotpluggable is 0. Kernel
> parameter bsp_hotplug can enable BSP hotplug feature.
> 
> Signed-off-by: Fenghua Yu <fenghua.yu@intel.com>
> ---
>  arch/x86/kernel/topology.c |   24 +++++++++++++++++-------
>  1 files changed, 17 insertions(+), 7 deletions(-)
> 
> diff --git a/arch/x86/kernel/topology.c b/arch/x86/kernel/topology.c
> index 76ee977..a3fc939 100644
> --- a/arch/x86/kernel/topology.c
> +++ b/arch/x86/kernel/topology.c
> @@ -35,18 +35,28 @@
>  static DEFINE_PER_CPU(struct x86_cpu, cpu_devices);
>  
>  #ifdef CONFIG_HOTPLUG_CPU
> +
> +static int bsp_hotpluggable;

__read_mostly

> +
> +static int __init enable_bsp_hotplug(char *str)
> +{
> +	bsp_hotpluggable = 1;
> +	return 1;
> +}
> +
> +__setup("bsp_hotplug", enable_bsp_hotplug);
> +
>  int __ref arch_register_cpu(int num)
>  {
>  	/*
> -	 * CPU0 cannot be offlined due to several
> -	 * restrictions and assumptions in kernel. This basically
> -	 * doesn't add a control file, one cannot attempt to offline
> -	 * BSP.
> +	 * Resume from suspend/hibernate depends on BSP. PIC interrupts depend
> +	 * on BSP.
>  	 *
> -	 * Also certain PCI quirks require not to enable hotplug control
> -	 * for all CPU's.
> +	 * If the BSP depencies are under control, one can tell kernel to
> +	 * enable BSP hotplug. This basically adds a control file and
> +	 * one can attempt to offline BSP.
>  	 */
> -	if (num)
> +	if (num || bsp_hotpluggable)
>  		per_cpu(cpu_devices, num).cpu.hotpluggable = 1;
>  
>  	return register_cpu(&per_cpu(cpu_devices, num).cpu, num);
> -- 
> 1.6.0.3
> 
> --
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/

Re: [PATCH v3 6/7] x86/i387.c: Thread xstate is initialized only on BSP once

[Konrad Rzeszutek Wilk]

On Wed, Nov 09, 2011 at 04:34:16PM -0800, Fenghua Yu wrote:
> From: Fenghua Yu <fenghua.yu@intel.com>
> 
> init_thread_xstate() is only called on BSP once to avoid to override
> xstate_size.
> 
> Signed-off-by: Fenghua Yu <fenghua.yu@intel.com>
> ---
>  arch/x86/kernel/i387.c |    9 ++++++++-
>  1 files changed, 8 insertions(+), 1 deletions(-)
> 
> diff --git a/arch/x86/kernel/i387.c b/arch/x86/kernel/i387.c
> index 739d859..f721a61 100644
> --- a/arch/x86/kernel/i387.c
> +++ b/arch/x86/kernel/i387.c
> @@ -93,6 +93,7 @@ void __cpuinit fpu_init(void)
>  {
>  	unsigned long cr0;
>  	unsigned long cr4_mask = 0;
> +	static int once = 1;
>  
>  	if (cpu_has_fxsr)
>  		cr4_mask |= X86_CR4_OSFXSR;
> @@ -107,8 +108,14 @@ void __cpuinit fpu_init(void)
>  		cr0 |= X86_CR0_EM;
>  	write_cr0(cr0);
>  
> -	if (!smp_processor_id())
> +	/*
> +	 * init_thread_xstate is only called on BSP once to avoid to override

"to avoid overriding xstate_size"

> +	 * xstate_size.
> +	 */
> +	if (!smp_processor_id() && once) {
> +		once = 0;
>  		init_thread_xstate();
> +	}
>  
>  	mxcsr_feature_mask_init();
>  	/* clean state in init */
> -- 
> 1.6.0.3
> 
> --
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/

Re: [PATCH v3 5/7] Documentations/cpu-hotplug.tx, kernel-parameters.txt: Add x86 CPU0 online/offline feature

[Konrad Rzeszutek Wilk]

On Wed, Nov 09, 2011 at 04:34:15PM -0800, Fenghua Yu wrote:
> From: Fenghua Yu <fenghua.yu@intel.com>
> 
> Add x86 CPU0 online/offline feature in the two documentations.
> 
> By default BSP is not pluggable. Kernel parameter bsp_hotplug enables BSP
> online/offline feature.
> 
> The documentations point out two BSP dependencies. First, resume from hibernate
> or suspend always starts from BSP. So hibernate and suspend are prevented if BSP
> is offline. Another dependency is PIC interrupts always go to BSP.
> 
> It's said that some machines may depend on CPU0 to poweroff/reboot. But I
> haven't seen such dependency on a few tested machines.
> 
> Please let me know if you see any CPU0 dependencies on your machine.
> 
> Signed-off-by: Fenghua Yu <fenghua.yu@intel.com>
> ---
> ---
>  Documentation/cpu-hotplug.txt       |   19 +++++++++++++++++++
>  Documentation/kernel-parameters.txt |   13 +++++++++++++
>  2 files changed, 32 insertions(+), 0 deletions(-)
> 
> diff --git a/Documentation/cpu-hotplug.txt b/Documentation/cpu-hotplug.txt
> index a20bfd4..93af1a0 100644
> --- a/Documentation/cpu-hotplug.txt
> +++ b/Documentation/cpu-hotplug.txt
> @@ -207,6 +207,25 @@ by making it not-removable.
>  
>  In such cases you will also notice that the online file is missing under cpu0.
>  
> +Q: Is CPU0 removable on X86?
> +A: Yes. If given kernel option bsp_hotplug, CPU0 is removable.
> +
> +But some features depend on BSP. Known dependencies are:
> +1. Resume from hibernate/suspend depends on BSP. Hibernate/suspend will fail if
> +BSP is offline and you need to online BSP before hibernate/suspend can continue.
> +2. PIC interrupts also depend on BSP. BSP can't be removed if a PIC interrupt
> +is detected.
> +
> +It's said poweroff/reboot may depend on BSP on some machines although I haven't
> +seen any poweroff/reboot failure so far after BSP is offline on a few tested
> +machines.
> +
> +Please let me know if you know or see any other dependencies of BSP.

You should include your email so folks can contact you.

> +
> +If the dependencies are under your control, you can turn on bsp_hotplug.
> +
> +--Fenghua Yu
> +
>  Q: How do i find out if a particular CPU is not removable?
>  A: Depending on the implementation, some architectures may show this by the
>  absence of the "online" file. This is done if it can be determined ahead of
> diff --git a/Documentation/kernel-parameters.txt b/Documentation/kernel-parameters.txt
> index a0c5c5f..55582db 100644
> --- a/Documentation/kernel-parameters.txt
> +++ b/Documentation/kernel-parameters.txt
> @@ -1842,6 +1842,19 @@ bytes respectively. Such letter suffixes can also be entirely omitted.
>  
>  	nox2apic	[X86-64,APIC] Do not enable x2APIC mode.
>  
> +	bsp_hotplug	[X86] BSP (aka CPU0) is hotpluggable.
> +			Some features depend on BSP. Known dependencies are:
> +			1. Resume from suspend/hibernate depends on BSP.
> +			Suspend/hibernate will fail if BSP is offline and you
> +			need to online BSP before suspend/hibernate.
> +			2. PIC interrupts also depend on BSP. BSP can't be
> +			removed if a PIC interrupt is detected.
> +			It's said poweroff/reboot may depend on BSP on some
> +			machines although I haven't seen such issues so far
> +			after BSP is offline on a few tested machines.
> +			If the dependencies are under your control, you can
> +			turn on bsp_hotplug.
> +
>  	nptcg=		[IA-64] Override max number of concurrent global TLB
>  			purges which is reported from either PAL_VM_SUMMARY or
>  			SAL PALO.
> -- 
> 1.6.0.3
> 
> --
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/

RE: [PATCH v3 1/7] x86/topology.c: Support functions for BSP online/offline

[Yu, Fenghua]

> > +
> > +static int bsp_hotpluggable;
> 
> __read_mostly

Actually bsp_hotpluggable is read only once during boot time. So if changing it, I would change it to __initdata.

Thanks.

-Fenghua