[PATCH 0/2] Drivers: hv: vmbus: use cpu_hotplug_enable/disable for CPU offlining prevention

[PATCH 0/2] Drivers: hv: vmbus: use cpu_hotplug_enable/disable for CPU offlining prevention

[Vitaly Kuznetsov]

Export cpu_hotplug_enable/cpu_hotplug_disable functions from cpu.c and use
them instead of altering the smp_ops.cpu_disable in Hyper-V vmbus module.

Vitaly Kuznetsov (2):
  cpu-hotplug: export cpu_hotplug_enable/cpu_hotplug_disable
  Drivers: hv: vmbus: use cpu_hotplug_enable/disable

 drivers/hv/vmbus_drv.c | 38 ++++----------------------------------
 kernel/cpu.c           |  3 ++-
 2 files changed, 6 insertions(+), 35 deletions(-)

-- 
1.9.3

[PATCH 1/2] cpu-hotplug: export cpu_hotplug_enable/cpu_hotplug_disable

[Vitaly Kuznetsov]

Loaded Hyper-V module will use these functions to disable CPU hotplug
under certain circumstances.

Signed-off-by: Vitaly Kuznetsov <vkuznets@redhat.com>
---
 kernel/cpu.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/kernel/cpu.c b/kernel/cpu.c
index 94bbe46..dc005e7 100644
--- a/kernel/cpu.c
+++ b/kernel/cpu.c
@@ -193,6 +193,7 @@ void cpu_hotplug_disable(void)
 	cpu_hotplug_disabled = 1;
 	cpu_maps_update_done();
 }
+EXPORT_SYMBOL_GPL(cpu_hotplug_disable);
 
 void cpu_hotplug_enable(void)
 {
@@ -200,7 +201,7 @@ void cpu_hotplug_enable(void)
 	cpu_hotplug_disabled = 0;
 	cpu_maps_update_done();
 }
-
+EXPORT_SYMBOL_GPL(cpu_hotplug_enable);
 #endif	/* CONFIG_HOTPLUG_CPU */
 
 /* Need to know about CPUs going up/down? */
-- 
1.9.3

Re: [PATCH 1/2] cpu-hotplug: export cpu_hotplug_enable/cpu_hotplug_disable

[Radim Krčmář]

2015-05-19 14:15+0200, Vitaly Kuznetsov:
> Loaded Hyper-V module will use these functions to disable CPU hotplug
> under certain circumstances.
> 
> Signed-off-by: Vitaly Kuznetsov <vkuznets@redhat.com>
> ---
>  kernel/cpu.c | 3 ++-
>  1 file changed, 2 insertions(+), 1 deletion(-)
> 
> diff --git a/kernel/cpu.c b/kernel/cpu.c
> index 94bbe46..dc005e7 100644
> --- a/kernel/cpu.c
> +++ b/kernel/cpu.c
> @@ -193,6 +193,7 @@ void cpu_hotplug_disable(void)
>  	cpu_hotplug_disabled = 1;

(I think it would be safer to make this into a counter, at least, so
  cpu_hotplug_disable() |
                        | cpu_hotplug_disable()
  cpu_hotplug_enable()  |
 wouldn't end up with enabled hotplug, when this switch is intended for
 hotplug-and-die situations.)

>  	cpu_maps_update_done();
>  }
> +EXPORT_SYMBOL_GPL(cpu_hotplug_disable);
>  
>  void cpu_hotplug_enable(void)
>  {
> @@ -200,7 +201,7 @@ void cpu_hotplug_enable(void)
>  	cpu_hotplug_disabled = 0;
>  	cpu_maps_update_done();
>  }
> -
> +EXPORT_SYMBOL_GPL(cpu_hotplug_enable);
>  #endif	/* CONFIG_HOTPLUG_CPU */
>  
>  /* Need to know about CPUs going up/down? */
> -- 
> 1.9.3
> 

Re: [PATCH 1/2] cpu-hotplug: export cpu_hotplug_enable/cpu_hotplug_disable

[Peter Zijlstra]

On Tue, May 19, 2015 at 02:15:41PM +0200, Vitaly Kuznetsov wrote:
> Loaded Hyper-V module will use these functions to disable CPU hotplug
> under certain circumstances.

What's wrong with get_online_cpus() ?

Re: [PATCH 1/2] cpu-hotplug: export cpu_hotplug_enable/cpu_hotplug_disable

[Vitaly Kuznetsov]

Peter Zijlstra <peterz@infradead.org> writes:

> On Tue, May 19, 2015 at 02:15:41PM +0200, Vitaly Kuznetsov wrote:
>> Loaded Hyper-V module will use these functions to disable CPU hotplug
>> under certain circumstances.
>
> What's wrong with get_online_cpus() ?

We need to disable cpu offlining permanently (till the module is
unloaded but this is highly unlikely). If we do get_online_cpus()
instead we get

# echo 0 > /sys/devices/system/cpu/cpu1/online 
<nothing happens, pemanent hang followed by the following:>

[  240.247109] INFO: task bash:972 blocked for more than 120 seconds.
[  240.248937]       Not tainted 4.1.0-rc3_bug1167380_test_rkr+ #602
[  240.250515] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
disables this message.
[  240.252379] bash            D ffff880021b83c18     0   972    939
0x00000004
[  240.255068]  ffff880021b83c18 0000000000000001 0000000000000046
ffff880021b83be8
[  240.258140]  ffffffff81fdcdb0 ffff880037dbcfc0 ffff880037203bd0
ffffffff81fdcdb8
[  240.261275]  ffff880021b84000 0000000000000001 0000000000000000
0000000000000001
[  240.264402] Call Trace:
[  240.265423]  [<ffffffff81aecf77>] schedule+0x37/0x90
[  240.266748]  [<ffffffff81081872>] cpu_hotplug_begin+0xd2/0xe0
[  240.268210]  [<ffffffff810817a5>] ? cpu_hotplug_begin+0x5/0xe0
[  240.269675]  [<ffffffff810cb860>] ? prepare_to_wait_event+0xf0/0xf0
[  240.271285]  [<ffffffff81ad561a>] _cpu_down+0x8a/0x300
[  240.275294]  [<ffffffff81ad58c6>] cpu_down+0x36/0x50
[  240.276680]  [<ffffffff8160f954>] cpu_subsys_offline+0x14/0x20
[  240.278145]  [<ffffffff8160a505>] device_offline+0xa5/0xd0
[  240.279618]  [<ffffffff8160a620>] online_store+0x50/0xa0
[  240.280991]  [<ffffffff81607678>] dev_attr_store+0x18/0x30
...

And if we do the cpu_hotplug_disable() call we get:
# echo 0 > /sys/devices/system/cpu/cpu1/online 
-bash: echo: write error: Device or resource busy

This behavior is preferable.

-- 
  Vitaly

[PATCH 2/2] Drivers: hv: vmbus: use cpu_hotplug_enable/disable

[Vitaly Kuznetsov]

Commit e513229b4c38 ("Drivers: hv: vmbus: prevent cpu offlining on newer
hypervisors") was alteringd smp_ops.cpu_disable to prevent CPU offlining.
We can bo better by using cpu_hotplug_enable/disable functions instead of
such hard-coding.

Reported-by: Radim Krčmář <rkrcmar@redhat.com>
Signed-off-by: Vitaly Kuznetsov <vkuznets@redhat.com>
---
 drivers/hv/vmbus_drv.c | 38 ++++----------------------------------
 1 file changed, 4 insertions(+), 34 deletions(-)

diff --git a/drivers/hv/vmbus_drv.c b/drivers/hv/vmbus_drv.c
index c85235e..e916ba2 100644
--- a/drivers/hv/vmbus_drv.c
+++ b/drivers/hv/vmbus_drv.c
@@ -763,38 +763,6 @@ static void vmbus_isr(void)
 	}
 }
 
-#ifdef CONFIG_HOTPLUG_CPU
-static int hyperv_cpu_disable(void)
-{
-	return -ENOSYS;
-}
-
-static void hv_cpu_hotplug_quirk(bool vmbus_loaded)
-{
-	static void *previous_cpu_disable;
-
-	/*
-	 * Offlining a CPU when running on newer hypervisors (WS2012R2, Win8,
-	 * ...) is not supported at this moment as channel interrupts are
-	 * distributed across all of them.
-	 */
-
-	if ((vmbus_proto_version == VERSION_WS2008) ||
-	    (vmbus_proto_version == VERSION_WIN7))
-		return;
-
-	if (vmbus_loaded) {
-		previous_cpu_disable = smp_ops.cpu_disable;
-		smp_ops.cpu_disable = hyperv_cpu_disable;
-		pr_notice("CPU offlining is not supported by hypervisor\n");
-	} else if (previous_cpu_disable)
-		smp_ops.cpu_disable = previous_cpu_disable;
-}
-#else
-static void hv_cpu_hotplug_quirk(bool vmbus_loaded)
-{
-}
-#endif
 
 /*
  * vmbus_bus_init -Main vmbus driver initialization routine.
@@ -836,7 +804,8 @@ static int vmbus_bus_init(int irq)
 	if (ret)
 		goto err_alloc;
 
-	hv_cpu_hotplug_quirk(true);
+	if (vmbus_proto_version > VERSION_WIN7)
+		cpu_hotplug_disable();
 
 	/*
 	 * Only register if the crash MSRs are available
@@ -1103,7 +1072,8 @@ static void __exit vmbus_exit(void)
 	for_each_online_cpu(cpu)
 		smp_call_function_single(cpu, hv_synic_cleanup, NULL, 1);
 	acpi_bus_unregister_driver(&vmbus_acpi_driver);
-	hv_cpu_hotplug_quirk(false);
+	if (vmbus_proto_version > VERSION_WIN7)
+		cpu_hotplug_enable();
 	vmbus_disconnect();
 }
 
-- 
1.9.3

RE: [PATCH 0/2] Drivers: hv: vmbus: use cpu_hotplug_enable/disable for CPU offlining prevention

[KY Srinivasan]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain; charset="utf-8", Size: 1163 bytes --]



> -----Original Message-----
> From: Vitaly Kuznetsov [mailto:vkuznets@redhat.com]
> Sent: Tuesday, May 19, 2015 5:16 AM
> To: devel@linuxdriverproject.org
> Cc: KY Srinivasan; Haiyang Zhang; linux-kernel@vger.kernel.org; Dexuan Cui;
> Ingo Molnar; Paul E. McKenney; Rafael J. Wysocki; Peter Zijlstra; Thomas
> Gleixner; Radim Krčmář
> Subject: [PATCH 0/2] Drivers: hv: vmbus: use cpu_hotplug_enable/disable
> for CPU offlining prevention
> 
> Export cpu_hotplug_enable/cpu_hotplug_disable functions from cpu.c and
> use
> them instead of altering the smp_ops.cpu_disable in Hyper-V vmbus
> module.
> 
> Vitaly Kuznetsov (2):
>   cpu-hotplug: export cpu_hotplug_enable/cpu_hotplug_disable
>   Drivers: hv: vmbus: use cpu_hotplug_enable/disable
> 
>  drivers/hv/vmbus_drv.c | 38 ++++----------------------------------
>  kernel/cpu.c           |  3 ++-
>  2 files changed, 6 insertions(+), 35 deletions(-)

Greg, 

Can you take both these patches through your tree.

Regards,

K. Y
> 
> --
> 1.9.3

ÿôèº{.nÇ+‰·Ÿ®‰­†+%ŠËÿ±éݶ\x17¥Šwÿº{.nÇ+‰·¥Š{±þG«éÿŠ{ayº\x1dʇڙë,j\a­¢f£¢·hšïêÿ‘êçz_è®\x03(­éšŽŠÝ¢j"ú\x1a¶^[m§ÿÿ¾\a«þG«éÿ¢¸?™¨è­Ú&£ø§~á¶iO•æ¬z·švØ^\x14\x04\x1a¶^[m§ÿÿÃ\fÿ¶ìÿ¢¸?–I¥