[PATCH v2] x86, hotplug: fix llc shared map unreleased during cpu hotplug

public inbox for linux-kernel@vger.kernel.org
 help / color / mirror / Atom feed

* [PATCH v2] x86, hotplug: fix llc shared map unreleased during cpu hotplug
@ 2014-07-22  8:04 Wanpeng Li
  2014-07-22  8:46 ` Chen, Gong
  2014-07-23  8:56 ` Yasuaki Ishimatsu
  0 siblings, 2 replies; 9+ messages in thread
From: Wanpeng Li @ 2014-07-22  8:04 UTC (permalink / raw)
  To: hpa
  Cc: Ingo Molnar, Peter Zijlstra, x86, Borislav Petkov,
	Yasuaki Ishimatsu, David Rientjes, Prarit Bhargava,
	Steven Rostedt, Jan Kiszka, Toshi Kani, linux-kernel, Wanpeng Li

[  220.262093] BUG: unable to handle kernel NULL pointer dereference at 0000000000000004
[  220.262104] IP: [<ffffffff810e7ac9>] find_busiest_group+0x2b9/0xa30
[  220.262111] PGD 5a9d5067 PUD 13067 PMD 0
[  220.262117] Oops: 0000 [#3] SMP
[...]
[  220.262245] Call Trace:
[  220.262252]  [<ffffffff810e8396>] load_balance+0x156/0x980
[  220.262259]  [<ffffffff816eeffe>] ? _raw_spin_unlock_irqrestore+0x2e/0xa0
[  220.262266]  [<ffffffff810e9aa3>] idle_balance+0xe3/0x150
[  220.262270]  [<ffffffff816ec4e7>] __schedule+0x797/0x8d0
[  220.262277]  [<ffffffff816ec934>] schedule+0x24/0x70
[  220.262283]  [<ffffffff816e9cd9>] schedule_timeout+0x119/0x1f0
[  220.262294]  [<ffffffff810bb6e0>] ? lock_timer_base+0x70/0x70
[  220.262301]  [<ffffffff816e9dc9>] schedule_timeout_uninterruptible+0x19/0x20
[  220.262308]  [<ffffffff810bd3e8>] msleep+0x18/0x20
[  220.262317]  [<ffffffff813aa11a>] lock_device_hotplug_sysfs+0x2a/0x50
[  220.262323]  [<ffffffff813aa16e>] online_store+0x2e/0x80
[  220.262358]  [<ffffffff813a873b>] dev_attr_store+0x1b/0x20
[  220.262366]  [<ffffffff812292fd>] sysfs_write_file+0xdd/0x160
[  220.262377]  [<ffffffff811b7e78>] vfs_write+0xc8/0x170
[  220.262384]  [<ffffffff811b83ca>] SyS_write+0x5a/0xa0
[  220.262388]  [<ffffffff816f76b9>] system_call_fastpath+0x16/0x1b

Last level cache shared map is built during cpu up and build sched domain 
routine takes advantage of it to setup sched domain cpu topology, however, 
llc shared map is unreleased during cpu disable which lead to invalid sched 
domain cpu topology. This patch fix it by release llc shared map correctly
during cpu disable.

Signed-off-by: Wanpeng Li <wanpeng.li@linux.intel.com>
---
v1 -> v2:
 * fix subject line

 arch/x86/kernel/smpboot.c | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/arch/x86/kernel/smpboot.c b/arch/x86/kernel/smpboot.c
index 5492798..0134ec7 100644
--- a/arch/x86/kernel/smpboot.c
+++ b/arch/x86/kernel/smpboot.c
@@ -1292,6 +1292,9 @@ static void remove_siblinginfo(int cpu)
 
 	for_each_cpu(sibling, cpu_sibling_mask(cpu))
 		cpumask_clear_cpu(cpu, cpu_sibling_mask(sibling));
+	for_each_cpu(sibling, cpu_llc_shared_mask(cpu))
+		cpumask_clear_cpu(cpu, cpu_llc_shared_mask(sibling));
+	cpumask_clear(cpu_llc_shared_mask(cpu));
 	cpumask_clear(cpu_sibling_mask(cpu));
 	cpumask_clear(cpu_core_mask(cpu));
 	c->phys_proc_id = 0;
-- 
1.9.1


^ permalink raw reply related	[flat|nested] 9+ messages in thread

* Re: [PATCH v2] x86, hotplug: fix llc shared map unreleased during cpu hotplug
  2014-07-22  8:04 [PATCH v2] x86, hotplug: fix llc shared map unreleased during cpu hotplug Wanpeng Li
@ 2014-07-22  8:46 ` Chen, Gong
  2014-07-22  9:30   ` Wanpeng Li
  2014-07-23  8:56 ` Yasuaki Ishimatsu
  1 sibling, 1 reply; 9+ messages in thread
From: Chen, Gong @ 2014-07-22  8:46 UTC (permalink / raw)
  To: Wanpeng Li
  Cc: hpa, Ingo Molnar, Peter Zijlstra, x86, Borislav Petkov,
	Yasuaki Ishimatsu, David Rientjes, Prarit Bhargava,
	Steven Rostedt, Jan Kiszka, Toshi Kani, linux-kernel

[-- Attachment #1: Type: text/plain, Size: 199 bytes --]

On Tue, Jul 22, 2014 at 04:04:52PM +0800, Wanpeng Li wrote:
> Subject: [PATCH v2] x86, hotplug: fix llc shared map unreleased during cpu
>  hotplug

See this link:
https://lkml.org/lkml/2014/7/17/78

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

^ permalink raw reply	[flat|nested] 9+ messages in thread

* Re: [PATCH v2] x86, hotplug: fix llc shared map unreleased during cpu hotplug
  2014-07-22  8:46 ` Chen, Gong
@ 2014-07-22  9:30   ` Wanpeng Li
  2014-07-22  9:44     ` Borislav Petkov
  0 siblings, 1 reply; 9+ messages in thread
From: Wanpeng Li @ 2014-07-22  9:30 UTC (permalink / raw)
  To: Chen, Gong
  Cc: hpa, Ingo Molnar, Peter Zijlstra, x86, Borislav Petkov,
	Yasuaki Ishimatsu, David Rientjes, Prarit Bhargava,
	Steven Rostedt, Jan Kiszka, Toshi Kani, linux-kernel, Wanpeng Li

On Tue, Jul 22, 2014 at 04:46:41AM -0400, Chen, Gong wrote:
>On Tue, Jul 22, 2014 at 04:04:52PM +0800, Wanpeng Li wrote:
>> Subject: [PATCH v2] x86, hotplug: fix llc shared map unreleased during cpu
>>  hotplug
>
>See this link:
>https://lkml.org/lkml/2014/7/17/78

Interesting, thanks for your pointing out.

Regards,
Wanpeng Li 


^ permalink raw reply	[flat|nested] 9+ messages in thread

* Re: [PATCH v2] x86, hotplug: fix llc shared map unreleased during cpu hotplug
  2014-07-22  9:30   ` Wanpeng Li
@ 2014-07-22  9:44     ` Borislav Petkov
  0 siblings, 0 replies; 9+ messages in thread
From: Borislav Petkov @ 2014-07-22  9:44 UTC (permalink / raw)
  To: Wanpeng Li
  Cc: Chen, Gong, hpa, Ingo Molnar, Peter Zijlstra, x86,
	Yasuaki Ishimatsu, David Rientjes, Prarit Bhargava,
	Steven Rostedt, Jan Kiszka, Toshi Kani, linux-kernel

On Tue, Jul 22, 2014 at 05:30:23PM +0800, Wanpeng Li wrote:
> On Tue, Jul 22, 2014 at 04:46:41AM -0400, Chen, Gong wrote:
> >On Tue, Jul 22, 2014 at 04:04:52PM +0800, Wanpeng Li wrote:
> >> Subject: [PATCH v2] x86, hotplug: fix llc shared map unreleased during cpu
> >>  hotplug
> >
> >See this link:
> >https://lkml.org/lkml/2014/7/17/78
> 
> Interesting, thanks for your pointing out.

Well, we might end up doing both.

Yasuaki's patch makes the core numbering stable which makes a lot of
sense in and of itself.

However, if some code somewhere relies implicitly on the fact that a set
bit in LLC mask means that core is online and goes and does stuff on it,
Wanpeng's reported splat will happen.

So probably we will have to have *stable* core numbers *and* clear the
LLC mask in order to be on the safe side of things.

Thanks.

-- 
Regards/Gruss,
    Boris.

Sent from a fat crate under my desk. Formatting is fine.
--

^ permalink raw reply	[flat|nested] 9+ messages in thread

* Re: [PATCH v2] x86, hotplug: fix llc shared map unreleased during cpu hotplug
  2014-07-22  8:04 [PATCH v2] x86, hotplug: fix llc shared map unreleased during cpu hotplug Wanpeng Li
  2014-07-22  8:46 ` Chen, Gong
@ 2014-07-23  8:56 ` Yasuaki Ishimatsu
  2014-07-29  7:06   ` Wanpeng Li
  1 sibling, 1 reply; 9+ messages in thread
From: Yasuaki Ishimatsu @ 2014-07-23  8:56 UTC (permalink / raw)
  To: Wanpeng Li
  Cc: hpa, Ingo Molnar, Peter Zijlstra, x86, Borislav Petkov,
	David Rientjes, Prarit Bhargava, Steven Rostedt, Jan Kiszka,
	Toshi Kani, linux-kernel

(2014/07/22 17:04), Wanpeng Li wrote:
> [  220.262093] BUG: unable to handle kernel NULL pointer dereference at 0000000000000004
> [  220.262104] IP: [<ffffffff810e7ac9>] find_busiest_group+0x2b9/0xa30
> [  220.262111] PGD 5a9d5067 PUD 13067 PMD 0
> [  220.262117] Oops: 0000 [#3] SMP
> [...]
> [  220.262245] Call Trace:
> [  220.262252]  [<ffffffff810e8396>] load_balance+0x156/0x980
> [  220.262259]  [<ffffffff816eeffe>] ? _raw_spin_unlock_irqrestore+0x2e/0xa0
> [  220.262266]  [<ffffffff810e9aa3>] idle_balance+0xe3/0x150
> [  220.262270]  [<ffffffff816ec4e7>] __schedule+0x797/0x8d0
> [  220.262277]  [<ffffffff816ec934>] schedule+0x24/0x70
> [  220.262283]  [<ffffffff816e9cd9>] schedule_timeout+0x119/0x1f0
> [  220.262294]  [<ffffffff810bb6e0>] ? lock_timer_base+0x70/0x70
> [  220.262301]  [<ffffffff816e9dc9>] schedule_timeout_uninterruptible+0x19/0x20
> [  220.262308]  [<ffffffff810bd3e8>] msleep+0x18/0x20
> [  220.262317]  [<ffffffff813aa11a>] lock_device_hotplug_sysfs+0x2a/0x50
> [  220.262323]  [<ffffffff813aa16e>] online_store+0x2e/0x80
> [  220.262358]  [<ffffffff813a873b>] dev_attr_store+0x1b/0x20
> [  220.262366]  [<ffffffff812292fd>] sysfs_write_file+0xdd/0x160
> [  220.262377]  [<ffffffff811b7e78>] vfs_write+0xc8/0x170
> [  220.262384]  [<ffffffff811b83ca>] SyS_write+0x5a/0xa0
> [  220.262388]  [<ffffffff816f76b9>] system_call_fastpath+0x16/0x1b
> 
> Last level cache shared map is built during cpu up and build sched domain
> routine takes advantage of it to setup sched domain cpu topology, however,
> llc shared map is unreleased during cpu disable which lead to invalid sched
> domain cpu topology. This patch fix it by release llc shared map correctly
> during cpu disable.
> 

I posted a latest patch as follows:
https://lkml.org/lkml/2014/7/22/1018

Could you confirm the patch fixes your issue?

Thanks,
Yasuaki Ishimatsu

> Signed-off-by: Wanpeng Li <wanpeng.li@linux.intel.com>
> ---
> v1 -> v2:
>   * fix subject line
> 
>   arch/x86/kernel/smpboot.c | 3 +++
>   1 file changed, 3 insertions(+)
> 
> diff --git a/arch/x86/kernel/smpboot.c b/arch/x86/kernel/smpboot.c
> index 5492798..0134ec7 100644
> --- a/arch/x86/kernel/smpboot.c
> +++ b/arch/x86/kernel/smpboot.c
> @@ -1292,6 +1292,9 @@ static void remove_siblinginfo(int cpu)
>   
>   	for_each_cpu(sibling, cpu_sibling_mask(cpu))
>   		cpumask_clear_cpu(cpu, cpu_sibling_mask(sibling));
> +	for_each_cpu(sibling, cpu_llc_shared_mask(cpu))
> +		cpumask_clear_cpu(cpu, cpu_llc_shared_mask(sibling));
> +	cpumask_clear(cpu_llc_shared_mask(cpu));
>   	cpumask_clear(cpu_sibling_mask(cpu));
>   	cpumask_clear(cpu_core_mask(cpu));
>   	c->phys_proc_id = 0;
> 



^ permalink raw reply	[flat|nested] 9+ messages in thread

* Re: [PATCH v2] x86, hotplug: fix llc shared map unreleased during cpu hotplug
  2014-07-23  8:56 ` Yasuaki Ishimatsu
@ 2014-07-29  7:06   ` Wanpeng Li
  2014-07-29  7:31     ` Yasuaki Ishimatsu
  0 siblings, 1 reply; 9+ messages in thread
From: Wanpeng Li @ 2014-07-29  7:06 UTC (permalink / raw)
  To: Yasuaki Ishimatsu
  Cc: hpa, Ingo Molnar, Peter Zijlstra, x86, Borislav Petkov,
	David Rientjes, Prarit Bhargava, Steven Rostedt, Jan Kiszka,
	Toshi Kani, linux-kernel, Konrad Rzeszutek Wilk, Zhang, Yang Z,
	Yong Wang

[-- Attachment #1: Type: text/plain, Size: 2923 bytes --]

Hi Yasuaki,
On Wed, Jul 23, 2014 at 05:56:07PM +0900, Yasuaki Ishimatsu wrote:
>(2014/07/22 17:04), Wanpeng Li wrote:
>> [  220.262093] BUG: unable to handle kernel NULL pointer dereference at 0000000000000004
>> [  220.262104] IP: [<ffffffff810e7ac9>] find_busiest_group+0x2b9/0xa30
>> [  220.262111] PGD 5a9d5067 PUD 13067 PMD 0
>> [  220.262117] Oops: 0000 [#3] SMP
>> [...]
>> [  220.262245] Call Trace:
>> [  220.262252]  [<ffffffff810e8396>] load_balance+0x156/0x980
>> [  220.262259]  [<ffffffff816eeffe>] ? _raw_spin_unlock_irqrestore+0x2e/0xa0
>> [  220.262266]  [<ffffffff810e9aa3>] idle_balance+0xe3/0x150
>> [  220.262270]  [<ffffffff816ec4e7>] __schedule+0x797/0x8d0
>> [  220.262277]  [<ffffffff816ec934>] schedule+0x24/0x70
>> [  220.262283]  [<ffffffff816e9cd9>] schedule_timeout+0x119/0x1f0
>> [  220.262294]  [<ffffffff810bb6e0>] ? lock_timer_base+0x70/0x70
>> [  220.262301]  [<ffffffff816e9dc9>] schedule_timeout_uninterruptible+0x19/0x20
>> [  220.262308]  [<ffffffff810bd3e8>] msleep+0x18/0x20
>> [  220.262317]  [<ffffffff813aa11a>] lock_device_hotplug_sysfs+0x2a/0x50
>> [  220.262323]  [<ffffffff813aa16e>] online_store+0x2e/0x80
>> [  220.262358]  [<ffffffff813a873b>] dev_attr_store+0x1b/0x20
>> [  220.262366]  [<ffffffff812292fd>] sysfs_write_file+0xdd/0x160
>> [  220.262377]  [<ffffffff811b7e78>] vfs_write+0xc8/0x170
>> [  220.262384]  [<ffffffff811b83ca>] SyS_write+0x5a/0xa0
>> [  220.262388]  [<ffffffff816f76b9>] system_call_fastpath+0x16/0x1b
>> 
>> Last level cache shared map is built during cpu up and build sched domain
>> routine takes advantage of it to setup sched domain cpu topology, however,
>> llc shared map is unreleased during cpu disable which lead to invalid sched
>> domain cpu topology. This patch fix it by release llc shared map correctly
>> during cpu disable.
>> 
>
>I posted a latest patch as follows:
>https://lkml.org/lkml/2014/7/22/1018
>
>Could you confirm the patch fixes your issue?

Sorry for the late, there is still call trace w/ your patch applied. The
call trace is in attachment.

Regards,
Wanpeng Li 

>
>Thanks,
>Yasuaki Ishimatsu
>
>> Signed-off-by: Wanpeng Li <wanpeng.li@linux.intel.com>
>> ---
>> v1 -> v2:
>>   * fix subject line
>> 
>>   arch/x86/kernel/smpboot.c | 3 +++
>>   1 file changed, 3 insertions(+)
>> 
>> diff --git a/arch/x86/kernel/smpboot.c b/arch/x86/kernel/smpboot.c
>> index 5492798..0134ec7 100644
>> --- a/arch/x86/kernel/smpboot.c
>> +++ b/arch/x86/kernel/smpboot.c
>> @@ -1292,6 +1292,9 @@ static void remove_siblinginfo(int cpu)
>>   
>>   	for_each_cpu(sibling, cpu_sibling_mask(cpu))
>>   		cpumask_clear_cpu(cpu, cpu_sibling_mask(sibling));
>> +	for_each_cpu(sibling, cpu_llc_shared_mask(cpu))
>> +		cpumask_clear_cpu(cpu, cpu_llc_shared_mask(sibling));
>> +	cpumask_clear(cpu_llc_shared_mask(cpu));
>>   	cpumask_clear(cpu_sibling_mask(cpu));
>>   	cpumask_clear(cpu_core_mask(cpu));
>>   	c->phys_proc_id = 0;
>> 
>

[-- Attachment #2: call trace.txt --]
[-- Type: text/plain, Size: 13979 bytes --]

when run "xl vcpu-set 0 2", the dom0 only report "broke affinity ..."
when run "xl vcpu-set 0 26", the call trace happens.

the dom0 call trace log as following:

[  295.464489] Broke affinity for irq 298
[  295.756205] Broke affinity for irq 299
[  295.767177] Broke affinity for irq 301
[  295.779177] Broke affinity for irq 303
[  366.283682] installing Xen timer for CPU 2
[  366.283749] cpu 2 spinlock event irq 103
[  366.310290] installing Xen timer for CPU 14
[  366.310347] cpu 14 spinlock event irq 110
[  366.312432] divide error: 0000 [#1] SMP
[  366.312449] Modules linked in: nfsv3 nfs_acl auth_rpcgss oid_registry nfsv4
d
[  366.312583] CPU: 14 PID: 63 Comm: ksoftirqd/14 Not tainted 3.15.6 #2
[  366.312598] Hardware name: Intel Corporation S2600WTT/S2600WTT, BIOS
GRNDSDP4
[  366.312623] task: ffff88017c8d2c10 ti: ffff88017c8f0000 task.ti:
ffff88017c80
[  366.312647] RIP: e030:[<ffffffff810ea5f9>]  [<ffffffff810ea5f9>]
find_busies0
[  366.312681] RSP: e02b:ffff88017c8f3ac8  EFLAGS: 00010046
[  366.312694] RAX: 0000000000000000 RBX: ffff88017c8f3bc8 RCX:
0000000000000000
[  366.312708] RDX: 0000000000000000 RSI: 0000000000000000 RDI:
0000000000000000
[  366.312724] RBP: ffff88017c8f3c38 R08: ffff880003fb3d00 R09:
0000000000000040
[  366.312742] R10: 0000000000000000 R11: 0000000000000000 R12:
0000000000013e00
[  366.312757] R13: ffff88017c8f3cb8 R14: ffff880003fb3ce0 R15:
0000000000000000
[  366.312783] FS:  0000000000000000(0000) GS:ffff880181bc0000(0000)
knlGS:00000
[  366.312803] CS:  e033 DS: 0000 ES: 0000 CR0: 0000000080050033
[  366.312817] CR2: 00007fad200d5000 CR3: 0000000001c14000 CR4:
0000000000042660
[  366.312836] Stack:
[  366.312843]  0000000000000000 ffff88017c8f3b18 0000000000002e7b
0000000000000
[  366.312868]  ffff880003fb3ce0 0000000000013df8 0000000000000200
0000000000010
[  366.312890]  0000000000000000 ffff880003fb3cf8 0000000000000000
0000000000000
[  366.312911] Call Trace:
[  366.312932]  [<ffffffff810eae37>] load_balance+0x177/0x9d0
[  366.312954]  [<ffffffff810df56b>] ? update_rq_clock+0x2b/0x50
[  366.312976]  [<ffffffff81058ea0>] ? xen_clocksource_read+0x20/0x30
[  366.312997]  [<ffffffff810edb8d>] pick_next_task_fair+0x1ed/0x430
[  366.313019]  [<ffffffff816ff0d3>] __schedule+0x113/0x870
[  366.313039]  [<ffffffff816ff9b4>] ? schedule+0x24/0x70
[  366.313059]  [<ffffffff816ff9b4>] schedule+0x24/0x70
[  366.313095]  [<ffffffff810dcd7c>] smpboot_thread_fn+0xbc/0x190
[  366.313112]  [<ffffffff810dccc0>] ? smpboot_create_threads+0x80/0x80
[  366.313135]  [<ffffffff810d565e>] kthread+0xce/0xf0
[  366.313155]  [<ffffffff810d5590>] ? kthread_freezable_should_stop+0x70/0x70
[  366.313174]  [<ffffffff8170c54c>] ret_from_fork+0x7c/0xb0
[  366.313190]  [<ffffffff810d5590>] ? kthread_freezable_should_stop+0x70/0x70
[  366.313204] Code: 0f 47 d1 eb 95 0f 1f 44 00 00 4d 89 ec 4d 89 f5 4c 8b b5 b
[  366.313372] RIP  [<ffffffff810ea5f9>] find_busiest_group+0x239/0x900
[  366.313391]  RSP <ffff88017c8f3ac8>
[  366.313406] ---[ end trace 42d3248df75182f3 ]---
[  366.313758] divide error: 0000 [#2] SMP
[  366.313776] Modules linked in: nfsv3 nfs_acl auth_rpcgss oid_registry nfsv4
d
[  366.313883] CPU: 14 PID: 63 Comm: ksoftirqd/14 Tainted: G      D      
3.15.2
[  366.313898] Hardware name: Intel Corporation S2600WTT/S2600WTT, BIOS
GRNDSDP4
[  366.313922] task: ffff88017c8d2c10 ti: ffff88017c8f0000 task.ti:
ffff88017c80
[  366.313940] RIP: e030:[<ffffffff810ea5f9>]  [<ffffffff810ea5f9>]
find_busies0
[  366.313966] RSP: e02b:ffff88017c8f3468  EFLAGS: 00010046
[  366.313979] RAX: 0000000000000000 RBX: ffff88017c8f3568 RCX:
0000000000000000
[  366.313993] RDX: 0000000000000000 RSI: 0000000000000000 RDI:
0000000000000000
[  366.314008] RBP: ffff88017c8f35d8 R08: ffff880003fb3d00 R09:
0000000000000040
[  366.314023] R10: 0000000000000000 R11: ffff880186148410 R12:
0000000000013e00
[  366.314042] R13: ffff88017c8f3658 R14: ffff880003fb3ce0 R15:
0000000000000000
[  366.314067] FS:  0000000000000000(0000) GS:ffff880181bc0000(0000)
knlGS:00000
[  366.314090] CS:  e033 DS: 0000 ES: 0000 CR0: 0000000080050033
[  366.314103] CR2: 00007fad200d5000 CR3: 0000000001c14000 CR4:
0000000000042660
[  366.314119] Stack:
[  366.314125]  ffff8801fc8f35c9 ffff88017c8f34b8 0000000000002e7b
00000000812ce
[  366.314150]  ffff880003fb3ce0 0000000000013df8 0000000000000200
0000000000010
[  366.314175]  000000006c106009 ffff880003fb3cf8 0000000000000000
0000000000000
[  366.314196] Call Trace:
[  366.314215]  [<ffffffff810eae37>] load_balance+0x177/0x9d0
[  366.314232]  [<ffffffff810df56b>] ? update_rq_clock+0x2b/0x50
[  366.314252]  [<ffffffff81058ea0>] ? xen_clocksource_read+0x20/0x30
[  366.314269]  [<ffffffff810edb8d>] pick_next_task_fair+0x1ed/0x430
[  366.314288]  [<ffffffff816ff0d3>] __schedule+0x113/0x870
[  366.314307]  [<ffffffff810b3a34>] ? release_task+0x304/0x480
[  366.314324]  [<ffffffff816ff9b4>] schedule+0x24/0x70
[  366.314340]  [<ffffffff810b42ac>] do_exit+0x6fc/0xac0
[  366.314356]  [<ffffffff81705008>] oops_end+0xa8/0x170
[  366.314371]  [<ffffffff810663b6>] die+0x56/0x90
[  366.314385]  [<ffffffff81704a23>] do_trap+0xc3/0x170
[  366.314402]  [<ffffffff8170812d>] ? __atomic_notifier_call_chain+0xd/0x10
[  366.314422]  [<ffffffff8106395b>] do_divide_error+0x9b/0xb0
[  366.314439]  [<ffffffff810ea5f9>] ? find_busiest_group+0x239/0x900
[  366.314456]  [<ffffffff8170dc0e>] divide_error+0x1e/0x30
[  366.314473]  [<ffffffff810ea5f9>] ? find_busiest_group+0x239/0x900
[  366.314491]  [<ffffffff810ea513>] ? find_busiest_group+0x153/0x900
[  366.314511]  [<ffffffff810eae37>] load_balance+0x177/0x9d0
[  366.314526]  [<ffffffff810df56b>] ? update_rq_clock+0x2b/0x50
[  366.314547]  [<ffffffff81058ea0>] ? xen_clocksource_read+0x20/0x30
[  366.314563]  [<ffffffff810edb8d>] pick_next_task_fair+0x1ed/0x430
[  366.314581]  [<ffffffff816ff0d3>] __schedule+0x113/0x870
[  366.314597]  [<ffffffff816ff9b4>] ? schedule+0x24/0x70
[  366.314613]  [<ffffffff816ff9b4>] schedule+0x24/0x70
[  366.314628]  [<ffffffff810dcd7c>] smpboot_thread_fn+0xbc/0x190
[  366.314650]  [<ffffffff810dccc0>] ? smpboot_create_threads+0x80/0x80
[  366.314668]  [<ffffffff810d565e>] kthread+0xce/0xf0
[  366.314684]  [<ffffffff810d5590>] ? kthread_freezable_should_stop+0x70/0x70
[  366.314701]  [<ffffffff8170c54c>] ret_from_fork+0x7c/0xb0
[  366.314717]  [<ffffffff810d5590>] ? kthread_freezable_should_stop+0x70/0x70
[  366.314735] Code: 0f 47 d1 eb 95 0f 1f 44 00 00 4d 89 ec 4d 89 f5 4c 8b b5 b
[  366.314891] RIP  [<ffffffff810ea5f9>] find_busiest_group+0x239/0x900
[  366.314909]  RSP <ffff88017c8f3468>
[  366.314927] ---[ end trace 42d3248df75182f4 ]---
[  366.314932] BUG: unable to handle kernel NULL pointer dereference at
0000000c
[  366.314938] IP: [<ffffffff810e86e7>] select_task_rq_fair+0x337/0x8c0
[  366.314942] PGD 0
[  366.314943] Oops: 0000 [#3] SMP
[  366.314960] Modules linked in: nfsv3 nfs_acl auth_rpcgss oid_registry nfsv4
d
[  366.314962] CPU: 1 PID: 8225 Comm: udevd Tainted: G      D       3.15.6 #2
[  366.314965] Hardware name: Intel Corporation S2600WTT/S2600WTT, BIOS
GRNDSDP4
[  366.314966] task: ffff8801771634e0 ti: ffff880002598000 task.ti:
ffff88000250
[  366.314972] RIP: e030:[<ffffffff810e86e7>]  [<ffffffff810e86e7>]
select_task0
[  366.314973] RSP: e02b:ffff88000259bd48  EFLAGS: 00010046
[  366.314974] RAX: 0000000000000000 RBX: 0000000000000000 RCX:
0000000000000019
[  366.314975] RDX: 0000000000000008 RSI: 0000000000000040 RDI:
0000000000000040
[  366.314979] RBP: ffff88000259be28 R08: ffff880003fb33f8 R09:
0000000000000000
[  366.314980] R10: 0000000000000000 R11: ffff88017cfe4338 R12:
0000000000000000
[  366.314981] R13: ffff880003fb33f8 R14: ffff880003fb33e0 R15:
0000000000000000
[  366.314988] FS:  00007fad200bb7a0(0000) GS:ffff880181a20000(0000)
knlGS:00000
[  366.314989] CS:  e033 DS: 0000 ES: 0000 CR0: 0000000080050033
[  366.314990] CR2: 000000000000000c CR3: 00000000030f2000 CR4:
0000000000042660
[  366.314991] Stack:
[  366.314993]  ffff88017c7de000 00000000ffffff9c ffff88000259be38
ffffffff811d5
[  366.314995]  0000000000013e00 0000000000013e00 ffff8801771637d8
000000000000d
[  366.314997]  ffff880003fb3420 0000000000001ade 0000000000013e00
0000000000018
[  366.314998] Call Trace:
[  366.315003]  [<ffffffff811d6a45>] ? do_filp_open+0x45/0xa0
[  366.315005]  [<ffffffff810dedd7>] sched_exec+0x47/0xc0
[  366.315009]  [<ffffffff811ccaca>] ? do_open_exec+0xaa/0xe0
[  366.315014]  [<ffffffff811cccee>] do_execve_common+0x1be/0x640
[  366.315019]  [<ffffffff811b5b77>] ? kmem_cache_alloc+0x37/0x120
[  366.315021]  [<ffffffff811cd202>] do_execve+0x32/0x40
[  366.315026]  [<ffffffff811cd23a>] SyS_execve+0x2a/0x40
[  366.315029]  [<ffffffff8170cba9>] stub_execve+0x69/0xa0
[  366.315055] Code: 48 8b 55 c0 4d 8b 36 4c 3b 72 10 74 43 48 89 45 b0 e9 6e f
[  366.315058] RIP  [<ffffffff810e86e7>] select_task_rq_fair+0x337/0x8c0
[  366.315058]  RSP <ffff88000259bd48>
[  366.315059] CR2: 000000000000000c
[  366.315060] ---[ end trace 42d3248df75182f5 ]---
[  366.315418] Fixing recursive fault but reboot is needed!
[  366.315538] BUG: unable to handle kernel NULL pointer dereference at
0000000c
[  366.315580] IP: [<ffffffff810e86e7>] select_task_rq_fair+0x337/0x8c0
[  366.315609] PGD 0
[  366.315616] Oops: 0000 [#4] SMP
[  366.315620] Modules linked in: nfsv3 nfs_acl auth_rpcgss oid_registry nfsv4
d
[  366.315660] CPU: 0 PID: 8220 Comm: udevd Tainted: G      D       3.15.6 #2
[  366.315666] Hardware name: Intel Corporation S2600WTT/S2600WTT, BIOS
GRNDSDP4
[  366.315673] task: ffff88017c680000 ti: ffff88007274c000 task.ti:
ffff88007270
[  366.315678] RIP: e030:[<ffffffff810e86e7>]  [<ffffffff810e86e7>]
select_task0
[  366.315687] RSP: e02b:ffff88007274fd48  EFLAGS: 00010046
[  366.315693] RAX: 0000000000000000 RBX: 0000000000000000 RCX:
0000000000000019
[  366.315699] RDX: 0000000000000008 RSI: 0000000000000040 RDI:
0000000000000040
[  366.315704] RBP: ffff88007274fe28 R08: ffff880003fb33f8 R09:
0000000000000000
[  366.315710] R10: 0000000000000000 R11: ffff88017cfe4338 R12:
0000000000000000
[  366.315715] R13: ffff880003fb33f8 R14: ffff880003fb33e0 R15:
0000000000000000
[  366.315724] FS:  00007fad200bb7a0(0000) GS:ffff880181a00000(0000)
knlGS:00000
[  366.315730] CS:  e033 DS: 0000 ES: 0000 CR0: 0000000080050033
[  366.315735] CR2: 000000000000000c CR3: 00000000725d7000 CR4:
0000000000042660
[  366.315740] Stack:
[  366.315743]  ffff88017c6e5000 00000000ffffff9c ffff88007274fe38
ffffffff811d5
[  366.315751]  0000000000013e00 0000000000013e00 ffff88017c6802f8
000000000000d
[  366.315759]  ffff880003fb3420 000000000000163e 0000000000013e00
0000000000018
[  366.315766] Call Trace:
[  366.315772]  [<ffffffff811d6a45>] ? do_filp_open+0x45/0xa0
[  366.315779]  [<ffffffff810dedd7>] sched_exec+0x47/0xc0
[  366.315787]  [<ffffffff811ccaca>] ? do_open_exec+0xaa/0xe0
[  366.315793]  [<ffffffff811cccee>] do_execve_common+0x1be/0x640
[  366.315801]  [<ffffffff811b5b77>] ? kmem_cache_alloc+0x37/0x120
[  366.315808]  [<ffffffff811cd202>] do_execve+0x32/0x40
[  366.315813]  [<ffffffff811cd23a>] SyS_execve+0x2a/0x40
[  366.315819]  [<ffffffff8170cba9>] stub_execve+0x69/0xa0
[  366.315824] Code: 48 8b 55 c0 4d 8b 36 4c 3b 72 10 74 43 48 89 45 b0 e9 6e f
[  366.315882] RIP  [<ffffffff810e86e7>] select_task_rq_fair+0x337/0x8c0
[  366.315890]  RSP <ffff88007274fd48>
[  366.315894] CR2: 000000000000000c
[  366.315899] ---[ end trace 42d3248df75182f6 ]---
[  366.317854] divide error: 0000 [#5] SMP
[  366.317869] Modules linked in: nfsv3 nfs_acl auth_rpcgss oid_registry nfsv4
d
[  366.317967] CPU: 14 PID: 6370 Comm: rsyslogd Tainted: G      D       3.15.6
2
[  366.317982] Hardware name: Intel Corporation S2600WTT/S2600WTT, BIOS
GRNDSDP4
[  366.318002] task: ffff880003098000 ti: ffff88017c04c000 task.ti:
ffff88017c00
[  366.318017] RIP: e030:[<ffffffff810ea5f9>]  [<ffffffff810ea5f9>]
find_busies0
[  366.318040] RSP: e02b:ffff88017c04fa78  EFLAGS: 00010046
[  366.318052] RAX: 0000000000000000 RBX: ffff88017c04fb78 RCX:
0000000000000000
[  366.318067] RDX: 0000000000000000 RSI: 0000000000000000 RDI:
0000000000000000
[  366.318082] RBP: ffff88017c04fbe8 R08: ffff880003fb3d00 R09:
0000000000000040
[  366.318096] R10: 0000000000000000 R11: 0000000000000293 R12:
0000000000013e00
[  366.318111] R13: ffff88017c04fc68 R14: ffff880003fb3ce0 R15:
0000000000000000
[  366.318135] FS:  00007f348d764700(0000) GS:ffff880181bc0000(0000)
knlGS:00000
[  366.318151] CS:  e033 DS: 0000 ES: 0000 CR0: 0000000080050033
[  366.318164] CR2: 00007fad200d5000 CR3: 0000000003271000 CR4:
0000000000042660
[  366.318179] Stack:
[  366.318186]  0000000000000001 ffff88017c04fac8 0000000000002e7b
00000000ffffc
[  366.318212]  ffff880003fb3ce0 0000000000013df8 0000000000000200
0000000000010
[  366.318236]  0000000085f9a800 ffff880003fb3cf8 0000000000000000
0000000000000
[  366.318258] Call Trace:
[  366.318274]  [<ffffffff810eae37>] load_balance+0x177/0x9d0
[  366.318290]  [<ffffffff810df56b>] ? update_rq_clock+0x2b/0x50
[  366.318306]  [<ffffffff81058ea0>] ? xen_clocksource_read+0x20/0x30
[  366.318323]  [<ffffffff810edb8d>] pick_next_task_fair+0x1ed/0x430
[  366.318342]  [<ffffffff816ff0d3>] __schedule+0x113/0x870
[  366.318357]  [<ffffffff81703ace>] ? _raw_spin_unlock_irqrestore+0x2e/0xa0
[  366.318375]  [<ffffffff816ff9b4>] schedule+0x24/0x70
[  366.318391]  [<ffffffff810fbb1a>] do_syslog+0x4ba/0x640
[  366.318406]  [<ffffffff810f2d00>] ? bit_waitqueue+0xe0/0xe0
[  366.318424]  [<ffffffff81235ea2>] kmsg_read+0x32/0x70
[  366.318439]  [<ffffffff812294de>] proc_reg_read+0x3e/0x70
[  366.318454]  [<ffffffff811c7405>] vfs_read+0xa5/0x180
[  366.318469]  [<ffffffff811c75c1>] SyS_read+0x51/0xc0
[  366.318484]  [<ffffffff8170c5f9>] system_call_fastpath+0x16/0x1b
[  366.318496] Code: 0f 47 d1 eb 95 0f 1f 44 00 00 4d 89 ec 4d 89 f5 4c 8b b5 b
[  366.318699] RIP  [<ffffffff810ea5f9>] find_busiest_group+0x239/0x900
[  366.318718]  RSP <ffff88017c04fa78>
[  366.318728] ---[ end trace 42d3248df75182f7 ]---

^ permalink raw reply	[flat|nested] 9+ messages in thread

* Re: [PATCH v2] x86, hotplug: fix llc shared map unreleased during cpu hotplug
  2014-07-29  7:06   ` Wanpeng Li
@ 2014-07-29  7:31     ` Yasuaki Ishimatsu
  2014-07-29  8:33       ` Borislav Petkov
  0 siblings, 1 reply; 9+ messages in thread
From: Yasuaki Ishimatsu @ 2014-07-29  7:31 UTC (permalink / raw)
  To: Wanpeng Li
  Cc: hpa, Ingo Molnar, Peter Zijlstra, x86, Borislav Petkov,
	David Rientjes, Prarit Bhargava, Steven Rostedt, Jan Kiszka,
	Toshi Kani, linux-kernel, Konrad Rzeszutek Wilk, Zhang, Yang Z,
	Yong Wang

Hi Wanpeng,

(2014/07/29 16:06), Wanpeng Li wrote:
> Hi Yasuaki,
> On Wed, Jul 23, 2014 at 05:56:07PM +0900, Yasuaki Ishimatsu wrote:
>> (2014/07/22 17:04), Wanpeng Li wrote:
>>> [  220.262093] BUG: unable to handle kernel NULL pointer dereference at 0000000000000004
>>> [  220.262104] IP: [<ffffffff810e7ac9>] find_busiest_group+0x2b9/0xa30
>>> [  220.262111] PGD 5a9d5067 PUD 13067 PMD 0
>>> [  220.262117] Oops: 0000 [#3] SMP
>>> [...]
>>> [  220.262245] Call Trace:
>>> [  220.262252]  [<ffffffff810e8396>] load_balance+0x156/0x980
>>> [  220.262259]  [<ffffffff816eeffe>] ? _raw_spin_unlock_irqrestore+0x2e/0xa0
>>> [  220.262266]  [<ffffffff810e9aa3>] idle_balance+0xe3/0x150
>>> [  220.262270]  [<ffffffff816ec4e7>] __schedule+0x797/0x8d0
>>> [  220.262277]  [<ffffffff816ec934>] schedule+0x24/0x70
>>> [  220.262283]  [<ffffffff816e9cd9>] schedule_timeout+0x119/0x1f0
>>> [  220.262294]  [<ffffffff810bb6e0>] ? lock_timer_base+0x70/0x70
>>> [  220.262301]  [<ffffffff816e9dc9>] schedule_timeout_uninterruptible+0x19/0x20
>>> [  220.262308]  [<ffffffff810bd3e8>] msleep+0x18/0x20
>>> [  220.262317]  [<ffffffff813aa11a>] lock_device_hotplug_sysfs+0x2a/0x50
>>> [  220.262323]  [<ffffffff813aa16e>] online_store+0x2e/0x80
>>> [  220.262358]  [<ffffffff813a873b>] dev_attr_store+0x1b/0x20
>>> [  220.262366]  [<ffffffff812292fd>] sysfs_write_file+0xdd/0x160
>>> [  220.262377]  [<ffffffff811b7e78>] vfs_write+0xc8/0x170
>>> [  220.262384]  [<ffffffff811b83ca>] SyS_write+0x5a/0xa0
>>> [  220.262388]  [<ffffffff816f76b9>] system_call_fastpath+0x16/0x1b
>>>
>>> Last level cache shared map is built during cpu up and build sched domain
>>> routine takes advantage of it to setup sched domain cpu topology, however,
>>> llc shared map is unreleased during cpu disable which lead to invalid sched
>>> domain cpu topology. This patch fix it by release llc shared map correctly
>>> during cpu disable.
>>>
>>
>> I posted a latest patch as follows:
>> https://lkml.org/lkml/2014/7/22/1018
>>
>> Could you confirm the patch fixes your issue?
>
> Sorry for the late, there is still call trace w/ your patch applied. The
> call trace is in attachment.

Thank you for reporting the result. As Borislav said, your v2 patch
is necessary for fixing your issue.

>
> Regards,
> Wanpeng Li
>
>>
>> Thanks,
>> Yasuaki Ishimatsu
>>
>>> Signed-off-by: Wanpeng Li <wanpeng.li@linux.intel.com>

Reviewed-by: Yasuaki Ishimatsu <isimatu.yasuaki@jp.fujitsu.com>

Thanks,
Yasuaki Ishimatsu

>>> ---
>>> v1 -> v2:
>>>    * fix subject line
>>>
>>>    arch/x86/kernel/smpboot.c | 3 +++
>>>    1 file changed, 3 insertions(+)
>>>
>>> diff --git a/arch/x86/kernel/smpboot.c b/arch/x86/kernel/smpboot.c
>>> index 5492798..0134ec7 100644
>>> --- a/arch/x86/kernel/smpboot.c
>>> +++ b/arch/x86/kernel/smpboot.c
>>> @@ -1292,6 +1292,9 @@ static void remove_siblinginfo(int cpu)
>>>
>>>    	for_each_cpu(sibling, cpu_sibling_mask(cpu))
>>>    		cpumask_clear_cpu(cpu, cpu_sibling_mask(sibling));
>>> +	for_each_cpu(sibling, cpu_llc_shared_mask(cpu))
>>> +		cpumask_clear_cpu(cpu, cpu_llc_shared_mask(sibling));
>>> +	cpumask_clear(cpu_llc_shared_mask(cpu));
>>>    	cpumask_clear(cpu_sibling_mask(cpu));
>>>    	cpumask_clear(cpu_core_mask(cpu));
>>>    	c->phys_proc_id = 0;
>>>
>>



^ permalink raw reply	[flat|nested] 9+ messages in thread

* Re: [PATCH v2] x86, hotplug: fix llc shared map unreleased during cpu hotplug
  2014-07-29  7:31     ` Yasuaki Ishimatsu
@ 2014-07-29  8:33       ` Borislav Petkov
  2014-07-29  9:02         ` Wanpeng Li
  0 siblings, 1 reply; 9+ messages in thread
From: Borislav Petkov @ 2014-07-29  8:33 UTC (permalink / raw)
  To: Yasuaki Ishimatsu, Wanpeng Li
  Cc: hpa, Ingo Molnar, Peter Zijlstra, x86, David Rientjes,
	Prarit Bhargava, Steven Rostedt, Jan Kiszka, Toshi Kani,
	linux-kernel, Konrad Rzeszutek Wilk, Zhang, Yang Z, Yong Wang

On Tue, Jul 29, 2014 at 04:31:30PM +0900, Yasuaki Ishimatsu wrote:
> >>I posted a latest patch as follows:
> >>https://lkml.org/lkml/2014/7/22/1018
> >>
> >>Could you confirm the patch fixes your issue?
> >
> >Sorry for the late, there is still call trace w/ your patch applied. The
> >call trace is in attachment.
> 
> Thank you for reporting the result. As Borislav said, your v2 patch
> is necessary for fixing your issue.

..

> >>>Signed-off-by: Wanpeng Li <wanpeng.li@linux.intel.com>
> 
> Reviewed-by: Yasuaki Ishimatsu <isimatu.yasuaki@jp.fujitsu.com>

Ok, Wanpeng, please send a cleaner version of your patch with a
simplified backtrace in it - the addresses will make no sense anyway so
you can do this:

Call Trace:
load_balance
? _raw_spin_unlock_irqrestore
idle_balance
...

so that we can still see how it happens.

Thanks.

-- 
Regards/Gruss,
    Boris.

Sent from a fat crate under my desk. Formatting is fine.
--

^ permalink raw reply	[flat|nested] 9+ messages in thread

* Re: [PATCH v2] x86, hotplug: fix llc shared map unreleased during cpu hotplug
  2014-07-29  8:33       ` Borislav Petkov
@ 2014-07-29  9:02         ` Wanpeng Li
  0 siblings, 0 replies; 9+ messages in thread
From: Wanpeng Li @ 2014-07-29  9:02 UTC (permalink / raw)
  To: Borislav Petkov
  Cc: Yasuaki Ishimatsu, Wanpeng Li, hpa, Ingo Molnar, Peter Zijlstra,
	x86, David Rientjes, Prarit Bhargava, Steven Rostedt, Jan Kiszka,
	Toshi Kani, linux-kernel, Konrad Rzeszutek Wilk, Zhang, Yang Z,
	Yong Wang

On Tue, Jul 29, 2014 at 10:33:59AM +0200, Borislav Petkov wrote:
>On Tue, Jul 29, 2014 at 04:31:30PM +0900, Yasuaki Ishimatsu wrote:
>> >>I posted a latest patch as follows:
>> >>https://lkml.org/lkml/2014/7/22/1018
>> >>
>> >>Could you confirm the patch fixes your issue?
>> >
>> >Sorry for the late, there is still call trace w/ your patch applied. The
>> >call trace is in attachment.
>> 
>> Thank you for reporting the result. As Borislav said, your v2 patch
>> is necessary for fixing your issue.
>
>..
>
>> >>>Signed-off-by: Wanpeng Li <wanpeng.li@linux.intel.com>
>> 
>> Reviewed-by: Yasuaki Ishimatsu <isimatu.yasuaki@jp.fujitsu.com>
>
>Ok, Wanpeng, please send a cleaner version of your patch with a
>simplified backtrace in it - the addresses will make no sense anyway so
>you can do this:
>
>Call Trace:
>load_balance
>? _raw_spin_unlock_irqrestore
>idle_balance
>...
>

Just send out the new version.

Regards,
Wanpeng Li 

>so that we can still see how it happens.
>
>Thanks.
>
>-- 
>Regards/Gruss,
>    Boris.
>
>Sent from a fat crate under my desk. Formatting is fine.
>--

^ permalink raw reply	[flat|nested] 9+ messages in thread

end of thread, other threads:[~2014-07-29  9:01 UTC | newest]

Thread overview: 9+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2014-07-22  8:04 [PATCH v2] x86, hotplug: fix llc shared map unreleased during cpu hotplug Wanpeng Li
2014-07-22  8:46 ` Chen, Gong
2014-07-22  9:30   ` Wanpeng Li
2014-07-22  9:44     ` Borislav Petkov
2014-07-23  8:56 ` Yasuaki Ishimatsu
2014-07-29  7:06   ` Wanpeng Li
2014-07-29  7:31     ` Yasuaki Ishimatsu
2014-07-29  8:33       ` Borislav Petkov
2014-07-29  9:02         ` Wanpeng Li

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox