* Regression: kexec/kdump boot hangs with x86/vector commits
@ 2017-12-13 2:52 Dave Young
2017-12-13 15:57 ` Yu Chen
0 siblings, 1 reply; 23+ messages in thread
From: Dave Young @ 2017-12-13 2:52 UTC (permalink / raw)
To: Thomas Gleixner
Cc: Juergen Gross, Tony Luck, Yu Chen, kexec, Dan Williams,
linux-kernel, mingo, Borislav Petkov, Boris Ostrovsky,
Arjan van de Ven, Rui Zhang
Hi,
Kexec reboot and kdump has broken on my laptop for long time with
4.15.0-rc1+ kernels. With the patch below an early panic been fixed:
https://patchwork.kernel.org/patch/10084289/
But still can not get a successful reboot, it looked like graphic
issue, but after bisecting the kernel, I got below:
[dyoung@dhcp-*-* linux]$ git bisect good
There are only 'skip'ped commits left to test.
The first bad commit could be any of:
2db1f959d9dc16035f2eb44ed5fdb2789b754d6a
4900be83602b6be07366d3e69f756c1959f4169a
We cannot bisect more!
These two commits can no be reverted because of code conflicts, thus
I reverted the whole series from Thomas (below commits), with those
x86/vector changes reverted, kexec reboot works fine.
Could you help to take a look, any thoughts? I can do the test
if you have some debug patch to try.
commit 90ad9e2d91067983f3328e21b306323877e5f48a
Author: Thomas Gleixner <tglx@linutronix.de>
Date: Wed Sep 13 23:29:49 2017 +0200
x86/io_apic: Reevaluate vector configuration on activate()
commit 2db1f959d9dc16035f2eb44ed5fdb2789b754d6a
Author: Thomas Gleixner <tglx@linutronix.de>
Date: Wed Sep 13 23:29:50 2017 +0200
x86/vector: Handle managed interrupts proper
commit 4900be83602b6be07366d3e69f756c1959f4169a
Author: Thomas Gleixner <tglx@linutronix.de>
Date: Wed Sep 13 23:29:51 2017 +0200
x86/vector/msi: Switch to global reservation mode
commit 464d12309e1b5829597793db551ae8ecaecf4036
Author: Thomas Gleixner <tglx@linutronix.de>
Date: Wed Sep 13 23:29:52 2017 +0200
x86/vector: Switch IOAPIC to global reservation mod
commit 2cffad7bad83157f89332872015f4305d2ac09ac
Author: Thomas Gleixner <tglx@linutronix.de>
Date: Wed Sep 13 23:29:53 2017 +0200
x86/irq: Simplify hotplug vector accounting
commit d6ffc6ac83b1f9f12652d89b9cb5bcbfbea7796c
Author: Thomas Gleixner <tglx@linutronix.de>
Date: Wed Sep 13 23:29:54 2017 +0200
x86/vector: Respect affinity mask in irq descriptor
commit 02edee152d6ea325c88898f3a702f5db2d78de7a
Author: Thomas Gleixner <tglx@linutronix.de>
Date: Thu Oct 12 11:05:28 2017 +0200
x86/apic/vector: Ignore set_affinity call for inactive interrupts
commit 0696d059f23c05f2dbc3b19ef50e5bdd175b782b
Author: Thomas Gleixner <tglx@linutronix.de>
Date: Mon Oct 16 16:16:19 2017 +0200
x86/vector: Use correct per cpu variable in free_moved_vector()
----------------
BTW, my git HEAD is below:
commit 43570f0383d6d5879ae585e6c3cf027ba321546f (origin/master, origin/HEAD, bisect)
Merge: 43f462f1c2e1 c14ca8386539
Author: Linus Torvalds <torvalds@linux-foundation.org>
Date: Tue Nov 28 16:22:10 2017 -0800
Merge branch 'linus' of git://git.kernel.org/pub/scm/linux/kernel/git/herbert/crypto-2.6
Thanks
Dave
_______________________________________________
kexec mailing list
kexec@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/kexec
^ permalink raw reply [flat|nested] 23+ messages in thread* Re: Regression: kexec/kdump boot hangs with x86/vector commits 2017-12-13 2:52 Regression: kexec/kdump boot hangs with x86/vector commits Dave Young @ 2017-12-13 15:57 ` Yu Chen 2017-12-14 9:24 ` Dave Young 0 siblings, 1 reply; 23+ messages in thread From: Yu Chen @ 2017-12-13 15:57 UTC (permalink / raw) To: Dave Young Cc: Juergen Gross, Tony Luck, Yu Chen, kexec, mingo, Dan Williams, linux-kernel, Rui Zhang, Borislav Petkov, Thomas Gleixner, Arjan van de Ven, Boris Ostrovsky On Wed, Dec 13, 2017 at 10:52:56AM +0800, Dave Young wrote: > Hi, > > Kexec reboot and kdump has broken on my laptop for long time with > 4.15.0-rc1+ kernels. With the patch below an early panic been fixed: > https://patchwork.kernel.org/patch/10084289/ > > But still can not get a successful reboot, it looked like graphic > issue, but after bisecting the kernel, I got below: > > [dyoung@dhcp-*-* linux]$ git bisect good > There are only 'skip'ped commits left to test. > The first bad commit could be any of: > 2db1f959d9dc16035f2eb44ed5fdb2789b754d6a > 4900be83602b6be07366d3e69f756c1959f4169a > We cannot bisect more! > > These two commits can no be reverted because of code conflicts, thus > I reverted the whole series from Thomas (below commits), with those > x86/vector changes reverted, kexec reboot works fine. > > Could you help to take a look, any thoughts? I can do the test > if you have some debug patch to try. Is it possible that the "second" kernel runs on non-zero CPU? If yes, what if some irqs are only delivered to cpu0? (use cpumask_of(0) directly) Thanks, Yu _______________________________________________ kexec mailing list kexec@lists.infradead.org http://lists.infradead.org/mailman/listinfo/kexec ^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: Regression: kexec/kdump boot hangs with x86/vector commits 2017-12-13 15:57 ` Yu Chen @ 2017-12-14 9:24 ` Dave Young 2018-01-04 3:15 ` Dave Young 0 siblings, 1 reply; 23+ messages in thread From: Dave Young @ 2017-12-14 9:24 UTC (permalink / raw) To: Yu Chen Cc: Juergen Gross, Tony Luck, kexec, mingo, Dan Williams, linux-kernel, Rui Zhang, Borislav Petkov, Thomas Gleixner, Arjan van de Ven, Boris Ostrovsky On 12/13/17 at 11:57pm, Yu Chen wrote: > On Wed, Dec 13, 2017 at 10:52:56AM +0800, Dave Young wrote: > > Hi, > > > > Kexec reboot and kdump has broken on my laptop for long time with > > 4.15.0-rc1+ kernels. With the patch below an early panic been fixed: > > https://patchwork.kernel.org/patch/10084289/ > > > > But still can not get a successful reboot, it looked like graphic > > issue, but after bisecting the kernel, I got below: > > > > [dyoung@dhcp-*-* linux]$ git bisect good > > There are only 'skip'ped commits left to test. > > The first bad commit could be any of: > > 2db1f959d9dc16035f2eb44ed5fdb2789b754d6a > > 4900be83602b6be07366d3e69f756c1959f4169a > > We cannot bisect more! > > > > These two commits can no be reverted because of code conflicts, thus > > I reverted the whole series from Thomas (below commits), with those > > x86/vector changes reverted, kexec reboot works fine. > > > > Could you help to take a look, any thoughts? I can do the test > > if you have some debug patch to try. > Is it possible that the "second" kernel runs on non-zero CPU? If yes, > what if some irqs are only delivered to cpu0? (use cpumask_of(0) > directly) Thanks for the reply. For kdump, yes, for kexec, I'm not sure. Here is some kexec kernel boot log: http://people.redhat.com/~ruyang/misc/kexec-regression.txt Copy the lockup call trace here: [ 23.779285] NMI watchdog: Watchdog detected hard LOCKUP on cpu 0 [ 23.779285] Modules linked in: arc4 rtsx_pci_sdmmc i915 iwlmvm kvm_intel mac8 0211 kvm irqbypass btusb btrtl btbcm intel_gtt btintel drm_kms_helper snd_hda_in tel syscopyarea bluetooth iwlwifi snd_hda_codec snd_hwdep snd_hda_core sysfillre ct snd_seq sysimgblt input_leds fb_sys_fops e1000e ecdh_generic cfg80211 snd_seq _device drm snd_pcm serio_raw ptp pcspkr thinkpad_acpi i2c_i801 snd_timer rtsx_p ci pps_core snd soundcore rfkill video [ 23.779307] CPU: 0 PID: 0 Comm: swapper/0 Not tainted 4.15.0-rc3+ #378 [ 23.779308] Hardware name: LENOVO 20ARS1BJ02/20ARS1BJ02, BIOS GJET92WW (2.42 ) 03/03/2017 [ 23.779312] RIP: 0010:poll_idle+0x2f/0x5f [ 23.779313] RSP: 0018:ffffffff81c03e80 EFLAGS: 00000246 [ 23.779314] RAX: ffffffff81c0f4c0 RBX: ffffffff81c6db80 RCX: 0000000000000000 [ 23.779315] RDX: 0000000000000000 RSI: ffffffff81c6db80 RDI: ffff88021f2201e8 [ 23.779316] RBP: ffff88021f2201e8 R08: 000000349a65b7dd R09: ffff88021f216db4 [ 23.779317] R10: ffffffff81c03e68 R11: 0000000000000000 R12: 0000000000000000 [ 23.779318] R13: ffffffff81c6db98 R14: 0000000000000000 R15: 0000000578a065b1 [ 23.779319] FS: 0000000000000000(0000) GS:ffff88021f200000(0000) knlGS:00000 00000000000 [ 23.779320] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 23.779321] CR2: 00007ffed1d0ee60 CR3: 000000021ec0a006 CR4: 00000000001606b0 [ 23.779322] Call Trace: [ 23.779328] cpuidle_enter_state+0x6a/0x2c0 [ 23.779333] do_idle+0x17b/0x1d0 [ 23.779335] cpu_startup_entry+0x6f/0x80 [ 23.779338] start_kernel+0x431/0x451 [ 23.779342] secondary_startup_64+0xa5/0xb0 [ 23.779344] Code: 00 fb 66 0f 1f 44 00 00 65 48 8b 04 25 40 c4 00 00 f0 80 48 02 20 48 8b 08 83 e1 08 74 0d eb 12 f3 90 65 48 8b 04 25 40 c4 00 00 <48> 8b 00 a8 08 74 ee 65 48 8b 04 25 40 c4 00 00 f0 80 60 02 df Thanks Dave > > Thanks, > Yu > _______________________________________________ kexec mailing list kexec@lists.infradead.org http://lists.infradead.org/mailman/listinfo/kexec ^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: Regression: kexec/kdump boot hangs with x86/vector commits 2017-12-14 9:24 ` Dave Young @ 2018-01-04 3:15 ` Dave Young 2018-01-17 7:22 ` kexec reboot fails with extra wbinvd introduced for AME SME Dave Young 0 siblings, 1 reply; 23+ messages in thread From: Dave Young @ 2018-01-04 3:15 UTC (permalink / raw) To: Yu Chen Cc: Juergen Gross, Tony Luck, kexec, mingo, Dan Williams, linux-kernel, Rui Zhang, Borislav Petkov, Thomas Gleixner, Arjan van de Ven, Boris Ostrovsky On 12/14/17 at 05:24pm, Dave Young wrote: > On 12/13/17 at 11:57pm, Yu Chen wrote: > > On Wed, Dec 13, 2017 at 10:52:56AM +0800, Dave Young wrote: > > > Hi, > > > > > > Kexec reboot and kdump has broken on my laptop for long time with > > > 4.15.0-rc1+ kernels. With the patch below an early panic been fixed: > > > https://patchwork.kernel.org/patch/10084289/ > > > > > > But still can not get a successful reboot, it looked like graphic > > > issue, but after bisecting the kernel, I got below: > > > > > > [dyoung@dhcp-*-* linux]$ git bisect good > > > There are only 'skip'ped commits left to test. > > > The first bad commit could be any of: > > > 2db1f959d9dc16035f2eb44ed5fdb2789b754d6a > > > 4900be83602b6be07366d3e69f756c1959f4169a > > > We cannot bisect more! > > > > > > These two commits can no be reverted because of code conflicts, thus > > > I reverted the whole series from Thomas (below commits), with those > > > x86/vector changes reverted, kexec reboot works fine. > > > > > > Could you help to take a look, any thoughts? I can do the test > > > if you have some debug patch to try. > > Is it possible that the "second" kernel runs on non-zero CPU? If yes, > > what if some irqs are only delivered to cpu0? (use cpumask_of(0) > > directly) > > Thanks for the reply. > > For kdump, yes, for kexec, I'm not sure. > > Here is some kexec kernel boot log: > http://people.redhat.com/~ruyang/misc/kexec-regression.txt > > Copy the lockup call trace here: > [ 23.779285] NMI watchdog: Watchdog detected hard LOCKUP on cpu 0 > [ 23.779285] Modules linked in: arc4 rtsx_pci_sdmmc i915 iwlmvm kvm_intel mac8 > 0211 kvm irqbypass btusb btrtl btbcm intel_gtt btintel drm_kms_helper snd_hda_in > tel syscopyarea bluetooth iwlwifi snd_hda_codec snd_hwdep snd_hda_core sysfillre > ct snd_seq sysimgblt input_leds fb_sys_fops e1000e ecdh_generic cfg80211 snd_seq > _device drm snd_pcm serio_raw ptp pcspkr thinkpad_acpi i2c_i801 snd_timer rtsx_p > ci pps_core snd soundcore rfkill video > [ 23.779307] CPU: 0 PID: 0 Comm: swapper/0 Not tainted 4.15.0-rc3+ #378 > [ 23.779308] Hardware name: LENOVO 20ARS1BJ02/20ARS1BJ02, BIOS GJET92WW (2.42 > ) 03/03/2017 > [ 23.779312] RIP: 0010:poll_idle+0x2f/0x5f > [ 23.779313] RSP: 0018:ffffffff81c03e80 EFLAGS: 00000246 > [ 23.779314] RAX: ffffffff81c0f4c0 RBX: ffffffff81c6db80 RCX: 0000000000000000 > [ 23.779315] RDX: 0000000000000000 RSI: ffffffff81c6db80 RDI: ffff88021f2201e8 > [ 23.779316] RBP: ffff88021f2201e8 R08: 000000349a65b7dd R09: ffff88021f216db4 > [ 23.779317] R10: ffffffff81c03e68 R11: 0000000000000000 R12: 0000000000000000 > [ 23.779318] R13: ffffffff81c6db98 R14: 0000000000000000 R15: 0000000578a065b1 > [ 23.779319] FS: 0000000000000000(0000) GS:ffff88021f200000(0000) knlGS:00000 > 00000000000 > [ 23.779320] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > [ 23.779321] CR2: 00007ffed1d0ee60 CR3: 000000021ec0a006 CR4: 00000000001606b0 > [ 23.779322] Call Trace: > [ 23.779328] cpuidle_enter_state+0x6a/0x2c0 > [ 23.779333] do_idle+0x17b/0x1d0 > [ 23.779335] cpu_startup_entry+0x6f/0x80 > [ 23.779338] start_kernel+0x431/0x451 > [ 23.779342] secondary_startup_64+0xa5/0xb0 > [ 23.779344] Code: 00 fb 66 0f 1f 44 00 00 65 48 8b 04 25 40 c4 00 00 f0 80 48 > 02 20 48 8b 08 83 e1 08 74 0d eb 12 f3 90 65 48 8b 04 25 40 c4 00 00 <48> 8b 00 > a8 08 74 ee 65 48 8b 04 25 40 c4 00 00 f0 80 60 02 df > Followup this issue, seems another commit from Thomas partially fixed this, kexec/kdump boot up successfully for me, but kexec after kexec (2nd kexec reboot cycle) failed, kernel hung early commit bc976233a872c0f20f018fb1e89264a541584e25 Author: Thomas Gleixner <tglx@linutronix.de> Date: Fri Dec 29 10:47:22 2017 +0100 genirq/msi, x86/vector: Prevent reservation mode for non maskable MSI Thanks Dave _______________________________________________ kexec mailing list kexec@lists.infradead.org http://lists.infradead.org/mailman/listinfo/kexec ^ permalink raw reply [flat|nested] 23+ messages in thread
* kexec reboot fails with extra wbinvd introduced for AME SME 2018-01-04 3:15 ` Dave Young @ 2018-01-17 7:22 ` Dave Young 2018-01-17 15:06 ` Tom Lendacky 2018-01-17 19:42 ` Linus Torvalds 0 siblings, 2 replies; 23+ messages in thread From: Dave Young @ 2018-01-17 7:22 UTC (permalink / raw) To: Yu Chen Cc: Juergen Gross, thomas.lendacky, Tony Luck, bhe, kexec, mingo, Dan Williams, linux-kernel, Rui Zhang, ebiederm, Borislav Petkov, torvalds, Thomas Gleixner, Arjan van de Ven, Boris Ostrovsky [Modify the subject since this is a new problem, original io vector issue has been fixed with one commit from Thomas] Add more cc according to below old discussion: https://lkml.org/lkml/2017/7/27/574 Tom, I'm not sure why you finally did not dynamically run wbinvd? On 01/04/18 at 11:15am, Dave Young wrote: > On 12/14/17 at 05:24pm, Dave Young wrote: > > On 12/13/17 at 11:57pm, Yu Chen wrote: > > > On Wed, Dec 13, 2017 at 10:52:56AM +0800, Dave Young wrote: > > > > Hi, > > > > > > > > Kexec reboot and kdump has broken on my laptop for long time with > > > > 4.15.0-rc1+ kernels. With the patch below an early panic been fixed: > > > > https://patchwork.kernel.org/patch/10084289/ > > > > > > > > But still can not get a successful reboot, it looked like graphic > > > > issue, but after bisecting the kernel, I got below: > > > > > > > > [dyoung@dhcp-*-* linux]$ git bisect good > > > > There are only 'skip'ped commits left to test. > > > > The first bad commit could be any of: > > > > 2db1f959d9dc16035f2eb44ed5fdb2789b754d6a > > > > 4900be83602b6be07366d3e69f756c1959f4169a > > > > We cannot bisect more! > > > > > > > > These two commits can no be reverted because of code conflicts, thus > > > > I reverted the whole series from Thomas (below commits), with those > > > > x86/vector changes reverted, kexec reboot works fine. > > > > > > > > Could you help to take a look, any thoughts? I can do the test > > > > if you have some debug patch to try. > > > Is it possible that the "second" kernel runs on non-zero CPU? If yes, > > > what if some irqs are only delivered to cpu0? (use cpumask_of(0) > > > directly) > > > > Thanks for the reply. > > > > For kdump, yes, for kexec, I'm not sure. > > > > Here is some kexec kernel boot log: > > http://people.redhat.com/~ruyang/misc/kexec-regression.txt > > > > Copy the lockup call trace here: > > [ 23.779285] NMI watchdog: Watchdog detected hard LOCKUP on cpu 0 > > [ 23.779285] Modules linked in: arc4 rtsx_pci_sdmmc i915 iwlmvm kvm_intel mac8 > > 0211 kvm irqbypass btusb btrtl btbcm intel_gtt btintel drm_kms_helper snd_hda_in > > tel syscopyarea bluetooth iwlwifi snd_hda_codec snd_hwdep snd_hda_core sysfillre > > ct snd_seq sysimgblt input_leds fb_sys_fops e1000e ecdh_generic cfg80211 snd_seq > > _device drm snd_pcm serio_raw ptp pcspkr thinkpad_acpi i2c_i801 snd_timer rtsx_p > > ci pps_core snd soundcore rfkill video > > [ 23.779307] CPU: 0 PID: 0 Comm: swapper/0 Not tainted 4.15.0-rc3+ #378 > > [ 23.779308] Hardware name: LENOVO 20ARS1BJ02/20ARS1BJ02, BIOS GJET92WW (2.42 > > ) 03/03/2017 > > [ 23.779312] RIP: 0010:poll_idle+0x2f/0x5f > > [ 23.779313] RSP: 0018:ffffffff81c03e80 EFLAGS: 00000246 > > [ 23.779314] RAX: ffffffff81c0f4c0 RBX: ffffffff81c6db80 RCX: 0000000000000000 > > [ 23.779315] RDX: 0000000000000000 RSI: ffffffff81c6db80 RDI: ffff88021f2201e8 > > [ 23.779316] RBP: ffff88021f2201e8 R08: 000000349a65b7dd R09: ffff88021f216db4 > > [ 23.779317] R10: ffffffff81c03e68 R11: 0000000000000000 R12: 0000000000000000 > > [ 23.779318] R13: ffffffff81c6db98 R14: 0000000000000000 R15: 0000000578a065b1 > > [ 23.779319] FS: 0000000000000000(0000) GS:ffff88021f200000(0000) knlGS:00000 > > 00000000000 > > [ 23.779320] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > > [ 23.779321] CR2: 00007ffed1d0ee60 CR3: 000000021ec0a006 CR4: 00000000001606b0 > > [ 23.779322] Call Trace: > > [ 23.779328] cpuidle_enter_state+0x6a/0x2c0 > > [ 23.779333] do_idle+0x17b/0x1d0 > > [ 23.779335] cpu_startup_entry+0x6f/0x80 > > [ 23.779338] start_kernel+0x431/0x451 > > [ 23.779342] secondary_startup_64+0xa5/0xb0 > > [ 23.779344] Code: 00 fb 66 0f 1f 44 00 00 65 48 8b 04 25 40 c4 00 00 f0 80 48 > > 02 20 48 8b 08 83 e1 08 74 0d eb 12 f3 90 65 48 8b 04 25 40 c4 00 00 <48> 8b 00 > > a8 08 74 ee 65 48 8b 04 25 40 c4 00 00 f0 80 60 02 df > > > > Followup this issue, seems another commit from Thomas partially fixed > this, kexec/kdump boot up successfully for me, but kexec after kexec > (2nd kexec reboot cycle) failed, kernel hung early The above kexec reboot hang is another problem, so Thomas has fully fixed previous report, thanks! For the kexec reboot hang, if I remove the wbinvd in stop_this_cpu() then kexec works fine. like this: diff --git a/arch/x86/kernel/process.c b/arch/x86/kernel/process.c index 832a6acd730f..6d7499730b27 100644 --- a/arch/x86/kernel/process.c +++ b/arch/x86/kernel/process.c @@ -380,20 +380,8 @@ void stop_this_cpu(void *dummy) disable_local_APIC(); mcheck_cpu_clear(this_cpu_ptr(&cpu_info)); - for (;;) { - /* - * Use wbinvd followed by hlt to stop the processor. This - * provides support for kexec on a processor that supports - * SME. With kexec, going from SME inactive to SME active - * requires clearing cache entries so that addresses without - * the encryption bit set don't corrupt the same physical - * address that has the encryption bit set when caches are - * flushed. To achieve this a wbinvd is performed followed by - * a hlt. Even if the processor is not in the kexec/SME - * scenario this only adds a wbinvd to a halting processor. - */ - asm volatile("wbinvd; hlt" : : : "memory"); - } + for (;;) + halt(); } /* But I have no idea why though, seeking for help and thoughts.. > > commit bc976233a872c0f20f018fb1e89264a541584e25 > Author: Thomas Gleixner <tglx@linutronix.de> > Date: Fri Dec 29 10:47:22 2017 +0100 > > genirq/msi, x86/vector: Prevent reservation mode for non maskable MSI > > Thanks > Dave Thanks Dave _______________________________________________ kexec mailing list kexec@lists.infradead.org http://lists.infradead.org/mailman/listinfo/kexec ^ permalink raw reply related [flat|nested] 23+ messages in thread
* Re: kexec reboot fails with extra wbinvd introduced for AME SME 2018-01-17 7:22 ` kexec reboot fails with extra wbinvd introduced for AME SME Dave Young @ 2018-01-17 15:06 ` Tom Lendacky 2018-01-18 1:50 ` Dave Young 2018-01-17 19:42 ` Linus Torvalds 1 sibling, 1 reply; 23+ messages in thread From: Tom Lendacky @ 2018-01-17 15:06 UTC (permalink / raw) To: Dave Young, Yu Chen Cc: Juergen Gross, Tony Luck, bhe, kexec, mingo, Dan Williams, linux-kernel, Rui Zhang, ebiederm, Borislav Petkov, torvalds, Thomas Gleixner, Arjan van de Ven, Boris Ostrovsky On 1/17/2018 1:22 AM, Dave Young wrote: > [Modify the subject since this is a new problem, original io vector > issue has been fixed with one commit from Thomas] > > Add more cc according to below old discussion: > https://lkml.org/lkml/2017/7/27/574 > > Tom, I'm not sure why you finally did not dynamically run wbinvd? That discussion was aimed at the wbinvd that was being performed in arch/x86/kernel/relocate_kernel_64.S, which is dynamically run based on a flag. > On 01/04/18 at 11:15am, Dave Young wrote: >> On 12/14/17 at 05:24pm, Dave Young wrote: >>> On 12/13/17 at 11:57pm, Yu Chen wrote: >>>> On Wed, Dec 13, 2017 at 10:52:56AM +0800, Dave Young wrote: >>>>> Hi, >>>>> >>>>> Kexec reboot and kdump has broken on my laptop for long time with >>>>> 4.15.0-rc1+ kernels. With the patch below an early panic been fixed: >>>>> https://patchwork.kernel.org/patch/10084289/ >>>>> >>>>> But still can not get a successful reboot, it looked like graphic >>>>> issue, but after bisecting the kernel, I got below: >>>>> >>>>> [dyoung@dhcp-*-* linux]$ git bisect good >>>>> There are only 'skip'ped commits left to test. >>>>> The first bad commit could be any of: >>>>> 2db1f959d9dc16035f2eb44ed5fdb2789b754d6a >>>>> 4900be83602b6be07366d3e69f756c1959f4169a >>>>> We cannot bisect more! >>>>> >>>>> These two commits can no be reverted because of code conflicts, thus >>>>> I reverted the whole series from Thomas (below commits), with those >>>>> x86/vector changes reverted, kexec reboot works fine. >>>>> >>>>> Could you help to take a look, any thoughts? I can do the test >>>>> if you have some debug patch to try. >>>> Is it possible that the "second" kernel runs on non-zero CPU? If yes, >>>> what if some irqs are only delivered to cpu0? (use cpumask_of(0) >>>> directly) >>> >>> Thanks for the reply. >>> >>> For kdump, yes, for kexec, I'm not sure. >>> >>> Here is some kexec kernel boot log: >>> http://people.redhat.com/~ruyang/misc/kexec-regression.txt >>> >>> Copy the lockup call trace here: >>> [ 23.779285] NMI watchdog: Watchdog detected hard LOCKUP on cpu 0 >>> [ 23.779285] Modules linked in: arc4 rtsx_pci_sdmmc i915 iwlmvm kvm_intel mac8 >>> 0211 kvm irqbypass btusb btrtl btbcm intel_gtt btintel drm_kms_helper snd_hda_in >>> tel syscopyarea bluetooth iwlwifi snd_hda_codec snd_hwdep snd_hda_core sysfillre >>> ct snd_seq sysimgblt input_leds fb_sys_fops e1000e ecdh_generic cfg80211 snd_seq >>> _device drm snd_pcm serio_raw ptp pcspkr thinkpad_acpi i2c_i801 snd_timer rtsx_p >>> ci pps_core snd soundcore rfkill video >>> [ 23.779307] CPU: 0 PID: 0 Comm: swapper/0 Not tainted 4.15.0-rc3+ #378 >>> [ 23.779308] Hardware name: LENOVO 20ARS1BJ02/20ARS1BJ02, BIOS GJET92WW (2.42 >>> ) 03/03/2017 >>> [ 23.779312] RIP: 0010:poll_idle+0x2f/0x5f >>> [ 23.779313] RSP: 0018:ffffffff81c03e80 EFLAGS: 00000246 >>> [ 23.779314] RAX: ffffffff81c0f4c0 RBX: ffffffff81c6db80 RCX: 0000000000000000 >>> [ 23.779315] RDX: 0000000000000000 RSI: ffffffff81c6db80 RDI: ffff88021f2201e8 >>> [ 23.779316] RBP: ffff88021f2201e8 R08: 000000349a65b7dd R09: ffff88021f216db4 >>> [ 23.779317] R10: ffffffff81c03e68 R11: 0000000000000000 R12: 0000000000000000 >>> [ 23.779318] R13: ffffffff81c6db98 R14: 0000000000000000 R15: 0000000578a065b1 >>> [ 23.779319] FS: 0000000000000000(0000) GS:ffff88021f200000(0000) knlGS:00000 >>> 00000000000 >>> [ 23.779320] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 >>> [ 23.779321] CR2: 00007ffed1d0ee60 CR3: 000000021ec0a006 CR4: 00000000001606b0 >>> [ 23.779322] Call Trace: >>> [ 23.779328] cpuidle_enter_state+0x6a/0x2c0 >>> [ 23.779333] do_idle+0x17b/0x1d0 >>> [ 23.779335] cpu_startup_entry+0x6f/0x80 >>> [ 23.779338] start_kernel+0x431/0x451 >>> [ 23.779342] secondary_startup_64+0xa5/0xb0 >>> [ 23.779344] Code: 00 fb 66 0f 1f 44 00 00 65 48 8b 04 25 40 c4 00 00 f0 80 48 >>> 02 20 48 8b 08 83 e1 08 74 0d eb 12 f3 90 65 48 8b 04 25 40 c4 00 00 <48> 8b 00 >>> a8 08 74 ee 65 48 8b 04 25 40 c4 00 00 f0 80 60 02 df >>> >> >> Followup this issue, seems another commit from Thomas partially fixed >> this, kexec/kdump boot up successfully for me, but kexec after kexec >> (2nd kexec reboot cycle) failed, kernel hung early > > The above kexec reboot hang is another problem, so Thomas has fully > fixed previous report, thanks! > > For the kexec reboot hang, if I remove the wbinvd in stop_this_cpu() > then kexec works fine. like this: > > diff --git a/arch/x86/kernel/process.c b/arch/x86/kernel/process.c > index 832a6acd730f..6d7499730b27 100644 > --- a/arch/x86/kernel/process.c > +++ b/arch/x86/kernel/process.c > @@ -380,20 +380,8 @@ void stop_this_cpu(void *dummy) > disable_local_APIC(); > mcheck_cpu_clear(this_cpu_ptr(&cpu_info)); > > - for (;;) { > - /* > - * Use wbinvd followed by hlt to stop the processor. This > - * provides support for kexec on a processor that supports > - * SME. With kexec, going from SME inactive to SME active > - * requires clearing cache entries so that addresses without > - * the encryption bit set don't corrupt the same physical > - * address that has the encryption bit set when caches are > - * flushed. To achieve this a wbinvd is performed followed by > - * a hlt. Even if the processor is not in the kexec/SME > - * scenario this only adds a wbinvd to a halting processor. > - */ > - asm volatile("wbinvd; hlt" : : : "memory"); > - } > + for (;;) > + halt(); > } > > /* > > But I have no idea why though, seeking for help and thoughts.. Yeah, I don't know why that works either. Thanks, Tom > >> >> commit bc976233a872c0f20f018fb1e89264a541584e25 >> Author: Thomas Gleixner <tglx@linutronix.de> >> Date: Fri Dec 29 10:47:22 2017 +0100 >> >> genirq/msi, x86/vector: Prevent reservation mode for non maskable MSI >> >> Thanks >> Dave > > Thanks > Dave > _______________________________________________ kexec mailing list kexec@lists.infradead.org http://lists.infradead.org/mailman/listinfo/kexec ^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: kexec reboot fails with extra wbinvd introduced for AME SME 2018-01-17 15:06 ` Tom Lendacky @ 2018-01-18 1:50 ` Dave Young 0 siblings, 0 replies; 23+ messages in thread From: Dave Young @ 2018-01-18 1:50 UTC (permalink / raw) To: Tom Lendacky Cc: Juergen Gross, Tony Luck, Yu Chen, bhe, kexec, mingo, Dan Williams, linux-kernel, Rui Zhang, ebiederm, Borislav Petkov, torvalds, Thomas Gleixner, Arjan van de Ven, Boris Ostrovsky On 01/17/18 at 09:06am, Tom Lendacky wrote: > On 1/17/2018 1:22 AM, Dave Young wrote: > > [Modify the subject since this is a new problem, original io vector > > issue has been fixed with one commit from Thomas] > > > > Add more cc according to below old discussion: > > https://lkml.org/lkml/2017/7/27/574 > > > > Tom, I'm not sure why you finally did not dynamically run wbinvd? > > That discussion was aimed at the wbinvd that was being performed > in arch/x86/kernel/relocate_kernel_64.S, which is dynamically > run based on a flag. Oh, I did not notice that, sorry for referring a wrong discussion. [snip] Thanks Dave _______________________________________________ kexec mailing list kexec@lists.infradead.org http://lists.infradead.org/mailman/listinfo/kexec ^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: kexec reboot fails with extra wbinvd introduced for AME SME 2018-01-17 7:22 ` kexec reboot fails with extra wbinvd introduced for AME SME Dave Young 2018-01-17 15:06 ` Tom Lendacky @ 2018-01-17 19:42 ` Linus Torvalds 2018-01-17 20:01 ` Tom Lendacky ` (2 more replies) 1 sibling, 3 replies; 23+ messages in thread From: Linus Torvalds @ 2018-01-17 19:42 UTC (permalink / raw) To: Dave Young Cc: Juergen Gross, Tom Lendacky, Tony Luck, Yu Chen, Baoquan He, Kexec Mailing List, Ingo Molnar, Dan Williams, Linux Kernel Mailing List, Rui Zhang, ebiederm, Borislav Petkov, Thomas Gleixner, Arjan van de Ven, Boris Ostrovsky On Tue, Jan 16, 2018 at 11:22 PM, Dave Young <dyoung@redhat.com> wrote: > > For the kexec reboot hang, if I remove the wbinvd in stop_this_cpu() > then kexec works fine. like this: Honestly, I think we should apply that patch regardless. Using 'wbinvd' should not be some "just because of random reasons". There are CPU's with errata on wbinvd, and the thing in general is slow and nasty. Doing the wbinvd in a loop sounds even stranger. If we're only doing it because of some SME issue, why isn't it dependent on SME? And why is it inside that loop at all? Anyway, does it work for you if you just do the wbinvd() once, outside the loop? Admittedly the loop shouldn't actually loop (hlt with interrupts disabled), but who the hell knows.. Some of the errata around SME have been about machine check exceptions or something. See commit a68e5c94f7d3 ("x86, hotplug: Move WBINVD back outside the play_dead loop") for another example where wbinvd was inside a loop and apparently caused some odd issues. Linus _______________________________________________ kexec mailing list kexec@lists.infradead.org http://lists.infradead.org/mailman/listinfo/kexec ^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: kexec reboot fails with extra wbinvd introduced for AME SME 2018-01-17 19:42 ` Linus Torvalds @ 2018-01-17 20:01 ` Tom Lendacky 2018-01-17 22:53 ` Tom Lendacky 2018-01-17 20:10 ` Linus Torvalds 2018-01-18 1:47 ` Dave Young 2 siblings, 1 reply; 23+ messages in thread From: Tom Lendacky @ 2018-01-17 20:01 UTC (permalink / raw) To: Linus Torvalds, Dave Young Cc: Juergen Gross, Tony Luck, Yu Chen, Baoquan He, Kexec Mailing List, Ingo Molnar, Dan Williams, Linux Kernel Mailing List, Rui Zhang, ebiederm, Borislav Petkov, Thomas Gleixner, Arjan van de Ven, Boris Ostrovsky On 1/17/2018 1:42 PM, Linus Torvalds wrote: > On Tue, Jan 16, 2018 at 11:22 PM, Dave Young <dyoung@redhat.com> wrote: >> >> For the kexec reboot hang, if I remove the wbinvd in stop_this_cpu() >> then kexec works fine. like this: > > Honestly, I think we should apply that patch regardless. > > Using 'wbinvd' should not be some "just because of random reasons". > There are CPU's with errata on wbinvd, and the thing in general is > slow and nasty. > > Doing the wbinvd in a loop sounds even stranger. > > If we're only doing it because of some SME issue, why isn't it > dependent on SME? And why is it inside that loop at all? My original patches did check for X86_FEATURE_SME and only do the wbinvd if SME was supported (although still in the loop). The general consensus was to just do the wbinvd no matter what and so it is as it is today. It can probably be outside of the loop. The issue I was seeing was memory corruption from the stack when using halt() with paravirt ops enabled. So a native_halt() should be used. > > Anyway, does it work for you if you just do the wbinvd() once, outside > the loop? Admittedly the loop shouldn't actually loop (hlt with > interrupts disabled), but who the hell knows.. Some of the errata > around SME have been about machine check exceptions or something. I think that should work as long as it's a native_wbinvd() call and it can also be conditional on boot_cpu_has(X86_FEATURE_SME). I'll do some testing. Thanks, Tom > > See commit a68e5c94f7d3 ("x86, hotplug: Move WBINVD back outside the > play_dead loop") for another example where wbinvd was inside a loop > and apparently caused some odd issues. > > Linus > _______________________________________________ kexec mailing list kexec@lists.infradead.org http://lists.infradead.org/mailman/listinfo/kexec ^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: kexec reboot fails with extra wbinvd introduced for AME SME 2018-01-17 20:01 ` Tom Lendacky @ 2018-01-17 22:53 ` Tom Lendacky 0 siblings, 0 replies; 23+ messages in thread From: Tom Lendacky @ 2018-01-17 22:53 UTC (permalink / raw) To: Linus Torvalds, Dave Young Cc: Juergen Gross, Tony Luck, Yu Chen, Baoquan He, Kexec Mailing List, Ingo Molnar, Dan Williams, Linux Kernel Mailing List, Rui Zhang, ebiederm, Borislav Petkov, Thomas Gleixner, Arjan van de Ven, Boris Ostrovsky On 1/17/2018 2:01 PM, Tom Lendacky wrote: > On 1/17/2018 1:42 PM, Linus Torvalds wrote: >> On Tue, Jan 16, 2018 at 11:22 PM, Dave Young <dyoung@redhat.com> wrote: >>> >>> For the kexec reboot hang, if I remove the wbinvd in stop_this_cpu() >>> then kexec works fine. like this: >> >> Honestly, I think we should apply that patch regardless. >> >> Using 'wbinvd' should not be some "just because of random reasons". >> There are CPU's with errata on wbinvd, and the thing in general is >> slow and nasty. >> >> Doing the wbinvd in a loop sounds even stranger. >> >> If we're only doing it because of some SME issue, why isn't it >> dependent on SME? And why is it inside that loop at all? > > My original patches did check for X86_FEATURE_SME and only do the > wbinvd if SME was supported (although still in the loop). The general > consensus was to just do the wbinvd no matter what and so it is as it is > today. > > It can probably be outside of the loop. The issue I was seeing was > memory corruption from the stack when using halt() with paravirt ops > enabled. So a native_halt() should be used. > >> >> Anyway, does it work for you if you just do the wbinvd() once, outside >> the loop? Admittedly the loop shouldn't actually loop (hlt with >> interrupts disabled), but who the hell knows.. Some of the errata >> around SME have been about machine check exceptions or something. > > I think that should work as long as it's a native_wbinvd() call and it > can also be conditional on boot_cpu_has(X86_FEATURE_SME). > > I'll do some testing. Looks like everything is good with the suggested changes. Patch to follow shortly. Thanks, Tom > > Thanks, > Tom > >> >> See commit a68e5c94f7d3 ("x86, hotplug: Move WBINVD back outside the >> play_dead loop") for another example where wbinvd was inside a loop >> and apparently caused some odd issues. >> >> Linus >> _______________________________________________ kexec mailing list kexec@lists.infradead.org http://lists.infradead.org/mailman/listinfo/kexec ^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: kexec reboot fails with extra wbinvd introduced for AME SME 2018-01-17 19:42 ` Linus Torvalds 2018-01-17 20:01 ` Tom Lendacky @ 2018-01-17 20:10 ` Linus Torvalds 2018-01-18 1:47 ` Dave Young 2 siblings, 0 replies; 23+ messages in thread From: Linus Torvalds @ 2018-01-17 20:10 UTC (permalink / raw) To: Dave Young Cc: Juergen Gross, Tom Lendacky, Tony Luck, Yu Chen, Baoquan He, Kexec Mailing List, Ingo Molnar, Dan Williams, Linux Kernel Mailing List, Rui Zhang, ebiederm, Borislav Petkov, Thomas Gleixner, Arjan van de Ven, Boris Ostrovsky On Wed, Jan 17, 2018 at 11:42 AM, Linus Torvalds <torvalds@linux-foundation.org> wrote: > > [ .. ] Some of the errata > around SME have been about machine check exceptions or something. That should be "some of the errata around wbinvd". I have no idea if there have been SME issues. That said, the really bad old wbinvd bug (which hung the system iirc) was for some old PPro CPU's. Googling around, the errata I find seem either irrelevant (eip corruption in 16-bit mode) or fairly mild (odd behavior wrt parity errors). So I have this memory of wbinvd having problems, but that memory does seem to be largely historical. But Dave Young clearly sees *something* odd going on. Linus _______________________________________________ kexec mailing list kexec@lists.infradead.org http://lists.infradead.org/mailman/listinfo/kexec ^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: kexec reboot fails with extra wbinvd introduced for AME SME 2018-01-17 19:42 ` Linus Torvalds 2018-01-17 20:01 ` Tom Lendacky 2018-01-17 20:10 ` Linus Torvalds @ 2018-01-18 1:47 ` Dave Young 2018-01-18 2:14 ` Linus Torvalds 2 siblings, 1 reply; 23+ messages in thread From: Dave Young @ 2018-01-18 1:47 UTC (permalink / raw) To: Linus Torvalds Cc: Juergen Gross, Tom Lendacky, Tony Luck, Yu Chen, Baoquan He, Kexec Mailing List, Ingo Molnar, Dan Williams, Linux Kernel Mailing List, Rui Zhang, ebiederm, Borislav Petkov, Thomas Gleixner, Arjan van de Ven, Boris Ostrovsky On 01/17/18 at 11:42am, Linus Torvalds wrote: > On Tue, Jan 16, 2018 at 11:22 PM, Dave Young <dyoung@redhat.com> wrote: > > > > For the kexec reboot hang, if I remove the wbinvd in stop_this_cpu() > > then kexec works fine. like this: > > Honestly, I think we should apply that patch regardless. > > Using 'wbinvd' should not be some "just because of random reasons". > There are CPU's with errata on wbinvd, and the thing in general is > slow and nasty. > > Doing the wbinvd in a loop sounds even stranger. > > If we're only doing it because of some SME issue, why isn't it > dependent on SME? And why is it inside that loop at all? > > Anyway, does it work for you if you just do the wbinvd() once, outside > the loop? Admittedly the loop shouldn't actually loop (hlt with > interrupts disabled), but who the hell knows.. Some of the errata > around SME have been about machine check exceptions or something. It does not work with just once wbinvd(), and it only works with removing the wbinvd() for me. Tom's new post works for me as well since my cpu is an Intel i5-4200U. > > See commit a68e5c94f7d3 ("x86, hotplug: Move WBINVD back outside the > play_dead loop") for another example where wbinvd was inside a loop > and apparently caused some odd issues. > > Linus Thanks for the help! Finally I get my laptop kexec & kdump works again after solving these issues. Thanks Dave _______________________________________________ kexec mailing list kexec@lists.infradead.org http://lists.infradead.org/mailman/listinfo/kexec ^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: kexec reboot fails with extra wbinvd introduced for AME SME 2018-01-18 1:47 ` Dave Young @ 2018-01-18 2:14 ` Linus Torvalds 2018-01-18 2:29 ` Dave Young ` (3 more replies) 0 siblings, 4 replies; 23+ messages in thread From: Linus Torvalds @ 2018-01-18 2:14 UTC (permalink / raw) To: Dave Young Cc: Juergen Gross, Tom Lendacky, Tony Luck, Yu Chen, Baoquan He, Kexec Mailing List, Ingo Molnar, Dan Williams, Linux Kernel Mailing List, Rui Zhang, ebiederm, Borislav Petkov, Thomas Gleixner, Arjan van de Ven, Boris Ostrovsky On Wed, Jan 17, 2018 at 5:47 PM, Dave Young <dyoung@redhat.com> wrote: > > It does not work with just once wbinvd(), and it only works with > removing the wbinvd() for me. Tom's new post works for me as well > since my cpu is an Intel i5-4200U. Intriguing. It's not like the wbinvd really should be that much of a deal. I think Tom's patch is fine and should be applied, but it does worry me a bit that even a single wbinvd makes that much of a difference for you. There is very little logical reason I can think of that a wbinvd should make any difference what-so-ever on an i5-4200U. I wonder if you have some system issues, and wbinvd just happens to trigger them. But I think we do wbinvd before a suspend-to-RAM too (it's "ACPI_FLUSH_CPU_CACHE()" in the ACPI code). And the dmr code dioes "wbinvd_on_all_cpus()" which does a cross-call etc. Would you mind experimenting a bit with that wbinvd? In particular, what happens if you enable it (so it's not hidden by the SME check), but you move it up to before interrupts are disabled? I'm wondering if there is some issue with MCE generation and wbinvd and whatever, and doing it when the CPU is down and interrupts are disabled causes some system issue.. Does anybody have any other ideas? Linus _______________________________________________ kexec mailing list kexec@lists.infradead.org http://lists.infradead.org/mailman/listinfo/kexec ^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: kexec reboot fails with extra wbinvd introduced for AME SME 2018-01-18 2:14 ` Linus Torvalds @ 2018-01-18 2:29 ` Dave Young 2018-01-18 5:14 ` Tom Lendacky 2018-01-18 2:47 ` Dave Young ` (2 subsequent siblings) 3 siblings, 1 reply; 23+ messages in thread From: Dave Young @ 2018-01-18 2:29 UTC (permalink / raw) To: Linus Torvalds Cc: Juergen Gross, Tom Lendacky, Tony Luck, Yu Chen, Baoquan He, Kexec Mailing List, Ingo Molnar, Dan Williams, Linux Kernel Mailing List, Rui Zhang, ebiederm, Borislav Petkov, Thomas Gleixner, Arjan van de Ven, Boris Ostrovsky On 01/17/18 at 06:14pm, Linus Torvalds wrote: > On Wed, Jan 17, 2018 at 5:47 PM, Dave Young <dyoung@redhat.com> wrote: > > > > It does not work with just once wbinvd(), and it only works with > > removing the wbinvd() for me. Tom's new post works for me as well > > since my cpu is an Intel i5-4200U. > > Intriguing. > > It's not like the wbinvd really should be that much of a deal. > > I think Tom's patch is fine and should be applied, but it does worry > me a bit that even a single wbinvd makes that much of a difference for > you. There is very little logical reason I can think of that a wbinvd > should make any difference what-so-ever on an i5-4200U. > > I wonder if you have some system issues, and wbinvd just happens to > trigger them. But I think we do wbinvd before a suspend-to-RAM too > (it's "ACPI_FLUSH_CPU_CACHE()" in the ACPI code). And the dmr code > dioes "wbinvd_on_all_cpus()" which does a cross-call etc. > > Would you mind experimenting a bit with that wbinvd? > > In particular, what happens if you enable it (so it's not hidden by > the SME check), but you move it up to before interrupts are disabled? Will play with it more. Actually I found the hang seems happens in code of arch/x86/kernel/relocate_kernel_64.S, there is another wbinvd there as well. > > I'm wondering if there is some issue with MCE generation and wbinvd > and whatever, and doing it when the CPU is down and interrupts are > disabled causes some system issue.. > > Does anybody have any other ideas? > > Linus _______________________________________________ kexec mailing list kexec@lists.infradead.org http://lists.infradead.org/mailman/listinfo/kexec ^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: kexec reboot fails with extra wbinvd introduced for AME SME 2018-01-18 2:29 ` Dave Young @ 2018-01-18 5:14 ` Tom Lendacky 0 siblings, 0 replies; 23+ messages in thread From: Tom Lendacky @ 2018-01-18 5:14 UTC (permalink / raw) To: Dave Young, Linus Torvalds Cc: Juergen Gross, Tony Luck, Yu Chen, Baoquan He, Kexec Mailing List, Ingo Molnar, Dan Williams, Linux Kernel Mailing List, Rui Zhang, ebiederm, Borislav Petkov, Thomas Gleixner, Arjan van de Ven, Boris Ostrovsky On 1/17/2018 8:29 PM, Dave Young wrote: > On 01/17/18 at 06:14pm, Linus Torvalds wrote: >> On Wed, Jan 17, 2018 at 5:47 PM, Dave Young <dyoung@redhat.com> wrote: >>> >>> It does not work with just once wbinvd(), and it only works with >>> removing the wbinvd() for me. Tom's new post works for me as well >>> since my cpu is an Intel i5-4200U. >> >> Intriguing. >> >> It's not like the wbinvd really should be that much of a deal. >> >> I think Tom's patch is fine and should be applied, but it does worry >> me a bit that even a single wbinvd makes that much of a difference for >> you. There is very little logical reason I can think of that a wbinvd >> should make any difference what-so-ever on an i5-4200U. >> >> I wonder if you have some system issues, and wbinvd just happens to >> trigger them. But I think we do wbinvd before a suspend-to-RAM too >> (it's "ACPI_FLUSH_CPU_CACHE()" in the ACPI code). And the dmr code >> dioes "wbinvd_on_all_cpus()" which does a cross-call etc. >> >> Would you mind experimenting a bit with that wbinvd? >> >> In particular, what happens if you enable it (so it's not hidden by >> the SME check), but you move it up to before interrupts are disabled? > > Will play with it more. Actually I found the hang seems happens > in code of arch/x86/kernel/relocate_kernel_64.S, there is another > wbinvd there as well. The wbinvd in arch/x86/kernel/relocate_kernel_64.S is only performed if SME is active, so that one won't be executed on an Intel chip. Thanks, Tom > >> >> I'm wondering if there is some issue with MCE generation and wbinvd >> and whatever, and doing it when the CPU is down and interrupts are >> disabled causes some system issue.. >> >> Does anybody have any other ideas? >> >> Linus _______________________________________________ kexec mailing list kexec@lists.infradead.org http://lists.infradead.org/mailman/listinfo/kexec ^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: kexec reboot fails with extra wbinvd introduced for AME SME 2018-01-18 2:14 ` Linus Torvalds 2018-01-18 2:29 ` Dave Young @ 2018-01-18 2:47 ` Dave Young 2018-01-18 2:56 ` Linus Torvalds 2018-01-18 2:53 ` Arjan van de Ven 2018-01-18 3:00 ` Arjan van de Ven 3 siblings, 1 reply; 23+ messages in thread From: Dave Young @ 2018-01-18 2:47 UTC (permalink / raw) To: Linus Torvalds Cc: Juergen Gross, Tom Lendacky, Tony Luck, Yu Chen, Baoquan He, Kexec Mailing List, Ingo Molnar, Dan Williams, Linux Kernel Mailing List, Rui Zhang, ebiederm, Borislav Petkov, Thomas Gleixner, Arjan van de Ven, Boris Ostrovsky On 01/17/18 at 06:14pm, Linus Torvalds wrote: > On Wed, Jan 17, 2018 at 5:47 PM, Dave Young <dyoung@redhat.com> wrote: > > > > It does not work with just once wbinvd(), and it only works with > > removing the wbinvd() for me. Tom's new post works for me as well > > since my cpu is an Intel i5-4200U. > > Intriguing. > > It's not like the wbinvd really should be that much of a deal. > > I think Tom's patch is fine and should be applied, but it does worry > me a bit that even a single wbinvd makes that much of a difference for > you. There is very little logical reason I can think of that a wbinvd > should make any difference what-so-ever on an i5-4200U. > > I wonder if you have some system issues, and wbinvd just happens to > trigger them. But I think we do wbinvd before a suspend-to-RAM too > (it's "ACPI_FLUSH_CPU_CACHE()" in the ACPI code). And the dmr code > dioes "wbinvd_on_all_cpus()" which does a cross-call etc. > > Would you mind experimenting a bit with that wbinvd? > > In particular, what happens if you enable it (so it's not hidden by > the SME check), but you move it up to before interrupts are disabled? Did several quick tests, probably need more tests, but till now the results are: void stop_this_cpu(void *dummy) { =====> add wbinvd here: kexec works local_irq_disable(); =====> add wbinvd here: kexec works /* * Remove this CPU: */ set_cpu_online(smp_processor_id(), false); =====> add wbinvd here: kexec does not work disable_local_APIC(); mcheck_cpu_clear(this_cpu_ptr(&cpu_info)); [snip] So it seems that it will not work after cpu offined.. > > I'm wondering if there is some issue with MCE generation and wbinvd > and whatever, and doing it when the CPU is down and interrupts are > disabled causes some system issue.. > > Does anybody have any other ideas? > > Linus _______________________________________________ kexec mailing list kexec@lists.infradead.org http://lists.infradead.org/mailman/listinfo/kexec ^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: kexec reboot fails with extra wbinvd introduced for AME SME 2018-01-18 2:47 ` Dave Young @ 2018-01-18 2:56 ` Linus Torvalds 0 siblings, 0 replies; 23+ messages in thread From: Linus Torvalds @ 2018-01-18 2:56 UTC (permalink / raw) To: Dave Young Cc: Juergen Gross, Tom Lendacky, Tony Luck, Yu Chen, Baoquan He, Kexec Mailing List, Ingo Molnar, Dan Williams, Linux Kernel Mailing List, Rui Zhang, ebiederm, Borislav Petkov, Thomas Gleixner, Arjan van de Ven, Boris Ostrovsky On Wed, Jan 17, 2018 at 6:47 PM, Dave Young <dyoung@redhat.com> wrote: > Did several quick tests, probably need more tests, but till now the > results are: > > void stop_this_cpu(void *dummy) > { > =====> add wbinvd here: kexec works > local_irq_disable(); > =====> add wbinvd here: kexec works > /* > * Remove this CPU: > */ > set_cpu_online(smp_processor_id(), false); > =====> add wbinvd here: kexec does not work Funky. > So it seems that it will not work after cpu offined.. Well, that set_cpu_online() call really just clears a bit in our '__cpu_online_mask' CPU mask. It doesn't really do anything to the *hardware*. But I do wonder if the wbinvd causes an SMI or something on your system. I _think_ wbinvd causes some external pin to be wiggled just to tell possible external cache hardware to flush too, and on a system level that could be tied to some random thing. And then if we get an SMI/NMI when we've marked the system offline, maybe we do something odd. Very odd. But maybe this makes somebody go "Duh, that's because of xyz.." Linus _______________________________________________ kexec mailing list kexec@lists.infradead.org http://lists.infradead.org/mailman/listinfo/kexec ^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: kexec reboot fails with extra wbinvd introduced for AME SME 2018-01-18 2:14 ` Linus Torvalds 2018-01-18 2:29 ` Dave Young 2018-01-18 2:47 ` Dave Young @ 2018-01-18 2:53 ` Arjan van de Ven 2018-01-18 2:57 ` Dave Young 2018-01-18 3:00 ` Arjan van de Ven 3 siblings, 1 reply; 23+ messages in thread From: Arjan van de Ven @ 2018-01-18 2:53 UTC (permalink / raw) To: Linus Torvalds, Dave Young Cc: Juergen Gross, Tom Lendacky, Tony Luck, Yu Chen, Baoquan He, Kexec Mailing List, Ingo Molnar, Dan Williams, Linux Kernel Mailing List, Rui Zhang, ebiederm, Borislav Petkov, Thomas Gleixner, Boris Ostrovsky > > Does anybody have any other ideas? wbinvd is thankfully not common, but also not rare (MTRR setup and a bunch of other cases) and in some other operating systems it happens even more than on Linux.. it's generally not totally broken like this. I can only imagine a machine check case where a write back to a bad cell causes some parity error or something... but it's odd that no other machine checks are reported? (can the user check for this please) _______________________________________________ kexec mailing list kexec@lists.infradead.org http://lists.infradead.org/mailman/listinfo/kexec ^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: kexec reboot fails with extra wbinvd introduced for AME SME 2018-01-18 2:53 ` Arjan van de Ven @ 2018-01-18 2:57 ` Dave Young 2018-01-18 3:00 ` Linus Torvalds 0 siblings, 1 reply; 23+ messages in thread From: Dave Young @ 2018-01-18 2:57 UTC (permalink / raw) To: Arjan van de Ven Cc: Juergen Gross, Tom Lendacky, Tony Luck, Yu Chen, Baoquan He, Kexec Mailing List, Ingo Molnar, Dan Williams, Linux Kernel Mailing List, Rui Zhang, ebiederm, Borislav Petkov, Thomas Gleixner, Linus Torvalds, Boris Ostrovsky On 01/17/18 at 06:53pm, Arjan van de Ven wrote: > > > > Does anybody have any other ideas? > > wbinvd is thankfully not common, but also not rare (MTRR setup and a bunch of other cases) > and in some other operating systems it happens even more than on Linux.. it's generally not totally broken like this. > > I can only imagine a machine check case where a write back to a bad cell causes some parity error > or something... but it's odd that no other machine checks are reported? > (can the user check for this please) Could you say more about how to check this? My .config disabled CONFIG_X86_MCE, should this be enabled? Thanks Dave _______________________________________________ kexec mailing list kexec@lists.infradead.org http://lists.infradead.org/mailman/listinfo/kexec ^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: kexec reboot fails with extra wbinvd introduced for AME SME 2018-01-18 2:57 ` Dave Young @ 2018-01-18 3:00 ` Linus Torvalds 2018-01-18 3:04 ` Dave Young 0 siblings, 1 reply; 23+ messages in thread From: Linus Torvalds @ 2018-01-18 3:00 UTC (permalink / raw) To: Dave Young Cc: Juergen Gross, Tom Lendacky, Tony Luck, Yu Chen, Baoquan He, Kexec Mailing List, Ingo Molnar, Dan Williams, Linux Kernel Mailing List, Rui Zhang, ebiederm, Borislav Petkov, Thomas Gleixner, Arjan van de Ven, Boris Ostrovsky On Wed, Jan 17, 2018 at 6:57 PM, Dave Young <dyoung@redhat.com> wrote: > > Could you say more about how to check this? My .config disabled > CONFIG_X86_MCE, should this be enabled? By all means, try it. Some pending machine check exception that we have *not* reacted to, and that is pending around kexec boot could very possibly be relevant. I forget your exact symptoms. Was it that the _first_ kexec works, but the second one hangs? Or was that the other unrelated issue? Or have I confused this with some other report entirely? Linus _______________________________________________ kexec mailing list kexec@lists.infradead.org http://lists.infradead.org/mailman/listinfo/kexec ^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: kexec reboot fails with extra wbinvd introduced for AME SME 2018-01-18 3:00 ` Linus Torvalds @ 2018-01-18 3:04 ` Dave Young 2018-01-18 3:31 ` Dave Young 0 siblings, 1 reply; 23+ messages in thread From: Dave Young @ 2018-01-18 3:04 UTC (permalink / raw) To: Linus Torvalds Cc: Juergen Gross, Tom Lendacky, Tony Luck, Yu Chen, Baoquan He, Kexec Mailing List, Ingo Molnar, Dan Williams, Linux Kernel Mailing List, Rui Zhang, ebiederm, Borislav Petkov, Thomas Gleixner, Arjan van de Ven, Boris Ostrovsky On 01/17/18 at 07:00pm, Linus Torvalds wrote: > On Wed, Jan 17, 2018 at 6:57 PM, Dave Young <dyoung@redhat.com> wrote: > > > > Could you say more about how to check this? My .config disabled > > CONFIG_X86_MCE, should this be enabled? > > By all means, try it. > > Some pending machine check exception that we have *not* reacted to, > and that is pending around kexec boot could very possibly be relevant. > > I forget your exact symptoms. Was it that the _first_ kexec works, but > the second one hangs? Or was that the other unrelated issue? Or have I > confused this with some other report entirely? Yes, 1st kexec works, 2nd hangs. Rebuilding with MCE enabled now. Thanks Dave > > Linus _______________________________________________ kexec mailing list kexec@lists.infradead.org http://lists.infradead.org/mailman/listinfo/kexec ^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: kexec reboot fails with extra wbinvd introduced for AME SME 2018-01-18 3:04 ` Dave Young @ 2018-01-18 3:31 ` Dave Young 0 siblings, 0 replies; 23+ messages in thread From: Dave Young @ 2018-01-18 3:31 UTC (permalink / raw) To: Linus Torvalds Cc: Juergen Gross, Tom Lendacky, Tony Luck, Yu Chen, Baoquan He, Kexec Mailing List, Ingo Molnar, Dan Williams, Linux Kernel Mailing List, Rui Zhang, ebiederm, Borislav Petkov, Thomas Gleixner, Arjan van de Ven, Boris Ostrovsky On 01/18/18 at 11:04am, Dave Young wrote: > On 01/17/18 at 07:00pm, Linus Torvalds wrote: > > On Wed, Jan 17, 2018 at 6:57 PM, Dave Young <dyoung@redhat.com> wrote: > > > > > > Could you say more about how to check this? My .config disabled > > > CONFIG_X86_MCE, should this be enabled? > > > > By all means, try it. > > > > Some pending machine check exception that we have *not* reacted to, > > and that is pending around kexec boot could very possibly be relevant. > > > > I forget your exact symptoms. Was it that the _first_ kexec works, but > > the second one hangs? Or was that the other unrelated issue? Or have I > > confused this with some other report entirely? > > Yes, 1st kexec works, 2nd hangs. > > Rebuilding with MCE enabled now. dmesg only shows "mce: CPU supports 7 MCE banks" Nothing special happens with MCE enabled , anyway thank all for the hints, I will continue to think about this and maybe try more experiments. > > Thanks > Dave > > > > Linus _______________________________________________ kexec mailing list kexec@lists.infradead.org http://lists.infradead.org/mailman/listinfo/kexec ^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: kexec reboot fails with extra wbinvd introduced for AME SME 2018-01-18 2:14 ` Linus Torvalds ` (2 preceding siblings ...) 2018-01-18 2:53 ` Arjan van de Ven @ 2018-01-18 3:00 ` Arjan van de Ven 3 siblings, 0 replies; 23+ messages in thread From: Arjan van de Ven @ 2018-01-18 3:00 UTC (permalink / raw) To: Linus Torvalds, Dave Young Cc: Juergen Gross, Tom Lendacky, Tony Luck, Yu Chen, Baoquan He, Kexec Mailing List, Ingo Molnar, Dan Williams, Linux Kernel Mailing List, Rui Zhang, ebiederm, Borislav Petkov, Thomas Gleixner, Boris Ostrovsky > Does anybody have any other ideas? the only other weird case that comes to mind; what happens if there's a line dirty in the caches, but the memory is now mapped uncached. (Which could happen if kexec does muck with MTRRs, CR0 or other similar things in weird ways)... not sure what happens in CPU, a machine check for cache inclusion violations is not beyond the imagination and might be lethal this would explain a kexec specific angle versus general normal (but rare) use of wbinvd. other weird case could be cached mmio (not common, but some gpus and the like can do it) with iommu/VT-D in the middle, and during kexec VT-D shutting down the iommu before the wbinvd. This would be... highly odd... but this report already is in highly odd space. _______________________________________________ kexec mailing list kexec@lists.infradead.org http://lists.infradead.org/mailman/listinfo/kexec ^ permalink raw reply [flat|nested] 23+ messages in thread
end of thread, other threads:[~2018-01-18 5:15 UTC | newest] Thread overview: 23+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2017-12-13 2:52 Regression: kexec/kdump boot hangs with x86/vector commits Dave Young 2017-12-13 15:57 ` Yu Chen 2017-12-14 9:24 ` Dave Young 2018-01-04 3:15 ` Dave Young 2018-01-17 7:22 ` kexec reboot fails with extra wbinvd introduced for AME SME Dave Young 2018-01-17 15:06 ` Tom Lendacky 2018-01-18 1:50 ` Dave Young 2018-01-17 19:42 ` Linus Torvalds 2018-01-17 20:01 ` Tom Lendacky 2018-01-17 22:53 ` Tom Lendacky 2018-01-17 20:10 ` Linus Torvalds 2018-01-18 1:47 ` Dave Young 2018-01-18 2:14 ` Linus Torvalds 2018-01-18 2:29 ` Dave Young 2018-01-18 5:14 ` Tom Lendacky 2018-01-18 2:47 ` Dave Young 2018-01-18 2:56 ` Linus Torvalds 2018-01-18 2:53 ` Arjan van de Ven 2018-01-18 2:57 ` Dave Young 2018-01-18 3:00 ` Linus Torvalds 2018-01-18 3:04 ` Dave Young 2018-01-18 3:31 ` Dave Young 2018-01-18 3:00 ` Arjan van de Ven
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).