From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1752056AbbKIUV5 (ORCPT ); Mon, 9 Nov 2015 15:21:57 -0500 Received: from bh-25.webhostbox.net ([208.91.199.152]:59004 "EHLO bh-25.webhostbox.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1750997AbbKIUVy (ORCPT ); Mon, 9 Nov 2015 15:21:54 -0500 Date: Mon, 9 Nov 2015 12:21:49 -0800 From: Guenter Roeck To: Jiang Liu Cc: fandongdong , Alex Williamson , Joerg Roedeljoro , linux-kernel@vger.kernel.org Subject: Re: Panic when cpu hot-remove Message-ID: <20151109202149.GA28496@roeck-us.net> References: <42BB8332972FC149B81C55A0D41E3A79C07469@jtjnmailbox06.home.langchao.com> <20150617115238.GC27750@8bytes.org> <1434551800.5628.5.camel@redhat.com> <558259BD.7080402@linux.intel.com> <558272E3.4000504@inspur.com> <55827927.4080504@inspur.com> <558BB7B8.7000402@linux.intel.com> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: <558BB7B8.7000402@linux.intel.com> User-Agent: Mutt/1.5.23 (2014-03-12) X-Authenticated_sender: guenter@roeck-us.net X-OutGoing-Spam-Status: No, score=-1.0 X-AntiAbuse: This header was added to track abuse, please include it with any abuse report X-AntiAbuse: Primary Hostname - bh-25.webhostbox.net X-AntiAbuse: Original Domain - vger.kernel.org X-AntiAbuse: Originator/Caller UID/GID - [47 12] / [47 12] X-AntiAbuse: Sender Address Domain - roeck-us.net X-Get-Message-Sender-Via: bh-25.webhostbox.net: authenticated_id: guenter@roeck-us.net X-Source: X-Source-Args: X-Source-Dir: Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Gerry, On Thu, Jun 25, 2015 at 04:11:36PM +0800, Jiang Liu wrote: > On 2015/6/18 15:54, fandongdong wrote: > > > > > > 在 2015/6/18 15:27, fandongdong 写道: > >> > >> > >> 在 2015/6/18 13:40, Jiang Liu 写道: > >>> On 2015/6/17 22:36, Alex Williamson wrote: > >>>> On Wed, 2015-06-17 at 13:52 +0200, Joerg Roedeljoro wrote: > >>>>> On Wed, Jun 17, 2015 at 10:42:49AM +0000, 范冬冬 wrote: > >>>>>> Hi maintainer, > >>>>>> > >>>>>> We found a problem that a panic happen when cpu was hot-removed. > >>>>>> We also trace the problem according to the calltrace information. > >>>>>> An endless loop happen because value head is not equal to value > >>>>>> tail forever in the function qi_check_fault( ). > >>>>>> The location code is as follows: > >>>>>> > >>>>>> > >>>>>> do { > >>>>>> if (qi->desc_status[head] == QI_IN_USE) > >>>>>> qi->desc_status[head] = QI_ABORT; > >>>>>> head = (head - 2 + QI_LENGTH) % QI_LENGTH; > >>>>>> } while (head != tail); > >>>>> Hmm, this code interates only over every second QI descriptor, and > >>>>> tail > >>>>> probably points to a descriptor that is not iterated over. > >>>>> > >>>>> Jiang, can you please have a look? > >>>> I think that part is normal, the way we use the queue is to always > >>>> submit a work operation followed by a wait operation so that we can > >>>> determine the work operation is complete. That's done via > >>>> qi_submit_sync(). We have had spurious reports of the queue getting > >>>> impossibly out of sync though. I saw one that was somehow linked to > >>>> the > >>>> I/O AT DMA engine. Roland Dreier saw something similar[1]. I'm not > >>>> sure if they're related to this, but maybe worth comparing. Thanks, > >>> Thanks, Alex and Joerg! > >>> > >>> Hi Dongdong, > >>> Could you please help to give some instructions about how to > >>> reproduce this issue? I will try to reproduce it if possible. > >>> Thanks! > >>> Gerry > >> Hi Gerry, > >> > >> We're running kernel 4.1.0 on a 4-socket system and we want to > >> offline socket 1. > >> Steps as follows: > >> > >> echo 1 > /sys/firmware/acpi/hotplug/force_remove > >> echo 1 > /sys/devices/LNXSYSTM:00/LNXSYBUS:00/ACPI0004:01/eject > Hi Dongdong, > I failed to reproduce this issue on my side. Some please help > to confirm? > 1) Is this issue reproducible on your side? > 2) Does this issue happen if you disable irqbalance service on you > system? > 3) Has the corresponding PCI host bridge been removed before removing > the socket? > > From the log message, we only noticed log messages for CPU and memory, > but not messages for PCI (IOMMU) devices. And this log message > "[ 149.976493] acpi ACPI0004:01: Still not present" > implies that the socket has been powered off during the ejection. > So the story may be that you powered off the socket while the host > bridge on the socket is still in use. > Thanks! > Gerry > Was this problem ever resolved ? We are seeing the same (or a similar) problem randomly with our hardware. No CPU hotplug is involved. Any idea what I can do (or help) to track down the problem ? Thanks, Guenter --- Sample traceback: [ 485.547997] Uhhuh. NMI received for unknown reason 29 on CPU 0. [ 485.633519] Do you have a strange power saving mode enabled? [ 485.715262] Kernel panic - not syncing: NMI: Not continuing^M [ 485.795750] CPU: 0 PID: 25109 Comm: cty Tainted: P W O 4.1.12-juniper-00687-g3de457e-dirty #1 [ 485.932825] Hardware name: Juniper Networks, Inc. 0576/HSW RCB PTX, BIOS NGRE_v0.44 04/07/2015 [ 486.057327] 0000000000000029 ffff88085f605df8 ffffffff80a9e179 0000000000000000 [ 486.164220] ffffffff80e53b4a ffff88085f605e78 ffffffff80a99b6f ffff88085f605e18^M [ 486.271116] ffffffff00000008 ffff88085f605e88 ffff88085f605e28 ffffffff81019a00 [ 486.378012] Call Trace: [ 486.413225] [] dump_stack+0x4f/0x7b [ 486.496228] [] panic+0xbb/0x1e9 [ 486.565393] [] unknown_nmi_error+0x9c/0xa0 [ 486.648394] [] default_do_nmi+0x19c/0x1c0 [ 486.730138] [] do_nmi+0xe6/0x160 [ 486.800564] [] end_repeat_nmi+0x1a/0x1e [ 486.879793] [] ? qi_submit_sync+0x186/0x3f0 [ 486.964051] [] ? qi_submit_sync+0x186/0x3f0 [ 487.048307] [] ? qi_submit_sync+0x186/0x3f0 [ 487.132564] <> [] modify_irte+0x93/0xd0 [ 487.219342] [] intel_ioapic_set_affinity+0x113/0x1a0 [ 487.314918] [] set_remapped_irq_affinity+0x20/0x30 [ 487.407979] [] irq_do_set_affinity+0x1c/0x50 [ 487.493494] [] setup_affinity+0x5d/0x80 [ 487.572725] [] __setup_irq+0x2c4/0x580 [ 487.650695] [] ? serial8250_modem_status+0xd0/0xd0 [ 487.743755] [] request_threaded_irq+0xf4/0x1b0 [ 487.831786] [] univ8250_setup_irq+0x24f/0x290 [ 487.918560] [] serial8250_do_startup+0x117/0x5f0 [ 488.009108] [] serial8250_startup+0x25/0x30 [ 488.093365] [] uart_startup.part.16+0x89/0x1f0 [ 488.181397] [] uart_open+0x115/0x160 [ 488.256852] [] ? check_tty_count+0x57/0xc0 [ 488.339854] [] tty_open+0xcc/0x610 [ 488.412793] [] ? kobj_lookup+0x112/0x170 [ 488.493283] [] chrdev_open+0x9f/0x1d0 [ 488.569992] [] do_dentry_open+0x217/0x340 [ 488.651735] [] ? cdev_put+0x30/0x30 [ 488.725934] [] vfs_open+0x57/0x60 [ 488.797616] [] do_last+0x3fb/0xee0 [ 488.870557] [] path_openat+0x80/0x640^M [ 488.947270] [] do_filp_open+0x3a/0x90 [ 489.023984] [] ? _raw_spin_unlock+0x18/0x40 [ 489.108240] [] ? __alloc_fd+0xa7/0x130 [ 489.186213] [] do_sys_open+0x129/0x220^M [ 489.264184] [] compat_SyS_open+0x1b/0x20 [ 489.344670] [] ia32_do_call+0x13/0x13 --- Similar traceback, but during PCIe hotplug: Call Trace: [] dump_stack+0x4f/0x7b^M [] panic+0xbb/0x1df [] unknown_nmi_error+0x9c/0xa0 [] default_do_nmi+0x19c/0x1c0 [] do_nmi+0xe6/0x160^M [] end_repeat_nmi+0x1a/0x1e [] ? qi_submit_sync+0x186/0x3f0 [] ? qi_submit_sync+0x186/0x3f0 [] ? qi_submit_sync+0x186/0x3f0 <> [] free_irte+0xe5/0x130 [] free_remapped_irq+0x2f/0x40^M [] arch_teardown_hwirq+0x23/0x70 [] irq_free_hwirqs+0x38/0x60 [] native_teardown_msi_irq+0x13/0x20 [] arch_teardown_msi_irq+0xf/0x20 [] default_teardown_msi_irqs+0x5f/0xa0 [] arch_teardown_msi_irqs+0xf/0x20 [] free_msi_irqs+0x89/0x1a0 [] pci_disable_msi+0x45/0x50 [] cleanup_service_irqs+0x25/0x40 [] pcie_port_device_remove+0x2e/0x40 [] pcie_portdrv_remove+0xe/0x10 --- Similar, but at another location in qi_submit_sync: Call Trace: [] dump_stack+0x4f/0x7b^M [] panic+0xbb/0x1df [] unknown_nmi_error+0x9c/0xa0 [] default_do_nmi+0x19c/0x1c0 [] do_nmi+0xe6/0x160^M [] end_repeat_nmi+0x1a/0x1e [] ? _raw_spin_lock+0x38/0x40^M [] ? _raw_spin_lock+0x38/0x40 [] ? _raw_spin_lock+0x38/0x40 <> [] qi_submit_sync+0x25d/0x3f0 [] free_irte+0xe5/0x130 [] free_remapped_irq+0x2f/0x40 [] arch_teardown_hwirq+0x23/0x70 [] irq_free_hwirqs+0x38/0x60 [] native_teardown_msi_irq+0x13/0x20 [] arch_teardown_msi_irq+0xf/0x20 [] default_teardown_msi_irqs+0x5f/0xa0 [] arch_teardown_msi_irqs+0xf/0x20 [] free_msi_irqs+0x89/0x1a0 [] pci_disable_msi+0x45/0x50 [] cleanup_service_irqs+0x25/0x40 [] pcie_port_device_remove+0x2e/0x40 [] pcie_portdrv_remove+0xe/0x10 [] pci_device_remove+0x3d/0xc0 The NMIs during PCIe hotplug seem to be more likely (possibly because our testing generates a large number of PCIe hotplug events). --- CPU information: processor : 0 vendor_id : GenuineIntel cpu family : 6 model : 63 model name : Genuine Intel(R) CPU @ 1.80GHz stepping : 1 microcode : 0x14