* [PATCH v2 0/1] igb: Do not bring the device up after non-fatal error @ 2024-09-24 21:06 Mohamed Khalfella 2024-09-24 21:06 ` [PATCH v2 1/1] " Mohamed Khalfella 0 siblings, 1 reply; 6+ messages in thread From: Mohamed Khalfella @ 2024-09-24 21:06 UTC (permalink / raw) To: Tony Nguyen, Przemek Kitszel, David S. Miller, Eric Dumazet, Jakub Kicinski, Paolo Abeni, Jeff Garzik, Yuanyuan Zhong, Auke Kok, Mohamed Khalfella, Simon Horman, Ying Hsu Cc: intel-wired-lan, netdev, linux-kernel Changes: - Changed dev_info() to dev_dbg() as suggested. v1: https://lore.kernel.org/all/20240923212218.116979-1-mkhalfella@purestorage.com/ Mohamed Khalfella (1): igb: Do not bring the device up after non-fatal error drivers/net/ethernet/intel/igb/igb_main.c | 4 ++++ 1 file changed, 4 insertions(+) -- 2.45.2 ^ permalink raw reply [flat|nested] 6+ messages in thread
* [PATCH v2 1/1] igb: Do not bring the device up after non-fatal error 2024-09-24 21:06 [PATCH v2 0/1] igb: Do not bring the device up after non-fatal error Mohamed Khalfella @ 2024-09-24 21:06 ` Mohamed Khalfella 2024-09-25 18:30 ` Simon Horman ` (2 more replies) 0 siblings, 3 replies; 6+ messages in thread From: Mohamed Khalfella @ 2024-09-24 21:06 UTC (permalink / raw) To: Tony Nguyen, Przemek Kitszel, David S. Miller, Eric Dumazet, Jakub Kicinski, Paolo Abeni, Auke Kok, Yuanyuan Zhong, Jeff Garzik, Mohamed Khalfella, Ying Hsu, Simon Horman Cc: intel-wired-lan, netdev, linux-kernel Commit 004d25060c78 ("igb: Fix igb_down hung on surprise removal") changed igb_io_error_detected() to ignore non-fatal pcie errors in order to avoid hung task that can happen when igb_down() is called multiple times. This caused an issue when processing transient non-fatal errors. igb_io_resume(), which is called after igb_io_error_detected(), assumes that device is brought down by igb_io_error_detected() if the interface is up. This resulted in panic with stacktrace below. [ T3256] igb 0000:09:00.0 haeth0: igb: haeth0 NIC Link is Down [ T292] pcieport 0000:00:1c.5: AER: Uncorrected (Non-Fatal) error received: 0000:09:00.0 [ T292] igb 0000:09:00.0: PCIe Bus Error: severity=Uncorrected (Non-Fatal), type=Transaction Layer, (Requester ID) [ T292] igb 0000:09:00.0: device [8086:1537] error status/mask=00004000/00000000 [ T292] igb 0000:09:00.0: [14] CmpltTO [ 200.105524,009][ T292] igb 0000:09:00.0: AER: TLP Header: 00000000 00000000 00000000 00000000 [ T292] pcieport 0000:00:1c.5: AER: broadcast error_detected message [ T292] igb 0000:09:00.0: Non-correctable non-fatal error reported. [ T292] pcieport 0000:00:1c.5: AER: broadcast mmio_enabled message [ T292] pcieport 0000:00:1c.5: AER: broadcast resume message [ T292] ------------[ cut here ]------------ [ T292] kernel BUG at net/core/dev.c:6539! [ T292] invalid opcode: 0000 [#1] PREEMPT SMP [ T292] RIP: 0010:napi_enable+0x37/0x40 [ T292] Call Trace: [ T292] <TASK> [ T292] ? die+0x33/0x90 [ T292] ? do_trap+0xdc/0x110 [ T292] ? napi_enable+0x37/0x40 [ T292] ? do_error_trap+0x70/0xb0 [ T292] ? napi_enable+0x37/0x40 [ T292] ? napi_enable+0x37/0x40 [ T292] ? exc_invalid_op+0x4e/0x70 [ T292] ? napi_enable+0x37/0x40 [ T292] ? asm_exc_invalid_op+0x16/0x20 [ T292] ? napi_enable+0x37/0x40 [ T292] igb_up+0x41/0x150 [ T292] igb_io_resume+0x25/0x70 [ T292] report_resume+0x54/0x70 [ T292] ? report_frozen_detected+0x20/0x20 [ T292] pci_walk_bus+0x6c/0x90 [ T292] ? aer_print_port_info+0xa0/0xa0 [ T292] pcie_do_recovery+0x22f/0x380 [ T292] aer_process_err_devices+0x110/0x160 [ T292] aer_isr+0x1c1/0x1e0 [ T292] ? disable_irq_nosync+0x10/0x10 [ T292] irq_thread_fn+0x1a/0x60 [ T292] irq_thread+0xe3/0x1a0 [ T292] ? irq_set_affinity_notifier+0x120/0x120 [ T292] ? irq_affinity_notify+0x100/0x100 [ T292] kthread+0xe2/0x110 [ T292] ? kthread_complete_and_exit+0x20/0x20 [ T292] ret_from_fork+0x2d/0x50 [ T292] ? kthread_complete_and_exit+0x20/0x20 [ T292] ret_from_fork_asm+0x11/0x20 [ T292] </TASK> To fix this issue igb_io_resume() checks if the interface is running and the device is not down this means igb_io_error_detected() did not bring the device down and there is no need to bring it up. Signed-off-by: Mohamed Khalfella <mkhalfella@purestorage.com> Reviewed-by: Yuanyuan Zhong<yzhong@purestorage.com> Fixes: 004d25060c78 ("igb: Fix igb_down hung on surprise removal") --- drivers/net/ethernet/intel/igb/igb_main.c | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/drivers/net/ethernet/intel/igb/igb_main.c b/drivers/net/ethernet/intel/igb/igb_main.c index 1ef4cb871452..f1d088168723 100644 --- a/drivers/net/ethernet/intel/igb/igb_main.c +++ b/drivers/net/ethernet/intel/igb/igb_main.c @@ -9651,6 +9651,10 @@ static void igb_io_resume(struct pci_dev *pdev) struct igb_adapter *adapter = netdev_priv(netdev); if (netif_running(netdev)) { + if (!test_bit(__IGB_DOWN, &adapter->state)) { + dev_dbg(&pdev->dev, "Resuming from non-fatal error, do nothing.\n"); + return; + } if (igb_up(adapter)) { dev_err(&pdev->dev, "igb_up failed after reset\n"); return; -- 2.45.2 ^ permalink raw reply related [flat|nested] 6+ messages in thread
* Re: [PATCH v2 1/1] igb: Do not bring the device up after non-fatal error 2024-09-24 21:06 ` [PATCH v2 1/1] " Mohamed Khalfella @ 2024-09-25 18:30 ` Simon Horman 2024-09-28 14:40 ` [Intel-wired-lan] " Pucha, HimasekharX Reddy 2024-10-02 12:53 ` Pucha, HimasekharX Reddy 2 siblings, 0 replies; 6+ messages in thread From: Simon Horman @ 2024-09-25 18:30 UTC (permalink / raw) To: Mohamed Khalfella Cc: Tony Nguyen, Przemek Kitszel, David S. Miller, Eric Dumazet, Jakub Kicinski, Paolo Abeni, Auke Kok, Yuanyuan Zhong, Jeff Garzik, Ying Hsu, intel-wired-lan, netdev, linux-kernel On Tue, Sep 24, 2024 at 03:06:01PM -0600, Mohamed Khalfella wrote: > Commit 004d25060c78 ("igb: Fix igb_down hung on surprise removal") > changed igb_io_error_detected() to ignore non-fatal pcie errors in order > to avoid hung task that can happen when igb_down() is called multiple > times. This caused an issue when processing transient non-fatal errors. > igb_io_resume(), which is called after igb_io_error_detected(), assumes > that device is brought down by igb_io_error_detected() if the interface > is up. This resulted in panic with stacktrace below. > > [ T3256] igb 0000:09:00.0 haeth0: igb: haeth0 NIC Link is Down > [ T292] pcieport 0000:00:1c.5: AER: Uncorrected (Non-Fatal) error received: 0000:09:00.0 > [ T292] igb 0000:09:00.0: PCIe Bus Error: severity=Uncorrected (Non-Fatal), type=Transaction Layer, (Requester ID) > [ T292] igb 0000:09:00.0: device [8086:1537] error status/mask=00004000/00000000 > [ T292] igb 0000:09:00.0: [14] CmpltTO [ 200.105524,009][ T292] igb 0000:09:00.0: AER: TLP Header: 00000000 00000000 00000000 00000000 > [ T292] pcieport 0000:00:1c.5: AER: broadcast error_detected message > [ T292] igb 0000:09:00.0: Non-correctable non-fatal error reported. > [ T292] pcieport 0000:00:1c.5: AER: broadcast mmio_enabled message > [ T292] pcieport 0000:00:1c.5: AER: broadcast resume message > [ T292] ------------[ cut here ]------------ > [ T292] kernel BUG at net/core/dev.c:6539! > [ T292] invalid opcode: 0000 [#1] PREEMPT SMP > [ T292] RIP: 0010:napi_enable+0x37/0x40 > [ T292] Call Trace: > [ T292] <TASK> > [ T292] ? die+0x33/0x90 > [ T292] ? do_trap+0xdc/0x110 > [ T292] ? napi_enable+0x37/0x40 > [ T292] ? do_error_trap+0x70/0xb0 > [ T292] ? napi_enable+0x37/0x40 > [ T292] ? napi_enable+0x37/0x40 > [ T292] ? exc_invalid_op+0x4e/0x70 > [ T292] ? napi_enable+0x37/0x40 > [ T292] ? asm_exc_invalid_op+0x16/0x20 > [ T292] ? napi_enable+0x37/0x40 > [ T292] igb_up+0x41/0x150 > [ T292] igb_io_resume+0x25/0x70 > [ T292] report_resume+0x54/0x70 > [ T292] ? report_frozen_detected+0x20/0x20 > [ T292] pci_walk_bus+0x6c/0x90 > [ T292] ? aer_print_port_info+0xa0/0xa0 > [ T292] pcie_do_recovery+0x22f/0x380 > [ T292] aer_process_err_devices+0x110/0x160 > [ T292] aer_isr+0x1c1/0x1e0 > [ T292] ? disable_irq_nosync+0x10/0x10 > [ T292] irq_thread_fn+0x1a/0x60 > [ T292] irq_thread+0xe3/0x1a0 > [ T292] ? irq_set_affinity_notifier+0x120/0x120 > [ T292] ? irq_affinity_notify+0x100/0x100 > [ T292] kthread+0xe2/0x110 > [ T292] ? kthread_complete_and_exit+0x20/0x20 > [ T292] ret_from_fork+0x2d/0x50 > [ T292] ? kthread_complete_and_exit+0x20/0x20 > [ T292] ret_from_fork_asm+0x11/0x20 > [ T292] </TASK> > > To fix this issue igb_io_resume() checks if the interface is running and > the device is not down this means igb_io_error_detected() did not bring > the device down and there is no need to bring it up. > > Signed-off-by: Mohamed Khalfella <mkhalfella@purestorage.com> > Reviewed-by: Yuanyuan Zhong<yzhong@purestorage.com> > Fixes: 004d25060c78 ("igb: Fix igb_down hung on surprise removal") Thanks for the update, this version looks good to me. Reviewed-by: Simon Horman <horms@kernel.org> ^ permalink raw reply [flat|nested] 6+ messages in thread
* RE: [Intel-wired-lan] [PATCH v2 1/1] igb: Do not bring the device up after non-fatal error 2024-09-24 21:06 ` [PATCH v2 1/1] " Mohamed Khalfella 2024-09-25 18:30 ` Simon Horman @ 2024-09-28 14:40 ` Pucha, HimasekharX Reddy 2024-09-30 18:27 ` Mohamed Khalfella 2024-10-02 12:53 ` Pucha, HimasekharX Reddy 2 siblings, 1 reply; 6+ messages in thread From: Pucha, HimasekharX Reddy @ 2024-09-28 14:40 UTC (permalink / raw) To: Mohamed Khalfella, Nguyen, Anthony L, Kitszel, Przemyslaw, David S. Miller, Eric Dumazet, Jakub Kicinski, Paolo Abeni, Auke Kok, Zhong, YuanYuan, Jeff Garzik, Ying Hsu, Simon Horman Cc: netdev@vger.kernel.org, intel-wired-lan@lists.osuosl.org, linux-kernel@vger.kernel.org >-----Original Message----- >From: Intel-wired-lan <intel-wired-lan-bounces@osuosl.org> On Behalf Of Mohamed Khalfella > Sent: Wednesday, September 25, 2024 2:36 AM > To: Nguyen, Anthony L <anthony.l.nguyen@intel.com>; Kitszel, Przemyslaw <przemyslaw.kitszel@intel.com>; David S. Miller <davem@davemloft.net>; Eric Dumazet <edumazet@google.com>; Jakub Kicinski <kuba@kernel.org>; Paolo Abeni <pabeni@redhat.com>; Auke Kok <auke-jan.h.kok@intel.com>; Zhong, YuanYuan <yzhong@purestorage.com>; Jeff Garzik <jgarzik@redhat.com>; Mohamed Khalfella <mkhalfella@purestorage.com>; Ying Hsu <yinghsu@chromium.org>; Simon Horman <horms@kernel.org> > Cc: netdev@vger.kernel.org; intel-wired-lan@lists.osuosl.org; linux-kernel@vger.kernel.org > Subject: [Intel-wired-lan] [PATCH v2 1/1] igb: Do not bring the device up after non-fatal error > >Commit 004d25060c78 ("igb: Fix igb_down hung on surprise removal") changed igb_io_error_detected() to ignore non-fatal pcie errors in order to avoid hung task that can happen when igb_down() is called multiple times. This caused an issue when processing transient non-fatal errors. > igb_io_resume(), which is called after igb_io_error_detected(), assumes that device is brought down by igb_io_error_detected() if the interface is up. This resulted in panic with stacktrace below. > > [ T3256] igb 0000:09:00.0 haeth0: igb: haeth0 NIC Link is Down [ T292] pcieport 0000:00:1c.5: AER: Uncorrected (Non-Fatal) error received: 0000:09:00.0 [ T292] igb 0000:09:00.0: PCIe Bus Error: severity=Uncorrected (Non-Fatal), type=Transaction Layer, (Requester ID) > [ T292] igb 0000:09:00.0: device [8086:1537] error status/mask=00004000/00000000 > [ T292] igb 0000:09:00.0: [14] CmpltTO [ 200.105524,009][ T292] igb 0000:09:00.0: AER: TLP Header: 00000000 00000000 00000000 00000000 > [ T292] pcieport 0000:00:1c.5: AER: broadcast error_detected message [ T292] igb 0000:09:00.0: Non-correctable non-fatal error reported. > [ T292] pcieport 0000:00:1c.5: AER: broadcast mmio_enabled message [ T292] pcieport 0000:00:1c.5: AER: broadcast resume message [ T292] ------------[ cut here ]------------ [ T292] kernel BUG at net/core/dev.c:6539! > [ T292] invalid opcode: 0000 [#1] PREEMPT SMP [ T292] RIP: 0010:napi_enable+0x37/0x40 [ T292] Call Trace: > [ T292] <TASK> > [ T292] ? die+0x33/0x90 > [ T292] ? do_trap+0xdc/0x110 > [ T292] ? napi_enable+0x37/0x40 > [ T292] ? do_error_trap+0x70/0xb0 > [ T292] ? napi_enable+0x37/0x40 > [ T292] ? napi_enable+0x37/0x40 > [ T292] ? exc_invalid_op+0x4e/0x70 > [ T292] ? napi_enable+0x37/0x40 > [ T292] ? asm_exc_invalid_op+0x16/0x20 [ T292] ? napi_enable+0x37/0x40 [ T292] igb_up+0x41/0x150 [ T292] igb_io_resume+0x25/0x70 [ T292] report_resume+0x54/0x70 [ T292] ? report_frozen_detected+0x20/0x20 [ T292] pci_walk_bus+0x6c/0x90 [ T292] ? aer_print_port_info+0xa0/0xa0 [ T292] pcie_do_recovery+0x22f/0x380 [ T292] aer_process_err_devices+0x110/0x160 > [ T292] aer_isr+0x1c1/0x1e0 > [ T292] ? disable_irq_nosync+0x10/0x10 [ T292] irq_thread_fn+0x1a/0x60 [ T292] irq_thread+0xe3/0x1a0 [ T292] ? irq_set_affinity_notifier+0x120/0x120 > [ T292] ? irq_affinity_notify+0x100/0x100 [ T292] kthread+0xe2/0x110 [ T292] ? kthread_complete_and_exit+0x20/0x20 > [ T292] ret_from_fork+0x2d/0x50 > [ T292] ? kthread_complete_and_exit+0x20/0x20 > [ T292] ret_from_fork_asm+0x11/0x20 > [ T292] </TASK> > > To fix this issue igb_io_resume() checks if the interface is running and the device is not down this means igb_io_error_detected() did not bring the device down and there is no need to bring it up. > > Signed-off-by: Mohamed Khalfella <mkhalfella@purestorage.com> > Reviewed-by: Yuanyuan Zhong<yzhong@purestorage.com> > Fixes: 004d25060c78 ("igb: Fix igb_down hung on surprise removal") > --- > drivers/net/ethernet/intel/igb/igb_main.c | 4 ++++ > 1 file changed, 4 insertions(+) Any reproductions steps for reproduction of these issue? ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [Intel-wired-lan] [PATCH v2 1/1] igb: Do not bring the device up after non-fatal error 2024-09-28 14:40 ` [Intel-wired-lan] " Pucha, HimasekharX Reddy @ 2024-09-30 18:27 ` Mohamed Khalfella 0 siblings, 0 replies; 6+ messages in thread From: Mohamed Khalfella @ 2024-09-30 18:27 UTC (permalink / raw) To: Pucha, HimasekharX Reddy Cc: Nguyen, Anthony L, Kitszel, Przemyslaw, David S. Miller, Eric Dumazet, Jakub Kicinski, Paolo Abeni, Auke Kok, Zhong, YuanYuan, Jeff Garzik, Ying Hsu, Simon Horman, netdev@vger.kernel.org, intel-wired-lan@lists.osuosl.org, linux-kernel@vger.kernel.org On 2024-09-28 14:40:05 +0000, Pucha, HimasekharX Reddy wrote: > >-----Original Message----- > >From: Intel-wired-lan <intel-wired-lan-bounces@osuosl.org> On Behalf Of Mohamed Khalfella > > Sent: Wednesday, September 25, 2024 2:36 AM > > To: Nguyen, Anthony L <anthony.l.nguyen@intel.com>; Kitszel, Przemyslaw <przemyslaw.kitszel@intel.com>; David S. Miller <davem@davemloft.net>; Eric Dumazet <edumazet@google.com>; Jakub Kicinski <kuba@kernel.org>; Paolo Abeni <pabeni@redhat.com>; Auke Kok <auke-jan.h.kok@intel.com>; Zhong, YuanYuan <yzhong@purestorage.com>; Jeff Garzik <jgarzik@redhat.com>; Mohamed Khalfella <mkhalfella@purestorage.com>; Ying Hsu <yinghsu@chromium.org>; Simon Horman <horms@kernel.org> > > Cc: netdev@vger.kernel.org; intel-wired-lan@lists.osuosl.org; linux-kernel@vger.kernel.org > > Subject: [Intel-wired-lan] [PATCH v2 1/1] igb: Do not bring the device up after non-fatal error > > > >Commit 004d25060c78 ("igb: Fix igb_down hung on surprise removal") changed igb_io_error_detected() to ignore non-fatal pcie errors in order to avoid hung task that can happen when igb_down() is called multiple times. This caused an issue when processing transient non-fatal errors. > > igb_io_resume(), which is called after igb_io_error_detected(), assumes that device is brought down by igb_io_error_detected() if the interface is up. This resulted in panic with stacktrace below. > > > > [ T3256] igb 0000:09:00.0 haeth0: igb: haeth0 NIC Link is Down [ T292] pcieport 0000:00:1c.5: AER: Uncorrected (Non-Fatal) error received: 0000:09:00.0 [ T292] igb 0000:09:00.0: PCIe Bus Error: severity=Uncorrected (Non-Fatal), type=Transaction Layer, (Requester ID) > > [ T292] igb 0000:09:00.0: device [8086:1537] error status/mask=00004000/00000000 > > [ T292] igb 0000:09:00.0: [14] CmpltTO [ 200.105524,009][ T292] igb 0000:09:00.0: AER: TLP Header: 00000000 00000000 00000000 00000000 > > [ T292] pcieport 0000:00:1c.5: AER: broadcast error_detected message [ T292] igb 0000:09:00.0: Non-correctable non-fatal error reported. > > [ T292] pcieport 0000:00:1c.5: AER: broadcast mmio_enabled message [ T292] pcieport 0000:00:1c.5: AER: broadcast resume message [ T292] ------------[ cut here ]------------ [ T292] kernel BUG at net/core/dev.c:6539! > > [ T292] invalid opcode: 0000 [#1] PREEMPT SMP [ T292] RIP: 0010:napi_enable+0x37/0x40 [ T292] Call Trace: > > [ T292] <TASK> > > [ T292] ? die+0x33/0x90 > > [ T292] ? do_trap+0xdc/0x110 > > [ T292] ? napi_enable+0x37/0x40 > > [ T292] ? do_error_trap+0x70/0xb0 > > [ T292] ? napi_enable+0x37/0x40 > > [ T292] ? napi_enable+0x37/0x40 > > [ T292] ? exc_invalid_op+0x4e/0x70 > > [ T292] ? napi_enable+0x37/0x40 > > [ T292] ? asm_exc_invalid_op+0x16/0x20 [ T292] ? napi_enable+0x37/0x40 [ T292] igb_up+0x41/0x150 [ T292] igb_io_resume+0x25/0x70 [ T292] report_resume+0x54/0x70 [ T292] ? report_frozen_detected+0x20/0x20 [ T292] pci_walk_bus+0x6c/0x90 [ T292] ? aer_print_port_info+0xa0/0xa0 [ T292] pcie_do_recovery+0x22f/0x380 [ T292] aer_process_err_devices+0x110/0x160 > > [ T292] aer_isr+0x1c1/0x1e0 > > [ T292] ? disable_irq_nosync+0x10/0x10 [ T292] irq_thread_fn+0x1a/0x60 [ T292] irq_thread+0xe3/0x1a0 [ T292] ? irq_set_affinity_notifier+0x120/0x120 > > [ T292] ? irq_affinity_notify+0x100/0x100 [ T292] kthread+0xe2/0x110 [ T292] ? kthread_complete_and_exit+0x20/0x20 > > [ T292] ret_from_fork+0x2d/0x50 > > [ T292] ? kthread_complete_and_exit+0x20/0x20 > > [ T292] ret_from_fork_asm+0x11/0x20 > > [ T292] </TASK> > > > > To fix this issue igb_io_resume() checks if the interface is running and the device is not down this means igb_io_error_detected() did not bring the device down and there is no need to bring it up. > > > > Signed-off-by: Mohamed Khalfella <mkhalfella@purestorage.com> > > Reviewed-by: Yuanyuan Zhong<yzhong@purestorage.com> > > Fixes: 004d25060c78 ("igb: Fix igb_down hung on surprise removal") > > --- > > drivers/net/ethernet/intel/igb/igb_main.c | 4 ++++ > > 1 file changed, 4 insertions(+) > > Any reproductions steps for reproduction of these issue? > I know of two way to reproduce this kernel panic on a kernel that does not have the patch above. 1- Using aer-inject: This works on both physical machine and vm. Here are the steps on a virtual machine. root@(none):~# lspci -t -v -s 03:00.0 -[0000:02]---00.0-[03]----00.0 Intel Corporation 82576 Gigabit Network Connection root@(none):~# cat > /tmp/uncor << EOF > AER > UNCOR_STATUS COMP_ABORT > HEADER_LOG 0 1 2 3 > EOF root@(none):~# modprobe aer_inject root@(none):~# /var/tmp/aer-inject --id=0000:03:00.0 /tmp/uncor This is the qemu command used to start the vm. You probably do not need all the options related to numa settings and iommu. Only the part related to pci setup should be enough. /usr/bin/qemu-system-x86_64 \ -kernel $SRCDIR/arch/x86/boot/bzImage \ -initrd $INITRAMFSIMG \ -append "rdinit=/startup.sh console=ttyS0,115200n8" \ -machine q35,accel=kvm,kernel-irqchip=split \ -nographic \ -chardev socket,id=gdb0,host=0.0.0.0,port=22004,telnet=on,server=on,wait=off \ -gdb chardev:gdb0 \ $GDB_WAIT \ -serial telnet:127.0.0.1:22003,server=on,wait=off \ -device pxb-pcie,id=pcie.1,bus_nr=2,bus=pcie.0 \ -device ioh3420,id=pcie_port1,bus=pcie.1,chassis=1 \ -netdev user,id=net0,hostfwd=tcp:127.0.0.1:22002-:22 \ -device igb,netdev=net0,id=net0,mac=52:54:00:b8:9c:58,bus=pcie_port1 \ -cpu host \ -smp 4 \ -m 8G \ -object memory-backend-ram,size=7G,id=m0 \ -object memory-backend-ram,size=1G,id=m1 \ -numa node,nodeid=0,memdev=m0,cpus=0-1 \ -numa node,nodeid=1,memdev=m1,cpus=2-3 \ -chardev socket,id=charmonitor,host=0.0.0.0,port=10001,telnet=on,server=on,wait=off \ -mon chardev=charmonitor,id=monitor \ -boot order=c \ -device intel-iommu,intremap=on 2- Using pcie_aer_inject_error Injecting pcie aer error from qemu monitor should be enough to trigger the kernel panic. This is using the qemu command above. (qemu) pcie_aer_inject_error pcie_port1 0x00004000 ^ permalink raw reply [flat|nested] 6+ messages in thread
* RE: [Intel-wired-lan] [PATCH v2 1/1] igb: Do not bring the device up after non-fatal error 2024-09-24 21:06 ` [PATCH v2 1/1] " Mohamed Khalfella 2024-09-25 18:30 ` Simon Horman 2024-09-28 14:40 ` [Intel-wired-lan] " Pucha, HimasekharX Reddy @ 2024-10-02 12:53 ` Pucha, HimasekharX Reddy 2 siblings, 0 replies; 6+ messages in thread From: Pucha, HimasekharX Reddy @ 2024-10-02 12:53 UTC (permalink / raw) To: Mohamed Khalfella, Nguyen, Anthony L, Kitszel, Przemyslaw, David S. Miller, Eric Dumazet, Jakub Kicinski, Paolo Abeni, Auke Kok, Zhong, YuanYuan, Jeff Garzik, Ying Hsu, Simon Horman Cc: netdev@vger.kernel.org, intel-wired-lan@lists.osuosl.org, linux-kernel@vger.kernel.org > -----Original Message----- > From: Intel-wired-lan <intel-wired-lan-bounces@osuosl.org> On Behalf Of Mohamed Khalfella > Sent: Wednesday, September 25, 2024 2:36 AM > To: Nguyen, Anthony L <anthony.l.nguyen@intel.com>; Kitszel, Przemyslaw <przemyslaw.kitszel@intel.com>; David S. Miller <davem@davemloft.net>; Eric Dumazet <edumazet@google.com>; Jakub Kicinski <kuba@kernel.org>; Paolo Abeni <pabeni@redhat.com>; Auke Kok <auke-jan.h.kok@intel.com>; Zhong, YuanYuan <yzhong@purestorage.com>; Jeff Garzik <jgarzik@redhat.com>; Mohamed Khalfella <mkhalfella@purestorage.com>; Ying Hsu <yinghsu@chromium.org>; Simon Horman <horms@kernel.org> > Cc: netdev@vger.kernel.org; intel-wired-lan@lists.osuosl.org; linux-kernel@vger.kernel.org > Subject: [Intel-wired-lan] [PATCH v2 1/1] igb: Do not bring the device up after non-fatal error > > Commit 004d25060c78 ("igb: Fix igb_down hung on surprise removal") changed igb_io_error_detected() to ignore non-fatal pcie errors in order to avoid hung task that can happen when igb_down() is called multiple times. This caused an issue when processing transient non-fatal errors. > igb_io_resume(), which is called after igb_io_error_detected(), assumes that device is brought down by igb_io_error_detected() if the interface is up. This resulted in panic with stacktrace below. > > [ T3256] igb 0000:09:00.0 haeth0: igb: haeth0 NIC Link is Down [ T292] pcieport 0000:00:1c.5: AER: Uncorrected (Non-Fatal) error received: 0000:09:00.0 [ T292] igb 0000:09:00.0: PCIe Bus Error: severity=Uncorrected (Non-Fatal), type=Transaction Layer, (Requester ID) > [ T292] igb 0000:09:00.0: device [8086:1537] error status/mask=00004000/00000000 > [ T292] igb 0000:09:00.0: [14] CmpltTO [ 200.105524,009][ T292] igb 0000:09:00.0: AER: TLP Header: 00000000 00000000 00000000 00000000 > [ T292] pcieport 0000:00:1c.5: AER: broadcast error_detected message [ T292] igb 0000:09:00.0: Non-correctable non-fatal error reported. > [ T292] pcieport 0000:00:1c.5: AER: broadcast mmio_enabled message [ T292] pcieport 0000:00:1c.5: AER: broadcast resume message [ T292] ------------[ cut here ]------------ [ T292] kernel BUG at net/core/dev.c:6539! > [ T292] invalid opcode: 0000 [#1] PREEMPT SMP [ T292] RIP: 0010:napi_enable+0x37/0x40 [ T292] Call Trace: > [ T292] <TASK> > [ T292] ? die+0x33/0x90 > [ T292] ? do_trap+0xdc/0x110 > [ T292] ? napi_enable+0x37/0x40 > [ T292] ? do_error_trap+0x70/0xb0 > [ T292] ? napi_enable+0x37/0x40 > [ T292] ? napi_enable+0x37/0x40 > [ T292] ? exc_invalid_op+0x4e/0x70 > [ T292] ? napi_enable+0x37/0x40 > [ T292] ? asm_exc_invalid_op+0x16/0x20 [ T292] ? napi_enable+0x37/0x40 [ T292] igb_up+0x41/0x150 [ T292] igb_io_resume+0x25/0x70 [ T292] report_resume+0x54/0x70 [ T292] ? report_frozen_detected+0x20/0x20 [ T292] pci_walk_bus+0x6c/0x90 [ T292] ? aer_print_port_info+0xa0/0xa0 [ T292] pcie_do_recovery+0x22f/0x380 [ T292] aer_process_err_devices+0x110/0x160 > [ T292] aer_isr+0x1c1/0x1e0 > [ T292] ? disable_irq_nosync+0x10/0x10 [ T292] irq_thread_fn+0x1a/0x60 [ T292] irq_thread+0xe3/0x1a0 [ T292] ? irq_set_affinity_notifier+0x120/0x120 > [ T292] ? irq_affinity_notify+0x100/0x100 [ T292] kthread+0xe2/0x110 [ T292] ? kthread_complete_and_exit+0x20/0x20 > [ T292] ret_from_fork+0x2d/0x50 > [ T292] ? kthread_complete_and_exit+0x20/0x20 > [ T292] ret_from_fork_asm+0x11/0x20 > [ T292] </TASK> > > To fix this issue igb_io_resume() checks if the interface is running and the device is not down this means igb_io_error_detected() did not bring the device down and there is no need to bring it up. > > Signed-off-by: Mohamed Khalfella <mkhalfella@purestorage.com> > Reviewed-by: Yuanyuan Zhong<yzhong@purestorage.com> > Fixes: 004d25060c78 ("igb: Fix igb_down hung on surprise removal") > --- > drivers/net/ethernet/intel/igb/igb_main.c | 4 ++++ > 1 file changed, 4 insertions(+) > Tested-by: Pucha Himasekhar Reddy <himasekharx.reddy.pucha@intel.com> (A Contingent worker at Intel) ^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2024-10-02 12:54 UTC | newest] Thread overview: 6+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2024-09-24 21:06 [PATCH v2 0/1] igb: Do not bring the device up after non-fatal error Mohamed Khalfella 2024-09-24 21:06 ` [PATCH v2 1/1] " Mohamed Khalfella 2024-09-25 18:30 ` Simon Horman 2024-09-28 14:40 ` [Intel-wired-lan] " Pucha, HimasekharX Reddy 2024-09-30 18:27 ` Mohamed Khalfella 2024-10-02 12:53 ` Pucha, HimasekharX Reddy
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).