* Crash in mlx4 shutdown with 4.9-rc3 @ 2016-11-04 14:29 Steve Wise 2016-11-05 13:15 ` Leon Romanovsky 0 siblings, 1 reply; 3+ messages in thread From: Steve Wise @ 2016-11-04 14:29 UTC (permalink / raw) To: yishaih-VPRAkNaXOzVWk0Htik3J/w; +Cc: linux-rdma-u79uwXL29TY76Z2rM5mHXA Hey Yishai, Is this by chance a known bug having a pending fix somewhere? I'm seeing it frequently when shutting down. I'm using 4.9-rc3 with memory debugging enabled... [59984.502834] mlx4_core 0000:81:00.0: mlx4_shutdown was called [59984.603599] mlx4_en 0000:81:00.0: removed PHC [59985.145590] general protection fault: 0000 [#1] SMP DEBUG_PAGEALLOC [59985.151990] Modules linked in: uio_pci_generic uio iw_cxgb4 cxgb4 nvmet_rdma nvmet null_blk brd rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm dm_mirror dm_region_hash dm_log dm_mod intel_rapl iosf_mbi sb_edac edac_core x86_pkg_temp_thermal coretemp ext4 kvm jbd2 irqbypass crct10dif_pclmul crc32_pclmul ghash_clmulni_intel mbcache aesni_intel lrw gf128mul iTCO_wdt glue_helper mei_me iTCO_vendor_support ablk_helper cryptd mxm_wmi ipmi_si i2c_i801 lpc_ich mei sg nfsd mfd_core i2c_smbus ipmi_msghandler pcspkr shpchp auth_rpcgss wmi nfs_acl lockd grace sunrpc ip_tables xfs libcrc32c libcxgb mlx4_ib ib_core mlx4_en sd_mod drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm mlx4_core igb drm ahci libahci ptp libata crc32c_intel pps_core dca nvme i2c_algo_bit nvme_core i2c_core [last unloaded: cxgb4] [59985.239258] CPU: 30 PID: 10937 Comm: kworker/30:1 Not tainted 4.9.0-rc3-debug+ #2 [59985.246992] Hardware name: Supermicro X9DR3-F/X9DR3-F, BIOS 3.2a 07/09/2015 [59985.254098] Workqueue: events linkwatch_event [59985.258600] task: ffff88105312c6c0 task.stack: ffffc90020204000 [59985.264657] RIP: 0010:[<ffffffffa05ae1ba>] [<ffffffffa05ae1ba>] mlx4_en_get_phys_port_id+0x1a/0x50 [mlx4_en] [59985.274874] RSP: 0018:ffffc90020207c30 EFLAGS: 00010286 [59985.280312] RAX: 6b6b6b6b6b6b6b6b RBX: ffff881048c220c0 RCX: 0000000000000000 [59985.287582] RDX: 0000000000000001 RSI: ffffc90020207cb0 RDI: ffff881037020000 [59985.294844] RBP: ffffc90020207c30 R08: 00000000000005f0 R09: ffff88102017e752 [59985.302100] R10: ffff88085f4090c0 R11: ffff88102017e678 R12: ffff881037020000 [59985.309356] R13: ffff88102017e678 R14: 0000000000000000 R15: 0000000000000000 [59985.316608] FS: 0000000000000000(0000) GS:ffff881057580000(0000) knlGS:0000000000000000 [59985.324936] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [59985.330805] CR2: 00007fff8fd82ff8 CR3: 0000000001c07000 CR4: 00000000000406e0 [59985.338072] Stack: [59985.340219] ffffc90020207c40 ffffffff81587a6e ffffc90020207d00 ffffffff815a36ce [59985.347950] ffff881048c220c0 ffffc90020207cd7 0000000000000000 0000000000000010 [59985.355684] 02000000ffffffff 000003e820000000 00000000000005dc 0000010000000000 [59985.363408] Call Trace: [59985.365994] [<ffffffff81587a6e>] dev_get_phys_port_id+0x1e/0x30 [59985.372123] [<ffffffff815a36ce>] rtnl_fill_ifinfo+0x4be/0xff0 [59985.378076] [<ffffffff815a53f3>] rtmsg_ifinfo_build_skb+0x73/0xe0 [59985.384377] [<ffffffff815a5476>] rtmsg_ifinfo.part.27+0x16/0x50 [59985.390505] [<ffffffff815a54c8>] rtmsg_ifinfo+0x18/0x20 [59985.395940] [<ffffffff8158a6c6>] netdev_state_change+0x46/0x50 [59985.401983] [<ffffffff815a5e78>] linkwatch_do_dev+0x38/0x50 [59985.407764] [<ffffffff815a6165>] __linkwatch_run_queue+0xf5/0x170 [59985.414067] [<ffffffff815a6205>] linkwatch_event+0x25/0x30 [59985.419764] [<ffffffff81099a82>] process_one_work+0x152/0x400 [59985.425716] [<ffffffff8109a325>] worker_thread+0x125/0x4b0 [59985.431409] [<ffffffff8109a200>] ? rescuer_thread+0x350/0x350 [59985.437366] [<ffffffff8109fc6a>] kthread+0xca/0xe0 [59985.442367] [<ffffffff8109fba0>] ? kthread_park+0x60/0x60 [59985.447978] [<ffffffff816a1285>] ret_from_fork+0x25/0x30 [59985.453497] Code: f0 5d c3 66 66 66 66 66 2e 0f 1f 84 00 00 00 00 00 66 66 66 66 90 55 48 8b 87 c0 08 00 00 48 63 97 9c d5 00 00 48 89 e5 48 8b 00 <48> 8b 94 d0 58 02 00 00 48 85 d2 74 1c c6 46 20 08 31 c0 88 54 [59985.474081] RIP [<ffffffffa05ae1ba>] mlx4_en_get_phys_port_id+0x1a/0x50 [mlx4_en] [59985.481915] RSP <ffffc90020207c30> [59985.485910] ---[ end trace 317937c8890959b8 ]--- [59990.228721] Kernel panic - not syncing: Fatal exception [59990.234181] Kernel Offset: disabled [59990.239944] ---[ end Kernel panic - not syncing: Fatal exception -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 3+ messages in thread
* Re: Crash in mlx4 shutdown with 4.9-rc3 2016-11-04 14:29 Crash in mlx4 shutdown with 4.9-rc3 Steve Wise @ 2016-11-05 13:15 ` Leon Romanovsky [not found] ` <20161105131513.GP3617-2ukJVAZIZ/Y@public.gmane.org> 0 siblings, 1 reply; 3+ messages in thread From: Leon Romanovsky @ 2016-11-05 13:15 UTC (permalink / raw) To: Steve Wise Cc: yishaih-VPRAkNaXOzVWk0Htik3J/w, linux-rdma-u79uwXL29TY76Z2rM5mHXA, Majd Dibbiny, Tariq Toukan [-- Attachment #1: Type: text/plain, Size: 4839 bytes --] On Fri, Nov 04, 2016 at 09:29:47AM -0500, Steve Wise wrote: > Hey Yishai, Is this by chance a known bug having a pending fix somewhere? I'm > seeing it frequently when shutting down. I'm using 4.9-rc3 with memory > debugging enabled... Hi Steve, We have a fix for this oops in our submission queue to netdev and it is now in final stages of verification. Tariq is planning to submit it on Sunday. Thanks > > [59984.502834] mlx4_core 0000:81:00.0: mlx4_shutdown was called > [59984.603599] mlx4_en 0000:81:00.0: removed PHC > [59985.145590] general protection fault: 0000 [#1] SMP DEBUG_PAGEALLOC > [59985.151990] Modules linked in: uio_pci_generic uio iw_cxgb4 cxgb4 nvmet_rdma > nvmet null_blk brd rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi > scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp ib_ipoib > rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm dm_mirror dm_region_hash > dm_log dm_mod intel_rapl iosf_mbi sb_edac edac_core x86_pkg_temp_thermal > coretemp ext4 kvm jbd2 irqbypass crct10dif_pclmul crc32_pclmul > ghash_clmulni_intel mbcache aesni_intel lrw gf128mul iTCO_wdt glue_helper mei_me > iTCO_vendor_support ablk_helper cryptd mxm_wmi ipmi_si i2c_i801 lpc_ich mei sg > nfsd mfd_core i2c_smbus ipmi_msghandler pcspkr shpchp auth_rpcgss wmi nfs_acl > lockd grace sunrpc ip_tables xfs libcrc32c libcxgb mlx4_ib ib_core mlx4_en > sd_mod drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm > mlx4_core igb drm ahci libahci ptp libata crc32c_intel pps_core dca nvme > i2c_algo_bit nvme_core i2c_core [last unloaded: cxgb4] > [59985.239258] CPU: 30 PID: 10937 Comm: kworker/30:1 Not tainted > 4.9.0-rc3-debug+ #2 > [59985.246992] Hardware name: Supermicro X9DR3-F/X9DR3-F, BIOS 3.2a 07/09/2015 > [59985.254098] Workqueue: events linkwatch_event > [59985.258600] task: ffff88105312c6c0 task.stack: ffffc90020204000 > [59985.264657] RIP: 0010:[<ffffffffa05ae1ba>] [<ffffffffa05ae1ba>] > mlx4_en_get_phys_port_id+0x1a/0x50 [mlx4_en] > [59985.274874] RSP: 0018:ffffc90020207c30 EFLAGS: 00010286 > [59985.280312] RAX: 6b6b6b6b6b6b6b6b RBX: ffff881048c220c0 RCX: 0000000000000000 > [59985.287582] RDX: 0000000000000001 RSI: ffffc90020207cb0 RDI: ffff881037020000 > [59985.294844] RBP: ffffc90020207c30 R08: 00000000000005f0 R09: ffff88102017e752 > [59985.302100] R10: ffff88085f4090c0 R11: ffff88102017e678 R12: ffff881037020000 > [59985.309356] R13: ffff88102017e678 R14: 0000000000000000 R15: 0000000000000000 > [59985.316608] FS: 0000000000000000(0000) GS:ffff881057580000(0000) > knlGS:0000000000000000 > [59985.324936] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > [59985.330805] CR2: 00007fff8fd82ff8 CR3: 0000000001c07000 CR4: 00000000000406e0 > [59985.338072] Stack: > [59985.340219] ffffc90020207c40 ffffffff81587a6e ffffc90020207d00 > ffffffff815a36ce > [59985.347950] ffff881048c220c0 ffffc90020207cd7 0000000000000000 > 0000000000000010 > [59985.355684] 02000000ffffffff 000003e820000000 00000000000005dc > 0000010000000000 > [59985.363408] Call Trace: > [59985.365994] [<ffffffff81587a6e>] dev_get_phys_port_id+0x1e/0x30 > [59985.372123] [<ffffffff815a36ce>] rtnl_fill_ifinfo+0x4be/0xff0 > [59985.378076] [<ffffffff815a53f3>] rtmsg_ifinfo_build_skb+0x73/0xe0 > [59985.384377] [<ffffffff815a5476>] rtmsg_ifinfo.part.27+0x16/0x50 > [59985.390505] [<ffffffff815a54c8>] rtmsg_ifinfo+0x18/0x20 > [59985.395940] [<ffffffff8158a6c6>] netdev_state_change+0x46/0x50 > [59985.401983] [<ffffffff815a5e78>] linkwatch_do_dev+0x38/0x50 > [59985.407764] [<ffffffff815a6165>] __linkwatch_run_queue+0xf5/0x170 > [59985.414067] [<ffffffff815a6205>] linkwatch_event+0x25/0x30 > [59985.419764] [<ffffffff81099a82>] process_one_work+0x152/0x400 > [59985.425716] [<ffffffff8109a325>] worker_thread+0x125/0x4b0 > [59985.431409] [<ffffffff8109a200>] ? rescuer_thread+0x350/0x350 > [59985.437366] [<ffffffff8109fc6a>] kthread+0xca/0xe0 > [59985.442367] [<ffffffff8109fba0>] ? kthread_park+0x60/0x60 > [59985.447978] [<ffffffff816a1285>] ret_from_fork+0x25/0x30 > [59985.453497] Code: f0 5d c3 66 66 66 66 66 2e 0f 1f 84 00 00 00 00 00 66 66 66 > 66 90 55 48 8b 87 c0 08 00 00 48 63 97 9c d5 00 00 48 89 e5 48 8b 00 <48> 8b 94 > d0 58 02 00 00 48 85 d2 74 1c c6 46 20 08 31 c0 88 54 > [59985.474081] RIP [<ffffffffa05ae1ba>] mlx4_en_get_phys_port_id+0x1a/0x50 > [mlx4_en] > [59985.481915] RSP <ffffc90020207c30> > [59985.485910] ---[ end trace 317937c8890959b8 ]--- > [59990.228721] Kernel panic - not syncing: Fatal exception > [59990.234181] Kernel Offset: disabled > [59990.239944] ---[ end Kernel panic - not syncing: Fatal exception > > -- > To unsubscribe from this list: send the line "unsubscribe linux-rdma" in > the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org > More majordomo info at http://vger.kernel.org/majordomo-info.html [-- Attachment #2: signature.asc --] [-- Type: application/pgp-signature, Size: 819 bytes --] ^ permalink raw reply [flat|nested] 3+ messages in thread
[parent not found: <20161105131513.GP3617-2ukJVAZIZ/Y@public.gmane.org>]
* Re: Crash in mlx4 shutdown with 4.9-rc3 [not found] ` <20161105131513.GP3617-2ukJVAZIZ/Y@public.gmane.org> @ 2016-11-06 16:12 ` Tariq Toukan 0 siblings, 0 replies; 3+ messages in thread From: Tariq Toukan @ 2016-11-06 16:12 UTC (permalink / raw) To: Leon Romanovsky, Steve Wise Cc: yishaih-VPRAkNaXOzVWk0Htik3J/w, linux-rdma-u79uwXL29TY76Z2rM5mHXA, Majd Dibbiny, Tariq Toukan Hi Steve, On 05/11/2016 3:15 PM, Leon Romanovsky wrote: > On Fri, Nov 04, 2016 at 09:29:47AM -0500, Steve Wise wrote: >> Hey Yishai, Is this by chance a known bug having a pending fix somewhere? I'm >> seeing it frequently when shutting down. I'm using 4.9-rc3 with memory >> debugging enabled... > Hi Steve, > > We have a fix for this oops in our submission queue to netdev and > it is now in final stages of verification. Tariq is planning to submit > it on Sunday. > > Thanks This crash happens because the lifetime of mlx4_en_priv->mdev is shorter than that of struct net_device. One WA is to add a check of netif_device_present in dev_get_phys_port_id. Something like this: --- a/net/core/dev.c +++ b/net/core/dev.c @@ -6601,6 +6601,8 @@ int dev_get_phys_port_id(struct net_device *dev, if (!ops->ndo_get_phys_port_id) return -EOPNOTSUPP; + if (!netif_device_present(dev)) + return -ENODEV; return ops->ndo_get_phys_port_id(dev, ppid); } EXPORT_SYMBOL(dev_get_phys_port_id); However, this causes other issues when combining with MTU change. In MTU change, netif_device_present returns false for a while, causing an unexpected failure of dev_get_phys_port_id. We're working on fixing this correctly, but that won't happen today. Regards, Tariq Toukan >> [59984.502834] mlx4_core 0000:81:00.0: mlx4_shutdown was called >> [59984.603599] mlx4_en 0000:81:00.0: removed PHC >> [59985.145590] general protection fault: 0000 [#1] SMP DEBUG_PAGEALLOC >> [59985.151990] Modules linked in: uio_pci_generic uio iw_cxgb4 cxgb4 nvmet_rdma >> nvmet null_blk brd rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi >> scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp ib_ipoib >> rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm dm_mirror dm_region_hash >> dm_log dm_mod intel_rapl iosf_mbi sb_edac edac_core x86_pkg_temp_thermal >> coretemp ext4 kvm jbd2 irqbypass crct10dif_pclmul crc32_pclmul >> ghash_clmulni_intel mbcache aesni_intel lrw gf128mul iTCO_wdt glue_helper mei_me >> iTCO_vendor_support ablk_helper cryptd mxm_wmi ipmi_si i2c_i801 lpc_ich mei sg >> nfsd mfd_core i2c_smbus ipmi_msghandler pcspkr shpchp auth_rpcgss wmi nfs_acl >> lockd grace sunrpc ip_tables xfs libcrc32c libcxgb mlx4_ib ib_core mlx4_en >> sd_mod drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm >> mlx4_core igb drm ahci libahci ptp libata crc32c_intel pps_core dca nvme >> i2c_algo_bit nvme_core i2c_core [last unloaded: cxgb4] >> [59985.239258] CPU: 30 PID: 10937 Comm: kworker/30:1 Not tainted >> 4.9.0-rc3-debug+ #2 >> [59985.246992] Hardware name: Supermicro X9DR3-F/X9DR3-F, BIOS 3.2a 07/09/2015 >> [59985.254098] Workqueue: events linkwatch_event >> [59985.258600] task: ffff88105312c6c0 task.stack: ffffc90020204000 >> [59985.264657] RIP: 0010:[<ffffffffa05ae1ba>] [<ffffffffa05ae1ba>] >> mlx4_en_get_phys_port_id+0x1a/0x50 [mlx4_en] >> [59985.274874] RSP: 0018:ffffc90020207c30 EFLAGS: 00010286 >> [59985.280312] RAX: 6b6b6b6b6b6b6b6b RBX: ffff881048c220c0 RCX: 0000000000000000 >> [59985.287582] RDX: 0000000000000001 RSI: ffffc90020207cb0 RDI: ffff881037020000 >> [59985.294844] RBP: ffffc90020207c30 R08: 00000000000005f0 R09: ffff88102017e752 >> [59985.302100] R10: ffff88085f4090c0 R11: ffff88102017e678 R12: ffff881037020000 >> [59985.309356] R13: ffff88102017e678 R14: 0000000000000000 R15: 0000000000000000 >> [59985.316608] FS: 0000000000000000(0000) GS:ffff881057580000(0000) >> knlGS:0000000000000000 >> [59985.324936] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 >> [59985.330805] CR2: 00007fff8fd82ff8 CR3: 0000000001c07000 CR4: 00000000000406e0 >> [59985.338072] Stack: >> [59985.340219] ffffc90020207c40 ffffffff81587a6e ffffc90020207d00 >> ffffffff815a36ce >> [59985.347950] ffff881048c220c0 ffffc90020207cd7 0000000000000000 >> 0000000000000010 >> [59985.355684] 02000000ffffffff 000003e820000000 00000000000005dc >> 0000010000000000 >> [59985.363408] Call Trace: >> [59985.365994] [<ffffffff81587a6e>] dev_get_phys_port_id+0x1e/0x30 >> [59985.372123] [<ffffffff815a36ce>] rtnl_fill_ifinfo+0x4be/0xff0 >> [59985.378076] [<ffffffff815a53f3>] rtmsg_ifinfo_build_skb+0x73/0xe0 >> [59985.384377] [<ffffffff815a5476>] rtmsg_ifinfo.part.27+0x16/0x50 >> [59985.390505] [<ffffffff815a54c8>] rtmsg_ifinfo+0x18/0x20 >> [59985.395940] [<ffffffff8158a6c6>] netdev_state_change+0x46/0x50 >> [59985.401983] [<ffffffff815a5e78>] linkwatch_do_dev+0x38/0x50 >> [59985.407764] [<ffffffff815a6165>] __linkwatch_run_queue+0xf5/0x170 >> [59985.414067] [<ffffffff815a6205>] linkwatch_event+0x25/0x30 >> [59985.419764] [<ffffffff81099a82>] process_one_work+0x152/0x400 >> [59985.425716] [<ffffffff8109a325>] worker_thread+0x125/0x4b0 >> [59985.431409] [<ffffffff8109a200>] ? rescuer_thread+0x350/0x350 >> [59985.437366] [<ffffffff8109fc6a>] kthread+0xca/0xe0 >> [59985.442367] [<ffffffff8109fba0>] ? kthread_park+0x60/0x60 >> [59985.447978] [<ffffffff816a1285>] ret_from_fork+0x25/0x30 >> [59985.453497] Code: f0 5d c3 66 66 66 66 66 2e 0f 1f 84 00 00 00 00 00 66 66 66 >> 66 90 55 48 8b 87 c0 08 00 00 48 63 97 9c d5 00 00 48 89 e5 48 8b 00 <48> 8b 94 >> d0 58 02 00 00 48 85 d2 74 1c c6 46 20 08 31 c0 88 54 >> [59985.474081] RIP [<ffffffffa05ae1ba>] mlx4_en_get_phys_port_id+0x1a/0x50 >> [mlx4_en] >> [59985.481915] RSP <ffffc90020207c30> >> [59985.485910] ---[ end trace 317937c8890959b8 ]--- >> [59990.228721] Kernel panic - not syncing: Fatal exception >> [59990.234181] Kernel Offset: disabled >> [59990.239944] ---[ end Kernel panic - not syncing: Fatal exception >> >> -- >> To unsubscribe from this list: send the line "unsubscribe linux-rdma" in >> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org >> More majordomo info at http://vger.kernel.org/majordomo-info.html -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 3+ messages in thread
end of thread, other threads:[~2016-11-06 16:12 UTC | newest]
Thread overview: 3+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2016-11-04 14:29 Crash in mlx4 shutdown with 4.9-rc3 Steve Wise
2016-11-05 13:15 ` Leon Romanovsky
[not found] ` <20161105131513.GP3617-2ukJVAZIZ/Y@public.gmane.org>
2016-11-06 16:12 ` Tariq Toukan
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox