* nvmeof rdma regression issue on 4.14.0-rc1 [not found] <1735134433.8514119.1505997532669.JavaMail.zimbra@redhat.com> @ 2017-09-21 12:47 ` Yi Zhang 2017-09-21 14:44 ` nvmeof rdma regression issue on 4.14.0-rc1 (or maybe mlx4?) Christoph Hellwig 0 siblings, 1 reply; 16+ messages in thread From: Yi Zhang @ 2017-09-21 12:47 UTC (permalink / raw) Hi With below steps, it's easy to reproduce this issue on host side, I cannot reproduce it on 4.13. So I guess it's one regression issue on 4.14.0-rc1, let me know if you need more info. host side: #connect the target target side: #nvmetcli clear #sleep 90 #nvmetcli restore /etc/rdma.json Host side log: [ 184.318537] nvme nvme0: new ctrl: NQN "nqn.2014-08.org.nvmexpress.discovery", addr 172.31.0.90:4420 [ 184.502114] nvme nvme0: creating 40 I/O queues. [ 185.017751] nvme nvme0: new ctrl: NQN "testnqn", addr 172.31.0.90:4420 [ 191.303659] nvme nvme0: rescanning [ 252.226387] nvme nvme0: Reconnecting in 10 seconds... [ 262.389578] nvme nvme0: Connect rejected: status 8 (invalid service ID). [ 262.397084] nvme nvme0: rdma_resolve_addr wait failed (-104). [ 262.403514] nvme nvme0: Failed reconnect attempt 1 [ 262.408875] nvme nvme0: Reconnecting in 10 seconds... [ 272.616908] nvme nvme0: Connect rejected: status 8 (invalid service ID). [ 272.624408] nvme nvme0: rdma_resolve_addr wait failed (-104). [ 272.630839] nvme nvme0: Failed reconnect attempt 2 [ 272.636230] nvme nvme0: Reconnecting in 10 seconds... [ 282.856896] nvme nvme0: Connect rejected: status 8 (invalid service ID). [ 282.864398] nvme nvme0: rdma_resolve_addr wait failed (-104). [ 282.870831] nvme nvme0: Failed reconnect attempt 3 [ 282.876232] nvme nvme0: Reconnecting in 10 seconds... [ 293.116196] nvme nvme0: creating 40 I/O queues. [ 293.209662] DMAR: ERROR: DMA PTE for vPFN 0xe0f59 already set (to 10369a9001 not 10115ed001) [ 293.219117] ------------[ cut here ]------------ [ 293.224284] WARNING: CPU: 14 PID: 751 at drivers/iommu/intel-iommu.c:2305 __domain_mapping+0x367/0x380 [ 293.234698] Modules linked in: nvme_rdma nvme_fabrics nvme_core sch_mqprio ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter bridge 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_rapl ipmi_ssif sb_edac x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm irqbypass crct10dif_pclmul crc32_pclmul ghash_clmulni_intel intel_cstate intel_uncore iTCO_wdt ipmi_si intel_rapl_perf iTCO_vendor_support ipmi_devintf dcdbas sg pcspkr ipmi_msghandler ioatdma mei_me mei dca shpchp lpc_ich acpi_pad acpi_power_meter wmi nfsd auth_rpcgss nfs_acl lockd grace sunrpc binfmt_misc ip_tables xfs libcrc32c mlx4_en sd_mod [ 293.313884] mgag200 i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm mlx4_core tg3 ahci libahci ptp libata i2c_core crc32c_intel devlink pps_core dm_mirror dm_region_hash dm_log dm_mod [ 293.335583] CPU: 14 PID: 751 Comm: kworker/u369:7 Not tainted 4.14.0-rc1 #2 [ 293.343374] Hardware name: Dell Inc. PowerEdge R430/03XKDV, BIOS 1.6.2 01/08/2016 [ 293.351750] Workqueue: nvme-wq nvme_rdma_reconnect_ctrl_work [nvme_rdma] [ 293.359249] task: ffff881032ecdd00 task.stack: ffffc900084d8000 [ 293.365873] RIP: 0010:__domain_mapping+0x367/0x380 [ 293.371230] RSP: 0018:ffffc900084dbc60 EFLAGS: 00010202 [ 293.377075] RAX: 0000000000000004 RBX: 00000010115ed001 RCX: 0000000000000000 [ 293.385056] RDX: 0000000000000000 RSI: ffff88103e7ce038 RDI: ffff88103e7ce038 [ 293.393040] RBP: ffffc900084dbcc0 R08: 0000000000000000 R09: 0000000000000000 [ 293.401024] R10: 00000000000002f7 R11: 00000000010115ed R12: ffff88103b9e1ac8 [ 293.409744] R13: 0000000000000001 R14: 0000000000000001 R15: 00000000000e0f59 [ 293.418456] FS: 0000000000000000(0000) GS:ffff88103e7c0000(0000) knlGS:0000000000000000 [ 293.428229] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 293.435391] CR2: 0000154ecabc9140 CR3: 0000001005709001 CR4: 00000000001606e0 [ 293.444112] Call Trace: [ 293.447594] __intel_map_single+0xeb/0x180 [ 293.452918] intel_map_page+0x39/0x40 [ 293.457765] mlx4_ib_alloc_mr+0x141/0x220 [mlx4_ib] [ 293.463965] ib_alloc_mr+0x26/0x50 [ib_core] [ 293.469471] nvme_rdma_reinit_request+0x3a/0x70 [nvme_rdma] [ 293.476433] ? nvme_rdma_free_ctrl+0xb0/0xb0 [nvme_rdma] [ 293.483100] blk_mq_reinit_tagset+0x5c/0x90 [ 293.488508] nvme_rdma_configure_io_queues+0x211/0x290 [nvme_rdma] [ 293.496152] nvme_rdma_reconnect_ctrl_work+0x5b/0xd0 [nvme_rdma] [ 293.503598] process_one_work+0x149/0x360 [ 293.508815] worker_thread+0x4d/0x3c0 [ 293.513638] kthread+0x109/0x140 [ 293.517973] ? rescuer_thread+0x380/0x380 [ 293.523176] ? kthread_park+0x60/0x60 [ 293.527993] ret_from_fork+0x25/0x30 [ 293.532705] Code: b2 aa 81 4c 89 5d a0 4c 89 4d a8 e8 87 e1 c0 ff 8b 05 fe 6e 87 00 4c 8b 4d a8 4c 8b 5d a0 85 c0 74 09 83 e8 01 89 05 e9 6e 87 00 <0f> ff e9 b8 fd ff ff e8 8d c7 ba ff 0f 1f 00 66 2e 0f 1f 84 00 [ 293.555293] ---[ end trace c324b9449c77b573 ]--- [ 293.562133] DMAR: ERROR: DMA PTE for vPFN 0xe0f59 already set (to 10369a9001 not 1002790001) [ 293.572343] ------------[ cut here ]------------ [ 293.578288] WARNING: CPU: 14 PID: 751 at drivers/iommu/intel-iommu.c:2305 __domain_mapping+0x367/0x380 [ 293.589473] Modules linked in: nvme_rdma nvme_fabrics nvme_core sch_mqprio ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter bridge 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_rapl ipmi_ssif sb_edac x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm irqbypass crct10dif_pclmul crc32_pclmul ghash_clmulni_intel intel_cstate intel_uncore iTCO_wdt ipmi_si intel_rapl_perf iTCO_vendor_support ipmi_devintf dcdbas sg pcspkr ipmi_msghandler ioatdma mei_me mei dca shpchp lpc_ich acpi_pad acpi_power_meter wmi nfsd auth_rpcgss nfs_acl lockd grace sunrpc binfmt_misc ip_tables xfs libcrc32c mlx4_en sd_mod [ 293.674124] mgag200 i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm mlx4_core tg3 ahci libahci ptp libata i2c_core crc32c_intel devlink pps_core dm_mirror dm_region_hash dm_log dm_mod [ 293.697394] CPU: 14 PID: 751 Comm: kworker/u369:7 Tainted: G W 4.14.0-rc1 #2 [ 293.707322] Hardware name: Dell Inc. PowerEdge R430/03XKDV, BIOS 1.6.2 01/08/2016 [ 293.716456] Workqueue: nvme-wq nvme_rdma_reconnect_ctrl_work [nvme_rdma] [ 293.724706] task: ffff881032ecdd00 task.stack: ffffc900084d8000 [ 293.732095] RIP: 0010:__domain_mapping+0x367/0x380 [ 293.738214] RSP: 0018:ffffc900084dbc60 EFLAGS: 00010206 [ 293.744810] RAX: 0000000000000003 RBX: 0000001002790001 RCX: 0000000000000000 [ 293.753541] RDX: 0000000000000000 RSI: ffff88103e7ce038 RDI: ffff88103e7ce038 [ 293.762280] RBP: ffffc900084dbcc0 R08: 0000000000000000 R09: 0000000000000000 [ 293.771011] R10: 00000000000003ff R11: 0000000001002790 R12: ffff88103b9e1ac8 [ 293.779752] R13: 0000000000000001 R14: 0000000000000001 R15: 00000000000e0f59 [ 293.788487] FS: 0000000000000000(0000) GS:ffff88103e7c0000(0000) knlGS:0000000000000000 [ 293.798295] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 293.805480] CR2: 0000154ecabc9140 CR3: 0000001005709001 CR4: 00000000001606e0 [ 293.814226] Call Trace: [ 293.817726] __intel_map_single+0xeb/0x180 [ 293.823058] intel_map_page+0x39/0x40 [ 293.827906] mlx4_ib_alloc_mr+0x141/0x220 [mlx4_ib] [ 293.834112] ib_alloc_mr+0x26/0x50 [ib_core] [ 293.839640] nvme_rdma_reinit_request+0x3a/0x70 [nvme_rdma] [ 293.846628] ? nvme_rdma_free_ctrl+0xb0/0xb0 [nvme_rdma] [ 293.853322] blk_mq_reinit_tagset+0x5c/0x90 [ 293.858757] nvme_rdma_configure_io_queues+0x211/0x290 [nvme_rdma] [ 293.866429] nvme_rdma_reconnect_ctrl_work+0x5b/0xd0 [nvme_rdma] [ 293.873891] process_one_work+0x149/0x360 [ 293.879132] worker_thread+0x4d/0x3c0 [ 293.883979] kthread+0x109/0x140 [ 293.888336] ? rescuer_thread+0x380/0x380 [ 293.893554] ? kthread_park+0x60/0x60 [ 293.898399] ret_from_fork+0x25/0x30 [ 293.903139] Code: b2 aa 81 4c 89 5d a0 4c 89 4d a8 e8 87 e1 c0 ff 8b 05 fe 6e 87 00 4c 8b 4d a8 4c 8b 5d a0 85 c0 74 09 83 e8 01 89 05 e9 6e 87 00 <0f> ff e9 b8 fd ff ff e8 8d c7 ba ff 0f 1f 00 66 2e 0f 1f 84 00 [ 293.925798] ---[ end trace c324b9449c77b574 ]--- [ 294.460186] nvme nvme0: Successfully reconnected [ 294.460243] nvme nvme0: MEMREG for CQE 0xffff88100663a640 failed with status memory management operation error (6) [ 325.850105] general protection fault: 0000 [#1] SMP [ 325.856354] Modules linked in: nvme_rdma nvme_fabrics nvme_core sch_mqprio ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter bridge 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_rapl ipmi_ssif sb_edac x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm irqbypass crct10dif_pclmul crc32_pclmul ghash_clmulni_intel intel_cstate intel_uncore iTCO_wdt ipmi_si intel_rapl_perf iTCO_vendor_support ipmi_devintf dcdbas sg pcspkr ipmi_msghandler ioatdma mei_me mei dca shpchp lpc_ich acpi_pad acpi_power_meter wmi nfsd auth_rpcgss nfs_acl lockd grace sunrpc binfmt_misc ip_tables xfs libcrc32c mlx4_en sd_mod [ 325.940982] mgag200 i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm mlx4_core tg3 ahci libahci ptp libata i2c_core crc32c_intel devlink pps_core dm_mirror dm_region_hash dm_log dm_mod [ 325.964245] CPU: 22 PID: 3707 Comm: git Tainted: G W 4.14.0-rc1 #2 [ 325.973198] Hardware name: Dell Inc. PowerEdge R430/03XKDV, BIOS 1.6.2 01/08/2016 [ 325.982332] task: ffff881036451740 task.stack: ffffc90009368000 [ 325.989746] RIP: 0010:xfs_attr_shortform_getvalue+0x25/0x120 [xfs] [ 325.997412] RSP: 0018:ffffc9000936b920 EFLAGS: 00010282 [ 326.004006] RAX: 63656a626f3a755f RBX: ffffc9000936b980 RCX: ffffc9000936b980 [ 326.012738] RDX: 0000000001000000 RSI: ffffc9000936b980 RDI: ffffc9000936b980 [ 326.021465] RBP: ffffc9000936b958 R08: ffffffff81a740c0 R09: 0000000000000000 [ 326.030194] R10: 0000000000000008 R11: ffff881002f52700 R12: ffffc9000936ba54 [ 326.038913] R13: ffff882023b9fc00 R14: 0000000000000000 R15: 0000000000000008 [ 326.047642] FS: 000014c3de00e740(0000) GS:ffff88103e8c0000(0000) knlGS:0000000000000000 [ 326.057445] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 326.064627] CR2: 00000000007b7d44 CR3: 000000018b0cb005 CR4: 00000000001606e0 [ 326.073370] Call Trace: [ 326.076885] xfs_attr_get_ilocked+0x63/0x70 [xfs] [ 326.082917] xfs_attr_get+0xca/0x120 [xfs] [ 326.088269] xfs_xattr_get+0x4c/0x70 [xfs] [ 326.093592] __vfs_getxattr+0x57/0x70 [ 326.098428] inode_doinit_with_dentry+0x33c/0x580 [ 326.104430] selinux_d_instantiate+0x1c/0x20 [ 326.109947] security_d_instantiate+0x32/0x50 [ 326.115558] d_splice_alias+0x4c/0x370 [ 326.120505] xfs_vn_lookup+0x87/0xb0 [xfs] [ 326.125819] lookup_slow+0xa2/0x160 [ 326.130448] walk_component+0x160/0x250 [ 326.135461] ? legitimize_path.isra.33+0x2e/0x60 [ 326.141346] path_lookupat+0x79/0x210 [ 326.146162] ? cpumask_any_but+0x31/0x40 [ 326.151267] filename_lookup+0xaf/0x190 [ 326.156276] ? kmem_cache_alloc+0x9c/0x1b0 [ 326.161576] ? getname_flags+0x4f/0x1f0 [ 326.166583] ? getname_flags+0x6f/0x1f0 [ 326.171586] user_path_at_empty+0x36/0x40 [ 326.176768] vfs_statx+0x77/0xe0 [ 326.181058] SYSC_newlstat+0x3d/0x70 [ 326.185726] ? __audit_syscall_entry+0xaf/0x100 [ 326.191463] ? syscall_trace_enter+0x1d0/0x2b0 [ 326.197090] ? __audit_syscall_exit+0x209/0x290 [ 326.202799] SyS_newlstat+0xe/0x10 [ 326.207226] do_syscall_64+0x67/0x180 [ 326.211926] entry_SYSCALL64_slow_path+0x25/0x25 [ 326.217674] RIP: 0033:0x14c3dd4961a5 [ 326.222237] RSP: 002b:00007fff730b3478 EFLAGS: 00000246 ORIG_RAX: 0000000000000006 [ 326.231250] RAX: ffffffffffffffda RBX: 0000000000ca3c50 RCX: 000014c3dd4961a5 [ 326.239774] RDX: 00007fff730b3530 RSI: 00007fff730b3530 RDI: 0000000000ca3ca0 [ 326.248281] RBP: 0000000000ca3ca0 R08: 0000000000000000 R09: 0000000000000000 [ 326.256770] R10: 0000000000000001 R11: 0000000000000246 R12: 00007fff730b3530 [ 326.265239] R13: 0000000000000000 R14: 0000000000000170 R15: 00007fff730b3530 [ 326.273684] Code: 84 00 00 00 00 00 0f 1f 44 00 00 55 48 89 f9 48 89 e5 41 57 41 56 41 55 41 54 53 48 83 ec 10 48 8b 47 38 48 8b 40 38 48 8b 40 18 <44> 0f b6 70 02 48 8d 58 04 b8 c3 ff ff ff 45 85 f6 0f 84 82 00 [ 326.295785] RIP: xfs_attr_shortform_getvalue+0x25/0x120 [xfs] RSP: ffffc9000936b920 [ 326.304910] ---[ end trace c324b9449c77b575 ]--- [ 326.362863] Kernel panic - not syncing: Fatal exception [ 326.362964] Kernel Offset: disabled [ 326.427058] ---[ end Kernel panic - not syncing: Fatal exception Best Regards, Yi Zhang ^ permalink raw reply [flat|nested] 16+ messages in thread
* nvmeof rdma regression issue on 4.14.0-rc1 (or maybe mlx4?) 2017-09-21 12:47 ` nvmeof rdma regression issue on 4.14.0-rc1 Yi Zhang @ 2017-09-21 14:44 ` Christoph Hellwig 2017-09-24 7:38 ` Sagi Grimberg 0 siblings, 1 reply; 16+ messages in thread From: Christoph Hellwig @ 2017-09-21 14:44 UTC (permalink / raw) Adding linux-rdma, the dma mappings happen in the mlx4 driver On Thu, Sep 21, 2017@08:47:31AM -0400, Yi Zhang wrote: > Hi > > With below steps, it's easy to reproduce this issue on host side, I cannot reproduce it on 4.13. > So I guess it's one regression issue on 4.14.0-rc1, let me know if you need more info. > > host side: > #connect the target > > target side: > #nvmetcli clear > #sleep 90 > #nvmetcli restore /etc/rdma.json > > Host side log: > > [ 184.318537] nvme nvme0: new ctrl: NQN "nqn.2014-08.org.nvmexpress.discovery", addr 172.31.0.90:4420 > [ 184.502114] nvme nvme0: creating 40 I/O queues. > [ 185.017751] nvme nvme0: new ctrl: NQN "testnqn", addr 172.31.0.90:4420 > [ 191.303659] nvme nvme0: rescanning > [ 252.226387] nvme nvme0: Reconnecting in 10 seconds... > [ 262.389578] nvme nvme0: Connect rejected: status 8 (invalid service ID). > [ 262.397084] nvme nvme0: rdma_resolve_addr wait failed (-104). > [ 262.403514] nvme nvme0: Failed reconnect attempt 1 > [ 262.408875] nvme nvme0: Reconnecting in 10 seconds... > [ 272.616908] nvme nvme0: Connect rejected: status 8 (invalid service ID). > [ 272.624408] nvme nvme0: rdma_resolve_addr wait failed (-104). > [ 272.630839] nvme nvme0: Failed reconnect attempt 2 > [ 272.636230] nvme nvme0: Reconnecting in 10 seconds... > [ 282.856896] nvme nvme0: Connect rejected: status 8 (invalid service ID). > [ 282.864398] nvme nvme0: rdma_resolve_addr wait failed (-104). > [ 282.870831] nvme nvme0: Failed reconnect attempt 3 > [ 282.876232] nvme nvme0: Reconnecting in 10 seconds... > [ 293.116196] nvme nvme0: creating 40 I/O queues. > [ 293.209662] DMAR: ERROR: DMA PTE for vPFN 0xe0f59 already set (to 10369a9001 not 10115ed001) > [ 293.219117] ------------[ cut here ]------------ > [ 293.224284] WARNING: CPU: 14 PID: 751 at drivers/iommu/intel-iommu.c:2305 __domain_mapping+0x367/0x380 > [ 293.234698] Modules linked in: nvme_rdma nvme_fabrics nvme_core sch_mqprio ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter bridge 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_rapl ipmi_ssif sb_edac x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm irqbypass crct10dif_pclmul crc32_pclmul ghash_clmulni_intel intel_cstate intel_uncore iTCO_wdt ipmi_si intel_rapl_perf iTCO_vendor_support ipmi_devintf dcdbas sg pcspkr ipmi_msghandler ioatdma mei_me mei dca shpchp lpc_ich acpi_pad acpi_power_meter wmi nfsd auth_rpcgss nfs_acl lockd grace sunrpc binfmt_misc ip_tables xfs libcrc32c mlx4_en sd_mod > [ 293.313884] mgag200 i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm mlx4_core tg3 ahci libahci ptp libata i2c_core crc32c_intel devlink pps_core dm_mirror dm_region_hash dm_log dm_mod > [ 293.335583] CPU: 14 PID: 751 Comm: kworker/u369:7 Not tainted 4.14.0-rc1 #2 > [ 293.343374] Hardware name: Dell Inc. PowerEdge R430/03XKDV, BIOS 1.6.2 01/08/2016 > [ 293.351750] Workqueue: nvme-wq nvme_rdma_reconnect_ctrl_work [nvme_rdma] > [ 293.359249] task: ffff881032ecdd00 task.stack: ffffc900084d8000 > [ 293.365873] RIP: 0010:__domain_mapping+0x367/0x380 > [ 293.371230] RSP: 0018:ffffc900084dbc60 EFLAGS: 00010202 > [ 293.377075] RAX: 0000000000000004 RBX: 00000010115ed001 RCX: 0000000000000000 > [ 293.385056] RDX: 0000000000000000 RSI: ffff88103e7ce038 RDI: ffff88103e7ce038 > [ 293.393040] RBP: ffffc900084dbcc0 R08: 0000000000000000 R09: 0000000000000000 > [ 293.401024] R10: 00000000000002f7 R11: 00000000010115ed R12: ffff88103b9e1ac8 > [ 293.409744] R13: 0000000000000001 R14: 0000000000000001 R15: 00000000000e0f59 > [ 293.418456] FS: 0000000000000000(0000) GS:ffff88103e7c0000(0000) knlGS:0000000000000000 > [ 293.428229] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > [ 293.435391] CR2: 0000154ecabc9140 CR3: 0000001005709001 CR4: 00000000001606e0 > [ 293.444112] Call Trace: > [ 293.447594] __intel_map_single+0xeb/0x180 > [ 293.452918] intel_map_page+0x39/0x40 > [ 293.457765] mlx4_ib_alloc_mr+0x141/0x220 [mlx4_ib] > [ 293.463965] ib_alloc_mr+0x26/0x50 [ib_core] > [ 293.469471] nvme_rdma_reinit_request+0x3a/0x70 [nvme_rdma] > [ 293.476433] ? nvme_rdma_free_ctrl+0xb0/0xb0 [nvme_rdma] > [ 293.483100] blk_mq_reinit_tagset+0x5c/0x90 > [ 293.488508] nvme_rdma_configure_io_queues+0x211/0x290 [nvme_rdma] > [ 293.496152] nvme_rdma_reconnect_ctrl_work+0x5b/0xd0 [nvme_rdma] > [ 293.503598] process_one_work+0x149/0x360 > [ 293.508815] worker_thread+0x4d/0x3c0 > [ 293.513638] kthread+0x109/0x140 > [ 293.517973] ? rescuer_thread+0x380/0x380 > [ 293.523176] ? kthread_park+0x60/0x60 > [ 293.527993] ret_from_fork+0x25/0x30 > [ 293.532705] Code: b2 aa 81 4c 89 5d a0 4c 89 4d a8 e8 87 e1 c0 ff 8b 05 fe 6e 87 00 4c 8b 4d a8 4c 8b 5d a0 85 c0 74 09 83 e8 01 89 05 e9 6e 87 00 <0f> ff e9 b8 fd ff ff e8 8d c7 ba ff 0f 1f 00 66 2e 0f 1f 84 00 > [ 293.555293] ---[ end trace c324b9449c77b573 ]--- > [ 293.562133] DMAR: ERROR: DMA PTE for vPFN 0xe0f59 already set (to 10369a9001 not 1002790001) > [ 293.572343] ------------[ cut here ]------------ > [ 293.578288] WARNING: CPU: 14 PID: 751 at drivers/iommu/intel-iommu.c:2305 __domain_mapping+0x367/0x380 > [ 293.589473] Modules linked in: nvme_rdma nvme_fabrics nvme_core sch_mqprio ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter bridge 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_rapl ipmi_ssif sb_edac x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm irqbypass crct10dif_pclmul crc32_pclmul ghash_clmulni_intel intel_cstate intel_uncore iTCO_wdt ipmi_si intel_rapl_perf iTCO_vendor_support ipmi_devintf dcdbas sg pcspkr ipmi_msghandler ioatdma mei_me mei dca shpchp lpc_ich acpi_pad acpi_power_meter wmi nfsd auth_rpcgss nfs_acl lockd grace sunrpc binfmt_misc ip_tables xfs libcrc32c mlx4_en sd_mod > [ 293.674124] mgag200 i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm mlx4_core tg3 ahci libahci ptp libata i2c_core crc32c_intel devlink pps_core dm_mirror dm_region_hash dm_log dm_mod > [ 293.697394] CPU: 14 PID: 751 Comm: kworker/u369:7 Tainted: G W 4.14.0-rc1 #2 > [ 293.707322] Hardware name: Dell Inc. PowerEdge R430/03XKDV, BIOS 1.6.2 01/08/2016 > [ 293.716456] Workqueue: nvme-wq nvme_rdma_reconnect_ctrl_work [nvme_rdma] > [ 293.724706] task: ffff881032ecdd00 task.stack: ffffc900084d8000 > [ 293.732095] RIP: 0010:__domain_mapping+0x367/0x380 > [ 293.738214] RSP: 0018:ffffc900084dbc60 EFLAGS: 00010206 > [ 293.744810] RAX: 0000000000000003 RBX: 0000001002790001 RCX: 0000000000000000 > [ 293.753541] RDX: 0000000000000000 RSI: ffff88103e7ce038 RDI: ffff88103e7ce038 > [ 293.762280] RBP: ffffc900084dbcc0 R08: 0000000000000000 R09: 0000000000000000 > [ 293.771011] R10: 00000000000003ff R11: 0000000001002790 R12: ffff88103b9e1ac8 > [ 293.779752] R13: 0000000000000001 R14: 0000000000000001 R15: 00000000000e0f59 > [ 293.788487] FS: 0000000000000000(0000) GS:ffff88103e7c0000(0000) knlGS:0000000000000000 > [ 293.798295] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > [ 293.805480] CR2: 0000154ecabc9140 CR3: 0000001005709001 CR4: 00000000001606e0 > [ 293.814226] Call Trace: > [ 293.817726] __intel_map_single+0xeb/0x180 > [ 293.823058] intel_map_page+0x39/0x40 > [ 293.827906] mlx4_ib_alloc_mr+0x141/0x220 [mlx4_ib] > [ 293.834112] ib_alloc_mr+0x26/0x50 [ib_core] > [ 293.839640] nvme_rdma_reinit_request+0x3a/0x70 [nvme_rdma] > [ 293.846628] ? nvme_rdma_free_ctrl+0xb0/0xb0 [nvme_rdma] > [ 293.853322] blk_mq_reinit_tagset+0x5c/0x90 > [ 293.858757] nvme_rdma_configure_io_queues+0x211/0x290 [nvme_rdma] > [ 293.866429] nvme_rdma_reconnect_ctrl_work+0x5b/0xd0 [nvme_rdma] > [ 293.873891] process_one_work+0x149/0x360 > [ 293.879132] worker_thread+0x4d/0x3c0 > [ 293.883979] kthread+0x109/0x140 > [ 293.888336] ? rescuer_thread+0x380/0x380 > [ 293.893554] ? kthread_park+0x60/0x60 > [ 293.898399] ret_from_fork+0x25/0x30 > [ 293.903139] Code: b2 aa 81 4c 89 5d a0 4c 89 4d a8 e8 87 e1 c0 ff 8b 05 fe 6e 87 00 4c 8b 4d a8 4c 8b 5d a0 85 c0 74 09 83 e8 01 89 05 e9 6e 87 00 <0f> ff e9 b8 fd ff ff e8 8d c7 ba ff 0f 1f 00 66 2e 0f 1f 84 00 > [ 293.925798] ---[ end trace c324b9449c77b574 ]--- > [ 294.460186] nvme nvme0: Successfully reconnected > [ 294.460243] nvme nvme0: MEMREG for CQE 0xffff88100663a640 failed with status memory management operation error (6) > [ 325.850105] general protection fault: 0000 [#1] SMP > [ 325.856354] Modules linked in: nvme_rdma nvme_fabrics nvme_core sch_mqprio ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter bridge 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_rapl ipmi_ssif sb_edac x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm irqbypass crct10dif_pclmul crc32_pclmul ghash_clmulni_intel intel_cstate intel_uncore iTCO_wdt ipmi_si intel_rapl_perf iTCO_vendor_support ipmi_devintf dcdbas sg pcspkr ipmi_msghandler ioatdma mei_me mei dca shpchp lpc_ich acpi_pad acpi_power_meter wmi nfsd auth_rpcgss nfs_acl lockd grace sunrpc binfmt_misc ip_tables xfs libcrc32c mlx4_en sd_mod > [ 325.940982] mgag200 i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm mlx4_core tg3 ahci libahci ptp libata i2c_core crc32c_intel devlink pps_core dm_mirror dm_region_hash dm_log dm_mod > [ 325.964245] CPU: 22 PID: 3707 Comm: git Tainted: G W 4.14.0-rc1 #2 > [ 325.973198] Hardware name: Dell Inc. PowerEdge R430/03XKDV, BIOS 1.6.2 01/08/2016 > [ 325.982332] task: ffff881036451740 task.stack: ffffc90009368000 > [ 325.989746] RIP: 0010:xfs_attr_shortform_getvalue+0x25/0x120 [xfs] > [ 325.997412] RSP: 0018:ffffc9000936b920 EFLAGS: 00010282 > [ 326.004006] RAX: 63656a626f3a755f RBX: ffffc9000936b980 RCX: ffffc9000936b980 > [ 326.012738] RDX: 0000000001000000 RSI: ffffc9000936b980 RDI: ffffc9000936b980 > [ 326.021465] RBP: ffffc9000936b958 R08: ffffffff81a740c0 R09: 0000000000000000 > [ 326.030194] R10: 0000000000000008 R11: ffff881002f52700 R12: ffffc9000936ba54 > [ 326.038913] R13: ffff882023b9fc00 R14: 0000000000000000 R15: 0000000000000008 > [ 326.047642] FS: 000014c3de00e740(0000) GS:ffff88103e8c0000(0000) knlGS:0000000000000000 > [ 326.057445] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > [ 326.064627] CR2: 00000000007b7d44 CR3: 000000018b0cb005 CR4: 00000000001606e0 > [ 326.073370] Call Trace: > [ 326.076885] xfs_attr_get_ilocked+0x63/0x70 [xfs] > [ 326.082917] xfs_attr_get+0xca/0x120 [xfs] > [ 326.088269] xfs_xattr_get+0x4c/0x70 [xfs] > [ 326.093592] __vfs_getxattr+0x57/0x70 > [ 326.098428] inode_doinit_with_dentry+0x33c/0x580 > [ 326.104430] selinux_d_instantiate+0x1c/0x20 > [ 326.109947] security_d_instantiate+0x32/0x50 > [ 326.115558] d_splice_alias+0x4c/0x370 > [ 326.120505] xfs_vn_lookup+0x87/0xb0 [xfs] > [ 326.125819] lookup_slow+0xa2/0x160 > [ 326.130448] walk_component+0x160/0x250 > [ 326.135461] ? legitimize_path.isra.33+0x2e/0x60 > [ 326.141346] path_lookupat+0x79/0x210 > [ 326.146162] ? cpumask_any_but+0x31/0x40 > [ 326.151267] filename_lookup+0xaf/0x190 > [ 326.156276] ? kmem_cache_alloc+0x9c/0x1b0 > [ 326.161576] ? getname_flags+0x4f/0x1f0 > [ 326.166583] ? getname_flags+0x6f/0x1f0 > [ 326.171586] user_path_at_empty+0x36/0x40 > [ 326.176768] vfs_statx+0x77/0xe0 > [ 326.181058] SYSC_newlstat+0x3d/0x70 > [ 326.185726] ? __audit_syscall_entry+0xaf/0x100 > [ 326.191463] ? syscall_trace_enter+0x1d0/0x2b0 > [ 326.197090] ? __audit_syscall_exit+0x209/0x290 > [ 326.202799] SyS_newlstat+0xe/0x10 > [ 326.207226] do_syscall_64+0x67/0x180 > [ 326.211926] entry_SYSCALL64_slow_path+0x25/0x25 > [ 326.217674] RIP: 0033:0x14c3dd4961a5 > [ 326.222237] RSP: 002b:00007fff730b3478 EFLAGS: 00000246 ORIG_RAX: 0000000000000006 > [ 326.231250] RAX: ffffffffffffffda RBX: 0000000000ca3c50 RCX: 000014c3dd4961a5 > [ 326.239774] RDX: 00007fff730b3530 RSI: 00007fff730b3530 RDI: 0000000000ca3ca0 > [ 326.248281] RBP: 0000000000ca3ca0 R08: 0000000000000000 R09: 0000000000000000 > [ 326.256770] R10: 0000000000000001 R11: 0000000000000246 R12: 00007fff730b3530 > [ 326.265239] R13: 0000000000000000 R14: 0000000000000170 R15: 00007fff730b3530 > [ 326.273684] Code: 84 00 00 00 00 00 0f 1f 44 00 00 55 48 89 f9 48 89 e5 41 57 41 56 41 55 41 54 53 48 83 ec 10 48 8b 47 38 48 8b 40 38 48 8b 40 18 <44> 0f b6 70 02 48 8d 58 04 b8 c3 ff ff ff 45 85 f6 0f 84 82 00 > [ 326.295785] RIP: xfs_attr_shortform_getvalue+0x25/0x120 [xfs] RSP: ffffc9000936b920 > [ 326.304910] ---[ end trace c324b9449c77b575 ]--- > [ 326.362863] Kernel panic - not syncing: Fatal exception > [ 326.362964] Kernel Offset: disabled > [ 326.427058] ---[ end Kernel panic - not syncing: Fatal exception > > > Best Regards, > Yi Zhang > > > > _______________________________________________ > Linux-nvme mailing list > Linux-nvme at lists.infradead.org > http://lists.infradead.org/mailman/listinfo/linux-nvme ---end quoted text--- ^ permalink raw reply [flat|nested] 16+ messages in thread
* nvmeof rdma regression issue on 4.14.0-rc1 (or maybe mlx4?) 2017-09-21 14:44 ` nvmeof rdma regression issue on 4.14.0-rc1 (or maybe mlx4?) Christoph Hellwig @ 2017-09-24 7:38 ` Sagi Grimberg 2017-09-24 9:28 ` Yi Zhang 0 siblings, 1 reply; 16+ messages in thread From: Sagi Grimberg @ 2017-09-24 7:38 UTC (permalink / raw) > Adding linux-rdma, the dma mappings happen in the mlx4 driver ... >> [ 293.209662] DMAR: ERROR: DMA PTE for vPFN 0xe0f59 already set (to 10369a9001 not 10115ed001) >> [ 293.219117] ------------[ cut here ]------------ >> [ 293.224284] WARNING: CPU: 14 PID: 751 at drivers/iommu/intel-iommu.c:2305 __domain_mapping+0x367/0x380 >> [ 293.234698] Modules linked in: nvme_rdma nvme_fabrics nvme_core sch_mqprio ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter bridge 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_rapl ipmi_ssif sb_edac x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm irqbypass crct10dif_pclmul crc32_pclmul ghash_clmulni_intel intel_cstate intel_uncore iTCO_wdt ipmi_si intel_rapl_perf iTCO_vendor_support ipmi_devintf dcdbas sg pcspkr ipmi_msghandler ioatdma mei_me mei dca shpchp lpc_ich acpi_pad acpi_power_meter wmi nfsd auth_rpcgss nfs_acl lockd grace sunrpc binfmt_misc ip_tables xfs libcrc32c mlx4_en sd_mod >> [ 293.313884] mgag200 i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm mlx4_core tg3 ahci libahci ptp libata i2c_core crc32c_intel devlink pps_core dm_mirror dm_region_hash dm_log dm_mod >> [ 293.335583] CPU: 14 PID: 751 Comm: kworker/u369:7 Not tainted 4.14.0-rc1 #2 >> [ 293.343374] Hardware name: Dell Inc. PowerEdge R430/03XKDV, BIOS 1.6.2 01/08/2016 >> [ 293.351750] Workqueue: nvme-wq nvme_rdma_reconnect_ctrl_work [nvme_rdma] >> [ 293.359249] task: ffff881032ecdd00 task.stack: ffffc900084d8000 >> [ 293.365873] RIP: 0010:__domain_mapping+0x367/0x380 >> [ 293.371230] RSP: 0018:ffffc900084dbc60 EFLAGS: 00010202 >> [ 293.377075] RAX: 0000000000000004 RBX: 00000010115ed001 RCX: 0000000000000000 >> [ 293.385056] RDX: 0000000000000000 RSI: ffff88103e7ce038 RDI: ffff88103e7ce038 >> [ 293.393040] RBP: ffffc900084dbcc0 R08: 0000000000000000 R09: 0000000000000000 >> [ 293.401024] R10: 00000000000002f7 R11: 00000000010115ed R12: ffff88103b9e1ac8 >> [ 293.409744] R13: 0000000000000001 R14: 0000000000000001 R15: 00000000000e0f59 >> [ 293.418456] FS: 0000000000000000(0000) GS:ffff88103e7c0000(0000) knlGS:0000000000000000 >> [ 293.428229] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 >> [ 293.435391] CR2: 0000154ecabc9140 CR3: 0000001005709001 CR4: 00000000001606e0 >> [ 293.444112] Call Trace: >> [ 293.447594] __intel_map_single+0xeb/0x180 >> [ 293.452918] intel_map_page+0x39/0x40 >> [ 293.457765] mlx4_ib_alloc_mr+0x141/0x220 [mlx4_ib] >> [ 293.463965] ib_alloc_mr+0x26/0x50 [ib_core] >> [ 293.469471] nvme_rdma_reinit_request+0x3a/0x70 [nvme_rdma] >> [ 293.476433] ? nvme_rdma_free_ctrl+0xb0/0xb0 [nvme_rdma] >> [ 293.483100] blk_mq_reinit_tagset+0x5c/0x90 >> [ 293.488508] nvme_rdma_configure_io_queues+0x211/0x290 [nvme_rdma] >> [ 293.496152] nvme_rdma_reconnect_ctrl_work+0x5b/0xd0 [nvme_rdma] >> [ 293.503598] process_one_work+0x149/0x360 >> [ 293.508815] worker_thread+0x4d/0x3c0 >> [ 293.513638] kthread+0x109/0x140 >> [ 293.517973] ? rescuer_thread+0x380/0x380 >> [ 293.523176] ? kthread_park+0x60/0x60 >> [ 293.527993] ret_from_fork+0x25/0x30 Is it possible that ib_dereg_mr failed? can you please apply the following patch and report if you see a warning? -- diff --git a/drivers/nvme/host/rdma.c b/drivers/nvme/host/rdma.c index 92a03ff5fb4d..ef50b58b0bb6 100644 --- a/drivers/nvme/host/rdma.c +++ b/drivers/nvme/host/rdma.c @@ -274,7 +274,7 @@ static int nvme_rdma_reinit_request(void *data, struct request *rq) struct nvme_rdma_request *req = blk_mq_rq_to_pdu(rq); int ret = 0; - ib_dereg_mr(req->mr); + WARN_ON_ONCE(ib_dereg_mr(req->mr)); req->mr = ib_alloc_mr(dev->pd, IB_MR_TYPE_MEM_REG, ctrl->max_fr_pages); -- ^ permalink raw reply related [flat|nested] 16+ messages in thread
* nvmeof rdma regression issue on 4.14.0-rc1 (or maybe mlx4?) 2017-09-24 7:38 ` Sagi Grimberg @ 2017-09-24 9:28 ` Yi Zhang [not found] ` <20170924103426.GB25094@mtr-leonro.local> 0 siblings, 1 reply; 16+ messages in thread From: Yi Zhang @ 2017-09-24 9:28 UTC (permalink / raw) > Is it possible that ib_dereg_mr failed? > It seems not, and finally the system get panic, here is the log: [ 104.373784] nvme nvme0: new ctrl: NQN "nqn.2014-08.org.nvmexpress.discovery", addr 172.31.0.90:4420 [ 104.564001] nvme nvme0: creating 40 I/O queues. [ 105.070022] nvme nvme0: new ctrl: NQN "testnqn", addr 172.31.0.90:4420 [ 144.135070] nvme nvme0: rescanning [ 204.383678] nvme nvme0: Reconnecting in 10 seconds... [ 214.506489] nvme nvme0: Connect rejected: status 8 (invalid service ID). [ 214.513996] nvme nvme0: rdma_resolve_addr wait failed (-104). [ 214.520426] nvme nvme0: Failed reconnect attempt 1 [ 214.525788] nvme nvme0: Reconnecting in 10 seconds... [ 224.733962] nvme nvme0: Connect rejected: status 8 (invalid service ID). [ 224.741464] nvme nvme0: rdma_resolve_addr wait failed (-104). [ 224.747898] nvme nvme0: Failed reconnect attempt 2 [ 224.753301] nvme nvme0: Reconnecting in 10 seconds... [ 234.973834] nvme nvme0: Connect rejected: status 8 (invalid service ID). [ 234.981335] nvme nvme0: rdma_resolve_addr wait failed (-104). [ 234.987768] nvme nvme0: Failed reconnect attempt 3 [ 234.993150] nvme nvme0: Reconnecting in 10 seconds... [ 245.233395] nvme nvme0: creating 40 I/O queues. [ 245.238480] DMAR: ERROR: DMA PTE for vPFN 0xe109b already set (to 10098cc002 not 103b85e003) [ 245.247940] ------------[ cut here ]------------ [ 245.253110] WARNING: CPU: 38 PID: 6 at drivers/iommu/intel-iommu.c:2305 __domain_mapping+0x367/0x380 [ 245.263329] Modules linked in: nvme_rdma nvme_fabrics nvme_core sch_mqprio ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter bridge 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ibd [ 245.342493] mgag200 i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm mlx4_core tg3 ahci libahci ptp libata crc32c_intel i2c_core pps_core devlink dm_mirror dm_region_hash dmd [ 245.364191] CPU: 38 PID: 6 Comm: kworker/u368:0 Not tainted 4.14.0-rc1+ #7 [ 245.371880] Hardware name: Dell Inc. PowerEdge R430/03XKDV, BIOS 1.6.2 01/08/2016 [ 245.380265] Workqueue: ib_addr process_one_req [ib_core] [ 245.386211] task: ffff88018cb245c0 task.stack: ffffc9000009c000 [ 245.392836] RIP: 0010:__domain_mapping+0x367/0x380 [ 245.398194] RSP: 0018:ffffc9000009fa98 EFLAGS: 00010202 [ 245.404039] RAX: 0000000000000004 RBX: 000000103b85e003 RCX: 0000000000000000 [ 245.412018] RDX: 0000000000000000 RSI: ffff88103eace038 RDI: ffff88103eace038 [ 245.420001] RBP: ffffc9000009faf8 R08: 0000000000000000 R09: 0000000000000000 [ 245.427983] R10: 00000000000002f7 R11: 000000000103b85e R12: ffff881009bc74d8 [ 245.436711] R13: 0000000000000001 R14: 0000000000000001 R15: 00000000000e109b [ 245.445419] FS: 0000000000000000(0000) GS:ffff88103eac0000(0000) knlGS:0000000000000000 [ 245.455199] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 245.462357] CR2: 000014940a9b7140 CR3: 00000010119b5000 CR4: 00000000001606e0 [ 245.471074] Call Trace: [ 245.474549] __intel_map_single+0xeb/0x180 [ 245.479868] intel_alloc_coherent+0xb5/0x130 [ 245.485388] mlx4_buf_alloc+0xe5/0x1c0 [mlx4_core] [ 245.491482] mlx4_ib_alloc_cq_buf.isra.9+0x38/0xd0 [mlx4_ib] [ 245.498540] mlx4_ib_create_cq+0x223/0x450 [mlx4_ib] [ 245.504822] ib_alloc_cq+0x49/0x170 [ib_core] [ 245.510413] nvme_rdma_cm_handler+0x3a2/0x7ab [nvme_rdma] [ 245.517179] ? cma_acquire_dev+0x1e3/0x3b0 [rdma_cm] [ 245.523456] addr_handler+0xa4/0x1c0 [rdma_cm] [ 245.529147] process_one_req+0x8d/0x120 [ib_core] [ 245.535132] process_one_work+0x149/0x360 [ 245.540334] worker_thread+0x4d/0x3c0 [ 245.545145] kthread+0x109/0x140 [ 245.549462] ? rescuer_thread+0x380/0x380 [ 245.554654] ? kthread_park+0x60/0x60 [ 245.559456] ret_from_fork+0x25/0x30 [ 245.564153] Code: fe aa 81 4c 89 5d a0 4c 89 4d a8 e8 87 e1 c0 ff 8b 05 fe 6e 87 00 4c 8b 4d a8 4c 8b 5d a0 85 c0 74 09 83 e8 01 89 05 e9 6e 87 00 <0f> ff e9 b8 fd ff ff e8 8d c7 ba ff 0f 1f 00 66 2e 0f 1f 8 [ 245.586712] ---[ end trace 56749c1831388ff8 ]--- [ 245.592920] mlx4_core 0000:04:00.0: dma_pool_free mlx4_cmd, cccccccccccccccc/ccd80eccccccf203 (bad dma) [ 245.604179] mlx4_core 0000:04:00.0: dma_pool_free mlx4_cmd, cccccccccccccccc/cccccccccccccccc (bad dma) [ 245.615647] general protection fault: 0000 [#1] SMP [ 245.621836] Modules linked in: nvme_rdma nvme_fabrics nvme_core sch_mqprio ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter bridge 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ibd [ 245.706171] mgag200 i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm mlx4_core tg3 ahci libahci ptp libata crc32c_intel i2c_core pps_core devlink dm_mirror dm_region_hash dmd [ 245.729344] CPU: 38 PID: 6 Comm: kworker/u368:0 Tainted: G W 4.14.0-rc1+ #7 [ 245.739128] Hardware name: Dell Inc. PowerEdge R430/03XKDV, BIOS 1.6.2 01/08/2016 [ 245.748234] Workqueue: ib_addr process_one_req [ib_core] [ 245.754905] task: ffff88018cb245c0 task.stack: ffffc9000009c000 [ 245.762256] RIP: 0010:prefetch_freepointer.isra.65+0x11/0x20 [ 245.769313] RSP: 0018:ffffc9000009fcc0 EFLAGS: 00010286 [ 245.775881] RAX: 0000000000000000 RBX: cccccccccccccccc RCX: 0000000000001793 [ 245.784591] RDX: 0000000000001792 RSI: cccccccccccccccc RDI: ffff88018fc07aa0 [ 245.793294] RBP: ffffc9000009fcc0 R08: 000000000001ed40 R09: ffff8810098cccc0 [ 245.802002] R10: ffffffff818a99e0 R11: 00000000010098cd R12: 00000000014080c0 [ 245.810706] R13: ffffffffa07bd1e0 R14: ffff88018fc07a80 R15: ffff88018fc07a80 [ 245.819409] FS: 0000000000000000(0000) GS:ffff88103eac0000(0000) knlGS:0000000000000000 [ 245.829184] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 245.836342] CR2: 000014940a9b7140 CR3: 00000010119b5000 CR4: 00000000001606e0 [ 245.845056] Call Trace: [ 245.848524] kmem_cache_alloc_trace+0xa0/0x1c0 [ 245.854220] nvme_rdma_cm_handler+0x4e0/0x7ab [nvme_rdma] [ 245.860990] addr_handler+0xa4/0x1c0 [rdma_cm] [ 245.866694] process_one_req+0x8d/0x120 [ib_core] [ 245.872687] process_one_work+0x149/0x360 [ 245.877899] worker_thread+0x4d/0x3c0 [ 245.882720] kthread+0x109/0x140 [ 245.887051] ? rescuer_thread+0x380/0x380 [ 245.892244] ? kthread_park+0x60/0x60 [ 245.897054] ret_from_fork+0x25/0x30 [ 245.901760] Code: 31 d2 e8 b3 ea ff ff 5b 41 5c 5d c3 0f 1f 40 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 55 48 85 f6 48 89 e5 74 0a 48 63 07 <48> 8b 04 06 0f 18 08 5d c3 66 0f 1f 44 00 00 0f 1f 44 00 0 [ 245.924349] RIP: prefetch_freepointer.isra.65+0x11/0x20 RSP: ffffc9000009fcc0 [ 245.933145] ---[ end trace 56749c1831388ff9 ]--- [ 245.942680] Kernel panic - not syncing: Fatal exception [ 245.950207] Kernel Offset: disabled [ 245.958566] ---[ end Kernel panic - not syncing: Fatal exception [ 245.966082] ------------[ cut here ]------------ [ 245.972014] WARNING: CPU: 38 PID: 6 at kernel/sched/core.c:1179 set_task_cpu+0x191/0x1a0 [ 245.981822] Modules linked in: nvme_rdma nvme_fabrics nvme_core sch_mqprio ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter bridge 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ibd [ 246.066533] mgag200 i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm mlx4_core tg3 ahci libahci ptp libata crc32c_intel i2c_core pps_core devlink dm_mirror dm_region_hash dmd [ 246.089836] CPU: 38 PID: 6 Comm: kworker/u368:0 Tainted: G D W 4.14.0-rc1+ #7 [ 246.099683] Hardware name: Dell Inc. PowerEdge R430/03XKDV, BIOS 1.6.2 01/08/2016 [ 246.108849] Workqueue: ib_addr process_one_req [ib_core] [ 246.115566] task: ffff88018cb245c0 task.stack: ffffc9000009c000 [ 246.122948] RIP: 0010:set_task_cpu+0x191/0x1a0 [ 246.128668] RSP: 0018:ffff88103eac3c38 EFLAGS: 00010046 [ 246.135255] RAX: 0000000000000100 RBX: ffff88207bf445c0 RCX: 0000000000000001 [ 246.143978] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88207bf445c0 [ 246.152699] RBP: ffff88103eac3c58 R08: 0000000000000001 R09: 0000000000000000 [ 246.161418] R10: 0000000000000001 R11: 0000000003e236eb R12: ffff88207bf4516c [ 246.170137] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000001b900 [ 246.178854] FS: 0000000000000000(0000) GS:ffff88103eac0000(0000) knlGS:0000000000000000 [ 246.188644] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 246.195812] CR2: 000014940a9b7140 CR3: 00000010119b5000 CR4: 00000000001606e0 [ 246.204540] Call Trace: [ 246.208027] <IRQ> [ 246.211016] try_to_wake_up+0x166/0x470 [ 246.216036] default_wake_function+0x12/0x20 [ 246.221537] __wake_up_common+0x8a/0x160 [ 246.226641] __wake_up_locked+0x16/0x20 [ 246.231643] ep_poll_callback+0xd0/0x300 [ 246.236727] __wake_up_common+0x8a/0x160 [ 246.241817] __wake_up_common_lock+0x7e/0xc0 [ 246.247291] __wake_up+0x13/0x20 [ 246.251596] wake_up_klogd_work_func+0x40/0x60 [ 246.257265] irq_work_run_list+0x4d/0x70 [ 246.262353] ? tick_sched_do_timer+0x70/0x70 [ 246.267830] irq_work_tick+0x40/0x50 [ 246.272530] update_process_times+0x42/0x60 [ 246.277912] tick_sched_handle+0x2d/0x60 [ 246.282987] tick_sched_timer+0x39/0x70 [ 246.287945] __hrtimer_run_queues+0xe5/0x230 [ 246.293371] hrtimer_interrupt+0xa8/0x1a0 [ 246.298509] smp_apic_timer_interrupt+0x5f/0x130 [ 246.304322] apic_timer_interrupt+0x9d/0xb0 [ 246.309640] </IRQ> [ 246.312633] RIP: 0010:panic+0x1fd/0x245 [ 246.317554] RSP: 0018:ffffc9000009fb18 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff10 [ 246.326659] RAX: 0000000000000034 RBX: 0000000000000200 RCX: 0000000000000006 [ 246.335268] RDX: 0000000000000000 RSI: 0000000000000086 RDI: ffff88103eace030 [ 246.343856] RBP: ffffc9000009fb88 R08: 0000000000000000 R09: 0000000000000877 [ 246.352424] R10: 00000000000003ff R11: 0000000000000001 R12: ffffffff81a3e1d8 [ 246.360975] R13: 0000000000000000 R14: 0000000000000000 R15: ffff88018fc07a80 [ 246.369508] ? panic+0x1f6/0x245 [ 246.373657] oops_end+0xb8/0xd0 [ 246.377676] die+0x42/0x50 [ 246.381194] do_general_protection+0xd2/0x160 [ 246.386540] ? nvme_rdma_cm_handler+0x4e0/0x7ab [nvme_rdma] [ 246.393238] general_protection+0x22/0x30 [ 246.398181] RIP: 0010:prefetch_freepointer.isra.65+0x11/0x20 [ 246.404964] RSP: 0018:ffffc9000009fcc0 EFLAGS: 00010286 [ 246.411258] RAX: 0000000000000000 RBX: cccccccccccccccc RCX: 0000000000001793 [ 246.419692] RDX: 0000000000001792 RSI: cccccccccccccccc RDI: ffff88018fc07aa0 [ 246.428115] RBP: ffffc9000009fcc0 R08: 000000000001ed40 R09: ffff8810098cccc0 [ 246.436543] R10: ffffffff818a99e0 R11: 00000000010098cd R12: 00000000014080c0 [ 246.444970] R13: ffffffffa07bd1e0 R14: ffff88018fc07a80 R15: ffff88018fc07a80 [ 246.453402] ? nvme_rdma_cm_handler+0x4e0/0x7ab [nvme_rdma] [ 246.460087] kmem_cache_alloc_trace+0xa0/0x1c0 [ 246.465511] nvme_rdma_cm_handler+0x4e0/0x7ab [nvme_rdma] [ 246.472004] addr_handler+0xa4/0x1c0 [rdma_cm] [ 246.477424] process_one_req+0x8d/0x120 [ib_core] [ 246.483128] process_one_work+0x149/0x360 [ 246.488045] worker_thread+0x4d/0x3c0 [ 246.492577] kthread+0x109/0x140 [ 246.496620] ? rescuer_thread+0x380/0x380 [ 246.501540] ? kthread_park+0x60/0x60 [ 246.506070] ret_from_fork+0x25/0x30 [ 246.510496] Code: ff 80 8b ac 08 00 00 04 e9 23 ff ff ff 0f ff e9 bf fe ff ff f7 83 84 00 00 00 fd ff ff ff 0f 84 c9 fe ff ff 0f ff e9 c2 fe ff ff <0f> ff e9 d1 fe ff ff 0f 1f 84 00 00 00 00 00 0f 1f 44 00 0 [ 246.532545] ---[ end trace 56749c1831388ffa ]--- > can you please apply the following patch and report if you see a warning? > -- > diff --git a/drivers/nvme/host/rdma.c b/drivers/nvme/host/rdma.c > index 92a03ff5fb4d..ef50b58b0bb6 100644 > --- a/drivers/nvme/host/rdma.c > +++ b/drivers/nvme/host/rdma.c > @@ -274,7 +274,7 @@ static int nvme_rdma_reinit_request(void *data, > struct request *rq) > struct nvme_rdma_request *req = blk_mq_rq_to_pdu(rq); > int ret = 0; > > - ib_dereg_mr(req->mr); > + WARN_ON_ONCE(ib_dereg_mr(req->mr)); > > req->mr = ib_alloc_mr(dev->pd, IB_MR_TYPE_MEM_REG, > ctrl->max_fr_pages); > -- > > _______________________________________________ > Linux-nvme mailing list > Linux-nvme at lists.infradead.org > http://lists.infradead.org/mailman/listinfo/linux-nvme ^ permalink raw reply [flat|nested] 16+ messages in thread
[parent not found: <20170924103426.GB25094@mtr-leonro.local>]
* nvmeof rdma regression issue on 4.14.0-rc1 (or maybe mlx4?) [not found] ` <20170924103426.GB25094@mtr-leonro.local> @ 2017-09-25 5:20 ` Yi Zhang 2017-09-25 7:06 ` Sagi Grimberg 0 siblings, 1 reply; 16+ messages in thread From: Yi Zhang @ 2017-09-25 5:20 UTC (permalink / raw) On 09/24/2017 06:34 PM, Leon Romanovsky wrote: > On Sun, Sep 24, 2017@05:28:30PM +0800, Yi Zhang wrote: >>> Is it possible that ib_dereg_mr failed? >>> >> It seems not, and finally the system get panic, here is the log: > I looked on the issue during the weekend and didn't see any suspicious > commit in the mlx4 alloc/mapping area. > > Can I ask you to perform git bisect to find the problematic change? Hi Sagi I did git bisect for this issue, seems it was introduced by your patch "Few more patches from the centralization set". Here is the testing on the patch, let me know if you need more info. BAD 148b4e7 nvme-rdma: stop queues instead of simply flipping their state BAD a57bd54 nvme-rdma: introduce configure/destroy io queues Log: [ 127.899255] nvme nvme0: new ctrl: NQN "nqn.2014-08.org.nvmexpress.discovery", addr 172.31.0.90:4420 [ 128.074263] nvme nvme0: creating 40 I/O queues. [ 128.581822] nvme nvme0: new ctrl: NQN "testnqn", addr 172.31.0.90:4420 [ 177.486110] print_req_error: I/O error, dev nvme0n1, sector 0 [ 191.256637] nvme nvme0: Reconnecting in 10 seconds... [ 201.855846] nvme nvme0: Connect rejected: status 8 (invalid service ID). [ 201.863353] nvme nvme0: rdma_resolve_addr wait failed (-104). [ 201.869824] nvme nvme0: Failed reconnect attempt 1 [ 201.875183] nvme nvme0: Reconnecting in 10 seconds... [ 212.087828] nvme nvme0: Connect rejected: status 8 (invalid service ID). [ 212.095330] nvme nvme0: rdma_resolve_addr wait failed (-104). [ 212.101766] nvme nvme0: Failed reconnect attempt 2 [ 212.107129] nvme nvme0: Reconnecting in 10 seconds... [ 222.328398] nvme nvme0: Connect rejected: status 8 (invalid service ID). [ 222.335900] nvme nvme0: rdma_resolve_addr wait failed (-104). [ 222.342335] nvme nvme0: Failed reconnect attempt 3 [ 222.347699] nvme nvme0: Reconnecting in 10 seconds... [ 232.567791] nvme nvme0: Connect rejected: status 8 (invalid service ID). [ 232.575292] nvme nvme0: rdma_resolve_addr wait failed (-104). [ 232.581730] nvme nvme0: Failed reconnect attempt 4 [ 232.587094] nvme nvme0: Reconnecting in 10 seconds... [ 242.827727] nvme nvme0: creating 40 I/O queues. [ 242.832810] DMAR: ERROR: DMA PTE for vPFN 0xe129b already set (to 103c692002 not 1000915003) [ 242.842265] ------------[ cut here ]------------ [ 242.847437] WARNING: CPU: 0 PID: 783 at drivers/iommu/intel-iommu.c:2299 __domain_mapping+0x363/0x370 [ 242.857755] Modules linked in: nvme_rdma nvme_fabrics nvme_core sch_mqprio ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter bridge 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ibd [ 242.936919] mgag200 i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm mlx4_core tg3 ahci libahci crc32c_intel ptp libata i2c_core pps_core devlink dm_mirror dm_region_hash dmd [ 242.958625] CPU: 0 PID: 783 Comm: kworker/u368:1 Not tainted 4.13.0-rc7.a57bd54+ #15 [ 242.967304] Hardware name: Dell Inc. PowerEdge R430/03XKDV, BIOS 1.6.2 01/08/2016 [ 242.975687] Workqueue: ib_addr process_one_req [ib_core] [ 242.981631] task: ffff881019491740 task.stack: ffffc9000b534000 [ 242.989011] RIP: 0010:__domain_mapping+0x363/0x370 [ 242.995108] RSP: 0018:ffffc9000b537a50 EFLAGS: 00010202 [ 243.001694] RAX: 0000000000000004 RBX: 0000001000915003 RCX: 0000000000000000 [ 243.010433] RDX: 0000000000000000 RSI: ffff88103e60e038 RDI: ffff88103e60e038 [ 243.019170] RBP: ffffc9000b537ab0 R08: 0000000000000000 R09: 0000000000000000 [ 243.027893] R10: 00000000000002f7 R11: 0000000001000915 R12: ffff88201ea5c4d8 [ 243.036632] R13: 0000000000000001 R14: 0000000000000001 R15: 00000000000e129b [ 243.045348] FS: 0000000000000000(0000) GS:ffff88103e600000(0000) knlGS:0000000000000000 [ 243.055142] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 243.062337] CR2: 00007f9d3b0fab70 CR3: 00000020245b7000 CR4: 00000000001406f0 [ 243.071076] Call Trace: [ 243.074564] __intel_map_single+0xeb/0x180 [ 243.079903] intel_alloc_coherent+0xb5/0x130 [ 243.085445] mlx4_buf_alloc+0xe5/0x1c0 [mlx4_core] [ 243.091555] mlx4_ib_alloc_cq_buf.isra.9+0x38/0xd0 [mlx4_ib] [ 243.098621] mlx4_ib_create_cq+0x223/0x440 [mlx4_ib] [ 243.104901] ? find_gid.isra.5+0x167/0x1f0 [ib_core] [ 243.111178] ib_alloc_cq+0x49/0x170 [ib_core] [ 243.116791] nvme_rdma_cm_handler+0x3e7/0x886 [nvme_rdma] [ 243.123557] ? cma_attach_to_dev+0x17/0x50 [rdma_cm] [ 243.129838] ? cma_acquire_dev+0x1e3/0x3b0 [rdma_cm] [ 243.136115] ? account_entity_dequeue+0xaa/0xe0 [ 243.141918] addr_handler+0xa4/0x1c0 [rdma_cm] [ 243.147604] process_one_req+0x8d/0x120 [ib_core] [ 243.153585] process_one_work+0x149/0x360 [ 243.158807] worker_thread+0x4d/0x3c0 [ 243.163618] kthread+0x109/0x140 [ 243.167936] ? rescuer_thread+0x380/0x380 [ 243.173131] ? kthread_park+0x60/0x60 [ 243.177930] ret_from_fork+0x25/0x30 [ 243.182641] Code: f1 a9 81 4c 89 5d a0 4c 89 4d a8 e8 0b 58 c1 ff 8b 05 f2 16 88 00 4c 8b 4d a8 4c 8b 5d a0 85 c0 74 09 83 e8 01 89 05 dd 16 88 00 <0f> ff e9 bc fd ff ff e8 21 3d bb ff 90 0f 1f 44 00 00 55 4 [ 243.205193] ---[ end trace 725c2de52628c061 ]--- [ 243.211723] mlx4_core 0000:04:00.0: dma_pool_free mlx4_cmd, cccccccccccccccc/cc242fcccccc2801 (bad dma) [ 243.211724] mlx4_core 0000:04:00.0: dma_pool_free mlx4_cmd, cccccccccccccccc/cccccccccccccccc (bad dma) [ 243.212312] general protection fault: 0000 [#1] SMP [ 243.212312] Modules linked in: nvme_rdma nvme_fabrics nvme_core sch_mqprio ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter bridge 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ibd [ 243.212339] mgag200 i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm mlx4_core tg3 ahci libahci crc32c_intel ptp libata i2c_core pps_core devlink dm_mirror dm_region_hash dmd [ 243.212353] CPU: 36 PID: 783 Comm: kworker/u368:1 Tainted: G W 4.13.0-rc7.a57bd54+ #15 [ 243.212353] Hardware name: Dell Inc. PowerEdge R430/03XKDV, BIOS 1.6.2 01/08/2016 [ 243.212360] Workqueue: ib_addr process_one_req [ib_core] [ 243.212361] task: ffff881019491740 task.stack: ffffc9000b534000 [ 243.212364] RIP: 0010:kmem_cache_alloc_trace+0x7d/0x1b0 [ 243.212364] RSP: 0018:ffffc9000b537c88 EFLAGS: 00010282 [ 243.212365] RAX: 0000000000000000 RBX: 00000000014080c0 RCX: 0000000000006b83 [ 243.212366] RDX: 0000000000006b82 RSI: 00000000014080c0 RDI: ffff88018fc07a80 [ 243.212366] RBP: ffffc9000b537cc0 R08: 000000000001ed40 R09: 0000000000000000 [ 243.212373] R10: ffff88018fc07a80 R11: 000000000103c693 R12: 00000000014080c0 [ 243.212374] R13: ffffffffa08b317f R14: ffff88018fc07a80 R15: cccccccccccccccc [ 243.212375] FS: 0000000000000000(0000) GS:ffff88103ea80000(0000) knlGS:0000000000000000 [ 243.212375] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 243.212376] CR2: 00007f736b47c000 CR3: 000000201b770000 CR4: 00000000001406e0 [ 243.212377] Call Trace: [ 243.212379] nvme_rdma_cm_handler+0x4ef/0x886 [nvme_rdma] [ 243.212382] ? cma_attach_to_dev+0x17/0x50 [rdma_cm] [ 243.212383] ? nvme_rdma_memreg_done+0x30/0x30 [nvme_rdma] [ 243.212385] addr_handler+0xa4/0x1c0 [rdma_cm] [ 243.212390] process_one_req+0x8d/0x120 [ib_core] [ 243.212398] process_one_work+0x149/0x360 [ 243.212399] worker_thread+0x4d/0x3c0 [ 243.212400] kthread+0x109/0x140 [ 243.212401] ? rescuer_thread+0x380/0x380 [ 243.212403] ? kthread_park+0x60/0x60 [ 243.212404] ret_from_fork+0x25/0x30 [ 243.212405] Code: 4c 03 05 6f 12 dd 7e 4d 8b 38 49 8b 40 10 4d 85 ff 0f 84 ec 00 00 00 48 85 c0 0f 84 e3 00 00 00 49 63 42 20 48 8d 4a 01 4d 8b 02 <49> 8b 1c 07 4c 89 f8 65 49 0f c7 08 0f 94 c0 84 c0 74 b9 4 [ 243.212420] RIP: kmem_cache_alloc_trace+0x7d/0x1b0 RSP: ffffc9000b537c88 [ 243.212519] ---[ end trace 725c2de52628c062 ]--- [ 243.216792] Kernel panic - not syncing: Fatal exception [ 243.216878] Kernel Offset: disabled [ 243.583898] ---[ end Kernel panic - not syncing: Fatal exception Panic after connection with below commits, detailed log here: https://pastebin.com/7z0XSGSd 31fdf18 nvme-rdma: reuse configure/destroy_admin_queue 3f02fff nvme-rdma: don't free tagset on resets 18398af nvme-rdma: disable the controller on resets b28a308 nvme-rdma: move tagset allocation to a dedicated routine good 34b6c23 nvme: Add admin_tagset pointer to nvme_ctrl Thanks Yi > Added Tariq to the thread. > > Thanks > >> ^ permalink raw reply [flat|nested] 16+ messages in thread
* nvmeof rdma regression issue on 4.14.0-rc1 (or maybe mlx4?) 2017-09-25 5:20 ` Yi Zhang @ 2017-09-25 7:06 ` Sagi Grimberg 2017-09-25 11:01 ` Yi Zhang 0 siblings, 1 reply; 16+ messages in thread From: Sagi Grimberg @ 2017-09-25 7:06 UTC (permalink / raw) > Hi Sagi > I did git bisect for this issue, seems it was introduced by your patch > "Few more patches from the centralization set". > Here is the testing on the patch, let me know if you need more info. Thanks for reporting. > BAD 148b4e7 nvme-rdma: stop queues instead of simply flipping their state > BAD a57bd54 nvme-rdma: introduce configure/destroy io queues > Log: > [ 127.899255] nvme nvme0: new ctrl: NQN > "nqn.2014-08.org.nvmexpress.discovery", addr 172.31.0.90:4420 > [ 128.074263] nvme nvme0: creating 40 I/O queues. > [ 128.581822] nvme nvme0: new ctrl: NQN "testnqn", addr 172.31.0.90:4420 > [ 177.486110] print_req_error: I/O error, dev nvme0n1, sector 0 > [ 191.256637] nvme nvme0: Reconnecting in 10 seconds... > [ 201.855846] nvme nvme0: Connect rejected: status 8 (invalid service ID). > [ 201.863353] nvme nvme0: rdma_resolve_addr wait failed (-104). > [ 201.869824] nvme nvme0: Failed reconnect attempt 1 > [ 201.875183] nvme nvme0: Reconnecting in 10 seconds... > [ 212.087828] nvme nvme0: Connect rejected: status 8 (invalid service ID). > [ 212.095330] nvme nvme0: rdma_resolve_addr wait failed (-104). > [ 212.101766] nvme nvme0: Failed reconnect attempt 2 > [ 212.107129] nvme nvme0: Reconnecting in 10 seconds... > [ 222.328398] nvme nvme0: Connect rejected: status 8 (invalid service ID). > [ 222.335900] nvme nvme0: rdma_resolve_addr wait failed (-104). > [ 222.342335] nvme nvme0: Failed reconnect attempt 3 > [ 222.347699] nvme nvme0: Reconnecting in 10 seconds... > [ 232.567791] nvme nvme0: Connect rejected: status 8 (invalid service ID). > [ 232.575292] nvme nvme0: rdma_resolve_addr wait failed (-104). > [ 232.581730] nvme nvme0: Failed reconnect attempt 4 > [ 232.587094] nvme nvme0: Reconnecting in 10 seconds... > [ 242.827727] nvme nvme0: creating 40 I/O queues. > [ 242.832810] DMAR: ERROR: DMA PTE for vPFN 0xe129b already set (to > 103c692002 not 1000915003) > [ 242.842265] ------------[ cut here ]------------ > [ 242.847437] WARNING: CPU: 0 PID: 783 at > drivers/iommu/intel-iommu.c:2299 __domain_mapping+0x363/0x370 > [ 242.857755] Modules linked in: nvme_rdma nvme_fabrics nvme_core > sch_mqprio ebtable_filter ebtables ip6table_filter ip6_tables > iptable_filter bridge 8021q garp mrp stp llc rpcrdma ib_isert > iscsi_target_mod ibd > [ 242.936919] mgag200 i2c_algo_bit drm_kms_helper syscopyarea > sysfillrect sysimgblt fb_sys_fops ttm drm mlx4_core tg3 ahci libahci > crc32c_intel ptp libata i2c_core pps_core devlink dm_mirror > dm_region_hash dmd > [ 242.958625] CPU: 0 PID: 783 Comm: kworker/u368:1 Not tainted > 4.13.0-rc7.a57bd54+ #15 > [ 242.967304] Hardware name: Dell Inc. PowerEdge R430/03XKDV, BIOS > 1.6.2 01/08/2016 > [ 242.975687] Workqueue: ib_addr process_one_req [ib_core] > [ 242.981631] task: ffff881019491740 task.stack: ffffc9000b534000 > [ 242.989011] RIP: 0010:__domain_mapping+0x363/0x370 > [ 242.995108] RSP: 0018:ffffc9000b537a50 EFLAGS: 00010202 > [ 243.001694] RAX: 0000000000000004 RBX: 0000001000915003 RCX: > 0000000000000000 > [ 243.010433] RDX: 0000000000000000 RSI: ffff88103e60e038 RDI: > ffff88103e60e038 > [ 243.019170] RBP: ffffc9000b537ab0 R08: 0000000000000000 R09: > 0000000000000000 > [ 243.027893] R10: 00000000000002f7 R11: 0000000001000915 R12: > ffff88201ea5c4d8 > [ 243.036632] R13: 0000000000000001 R14: 0000000000000001 R15: > 00000000000e129b > [ 243.045348] FS: 0000000000000000(0000) GS:ffff88103e600000(0000) > knlGS:0000000000000000 > [ 243.055142] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > [ 243.062337] CR2: 00007f9d3b0fab70 CR3: 00000020245b7000 CR4: > 00000000001406f0 > [ 243.071076] Call Trace: > [ 243.074564] __intel_map_single+0xeb/0x180 > [ 243.079903] intel_alloc_coherent+0xb5/0x130 > [ 243.085445] mlx4_buf_alloc+0xe5/0x1c0 [mlx4_core] > [ 243.091555] mlx4_ib_alloc_cq_buf.isra.9+0x38/0xd0 [mlx4_ib] > [ 243.098621] mlx4_ib_create_cq+0x223/0x440 [mlx4_ib] > [ 243.104901] ? find_gid.isra.5+0x167/0x1f0 [ib_core] > [ 243.111178] ib_alloc_cq+0x49/0x170 [ib_core] > [ 243.116791] nvme_rdma_cm_handler+0x3e7/0x886 [nvme_rdma] > [ 243.123557] ? cma_attach_to_dev+0x17/0x50 [rdma_cm] > [ 243.129838] ? cma_acquire_dev+0x1e3/0x3b0 [rdma_cm] > [ 243.136115] ? account_entity_dequeue+0xaa/0xe0 > [ 243.141918] addr_handler+0xa4/0x1c0 [rdma_cm] > [ 243.147604] process_one_req+0x8d/0x120 [ib_core] > [ 243.153585] process_one_work+0x149/0x360 > [ 243.158807] worker_thread+0x4d/0x3c0 > [ 243.163618] kthread+0x109/0x140 > [ 243.167936] ? rescuer_thread+0x380/0x380 > [ 243.173131] ? kthread_park+0x60/0x60 > [ 243.177930] ret_from_fork+0x25/0x30 I'm not able to reproduce this. I still don't understand how we call ib_alloc_cq and get to a mapped buffer, its calling dma_alloc_coherent. Can you try the following patch and report what's going on? -- diff --git a/drivers/nvme/host/rdma.c b/drivers/nvme/host/rdma.c index 92a03ff5fb4d..3befaa0c53ff 100644 --- a/drivers/nvme/host/rdma.c +++ b/drivers/nvme/host/rdma.c @@ -927,10 +927,6 @@ static void nvme_rdma_reconnect_ctrl_work(struct work_struct *work) ++ctrl->ctrl.nr_reconnects; - if (ctrl->ctrl.queue_count > 1) - nvme_rdma_destroy_io_queues(ctrl, false); - - nvme_rdma_destroy_admin_queue(ctrl, false); ret = nvme_rdma_configure_admin_queue(ctrl, false); if (ret) goto requeue; @@ -938,7 +934,7 @@ static void nvme_rdma_reconnect_ctrl_work(struct work_struct *work) if (ctrl->ctrl.queue_count > 1) { ret = nvme_rdma_configure_io_queues(ctrl, false); if (ret) - goto requeue; + goto destroy_admin; } changed = nvme_change_ctrl_state(&ctrl->ctrl, NVME_CTRL_LIVE); @@ -956,6 +952,8 @@ static void nvme_rdma_reconnect_ctrl_work(struct work_struct *work) return; +destroy_admin: + nvme_rdma_destroy_admin_queue(ctrl, false); requeue: dev_info(ctrl->ctrl.device, "Failed reconnect attempt %d\n", ctrl->ctrl.nr_reconnects); @@ -990,6 +988,10 @@ static void nvme_rdma_error_recovery_work(struct work_struct *work) blk_mq_unquiesce_queue(ctrl->ctrl.admin_q); nvme_start_queues(&ctrl->ctrl); + if (ctrl->ctrl.queue_count > 1) + nvme_rdma_destroy_io_queues(ctrl, false); + nvme_rdma_destroy_admin_queue(ctrl, false); + nvme_rdma_reconnect_or_remove(ctrl); } -- I guess we could see weird phenomenons if we have a resource leak, but I ran kmemleak and could not get anything in this area... > Panic after connection with below commits, detailed log here: > https://pastebin.com/7z0XSGSd > 31fdf18 nvme-rdma: reuse configure/destroy_admin_queue > 3f02fff nvme-rdma: don't free tagset on resets > 18398af nvme-rdma: disable the controller on resets > b28a308 nvme-rdma: move tagset allocation to a dedicated routine > > good 34b6c23 nvme: Add admin_tagset pointer to nvme_ctrl Is that a reproducible panic? I'm not seeing this at all. Can you run gdb on nvme-rdma.ko $ l *(nvme_rdma_create_ctrl+0x37d) ^ permalink raw reply related [flat|nested] 16+ messages in thread
* nvmeof rdma regression issue on 4.14.0-rc1 (or maybe mlx4?) 2017-09-25 7:06 ` Sagi Grimberg @ 2017-09-25 11:01 ` Yi Zhang 2017-09-25 11:15 ` Sagi Grimberg 2017-10-02 12:51 ` Sagi Grimberg 0 siblings, 2 replies; 16+ messages in thread From: Yi Zhang @ 2017-09-25 11:01 UTC (permalink / raw) Hi Sagi Pls see my comments bellow. > I'm not able to reproduce this. > I've reproduce this on below two environment, and the target side I used is nullbk [root at rdma-virt-00 ~]$ lspci | grep -i mel 04:00.0 Network controller: Mellanox Technologies MT27520 Family [ConnectX-3 Pro] 04:00.1 Network controller: Mellanox Technologies MT27500/MT27520 Family [ConnectX-3/ConnectX-3 Pro Virtual Function] 04:00.2 Network controller: Mellanox Technologies MT27500/MT27520 Family [ConnectX-3/ConnectX-3 Pro Virtual Function] 04:00.3 Network controller: Mellanox Technologies MT27500/MT27520 Family [ConnectX-3/ConnectX-3 Pro Virtual Function] 04:00.4 Network controller: Mellanox Technologies MT27500/MT27520 Family [ConnectX-3/ConnectX-3 Pro Virtual Function] 04:00.5 Network controller: Mellanox Technologies MT27500/MT27520 Family [ConnectX-3/ConnectX-3 Pro Virtual Function] 04:00.6 Network controller: Mellanox Technologies MT27500/MT27520 Family [ConnectX-3/ConnectX-3 Pro Virtual Function] 04:00.7 Network controller: Mellanox Technologies MT27500/MT27520 Family [ConnectX-3/ConnectX-3 Pro Virtual Function] 06:00.0 Network controller: Mellanox Technologies MT27520 Family [ConnectX-3 Pro] 06:00.1 Network controller: Mellanox Technologies MT27500/MT27520 Family [ConnectX-3/ConnectX-3 Pro Virtual Function] 06:00.2 Network controller: Mellanox Technologies MT27500/MT27520 Family [ConnectX-3/ConnectX-3 Pro Virtual Function] 06:00.3 Network controller: Mellanox Technologies MT27500/MT27520 Family [ConnectX-3/ConnectX-3 Pro Virtual Function] 06:00.4 Network controller: Mellanox Technologies MT27500/MT27520 Family [ConnectX-3/ConnectX-3 Pro Virtual Function] 06:00.5 Network controller: Mellanox Technologies MT27500/MT27520 Family [ConnectX-3/ConnectX-3 Pro Virtual Function] 06:00.6 Network controller: Mellanox Technologies MT27500/MT27520 Family [ConnectX-3/ConnectX-3 Pro Virtual Function] 06:00.7 Network controller: Mellanox Technologies MT27500/MT27520 Family [ConnectX-3/ConnectX-3 Pro Virtual Function] [root at rdma-virt-03 ~]$ lspci | grep -i mel 04:00.0 Infiniband controller: Mellanox Technologies MT27700 Family [ConnectX-4] 04:00.1 Infiniband controller: Mellanox Technologies MT27700 Family [ConnectX-4] 05:00.0 Ethernet controller: Mellanox Technologies MT27710 Family [ConnectX-4 Lx] 05:00.1 Ethernet controller: Mellanox Technologies MT27710 Family [ConnectX-4 Lx] > I still don't understand how we call ib_alloc_cq and get to > a mapped buffer, its calling dma_alloc_coherent. > > Can you try the following patch and report what's going on? > -- > Here is the log, didn't get the error log, but the nvme0n1 device node also doesn't exist on host. [ 144.993253] nvme nvme0: new ctrl: NQN "nqn.2014-08.org.nvmexpress.discovery", addr 172.31.0.90:4420 [ 145.177205] nvme nvme0: creating 40 I/O queues. [ 145.711199] nvme nvme0: new ctrl: NQN "testnqn", addr 172.31.0.90:4420 [ 156.482770] nvme nvme0: Reconnecting in 10 seconds... [ 166.881540] nvme nvme0: Connect rejected: status 8 (invalid service ID). [ 166.889043] nvme nvme0: rdma_resolve_addr wait failed (-104). [ 166.895478] nvme nvme0: Failed reconnect attempt 1 [ 166.900840] nvme nvme0: Reconnecting in 10 seconds... [ 177.120933] nvme nvme0: Connect rejected: status 8 (invalid service ID). [ 177.128434] nvme nvme0: rdma_resolve_addr wait failed (-104). [ 177.134866] nvme nvme0: Failed reconnect attempt 2 [ 177.140227] nvme nvme0: Reconnecting in 10 seconds... [ 187.360819] nvme nvme0: Connect rejected: status 8 (invalid service ID). [ 187.368321] nvme nvme0: rdma_resolve_addr wait failed (-104). [ 187.374752] nvme nvme0: Failed reconnect attempt 3 [ 187.380113] nvme nvme0: Reconnecting in 10 seconds... [ 197.620302] nvme nvme0: creating 40 I/O queues. [ 198.227963] nvme nvme0: Successfully reconnected [ 198.228046] nvme nvme0: identifiers changed for nsid 1 [root at rdma-virt-01 ~]$ lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT sda 8:0 0 465.8G 0 disk ??sda2 8:2 0 464.8G 0 part ? ??rhelaa_rdma--virt--01-swap 253:1 0 4G 0 lvm [SWAP] ? ??rhelaa_rdma--virt--01-home 253:2 0 410.8G 0 lvm /home ? ??rhelaa_rdma--virt--01-root 253:0 0 50G 0 lvm / ??sda1 8:1 0 1G 0 part /boot > -- > > I guess we could see weird phenomenons if we have a resource leak, > but I ran kmemleak and could not get anything in this area... > >> Panic after connection with below commits, detailed log here: >> https://pastebin.com/7z0XSGSd >> 31fdf18 nvme-rdma: reuse configure/destroy_admin_queue >> 3f02fff nvme-rdma: don't free tagset on resets >> 18398af nvme-rdma: disable the controller on resets >> b28a308 nvme-rdma: move tagset allocation to a dedicated routine >> >> good 34b6c23 nvme: Add admin_tagset pointer to nvme_ctrl > > Is that a reproducible panic? I'm not seeing this at all. > Yes, I can reproduce every time. And the target side kernel version is 4.14.0-rc1 during the panic occurred. > Can you run gdb on nvme-rdma.ko > $ l *(nvme_rdma_create_ctrl+0x37d) > [root at rdma-virt-01 linux ((31fdf18...))]$ gdb /usr/lib/modules/4.13.0-rc7.31fdf18+/kernel/drivers/nvme/host/nvme-rdma.ko GNU gdb (GDB) Red Hat Enterprise Linux 7.6.1-100.el7 Copyright (C) 2013 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-redhat-linux-gnu". For bug reporting instructions, please see: <http://www.gnu.org/software/gdb/bugs/>... Reading symbols from /usr/lib/modules/4.13.0-rc7.31fdf18+/kernel/drivers/nvme/host/nvme-rdma.ko...done. (gdb) l *(nvme_rdma_create_ctrl+0x37d) 0x297d is in nvme_rdma_create_ctrl (drivers/nvme/host/rdma.c:656). 651 struct nvme_rdma_ctrl *ctrl = to_rdma_ctrl(nctrl); 652 struct blk_mq_tag_set *set = admin ? 653 &ctrl->admin_tag_set : &ctrl->tag_set; 654 655 blk_mq_free_tag_set(set); 656 nvme_rdma_dev_put(ctrl->device); 657 } 658 659 static struct blk_mq_tag_set *nvme_rdma_alloc_tagset(struct nvme_ctrl *nctrl, 660 bool admin) (gdb) > _______________________________________________ > Linux-nvme mailing list > Linux-nvme at lists.infradead.org > http://lists.infradead.org/mailman/listinfo/linux-nvme ^ permalink raw reply [flat|nested] 16+ messages in thread
* nvmeof rdma regression issue on 4.14.0-rc1 (or maybe mlx4?) 2017-09-25 11:01 ` Yi Zhang @ 2017-09-25 11:15 ` Sagi Grimberg 2017-10-02 12:51 ` Sagi Grimberg 1 sibling, 0 replies; 16+ messages in thread From: Sagi Grimberg @ 2017-09-25 11:15 UTC (permalink / raw) > Here is the log, didn't get the error log, but the nvme0n1 device node > also doesn't exist on host. That's because the identifiers changed maybe? did you have nvm0n2 instead? > Yes, I can reproduce every time. And the target side kernel version is > 4.14.0-rc1 during the panic occurred. What is the host side kernel version? doesn't look like 4.14.0-rc1 ^ permalink raw reply [flat|nested] 16+ messages in thread
* nvmeof rdma regression issue on 4.14.0-rc1 (or maybe mlx4?) 2017-09-25 11:01 ` Yi Zhang 2017-09-25 11:15 ` Sagi Grimberg @ 2017-10-02 12:51 ` Sagi Grimberg 2017-10-12 8:24 ` Yi Zhang 1 sibling, 1 reply; 16+ messages in thread From: Sagi Grimberg @ 2017-10-02 12:51 UTC (permalink / raw) Hi, >>> Panic after connection with below commits, detailed log here: >>> https://pastebin.com/7z0XSGSd >>> 31fdf18???? nvme-rdma: reuse configure/destroy_admin_queue >>> 3f02fff?????? nvme-rdma: don't free tagset on resets >>> 18398af??? nvme-rdma: disable the controller on resets >>> b28a308?? nvme-rdma: move tagset allocation to a dedicated routine >>> >>> good??? 34b6c23 nvme: Add admin_tagset pointer to nvme_ctrl >> >> Is that a reproducible panic? I'm not seeing this at all. >> > > Yes, I can reproduce every time. And the target side kernel version is > 4.14.0-rc1 during the panic occurred. > >> Can you run gdb on nvme-rdma.ko >> $ l *(nvme_rdma_create_ctrl+0x37d) >> > [root at rdma-virt-01 linux ((31fdf18...))]$ gdb > /usr/lib/modules/4.13.0-rc7.31fdf18+/kernel/drivers/nvme/host/nvme-rdma.ko > GNU gdb (GDB) Red Hat Enterprise Linux 7.6.1-100.el7 > Copyright (C) 2013 Free Software Foundation, Inc. > License GPLv3+: GNU GPL version 3 or later > <http://gnu.org/licenses/gpl.html> > This is free software: you are free to change and redistribute it. > There is NO WARRANTY, to the extent permitted by law.? Type "show copying" > and "show warranty" for details. > This GDB was configured as "x86_64-redhat-linux-gnu". > For bug reporting instructions, please see: > <http://www.gnu.org/software/gdb/bugs/>... > Reading symbols from > /usr/lib/modules/4.13.0-rc7.31fdf18+/kernel/drivers/nvme/host/nvme-rdma.ko...done. > > (gdb) l *(nvme_rdma_create_ctrl+0x37d) > 0x297d is in nvme_rdma_create_ctrl (drivers/nvme/host/rdma.c:656). > 651??????? struct nvme_rdma_ctrl *ctrl = to_rdma_ctrl(nctrl); > 652??????? struct blk_mq_tag_set *set = admin ? > 653??????????????? &ctrl->admin_tag_set : &ctrl->tag_set; > 654 > 655??????? blk_mq_free_tag_set(set); > 656??????? nvme_rdma_dev_put(ctrl->device); > 657??? } > 658 > 659??? static struct blk_mq_tag_set *nvme_rdma_alloc_tagset(struct > nvme_ctrl *nctrl, > 660??????????? bool admin) > (gdb) Lets take this one step at a time, starting with this issue. First, there is a reason why a simple create_ctrl fails, can we isolate exactly which call fails? Was something else going on that might have made the simple create_ctrl fail? We don't see any "rdma_resolve_addr failed" or "failed to connect queue" messages but we do see "creating I/O queues" which means that we either failed at IO tagset allocation or initializing connect_q. We have a missing error code assignment so can you try the following patch: -- diff --git a/drivers/nvme/host/rdma.c b/drivers/nvme/host/rdma.c index 58983000964b..98dd51e630bd 100644 --- a/drivers/nvme/host/rdma.c +++ b/drivers/nvme/host/rdma.c @@ -765,8 +765,10 @@ static int nvme_rdma_configure_admin_queue(struct nvme_rdma_ctrl *ctrl, if (new) { ctrl->ctrl.admin_tagset = nvme_rdma_alloc_tagset(&ctrl->ctrl, true); - if (IS_ERR(ctrl->ctrl.admin_tagset)) + if (IS_ERR(ctrl->ctrl.admin_tagset)) { + error = PTR_ERR(ctrl->ctrl.admin_tagset); goto out_free_queue; + } ctrl->ctrl.admin_q = blk_mq_init_queue(&ctrl->admin_tag_set); if (IS_ERR(ctrl->ctrl.admin_q)) { @@ -846,8 +848,10 @@ static int nvme_rdma_configure_io_queues(struct nvme_rdma_ctrl *ctrl, bool new) if (new) { ctrl->ctrl.tagset = nvme_rdma_alloc_tagset(&ctrl->ctrl, false); - if (IS_ERR(ctrl->ctrl.tagset)) + if (IS_ERR(ctrl->ctrl.tagset)) { + ret = PTR_ERR(ctrl->ctrl.tagset); goto out_free_io_queues; + } ctrl->ctrl.connect_q = blk_mq_init_queue(&ctrl->tag_set); if (IS_ERR(ctrl->ctrl.connect_q)) { -- Also, can you add the following debug messages to find out what failed? -- diff --git a/drivers/nvme/host/rdma.c b/drivers/nvme/host/rdma.c index 58983000964b..e46475100eea 100644 --- a/drivers/nvme/host/rdma.c +++ b/drivers/nvme/host/rdma.c @@ -676,6 +676,12 @@ static void nvme_rdma_free_tagset(struct nvme_ctrl *nctrl, bool admin) struct blk_mq_tag_set *set = admin ? &ctrl->admin_tag_set : &ctrl->tag_set; + if (set == &ctrl->tag_set) { + pr_err("%s: freeing IO tagset\n", __func__); + } else { + pr_err("%s: freeing ADMIN tagset\n", __func__); + } + blk_mq_free_tag_set(set); nvme_rdma_dev_put(ctrl->device); } -- ^ permalink raw reply related [flat|nested] 16+ messages in thread
* nvmeof rdma regression issue on 4.14.0-rc1 (or maybe mlx4?) 2017-10-02 12:51 ` Sagi Grimberg @ 2017-10-12 8:24 ` Yi Zhang 2017-10-16 10:21 ` Sagi Grimberg 0 siblings, 1 reply; 16+ messages in thread From: Yi Zhang @ 2017-10-12 8:24 UTC (permalink / raw) > Lets take this one step at a time, starting with this issue. > > First, there is a reason why a simple create_ctrl fails, can we isolate > exactly which call fails? Was something else going on that might have > made the simple create_ctrl fail? > > We don't see any "rdma_resolve_addr failed" or "failed to connect queue" > messages but we do see "creating I/O queues" which means that we either > failed at IO tagset allocation or initializing connect_q. > > We have a missing error code assignment so can you try the following > patch: Hi Sagi Sorry for the late response, here is the log with your patch, let me know if you need more test. [ 236.980239] nvme nvme0: new ctrl: NQN "nqn.2014-08.org.nvmexpress.discovery", addr 172.31.0.90:4420 [ 237.131416] nvme_rdma: nvme_rdma_free_tagset: freeing ADMIN tagset [ 237.173614] nvme nvme0: creating 40 I/O queues. [ 237.688843] nvme nvme0: new ctrl: NQN "testnqn", addr 172.31.0.90:4420 [ 243.384596] nvme nvme0: rescanning [ 243.527922] nvme nvme0: Reconnecting in 10 seconds... [ 254.010547] nvme nvme0: Connect rejected: status 8 (invalid service ID). [ 254.018052] nvme nvme0: rdma_resolve_addr wait failed (-104). [ 254.024482] nvme nvme0: Failed reconnect attempt 1 [ 254.029844] nvme nvme0: Reconnecting in 10 seconds... [ 264.238223] nvme nvme0: Connect rejected: status 8 (invalid service ID). [ 264.245727] nvme nvme0: rdma_resolve_addr wait failed (-104). [ 264.252159] nvme nvme0: Failed reconnect attempt 2 [ 264.257545] nvme nvme0: Reconnecting in 10 seconds... [ 274.477977] nvme nvme0: Connect rejected: status 8 (invalid service ID). [ 274.485470] nvme nvme0: rdma_resolve_addr wait failed (-104). [ 274.491905] nvme nvme0: Failed reconnect attempt 3 [ 274.497287] nvme nvme0: Reconnecting in 10 seconds... [ 284.737691] nvme nvme0: creating 40 I/O queues. [ 284.745163] DMAR: ERROR: DMA PTE for vPFN 0xe0e9d already set (to 100a1db001 not 1011432003) [ 284.754617] ------------[ cut here ]------------ [ 284.759790] WARNING: CPU: 12 PID: 6 at drivers/iommu/intel-iommu.c:2305 __domain_mapping+0x367/0x380 [ 284.770010] Modules linked in: nvme_rdma nvme_fabrics nvme_core sch_mqprio ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter bridge 8021q garp mrp stp llc rpcrdma id [ 284.849181] mgag200 i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm tg3 ahci mlx4_core libahci ptp libata crc32c_intel i2c_core pps_core devlink d [ 284.870886] CPU: 12 PID: 6 Comm: kworker/u368:0 Not tainted 4.14.0-rc4.10.02.sagi+ #30 [ 284.879746] Hardware name: Dell Inc. PowerEdge R430/03XKDV, BIOS 1.6.2 01/08/2016 [ 284.888133] Workqueue: ib_addr process_one_req [ib_core] [ 284.894074] task: ffff88018cb50000 task.stack: ffffc9000009c000 [ 284.900699] RIP: 0010:__domain_mapping+0x367/0x380 [ 284.906051] RSP: 0018:ffffc9000009fa98 EFLAGS: 00010202 [ 284.911899] RAX: 0000000000000004 RBX: 0000001011432003 RCX: 0000000000000000 [ 284.920643] RDX: 0000000000000000 RSI: ffff88103e78e058 RDI: ffff88103e78e058 [ 284.929371] RBP: ffffc9000009faf8 R08: 0000000000000000 R09: 0000000000000000 [ 284.938088] R10: 00000000000002f7 R11: 0000000001011432 R12: ffff88201ab634e8 [ 284.946814] R13: 0000000000000001 R14: 0000000000000001 R15: 00000000000e0e9d [ 284.955533] FS: 0000000000000000(0000) GS:ffff88103e780000(0000) knlGS:0000000000000000 [ 284.965318] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 284.972491] CR2: 00007f9348b7b000 CR3: 000000202f7e7005 CR4: 00000000001606e0 [ 284.981227] Call Trace: [ 284.984712] __intel_map_single+0xeb/0x180 [ 284.990059] intel_alloc_coherent+0xb5/0x130 [ 284.995584] mlx4_buf_alloc+0xe5/0x1c0 [mlx4_core] [ 285.001690] mlx4_ib_alloc_cq_buf.isra.9+0x38/0xd0 [mlx4_ib] [ 285.008754] mlx4_ib_create_cq+0x223/0x450 [mlx4_ib] [ 285.015027] ib_alloc_cq+0x49/0x170 [ib_core] [ 285.020634] nvme_rdma_cm_handler+0x3a2/0x7ab [nvme_rdma] [ 285.027416] ? cma_acquire_dev+0x1e3/0x3b0 [rdma_cm] [ 285.033689] addr_handler+0xa4/0x1c0 [rdma_cm] [ 285.039417] process_one_req+0x8d/0x120 [ib_core] [ 285.045408] process_one_work+0x149/0x360 [ 285.050620] worker_thread+0x4d/0x3c0 [ 285.055432] kthread+0x109/0x140 [ 285.059780] ? rescuer_thread+0x380/0x380 [ 285.064987] ? kthread_park+0x60/0x60 [ 285.069805] ret_from_fork+0x25/0x30 [ 285.074511] Code: 10 ab 81 4c 89 5d a0 4c 89 4d a8 e8 27 ac c0 ff 8b 05 fe 39 87 00 4c 8b 4d a8 4c 8b 5d a0 85 c0 74 09 83 e8 01 89 05 e9 39 87 00 <0f> ff e9 b8 fd ff ff e8 ed 8e b [ 285.097088] ---[ end trace 501470fecce57311 ]--- [ 285.103299] DMAR: DRHD: handling fault status reg 2 [ 285.103761] DMAR: ERROR: DMA PTE for vPFN 0xe0e9d already set (to 100a1db001 not 100526b002) [ 285.103766] ------------[ cut here ]------------ [ 285.103768] WARNING: CPU: 12 PID: 6 at drivers/iommu/intel-iommu.c:2305 __domain_mapping+0x367/0x380 [ 285.103768] Modules linked in: nvme_rdma nvme_fabrics nvme_core sch_mqprio ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter bridge 8021q garp mrp stp llc rpcrdma id [ 285.103785] mgag200 i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm tg3 ahci mlx4_core libahci ptp libata crc32c_intel i2c_core pps_core devlink d [ 285.103791] CPU: 12 PID: 6 Comm: kworker/u368:0 Tainted: G W 4.14.0-rc4.10.02.sagi+ #30 [ 285.103792] Hardware name: Dell Inc. PowerEdge R430/03XKDV, BIOS 1.6.2 01/08/2016 [ 285.103797] Workqueue: ib_addr process_one_req [ib_core] [ 285.103798] task: ffff88018cb50000 task.stack: ffffc9000009c000 [ 285.103799] RIP: 0010:__domain_mapping+0x367/0x380 [ 285.103800] RSP: 0018:ffffc9000009fc48 EFLAGS: 00010206 [ 285.103800] RAX: 0000000000000003 RBX: 000000100526b002 RCX: 0000000000000000 [ 285.103801] RDX: 0000000000000000 RSI: 0000000000000002 RDI: 0000000000000282 [ 285.103801] RBP: ffffc9000009fca8 R08: 0000000000000050 R09: 0000000000000000 [ 285.103802] R10: 0000000000000860 R11: 000000000100526b R12: ffff88201ab634e8 [ 285.103803] R13: 0000000000000001 R14: 0000000000000001 R15: 00000000000e0e9d [ 285.103803] FS: 0000000000000000(0000) GS:ffff88103e780000(0000) knlGS:0000000000000000 [ 285.103804] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 285.103805] CR2: 00007f9348b7b000 CR3: 000000202f7e7005 CR4: 00000000001606e0 [ 285.103805] Call Trace: [ 285.103808] __intel_map_single+0xeb/0x180 [ 285.103809] intel_map_page+0x39/0x40 [ 285.103811] nvme_rdma_cm_handler+0x47b/0x7ab [nvme_rdma] [ 285.103813] addr_handler+0xa4/0x1c0 [rdma_cm] [ 285.103817] process_one_req+0x8d/0x120 [ib_core] [ 285.103818] process_one_work+0x149/0x360 [ 285.103819] worker_thread+0x4d/0x3c0 [ 285.103821] kthread+0x109/0x140 [ 285.103822] ? rescuer_thread+0x380/0x380 [ 285.103823] ? kthread_park+0x60/0x60 [ 285.103825] ret_from_fork+0x25/0x30 [ 285.103826] Code: 10 ab 81 4c 89 5d a0 4c 89 4d a8 e8 27 ac c0 ff 8b 05 fe 39 87 00 4c 8b 4d a8 4c 8b 5d a0 85 c0 74 09 83 e8 01 89 05 e9 39 87 00 <0f> ff e9 b8 fd ff ff e8 ed 8e b [ 285.103839] ---[ end trace 501470fecce57312 ]--- [ 285.190580] DMAR: ERROR: DMA PTE for vPFN 0xe0e9d already set (to 100a1db001 not 100d51b001) [ 285.190586] ------------[ cut here ]------------ [ 285.190590] WARNING: CPU: 30 PID: 757 at drivers/iommu/intel-iommu.c:2305 __domain_mapping+0x367/0x380 [ 285.190590] Modules linked in: nvme_rdma nvme_fabrics nvme_core sch_mqprio ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter bridge 8021q garp mrp stp llc rpcrdma id [ 285.190609] mgag200 i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm tg3 ahci mlx4_core libahci ptp libata crc32c_intel i2c_core pps_core devlink d [ 285.190616] CPU: 30 PID: 757 Comm: kworker/u369:7 Tainted: G W 4.14.0-rc4.10.02.sagi+ #30 [ 285.190617] Hardware name: Dell Inc. PowerEdge R430/03XKDV, BIOS 1.6.2 01/08/2016 [ 285.190619] Workqueue: nvme-wq nvme_rdma_reconnect_ctrl_work [nvme_rdma] [ 285.190620] task: ffff881032bd8000 task.stack: ffffc9000851c000 [ 285.190622] RIP: 0010:__domain_mapping+0x367/0x380 [ 285.190622] RSP: 0018:ffffc9000851fc60 EFLAGS: 00010202 [ 285.190623] RAX: 0000000000000002 RBX: 000000100d51b001 RCX: 0000000000000000 [ 285.190624] RDX: 0000000000000000 RSI: 0000000000000002 RDI: 0000000000000292 [ 285.190624] RBP: ffffc9000851fcc0 R08: 0000000000000050 R09: 0000000000000000 [ 285.190625] R10: 0000000000000881 R11: 000000000100d51b R12: ffff88201ab634e8 [ 285.190625] R13: 0000000000000001 R14: 0000000000000001 R15: 00000000000e0e9d [ 285.190626] FS: 0000000000000000(0000) GS:ffff88103e9c0000(0000) knlGS:0000000000000000 [ 285.190626] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 285.190627] CR2: 00007f4a9e61fb70 CR3: 0000001011bdd001 CR4: 00000000001606e0 [ 285.190628] Call Trace: [ 285.190630] __intel_map_single+0xeb/0x180 [ 285.190632] intel_map_page+0x39/0x40 [ 285.190636] mlx4_ib_alloc_mr+0x141/0x220 [mlx4_ib] [ 285.190642] ib_alloc_mr+0x26/0x50 [ib_core] [ 285.190644] nvme_rdma_reinit_request+0x3a/0x70 [nvme_rdma] [ 285.190645] ? nvme_rdma_free_ctrl+0xb0/0xb0 [nvme_rdma] [ 285.190647] blk_mq_reinit_tagset+0x5c/0x90 [ 285.190649] nvme_rdma_configure_io_queues+0x204/0x280 [nvme_rdma] [ 285.190650] nvme_rdma_reconnect_ctrl_work+0x5b/0xe0 [nvme_rdma] [ 285.190652] process_one_work+0x149/0x360 [ 285.190653] worker_thread+0x4d/0x3c0 [ 285.190655] kthread+0x109/0x140 [ 285.190656] ? rescuer_thread+0x380/0x380 [ 285.190657] ? kthread_park+0x60/0x60 [ 285.190660] ret_from_fork+0x25/0x30 [ 285.190660] Code: 10 ab 81 4c 89 5d a0 4c 89 4d a8 e8 27 ac c0 ff 8b 05 fe 39 87 00 4c 8b 4d a8 4c 8b 5d a0 85 c0 74 09 83 e8 01 89 05 e9 39 87 00 <0f> ff e9 b8 fd ff ff e8 ed 8e b [ 285.190673] ---[ end trace 501470fecce57313 ]--- [ 285.819126] DMAR: [DMA Write] Request device [04:00.0] fault addr e0e9d000 [fault reason 05] PTE Write access is not set [ 286.035086] nvme nvme0: Successfully reconnected [ 286.035143] nvme nvme0: MEMREG for CQE 0xffff88100a1e6f80 failed with status memory management operation error (6) [ 286.187508] nvme nvme0: Reconnecting in 10 seconds... [ 296.270043] nvme nvme0: creating 40 I/O queues. [ 296.336505] DMAR: ERROR: DMA PTE for vPFN 0xe0e9d already set (to 1012082002 not 1012082002) [ 296.346704] ------------[ cut here ]------------ [ 296.352618] WARNING: CPU: 32 PID: 815 at drivers/iommu/intel-iommu.c:2305 __domain_mapping+0x367/0x380 [ 296.363755] Modules linked in: nvme_rdma nvme_fabrics nvme_core sch_mqprio ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter bridge 8021q garp mrp stp llc rpcrdma id [ 296.447983] mgag200 i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm tg3 ahci mlx4_core libahci ptp libata crc32c_intel i2c_core pps_core devlink d [ 296.471170] CPU: 32 PID: 815 Comm: kworker/u368:1 Tainted: G W 4.14.0-rc4.10.02.sagi+ #30 [ 296.482344] Hardware name: Dell Inc. PowerEdge R430/03XKDV, BIOS 1.6.2 01/08/2016 [ 296.491493] Workqueue: ib_addr process_one_req [ib_core] [ 296.498221] task: ffff8810373d2e80 task.stack: ffffc9000849c000 [ 296.505613] RIP: 0010:__domain_mapping+0x367/0x380 [ 296.511751] RSP: 0018:ffffc9000849fc48 EFLAGS: 00010202 [ 296.518380] RAX: 0000000000000001 RBX: 0000001012082002 RCX: 0000000000000000 [ 296.527146] RDX: 0000000000000000 RSI: ffff88103ea0e058 RDI: ffff88103ea0e058 [ 296.535918] RBP: ffffc9000849fca8 R08: 0000000000000000 R09: 0000000000000000 [ 296.544691] R10: 00000000000003ff R11: 0000000001012082 R12: ffff88201ab634e8 [ 296.553462] R13: 0000000000000001 R14: 0000000000000001 R15: 00000000000e0e9d [ 296.562240] FS: 0000000000000000(0000) GS:ffff88103ea00000(0000) knlGS:0000000000000000 [ 296.572078] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 296.579292] CR2: 00007fdc7ae90000 CR3: 0000001034df0002 CR4: 00000000001606e0 [ 296.588070] Call Trace: [ 296.591620] __intel_map_single+0xeb/0x180 [ 296.597012] intel_map_page+0x39/0x40 [ 296.601912] nvme_rdma_cm_handler+0x47b/0x7ab [nvme_rdma] [ 296.608760] addr_handler+0xa4/0x1c0 [rdma_cm] [ 296.614540] process_one_req+0x8d/0x120 [ib_core] [ 296.620602] process_one_work+0x149/0x360 [ 296.625883] worker_thread+0x4d/0x3c0 [ 296.630779] kthread+0x109/0x140 [ 296.635177] ? rescuer_thread+0x380/0x380 [ 296.640453] ? kthread_park+0x60/0x60 [ 296.645341] ret_from_fork+0x25/0x30 [ 296.650132] Code: 10 ab 81 4c 89 5d a0 4c 89 4d a8 e8 27 ac c0 ff 8b 05 fe 39 87 00 4c 8b 4d a8 4c 8b 5d a0 85 c0 74 09 83 e8 01 89 05 e9 39 87 00 <0f> ff e9 b8 fd ff ff e8 ed 8e b [ 296.672876] ---[ end trace 501470fecce57314 ]--- [ 297.231475] nvme nvme0: Successfully reconnected [ 297.231523] nvme nvme0: MEMREG for CQE 0xffff88100a1e6f80 failed with status memory management operation error (6) [ 297.378026] nvme nvme0: Reconnecting in 10 seconds... [ 307.765849] ------------[ cut here ]------------ [ 307.771793] kernel BUG at drivers/iommu/iova.c:836! [ 307.778007] invalid opcode: 0000 [#1] SMP [ 307.783257] Modules linked in: nvme_rdma nvme_fabrics nvme_core sch_mqprio ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter bridge 8021q garp mrp stp llc rpcrdma id [ 307.867936] mgag200 i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm tg3 ahci mlx4_core libahci ptp libata crc32c_intel i2c_core pps_core devlink d [ 307.891229] CPU: 30 PID: 757 Comm: kworker/u369:7 Tainted: G W 4.14.0-rc4.10.02.sagi+ #30 [ 307.902427] Hardware name: Dell Inc. PowerEdge R430/03XKDV, BIOS 1.6.2 01/08/2016 [ 307.911579] Workqueue: nvme-wq nvme_rdma_reconnect_ctrl_work [nvme_rdma] [ 307.919847] task: ffff881032bd8000 task.stack: ffffc9000851c000 [ 307.927232] RIP: 0010:iova_magazine_free_pfns+0x76/0x80 [ 307.933827] RSP: 0018:ffffc9000851fc58 EFLAGS: 00010046 [ 307.940415] RAX: 0000000000000000 RBX: 0000000000000017 RCX: 0000000100400038 [ 307.949132] RDX: 00000000000e0e9c RSI: 00000000000e0e9d RDI: ffff8810372c4e58 [ 307.957857] RBP: ffffc9000851fc78 R08: ffff88202fd41a40 R09: 0000000100400038 [ 307.966586] R10: 000000002fd41c01 R11: ffff88202fd41a40 R12: ffff880ffda45000 [ 307.975308] R13: ffff8810372c4e58 R14: 0000000000000046 R15: ffff880ffda45000 [ 307.984032] FS: 0000000000000000(0000) GS:ffff88103e9c0000(0000) knlGS:0000000000000000 [ 307.993834] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 308.001012] CR2: 00007f4a9e61fb70 CR3: 0000001011bdd001 CR4: 00000000001606e0 [ 308.009749] Call Trace: [ 308.013243] free_iova_fast+0xc9/0x1d0 [ 308.018198] fq_ring_free+0x78/0xc0 [ 308.022845] queue_iova+0x111/0x130 [ 308.027478] intel_unmap+0xa5/0x130 [ 308.032118] intel_unmap_page+0xe/0x10 [ 308.037044] nvme_rdma_free_ring.constprop.61+0x4a/0x90 [nvme_rdma] [ 308.044789] nvme_rdma_destroy_queue_ib+0x3a/0x50 [nvme_rdma] [ 308.051951] nvme_rdma_free_queue+0x1a/0x30 [nvme_rdma] [ 308.058529] nvme_rdma_destroy_io_queues+0x80/0xb0 [nvme_rdma] [ 308.065786] nvme_rdma_reconnect_ctrl_work+0x30/0xe0 [nvme_rdma] [ 308.073240] process_one_work+0x149/0x360 [ 308.078470] worker_thread+0x4d/0x3c0 [ 308.083299] kthread+0x109/0x140 [ 308.087652] ? rescuer_thread+0x380/0x380 [ 308.092878] ? kthread_park+0x60/0x60 [ 308.097699] ret_from_fork+0x25/0x30 [ 308.102407] Code: ef e8 ef fe ff ff 48 63 d3 49 3b 14 24 72 d7 4c 89 f6 4c 89 ef e8 3b 3d 2b 00 49 c7 04 24 00 00 00 00 5b 41 5c 41 5d 41 5e 5d c3 <0f> 0b 0f 1f 84 00 00 00 00 00 0 [ 308.124980] RIP: iova_magazine_free_pfns+0x76/0x80 RSP: ffffc9000851fc58 [ 308.133265] ---[ end trace 501470fecce57315 ]--- [ 308.142811] Kernel panic - not syncing: Fatal exception [ 308.149425] Kernel Offset: disabled [ 308.158679] ---[ end Kernel panic - not syncing: Fatal exception [ 308.166135] ------------[ cut here ]------------ [ 308.172010] WARNING: CPU: 30 PID: 757 at kernel/sched/core.c:1179 set_task_cpu+0x191/0x1a0 [ 308.181959] Modules linked in: nvme_rdma nvme_fabrics nvme_core sch_mqprio ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter bridge 8021q garp mrp stp llc rpcrdma id [ 308.266001] mgag200 i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm tg3 ahci mlx4_core libahci ptp libata crc32c_intel i2c_core pps_core devlink d [ 308.289112] CPU: 30 PID: 757 Comm: kworker/u369:7 Tainted: G D W 4.14.0-rc4.10.02.sagi+ #30 [ 308.300236] Hardware name: Dell Inc. PowerEdge R430/03XKDV, BIOS 1.6.2 01/08/2016 [ 308.309330] Workqueue: nvme-wq nvme_rdma_reconnect_ctrl_work [nvme_rdma] [ 308.317552] task: ffff881032bd8000 task.stack: ffffc9000851c000 [ 308.324905] RIP: 0010:set_task_cpu+0x191/0x1a0 [ 308.330621] RSP: 0018:ffff88103e9c3c38 EFLAGS: 00010046 [ 308.337191] RAX: 0000000000000200 RBX: ffff881034cc5d00 RCX: 0000000000000001 [ 308.345896] RDX: 0000000000000001 RSI: 0000000000000000 RDI: ffff881034cc5d00 [ 308.354604] RBP: ffff88103e9c3c58 R08: 0000000000000000 R09: 0000000000000000 [ 308.363314] R10: 00000000000003ff R11: 0000000000000001 R12: ffff881034cc68ac [ 308.372018] R13: 0000000000000000 R14: 0000000000000000 R15: 000000000001b900 [ 308.380720] FS: 0000000000000000(0000) GS:ffff88103e9c0000(0000) knlGS:0000000000000000 [ 308.390495] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 308.397632] CR2: 00007f4a9e61fb70 CR3: 0000001011bdd001 CR4: 00000000001606e0 [ 308.406347] Call Trace: [ 308.409809] <IRQ> [ 308.412789] try_to_wake_up+0x166/0x470 [ 308.417816] default_wake_function+0x12/0x20 [ 308.423325] __wake_up_common+0x8a/0x160 [ 308.428438] __wake_up_locked+0x16/0x20 [ 308.433450] ep_poll_callback+0xd0/0x300 [ 308.438562] __wake_up_common+0x8a/0x160 [ 308.443664] __wake_up_common_lock+0x7e/0xc0 [ 308.449150] __wake_up+0x13/0x20 [ 308.453455] wake_up_klogd_work_func+0x40/0x60 [ 308.459124] irq_work_run_list+0x4d/0x70 [ 308.464217] ? tick_sched_do_timer+0x70/0x70 [ 308.469701] irq_work_tick+0x40/0x50 [ 308.474409] update_process_times+0x42/0x60 [ 308.479800] tick_sched_handle+0x2d/0x60 [ 308.484881] tick_sched_timer+0x39/0x70 [ 308.489852] __hrtimer_run_queues+0xe5/0x230 [ 308.495291] hrtimer_interrupt+0xa8/0x1a0 [ 308.500466] smp_apic_timer_interrupt+0x5f/0x130 [ 308.506320] apic_timer_interrupt+0x9d/0xb0 [ 308.511661] </IRQ> [ 308.514658] RIP: 0010:panic+0x1fd/0x245 [ 308.519603] RSP: 0018:ffffc9000851f9a8 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff10 [ 308.528740] RAX: 0000000000000034 RBX: 0000000000000000 RCX: 0000000000000006 [ 308.537352] RDX: 0000000000000000 RSI: 0000000000000086 RDI: ffff88103e9ce050 [ 308.545943] RBP: ffffc9000851fa18 R08: 0000000000000000 R09: 00000000000008f9 [ 308.554511] R10: 00000000000003ff R11: 0000000000000001 R12: ffffffff81a3f340 [ 308.563067] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000004 [ 308.571619] ? panic+0x1f6/0x245 [ 308.575773] oops_end+0xb8/0xd0 [ 308.579822] die+0x42/0x50 [ 308.583348] do_trap+0x16a/0x180 [ 308.587442] do_error_trap+0x89/0x110 [ 308.592007] ? iova_magazine_free_pfns+0x76/0x80 [ 308.597624] ? rem_res_range+0x1ae/0x200 [mlx4_core] [ 308.603625] do_invalid_op+0x20/0x30 [ 308.608063] invalid_op+0x18/0x20 [ 308.612198] RIP: 0010:iova_magazine_free_pfns+0x76/0x80 [ 308.618473] RSP: 0018:ffffc9000851fc58 EFLAGS: 00010046 [ 308.624735] RAX: 0000000000000000 RBX: 0000000000000017 RCX: 0000000100400038 [ 308.633146] RDX: 00000000000e0e9c RSI: 00000000000e0e9d RDI: ffff8810372c4e58 [ 308.641555] RBP: ffffc9000851fc78 R08: ffff88202fd41a40 R09: 0000000100400038 [ 308.649969] R10: 000000002fd41c01 R11: ffff88202fd41a40 R12: ffff880ffda45000 [ 308.658387] R13: ffff8810372c4e58 R14: 0000000000000046 R15: ffff880ffda45000 [ 308.666799] ? iova_magazine_free_pfns+0x3e/0x80 [ 308.672399] free_iova_fast+0xc9/0x1d0 [ 308.677046] fq_ring_free+0x78/0xc0 [ 308.681370] queue_iova+0x111/0x130 [ 308.685691] intel_unmap+0xa5/0x130 [ 308.690011] intel_unmap_page+0xe/0x10 [ 308.694626] nvme_rdma_free_ring.constprop.61+0x4a/0x90 [nvme_rdma] [ 308.702056] nvme_rdma_destroy_queue_ib+0x3a/0x50 [nvme_rdma] [ 308.708909] nvme_rdma_free_queue+0x1a/0x30 [nvme_rdma] [ 308.715182] nvme_rdma_destroy_io_queues+0x80/0xb0 [nvme_rdma] [ 308.722148] nvme_rdma_reconnect_ctrl_work+0x30/0xe0 [nvme_rdma] [ 308.729311] process_one_work+0x149/0x360 [ 308.734242] worker_thread+0x4d/0x3c0 [ 308.738783] kthread+0x109/0x140 [ 308.742840] ? rescuer_thread+0x380/0x380 [ 308.747774] ? kthread_park+0x60/0x60 [ 308.752309] ret_from_fork+0x25/0x30 [ 308.756747] Code: ff 80 8b ac 08 00 00 04 e9 23 ff ff ff 0f ff e9 bf fe ff ff f7 83 84 00 00 00 fd ff ff ff 0f 84 c9 fe ff ff 0f ff e9 c2 fe ff ff <0f> ff e9 d1 fe ff ff 0f 1f 84 0 [ 308.778800] ---[ end trace 501470fecce57316 ]--- ^ permalink raw reply [flat|nested] 16+ messages in thread
* nvmeof rdma regression issue on 4.14.0-rc1 (or maybe mlx4?) 2017-10-12 8:24 ` Yi Zhang @ 2017-10-16 10:21 ` Sagi Grimberg 2017-10-19 6:33 ` Yi Zhang 0 siblings, 1 reply; 16+ messages in thread From: Sagi Grimberg @ 2017-10-16 10:21 UTC (permalink / raw) >> Lets take this one step at a time, starting with this issue. >> >> First, there is a reason why a simple create_ctrl fails, can we isolate >> exactly which call fails? Was something else going on that might have >> made the simple create_ctrl fail? >> >> We don't see any "rdma_resolve_addr failed" or "failed to connect queue" >> messages but we do see "creating I/O queues" which means that we either >> failed at IO tagset allocation or initializing connect_q. >> >> We have a missing error code assignment so can you try the following >> patch: > > Hi Sagi > Sorry for the late response, here is the log with your patch, let me > know if you need more test. Hi Yi, I was referring to the bug you reported on a simple create_ctrl failed: https://pastebin.com/7z0XSGSd Does it reproduce? ^ permalink raw reply [flat|nested] 16+ messages in thread
* nvmeof rdma regression issue on 4.14.0-rc1 (or maybe mlx4?) 2017-10-16 10:21 ` Sagi Grimberg @ 2017-10-19 6:33 ` Yi Zhang 2017-10-19 6:55 ` Sagi Grimberg 0 siblings, 1 reply; 16+ messages in thread From: Yi Zhang @ 2017-10-19 6:33 UTC (permalink / raw) On 10/16/2017 06:21 PM, Sagi Grimberg wrote: >>> Lets take this one step at a time, starting with this issue. >>> >>> First, there is a reason why a simple create_ctrl fails, can we isolate >>> exactly which call fails? Was something else going on that might have >>> made the simple create_ctrl fail? >>> >>> We don't see any "rdma_resolve_addr failed" or "failed to connect >>> queue" >>> messages but we do see "creating I/O queues" which means that we either >>> failed at IO tagset allocation or initializing connect_q. >>> >>> We have a missing error code assignment so can you try the following >>> patch: >> >> Hi Sagi >> Sorry for the late response, here is the log with your patch, let me >> know if you need more test. > > Hi Yi, > > I was referring to the bug you reported on a simple create_ctrl failed: > https://pastebin.com/7z0XSGSd > > Does it reproduce? > yes, this issue was reproduced during "git bisect" with below patch commit 31fdf18401703df246e20fc0382ed3594e6cf8d8 Author: Sagi Grimberg <sagi at grimberg.me> Date: Mon Aug 28 21:40:06 2017 +0200 nvme-rdma: reuse configure/destroy_admin_queue No need to open-code it. Signed-off-by: Sagi Grimberg <sagi at grimberg.me> Signed-off-by: Christoph Hellwig <hch at lst.de> > _______________________________________________ > Linux-nvme mailing list > Linux-nvme at lists.infradead.org > http://lists.infradead.org/mailman/listinfo/linux-nvme ^ permalink raw reply [flat|nested] 16+ messages in thread
* nvmeof rdma regression issue on 4.14.0-rc1 (or maybe mlx4?) 2017-10-19 6:33 ` Yi Zhang @ 2017-10-19 6:55 ` Sagi Grimberg 2017-10-19 8:23 ` Yi Zhang 0 siblings, 1 reply; 16+ messages in thread From: Sagi Grimberg @ 2017-10-19 6:55 UTC (permalink / raw) >> Hi Yi, >> >> I was referring to the bug you reported on a simple create_ctrl failed: >> https://pastebin.com/7z0XSGSd >> >> Does it reproduce? >> > yes, this issue was reproduced during "git bisect" with below patch OK, if this does not reproduce with the latest code, lets put it aside for now. So as for the error you see, can you please try the following patch? -- diff --git a/drivers/nvme/host/rdma.c b/drivers/nvme/host/rdma.c index 405895b1dff2..916658e010ff 100644 --- a/drivers/nvme/host/rdma.c +++ b/drivers/nvme/host/rdma.c @@ -572,6 +572,11 @@ static void nvme_rdma_free_queue(struct nvme_rdma_queue *queue) if (!test_and_clear_bit(NVME_RDMA_Q_ALLOCATED, &queue->flags)) return; + if(nvme_rdma_queue_idx(queue) == 0) + nvme_rdma_free_qe(queue->device->dev, + &queue->ctrl->async_event_sqe, + sizeof(struct nvme_command), DMA_TO_DEVICE); + nvme_rdma_destroy_queue_ib(queue); rdma_destroy_id(queue->cm_id); } @@ -739,8 +744,6 @@ static struct blk_mq_tag_set *nvme_rdma_alloc_tagset(struct nvme_ctrl *nctrl, static void nvme_rdma_destroy_admin_queue(struct nvme_rdma_ctrl *ctrl, bool remove) { - nvme_rdma_free_qe(ctrl->queues[0].device->dev, &ctrl->async_event_sqe, - sizeof(struct nvme_command), DMA_TO_DEVICE); nvme_rdma_stop_queue(&ctrl->queues[0]); if (remove) { blk_cleanup_queue(ctrl->ctrl.admin_q); -- ^ permalink raw reply related [flat|nested] 16+ messages in thread
* nvmeof rdma regression issue on 4.14.0-rc1 (or maybe mlx4?) 2017-10-19 6:55 ` Sagi Grimberg @ 2017-10-19 8:23 ` Yi Zhang 2017-10-19 9:44 ` Sagi Grimberg 0 siblings, 1 reply; 16+ messages in thread From: Yi Zhang @ 2017-10-19 8:23 UTC (permalink / raw) On 10/19/2017 02:55 PM, Sagi Grimberg wrote: > >>> Hi Yi, >>> >>> I was referring to the bug you reported on a simple create_ctrl failed: >>> https://pastebin.com/7z0XSGSd >>> >>> Does it reproduce? >>> >> yes, this issue was reproduced during "git bisect" with below patch > > OK, if this does not reproduce with the latest code, lets put it aside > for now. > > So as for the error you see, can you please try the following patch? Hi Sagi With this patch, no such error log found on host side, but I found there is no nvme0n1 device node even get "nvme nvme0: Successfully reconnected" on host. Host side: [ 98.181089] nvme nvme0: new ctrl: NQN "nqn.2014-08.org.nvmexpress.discovery", addr 172.31.0.90:4420 [ 98.329464] nvme nvme0: creating 40 I/O queues. [ 98.835409] nvme nvme0: new ctrl: NQN "testnqn", addr 172.31.0.90:4420 [ 107.873586] nvme nvme0: Reconnecting in 10 seconds... [ 118.505937] nvme nvme0: Connect rejected: status 8 (invalid service ID). [ 118.513443] nvme nvme0: rdma_resolve_addr wait failed (-104). [ 118.519875] nvme nvme0: Failed reconnect attempt 1 [ 118.525241] nvme nvme0: Reconnecting in 10 seconds... [ 128.733311] nvme nvme0: Connect rejected: status 8 (invalid service ID). [ 128.740812] nvme nvme0: rdma_resolve_addr wait failed (-104). [ 128.747247] nvme nvme0: Failed reconnect attempt 2 [ 128.752609] nvme nvme0: Reconnecting in 10 seconds... [ 138.973404] nvme nvme0: Connect rejected: status 8 (invalid service ID). [ 138.980904] nvme nvme0: rdma_resolve_addr wait failed (-104). [ 138.987329] nvme nvme0: Failed reconnect attempt 3 [ 138.992691] nvme nvme0: Reconnecting in 10 seconds... [ 149.232610] nvme nvme0: creating 40 I/O queues. [ 149.831443] nvme nvme0: Successfully reconnected [ 149.831519] nvme nvme0: identifiers changed for nsid 1 [root at rdma-virt-01 linux ((dafb1b2...))]$ lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT sda 8:0 0 465.8G 0 disk ??sda2 8:2 0 464.8G 0 part ? ??rhelaa_rdma--virt--01-swap 253:1 0 4G 0 lvm [SWAP] ? ??rhelaa_rdma--virt--01-home 253:2 0 410.8G 0 lvm /home ? ??rhelaa_rdma--virt--01-root 253:0 0 50G 0 lvm / ??sda1 8:1 0 1G 0 part /boot > -- > diff --git a/drivers/nvme/host/rdma.c b/drivers/nvme/host/rdma.c > index 405895b1dff2..916658e010ff 100644 > --- a/drivers/nvme/host/rdma.c > +++ b/drivers/nvme/host/rdma.c > @@ -572,6 +572,11 @@ static void nvme_rdma_free_queue(struct > nvme_rdma_queue *queue) > if (!test_and_clear_bit(NVME_RDMA_Q_ALLOCATED, &queue->flags)) > return; > > + if(nvme_rdma_queue_idx(queue) == 0) > + nvme_rdma_free_qe(queue->device->dev, > + &queue->ctrl->async_event_sqe, > + sizeof(struct nvme_command), DMA_TO_DEVICE); > + > nvme_rdma_destroy_queue_ib(queue); > rdma_destroy_id(queue->cm_id); > } > @@ -739,8 +744,6 @@ static struct blk_mq_tag_set > *nvme_rdma_alloc_tagset(struct nvme_ctrl *nctrl, > static void nvme_rdma_destroy_admin_queue(struct nvme_rdma_ctrl *ctrl, > bool remove) > { > - nvme_rdma_free_qe(ctrl->queues[0].device->dev, > &ctrl->async_event_sqe, > - sizeof(struct nvme_command), DMA_TO_DEVICE); > nvme_rdma_stop_queue(&ctrl->queues[0]); > if (remove) { > blk_cleanup_queue(ctrl->ctrl.admin_q); > -- > > _______________________________________________ > Linux-nvme mailing list > Linux-nvme at lists.infradead.org > http://lists.infradead.org/mailman/listinfo/linux-nvme ^ permalink raw reply [flat|nested] 16+ messages in thread
* nvmeof rdma regression issue on 4.14.0-rc1 (or maybe mlx4?) 2017-10-19 8:23 ` Yi Zhang @ 2017-10-19 9:44 ` Sagi Grimberg 2017-10-19 11:13 ` Yi Zhang 0 siblings, 1 reply; 16+ messages in thread From: Sagi Grimberg @ 2017-10-19 9:44 UTC (permalink / raw) > Hi Sagi > With this patch, no such error log found on host side, Awsome. that's the culprit... > but I found there > is no nvme0n1 device node even get "nvme nvme0: Successfully > reconnected" on host. That is expected because you did not persist a namespace UUID which caused the kernel to generate a random one. That confused the host as it got the same namespace ID with a different UUID. Can you please set a uuid when you rerun the test? ^ permalink raw reply [flat|nested] 16+ messages in thread
* nvmeof rdma regression issue on 4.14.0-rc1 (or maybe mlx4?) 2017-10-19 9:44 ` Sagi Grimberg @ 2017-10-19 11:13 ` Yi Zhang 0 siblings, 0 replies; 16+ messages in thread From: Yi Zhang @ 2017-10-19 11:13 UTC (permalink / raw) On 10/19/2017 05:44 PM, Sagi Grimberg wrote: > >> Hi Sagi >> With this patch, no such error log found on host side, > > > Awsome. that's the culprit... > >> but I found there is no nvme0n1 device node even get "nvme nvme0: >> Successfully reconnected" on host. > > That is expected because you did not persist a namespace UUID > which caused the kernel to generate a random one. That confused > the host as it got the same namespace ID with a different UUID. > > Can you please set a uuid when you rerun the test? > I tried add uuid filed on rdma.json, it works well now. [root at rdma-virt-00 ~]$ cat /etc/rdma.json { "hosts": [ { "nqn": "hostnqn" } ], "ports": [ { "addr": { "adrfam": "ipv4", "traddr": "172.31.0.90", "treq": "not specified", "trsvcid": "4420", "trtype": "rdma" }, "portid": 2, "referrals": [], "subsystems": [ "testnqn" ] } ], "subsystems": [ { "allowed_hosts": [], "attr": { "allow_any_host": "1" }, "namespaces": [ { "device": { "nguid": "ef90689c-6c46-d44c-89c1-4067801309a8", "path": "/dev/nullb0", "uuid": "00000000-0000-0000-0000-000000000001" }, "enable": 1, "nsid": 1 } ], "nqn": "testnqn" } ] } > _______________________________________________ > Linux-nvme mailing list > Linux-nvme at lists.infradead.org > http://lists.infradead.org/mailman/listinfo/linux-nvme ^ permalink raw reply [flat|nested] 16+ messages in thread
end of thread, other threads:[~2017-10-19 11:13 UTC | newest]
Thread overview: 16+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
[not found] <1735134433.8514119.1505997532669.JavaMail.zimbra@redhat.com>
2017-09-21 12:47 ` nvmeof rdma regression issue on 4.14.0-rc1 Yi Zhang
2017-09-21 14:44 ` nvmeof rdma regression issue on 4.14.0-rc1 (or maybe mlx4?) Christoph Hellwig
2017-09-24 7:38 ` Sagi Grimberg
2017-09-24 9:28 ` Yi Zhang
[not found] ` <20170924103426.GB25094@mtr-leonro.local>
2017-09-25 5:20 ` Yi Zhang
2017-09-25 7:06 ` Sagi Grimberg
2017-09-25 11:01 ` Yi Zhang
2017-09-25 11:15 ` Sagi Grimberg
2017-10-02 12:51 ` Sagi Grimberg
2017-10-12 8:24 ` Yi Zhang
2017-10-16 10:21 ` Sagi Grimberg
2017-10-19 6:33 ` Yi Zhang
2017-10-19 6:55 ` Sagi Grimberg
2017-10-19 8:23 ` Yi Zhang
2017-10-19 9:44 ` Sagi Grimberg
2017-10-19 11:13 ` Yi Zhang
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).