Re: nvmeof rdma regression issue on 4.14.0-rc1 (or maybe mlx4?)

Re: nvmeof rdma regression issue on 4.14.0-rc1 (or maybe mlx4?)

[Christoph Hellwig]

Adding linux-rdma, the dma mappings happen in the mlx4 driver

On Thu, Sep 21, 2017 at 08:47:31AM -0400, Yi Zhang wrote:
> Hi
> 
> With below steps, it's easy to reproduce this issue on host side, I cannot reproduce it on 4.13.
> So I guess it's one regression issue on 4.14.0-rc1, let me know if you need more info.
> 
> host side: 
> #connect the target
> 
> target side:
> #nvmetcli clear
> #sleep 90
> #nvmetcli restore /etc/rdma.json
> 
> Host side log:
> 
> [  184.318537] nvme nvme0: new ctrl: NQN "nqn.2014-08.org.nvmexpress.discovery", addr 172.31.0.90:4420
> [  184.502114] nvme nvme0: creating 40 I/O queues.
> [  185.017751] nvme nvme0: new ctrl: NQN "testnqn", addr 172.31.0.90:4420
> [  191.303659] nvme nvme0: rescanning
> [  252.226387] nvme nvme0: Reconnecting in 10 seconds...
> [  262.389578] nvme nvme0: Connect rejected: status 8 (invalid service ID).
> [  262.397084] nvme nvme0: rdma_resolve_addr wait failed (-104).
> [  262.403514] nvme nvme0: Failed reconnect attempt 1
> [  262.408875] nvme nvme0: Reconnecting in 10 seconds...
> [  272.616908] nvme nvme0: Connect rejected: status 8 (invalid service ID).
> [  272.624408] nvme nvme0: rdma_resolve_addr wait failed (-104).
> [  272.630839] nvme nvme0: Failed reconnect attempt 2
> [  272.636230] nvme nvme0: Reconnecting in 10 seconds...
> [  282.856896] nvme nvme0: Connect rejected: status 8 (invalid service ID).
> [  282.864398] nvme nvme0: rdma_resolve_addr wait failed (-104).
> [  282.870831] nvme nvme0: Failed reconnect attempt 3
> [  282.876232] nvme nvme0: Reconnecting in 10 seconds...
> [  293.116196] nvme nvme0: creating 40 I/O queues.
> [  293.209662] DMAR: ERROR: DMA PTE for vPFN 0xe0f59 already set (to 10369a9001 not 10115ed001)
> [  293.219117] ------------[ cut here ]------------
> [  293.224284] WARNING: CPU: 14 PID: 751 at drivers/iommu/intel-iommu.c:2305 __domain_mapping+0x367/0x380
> [  293.234698] Modules linked in: nvme_rdma nvme_fabrics nvme_core sch_mqprio ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter bridge 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_rapl ipmi_ssif sb_edac x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm irqbypass crct10dif_pclmul crc32_pclmul ghash_clmulni_intel intel_cstate intel_uncore iTCO_wdt ipmi_si intel_rapl_perf iTCO_vendor_support ipmi_devintf dcdbas sg pcspkr ipmi_msghandler ioatdma mei_me mei dca shpchp lpc_ich acpi_pad acpi_power_meter wmi nfsd auth_rpcgss nfs_acl lockd grace sunrpc binfmt_misc ip_tables xfs libcrc32c 
 mlx4_en sd_mod
> [  293.313884]  mgag200 i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm mlx4_core tg3 ahci libahci ptp libata i2c_core crc32c_intel devlink pps_core dm_mirror dm_region_hash dm_log dm_mod
> [  293.335583] CPU: 14 PID: 751 Comm: kworker/u369:7 Not tainted 4.14.0-rc1 #2
> [  293.343374] Hardware name: Dell Inc. PowerEdge R430/03XKDV, BIOS 1.6.2 01/08/2016
> [  293.351750] Workqueue: nvme-wq nvme_rdma_reconnect_ctrl_work [nvme_rdma]
> [  293.359249] task: ffff881032ecdd00 task.stack: ffffc900084d8000
> [  293.365873] RIP: 0010:__domain_mapping+0x367/0x380
> [  293.371230] RSP: 0018:ffffc900084dbc60 EFLAGS: 00010202
> [  293.377075] RAX: 0000000000000004 RBX: 00000010115ed001 RCX: 0000000000000000
> [  293.385056] RDX: 0000000000000000 RSI: ffff88103e7ce038 RDI: ffff88103e7ce038
> [  293.393040] RBP: ffffc900084dbcc0 R08: 0000000000000000 R09: 0000000000000000
> [  293.401024] R10: 00000000000002f7 R11: 00000000010115ed R12: ffff88103b9e1ac8
> [  293.409744] R13: 0000000000000001 R14: 0000000000000001 R15: 00000000000e0f59
> [  293.418456] FS:  0000000000000000(0000) GS:ffff88103e7c0000(0000) knlGS:0000000000000000
> [  293.428229] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> [  293.435391] CR2: 0000154ecabc9140 CR3: 0000001005709001 CR4: 00000000001606e0
> [  293.444112] Call Trace:
> [  293.447594]  __intel_map_single+0xeb/0x180
> [  293.452918]  intel_map_page+0x39/0x40
> [  293.457765]  mlx4_ib_alloc_mr+0x141/0x220 [mlx4_ib]
> [  293.463965]  ib_alloc_mr+0x26/0x50 [ib_core]
> [  293.469471]  nvme_rdma_reinit_request+0x3a/0x70 [nvme_rdma]
> [  293.476433]  ? nvme_rdma_free_ctrl+0xb0/0xb0 [nvme_rdma]
> [  293.483100]  blk_mq_reinit_tagset+0x5c/0x90
> [  293.488508]  nvme_rdma_configure_io_queues+0x211/0x290 [nvme_rdma]
> [  293.496152]  nvme_rdma_reconnect_ctrl_work+0x5b/0xd0 [nvme_rdma]
> [  293.503598]  process_one_work+0x149/0x360
> [  293.508815]  worker_thread+0x4d/0x3c0
> [  293.513638]  kthread+0x109/0x140
> [  293.517973]  ? rescuer_thread+0x380/0x380
> [  293.523176]  ? kthread_park+0x60/0x60
> [  293.527993]  ret_from_fork+0x25/0x30
> [  293.532705] Code: b2 aa 81 4c 89 5d a0 4c 89 4d a8 e8 87 e1 c0 ff 8b 05 fe 6e 87 00 4c 8b 4d a8 4c 8b 5d a0 85 c0 74 09 83 e8 01 89 05 e9 6e 87 00 <0f> ff e9 b8 fd ff ff e8 8d c7 ba ff 0f 1f 00 66 2e 0f 1f 84 00 
> [  293.555293] ---[ end trace c324b9449c77b573 ]---
> [  293.562133] DMAR: ERROR: DMA PTE for vPFN 0xe0f59 already set (to 10369a9001 not 1002790001)
> [  293.572343] ------------[ cut here ]------------
> [  293.578288] WARNING: CPU: 14 PID: 751 at drivers/iommu/intel-iommu.c:2305 __domain_mapping+0x367/0x380
> [  293.589473] Modules linked in: nvme_rdma nvme_fabrics nvme_core sch_mqprio ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter bridge 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_rapl ipmi_ssif sb_edac x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm irqbypass crct10dif_pclmul crc32_pclmul ghash_clmulni_intel intel_cstate intel_uncore iTCO_wdt ipmi_si intel_rapl_perf iTCO_vendor_support ipmi_devintf dcdbas sg pcspkr ipmi_msghandler ioatdma mei_me mei dca shpchp lpc_ich acpi_pad acpi_power_meter wmi nfsd auth_rpcgss nfs_acl lockd grace sunrpc binfmt_misc ip_tables xfs libcrc32c 
 mlx4_en sd_mod
> [  293.674124]  mgag200 i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm mlx4_core tg3 ahci libahci ptp libata i2c_core crc32c_intel devlink pps_core dm_mirror dm_region_hash dm_log dm_mod
> [  293.697394] CPU: 14 PID: 751 Comm: kworker/u369:7 Tainted: G        W       4.14.0-rc1 #2
> [  293.707322] Hardware name: Dell Inc. PowerEdge R430/03XKDV, BIOS 1.6.2 01/08/2016
> [  293.716456] Workqueue: nvme-wq nvme_rdma_reconnect_ctrl_work [nvme_rdma]
> [  293.724706] task: ffff881032ecdd00 task.stack: ffffc900084d8000
> [  293.732095] RIP: 0010:__domain_mapping+0x367/0x380
> [  293.738214] RSP: 0018:ffffc900084dbc60 EFLAGS: 00010206
> [  293.744810] RAX: 0000000000000003 RBX: 0000001002790001 RCX: 0000000000000000
> [  293.753541] RDX: 0000000000000000 RSI: ffff88103e7ce038 RDI: ffff88103e7ce038
> [  293.762280] RBP: ffffc900084dbcc0 R08: 0000000000000000 R09: 0000000000000000
> [  293.771011] R10: 00000000000003ff R11: 0000000001002790 R12: ffff88103b9e1ac8
> [  293.779752] R13: 0000000000000001 R14: 0000000000000001 R15: 00000000000e0f59
> [  293.788487] FS:  0000000000000000(0000) GS:ffff88103e7c0000(0000) knlGS:0000000000000000
> [  293.798295] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> [  293.805480] CR2: 0000154ecabc9140 CR3: 0000001005709001 CR4: 00000000001606e0
> [  293.814226] Call Trace:
> [  293.817726]  __intel_map_single+0xeb/0x180
> [  293.823058]  intel_map_page+0x39/0x40
> [  293.827906]  mlx4_ib_alloc_mr+0x141/0x220 [mlx4_ib]
> [  293.834112]  ib_alloc_mr+0x26/0x50 [ib_core]
> [  293.839640]  nvme_rdma_reinit_request+0x3a/0x70 [nvme_rdma]
> [  293.846628]  ? nvme_rdma_free_ctrl+0xb0/0xb0 [nvme_rdma]
> [  293.853322]  blk_mq_reinit_tagset+0x5c/0x90
> [  293.858757]  nvme_rdma_configure_io_queues+0x211/0x290 [nvme_rdma]
> [  293.866429]  nvme_rdma_reconnect_ctrl_work+0x5b/0xd0 [nvme_rdma]
> [  293.873891]  process_one_work+0x149/0x360
> [  293.879132]  worker_thread+0x4d/0x3c0
> [  293.883979]  kthread+0x109/0x140
> [  293.888336]  ? rescuer_thread+0x380/0x380
> [  293.893554]  ? kthread_park+0x60/0x60
> [  293.898399]  ret_from_fork+0x25/0x30
> [  293.903139] Code: b2 aa 81 4c 89 5d a0 4c 89 4d a8 e8 87 e1 c0 ff 8b 05 fe 6e 87 00 4c 8b 4d a8 4c 8b 5d a0 85 c0 74 09 83 e8 01 89 05 e9 6e 87 00 <0f> ff e9 b8 fd ff ff e8 8d c7 ba ff 0f 1f 00 66 2e 0f 1f 84 00 
> [  293.925798] ---[ end trace c324b9449c77b574 ]---
> [  294.460186] nvme nvme0: Successfully reconnected
> [  294.460243] nvme nvme0: MEMREG for CQE 0xffff88100663a640 failed with status memory management operation error (6)
> [  325.850105] general protection fault: 0000 [#1] SMP
> [  325.856354] Modules linked in: nvme_rdma nvme_fabrics nvme_core sch_mqprio ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter bridge 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_rapl ipmi_ssif sb_edac x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm irqbypass crct10dif_pclmul crc32_pclmul ghash_clmulni_intel intel_cstate intel_uncore iTCO_wdt ipmi_si intel_rapl_perf iTCO_vendor_support ipmi_devintf dcdbas sg pcspkr ipmi_msghandler ioatdma mei_me mei dca shpchp lpc_ich acpi_pad acpi_power_meter wmi nfsd auth_rpcgss nfs_acl lockd grace sunrpc binfmt_misc ip_tables xfs libcrc32c 
 mlx4_en sd_mod
> [  325.940982]  mgag200 i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm mlx4_core tg3 ahci libahci ptp libata i2c_core crc32c_intel devlink pps_core dm_mirror dm_region_hash dm_log dm_mod
> [  325.964245] CPU: 22 PID: 3707 Comm: git Tainted: G        W       4.14.0-rc1 #2
> [  325.973198] Hardware name: Dell Inc. PowerEdge R430/03XKDV, BIOS 1.6.2 01/08/2016
> [  325.982332] task: ffff881036451740 task.stack: ffffc90009368000
> [  325.989746] RIP: 0010:xfs_attr_shortform_getvalue+0x25/0x120 [xfs]
> [  325.997412] RSP: 0018:ffffc9000936b920 EFLAGS: 00010282
> [  326.004006] RAX: 63656a626f3a755f RBX: ffffc9000936b980 RCX: ffffc9000936b980
> [  326.012738] RDX: 0000000001000000 RSI: ffffc9000936b980 RDI: ffffc9000936b980
> [  326.021465] RBP: ffffc9000936b958 R08: ffffffff81a740c0 R09: 0000000000000000
> [  326.030194] R10: 0000000000000008 R11: ffff881002f52700 R12: ffffc9000936ba54
> [  326.038913] R13: ffff882023b9fc00 R14: 0000000000000000 R15: 0000000000000008
> [  326.047642] FS:  000014c3de00e740(0000) GS:ffff88103e8c0000(0000) knlGS:0000000000000000
> [  326.057445] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> [  326.064627] CR2: 00000000007b7d44 CR3: 000000018b0cb005 CR4: 00000000001606e0
> [  326.073370] Call Trace:
> [  326.076885]  xfs_attr_get_ilocked+0x63/0x70 [xfs]
> [  326.082917]  xfs_attr_get+0xca/0x120 [xfs]
> [  326.088269]  xfs_xattr_get+0x4c/0x70 [xfs]
> [  326.093592]  __vfs_getxattr+0x57/0x70
> [  326.098428]  inode_doinit_with_dentry+0x33c/0x580
> [  326.104430]  selinux_d_instantiate+0x1c/0x20
> [  326.109947]  security_d_instantiate+0x32/0x50
> [  326.115558]  d_splice_alias+0x4c/0x370
> [  326.120505]  xfs_vn_lookup+0x87/0xb0 [xfs]
> [  326.125819]  lookup_slow+0xa2/0x160
> [  326.130448]  walk_component+0x160/0x250
> [  326.135461]  ? legitimize_path.isra.33+0x2e/0x60
> [  326.141346]  path_lookupat+0x79/0x210
> [  326.146162]  ? cpumask_any_but+0x31/0x40
> [  326.151267]  filename_lookup+0xaf/0x190
> [  326.156276]  ? kmem_cache_alloc+0x9c/0x1b0
> [  326.161576]  ? getname_flags+0x4f/0x1f0
> [  326.166583]  ? getname_flags+0x6f/0x1f0
> [  326.171586]  user_path_at_empty+0x36/0x40
> [  326.176768]  vfs_statx+0x77/0xe0
> [  326.181058]  SYSC_newlstat+0x3d/0x70
> [  326.185726]  ? __audit_syscall_entry+0xaf/0x100
> [  326.191463]  ? syscall_trace_enter+0x1d0/0x2b0
> [  326.197090]  ? __audit_syscall_exit+0x209/0x290
> [  326.202799]  SyS_newlstat+0xe/0x10
> [  326.207226]  do_syscall_64+0x67/0x180
> [  326.211926]  entry_SYSCALL64_slow_path+0x25/0x25
> [  326.217674] RIP: 0033:0x14c3dd4961a5
> [  326.222237] RSP: 002b:00007fff730b3478 EFLAGS: 00000246 ORIG_RAX: 0000000000000006
> [  326.231250] RAX: ffffffffffffffda RBX: 0000000000ca3c50 RCX: 000014c3dd4961a5
> [  326.239774] RDX: 00007fff730b3530 RSI: 00007fff730b3530 RDI: 0000000000ca3ca0
> [  326.248281] RBP: 0000000000ca3ca0 R08: 0000000000000000 R09: 0000000000000000
> [  326.256770] R10: 0000000000000001 R11: 0000000000000246 R12: 00007fff730b3530
> [  326.265239] R13: 0000000000000000 R14: 0000000000000170 R15: 00007fff730b3530
> [  326.273684] Code: 84 00 00 00 00 00 0f 1f 44 00 00 55 48 89 f9 48 89 e5 41 57 41 56 41 55 41 54 53 48 83 ec 10 48 8b 47 38 48 8b 40 38 48 8b 40 18 <44> 0f b6 70 02 48 8d 58 04 b8 c3 ff ff ff 45 85 f6 0f 84 82 00 
> [  326.295785] RIP: xfs_attr_shortform_getvalue+0x25/0x120 [xfs] RSP: ffffc9000936b920
> [  326.304910] ---[ end trace c324b9449c77b575 ]---
> [  326.362863] Kernel panic - not syncing: Fatal exception
> [  326.362964] Kernel Offset: disabled
> [  326.427058] ---[ end Kernel panic - not syncing: Fatal exception
> 
> 
> Best Regards,
>   Yi Zhang
> 
> 
> 
> _______________________________________________
> Linux-nvme mailing list
> Linux-nvme-IAPFreCvJWM7uuMidbF8XUB+6BGkLq7r@public.gmane.org
> http://lists.infradead.org/mailman/listinfo/linux-nvme
---end quoted text---

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: nvmeof rdma regression issue on 4.14.0-rc1 (or maybe mlx4?)

[Sagi Grimberg]

> Adding linux-rdma, the dma mappings happen in the mlx4 driver

...

>> [  293.209662] DMAR: ERROR: DMA PTE for vPFN 0xe0f59 already set (to 10369a9001 not 10115ed001)
>> [  293.219117] ------------[ cut here ]------------
>> [  293.224284] WARNING: CPU: 14 PID: 751 at drivers/iommu/intel-iommu.c:2305 __domain_mapping+0x367/0x380
>> [  293.234698] Modules linked in: nvme_rdma nvme_fabrics nvme_core sch_mqprio ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter bridge 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_rapl ipmi_ssif sb_edac x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm irqbypass crct10dif_pclmul crc32_pclmul ghash_clmulni_intel intel_cstate intel_uncore iTCO_wdt ipmi_si intel_rapl_perf iTCO_vendor_support ipmi_devintf dcdbas sg pcspkr ipmi_msghandler ioatdma mei_me mei dca shpchp lpc_ich acpi_pad acpi_power_meter wmi nfsd auth_rpcgss nfs_acl lockd grace sunrpc binfmt_misc ip_tables xfs libcrc32c
  mlx4_en sd_mod
>> [  293.313884]  mgag200 i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm mlx4_core tg3 ahci libahci ptp libata i2c_core crc32c_intel devlink pps_core dm_mirror dm_region_hash dm_log dm_mod
>> [  293.335583] CPU: 14 PID: 751 Comm: kworker/u369:7 Not tainted 4.14.0-rc1 #2
>> [  293.343374] Hardware name: Dell Inc. PowerEdge R430/03XKDV, BIOS 1.6.2 01/08/2016
>> [  293.351750] Workqueue: nvme-wq nvme_rdma_reconnect_ctrl_work [nvme_rdma]
>> [  293.359249] task: ffff881032ecdd00 task.stack: ffffc900084d8000
>> [  293.365873] RIP: 0010:__domain_mapping+0x367/0x380
>> [  293.371230] RSP: 0018:ffffc900084dbc60 EFLAGS: 00010202
>> [  293.377075] RAX: 0000000000000004 RBX: 00000010115ed001 RCX: 0000000000000000
>> [  293.385056] RDX: 0000000000000000 RSI: ffff88103e7ce038 RDI: ffff88103e7ce038
>> [  293.393040] RBP: ffffc900084dbcc0 R08: 0000000000000000 R09: 0000000000000000
>> [  293.401024] R10: 00000000000002f7 R11: 00000000010115ed R12: ffff88103b9e1ac8
>> [  293.409744] R13: 0000000000000001 R14: 0000000000000001 R15: 00000000000e0f59
>> [  293.418456] FS:  0000000000000000(0000) GS:ffff88103e7c0000(0000) knlGS:0000000000000000
>> [  293.428229] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
>> [  293.435391] CR2: 0000154ecabc9140 CR3: 0000001005709001 CR4: 00000000001606e0
>> [  293.444112] Call Trace:
>> [  293.447594]  __intel_map_single+0xeb/0x180
>> [  293.452918]  intel_map_page+0x39/0x40
>> [  293.457765]  mlx4_ib_alloc_mr+0x141/0x220 [mlx4_ib]
>> [  293.463965]  ib_alloc_mr+0x26/0x50 [ib_core]
>> [  293.469471]  nvme_rdma_reinit_request+0x3a/0x70 [nvme_rdma]
>> [  293.476433]  ? nvme_rdma_free_ctrl+0xb0/0xb0 [nvme_rdma]
>> [  293.483100]  blk_mq_reinit_tagset+0x5c/0x90
>> [  293.488508]  nvme_rdma_configure_io_queues+0x211/0x290 [nvme_rdma]
>> [  293.496152]  nvme_rdma_reconnect_ctrl_work+0x5b/0xd0 [nvme_rdma]
>> [  293.503598]  process_one_work+0x149/0x360
>> [  293.508815]  worker_thread+0x4d/0x3c0
>> [  293.513638]  kthread+0x109/0x140
>> [  293.517973]  ? rescuer_thread+0x380/0x380
>> [  293.523176]  ? kthread_park+0x60/0x60
>> [  293.527993]  ret_from_fork+0x25/0x30

Is it possible that ib_dereg_mr failed?

can you please apply the following patch and report if you see a warning?
--
diff --git a/drivers/nvme/host/rdma.c b/drivers/nvme/host/rdma.c
index 92a03ff5fb4d..ef50b58b0bb6 100644
--- a/drivers/nvme/host/rdma.c
+++ b/drivers/nvme/host/rdma.c
@@ -274,7 +274,7 @@ static int nvme_rdma_reinit_request(void *data, 
struct request *rq)
         struct nvme_rdma_request *req = blk_mq_rq_to_pdu(rq);
         int ret = 0;

-       ib_dereg_mr(req->mr);
+       WARN_ON_ONCE(ib_dereg_mr(req->mr));

         req->mr = ib_alloc_mr(dev->pd, IB_MR_TYPE_MEM_REG,
                         ctrl->max_fr_pages);
--
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: nvmeof rdma regression issue on 4.14.0-rc1 (or maybe mlx4?)

[Yi Zhang]

> Is it possible that ib_dereg_mr failed?
>
It seems not, and finally the system get panic, here is the log:

[  104.373784] nvme nvme0: new ctrl: NQN 
"nqn.2014-08.org.nvmexpress.discovery", addr 172.31.0.90:4420
[  104.564001] nvme nvme0: creating 40 I/O queues.
[  105.070022] nvme nvme0: new ctrl: NQN "testnqn", addr 172.31.0.90:4420
[  144.135070] nvme nvme0: rescanning
[  204.383678] nvme nvme0: Reconnecting in 10 seconds...
[  214.506489] nvme nvme0: Connect rejected: status 8 (invalid service ID).
[  214.513996] nvme nvme0: rdma_resolve_addr wait failed (-104).
[  214.520426] nvme nvme0: Failed reconnect attempt 1
[  214.525788] nvme nvme0: Reconnecting in 10 seconds...
[  224.733962] nvme nvme0: Connect rejected: status 8 (invalid service ID).
[  224.741464] nvme nvme0: rdma_resolve_addr wait failed (-104).
[  224.747898] nvme nvme0: Failed reconnect attempt 2
[  224.753301] nvme nvme0: Reconnecting in 10 seconds...
[  234.973834] nvme nvme0: Connect rejected: status 8 (invalid service ID).
[  234.981335] nvme nvme0: rdma_resolve_addr wait failed (-104).
[  234.987768] nvme nvme0: Failed reconnect attempt 3
[  234.993150] nvme nvme0: Reconnecting in 10 seconds...
[  245.233395] nvme nvme0: creating 40 I/O queues.
[  245.238480] DMAR: ERROR: DMA PTE for vPFN 0xe109b already set (to 
10098cc002 not 103b85e003)
[  245.247940] ------------[ cut here ]------------
[  245.253110] WARNING: CPU: 38 PID: 6 at 
drivers/iommu/intel-iommu.c:2305 __domain_mapping+0x367/0x380
[  245.263329] Modules linked in: nvme_rdma nvme_fabrics nvme_core 
sch_mqprio ebtable_filter ebtables ip6table_filter ip6_tables 
iptable_filter bridge 8021q garp mrp stp llc rpcrdma ib_isert 
iscsi_target_mod ibd
[  245.342493]  mgag200 i2c_algo_bit drm_kms_helper syscopyarea 
sysfillrect sysimgblt fb_sys_fops ttm drm mlx4_core tg3 ahci libahci ptp 
libata crc32c_intel i2c_core pps_core devlink dm_mirror dm_region_hash dmd
[  245.364191] CPU: 38 PID: 6 Comm: kworker/u368:0 Not tainted 
4.14.0-rc1+ #7
[  245.371880] Hardware name: Dell Inc. PowerEdge R430/03XKDV, BIOS 
1.6.2 01/08/2016
[  245.380265] Workqueue: ib_addr process_one_req [ib_core]
[  245.386211] task: ffff88018cb245c0 task.stack: ffffc9000009c000
[  245.392836] RIP: 0010:__domain_mapping+0x367/0x380
[  245.398194] RSP: 0018:ffffc9000009fa98 EFLAGS: 00010202
[  245.404039] RAX: 0000000000000004 RBX: 000000103b85e003 RCX: 
0000000000000000
[  245.412018] RDX: 0000000000000000 RSI: ffff88103eace038 RDI: 
ffff88103eace038
[  245.420001] RBP: ffffc9000009faf8 R08: 0000000000000000 R09: 
0000000000000000
[  245.427983] R10: 00000000000002f7 R11: 000000000103b85e R12: 
ffff881009bc74d8
[  245.436711] R13: 0000000000000001 R14: 0000000000000001 R15: 
00000000000e109b
[  245.445419] FS:  0000000000000000(0000) GS:ffff88103eac0000(0000) 
knlGS:0000000000000000
[  245.455199] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  245.462357] CR2: 000014940a9b7140 CR3: 00000010119b5000 CR4: 
00000000001606e0
[  245.471074] Call Trace:
[  245.474549]  __intel_map_single+0xeb/0x180
[  245.479868]  intel_alloc_coherent+0xb5/0x130
[  245.485388]  mlx4_buf_alloc+0xe5/0x1c0 [mlx4_core]
[  245.491482]  mlx4_ib_alloc_cq_buf.isra.9+0x38/0xd0 [mlx4_ib]
[  245.498540]  mlx4_ib_create_cq+0x223/0x450 [mlx4_ib]
[  245.504822]  ib_alloc_cq+0x49/0x170 [ib_core]
[  245.510413]  nvme_rdma_cm_handler+0x3a2/0x7ab [nvme_rdma]
[  245.517179]  ? cma_acquire_dev+0x1e3/0x3b0 [rdma_cm]
[  245.523456]  addr_handler+0xa4/0x1c0 [rdma_cm]
[  245.529147]  process_one_req+0x8d/0x120 [ib_core]
[  245.535132]  process_one_work+0x149/0x360
[  245.540334]  worker_thread+0x4d/0x3c0
[  245.545145]  kthread+0x109/0x140
[  245.549462]  ? rescuer_thread+0x380/0x380
[  245.554654]  ? kthread_park+0x60/0x60
[  245.559456]  ret_from_fork+0x25/0x30
[  245.564153] Code: fe aa 81 4c 89 5d a0 4c 89 4d a8 e8 87 e1 c0 ff 8b 
05 fe 6e 87 00 4c 8b 4d a8 4c 8b 5d a0 85 c0 74 09 83 e8 01 89 05 e9 6e 
87 00 <0f> ff e9 b8 fd ff ff e8 8d c7 ba ff 0f 1f 00 66 2e 0f 1f 8
[  245.586712] ---[ end trace 56749c1831388ff8 ]---
[  245.592920] mlx4_core 0000:04:00.0: dma_pool_free mlx4_cmd, 
cccccccccccccccc/ccd80eccccccf203 (bad dma)
[  245.604179] mlx4_core 0000:04:00.0: dma_pool_free mlx4_cmd, 
cccccccccccccccc/cccccccccccccccc (bad dma)
[  245.615647] general protection fault: 0000 [#1] SMP
[  245.621836] Modules linked in: nvme_rdma nvme_fabrics nvme_core 
sch_mqprio ebtable_filter ebtables ip6table_filter ip6_tables 
iptable_filter bridge 8021q garp mrp stp llc rpcrdma ib_isert 
iscsi_target_mod ibd
[  245.706171]  mgag200 i2c_algo_bit drm_kms_helper syscopyarea 
sysfillrect sysimgblt fb_sys_fops ttm drm mlx4_core tg3 ahci libahci ptp 
libata crc32c_intel i2c_core pps_core devlink dm_mirror dm_region_hash dmd
[  245.729344] CPU: 38 PID: 6 Comm: kworker/u368:0 Tainted: G W       
4.14.0-rc1+ #7
[  245.739128] Hardware name: Dell Inc. PowerEdge R430/03XKDV, BIOS 
1.6.2 01/08/2016
[  245.748234] Workqueue: ib_addr process_one_req [ib_core]
[  245.754905] task: ffff88018cb245c0 task.stack: ffffc9000009c000
[  245.762256] RIP: 0010:prefetch_freepointer.isra.65+0x11/0x20
[  245.769313] RSP: 0018:ffffc9000009fcc0 EFLAGS: 00010286
[  245.775881] RAX: 0000000000000000 RBX: cccccccccccccccc RCX: 
0000000000001793
[  245.784591] RDX: 0000000000001792 RSI: cccccccccccccccc RDI: 
ffff88018fc07aa0
[  245.793294] RBP: ffffc9000009fcc0 R08: 000000000001ed40 R09: 
ffff8810098cccc0
[  245.802002] R10: ffffffff818a99e0 R11: 00000000010098cd R12: 
00000000014080c0
[  245.810706] R13: ffffffffa07bd1e0 R14: ffff88018fc07a80 R15: 
ffff88018fc07a80
[  245.819409] FS:  0000000000000000(0000) GS:ffff88103eac0000(0000) 
knlGS:0000000000000000
[  245.829184] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  245.836342] CR2: 000014940a9b7140 CR3: 00000010119b5000 CR4: 
00000000001606e0
[  245.845056] Call Trace:
[  245.848524]  kmem_cache_alloc_trace+0xa0/0x1c0
[  245.854220]  nvme_rdma_cm_handler+0x4e0/0x7ab [nvme_rdma]
[  245.860990]  addr_handler+0xa4/0x1c0 [rdma_cm]
[  245.866694]  process_one_req+0x8d/0x120 [ib_core]
[  245.872687]  process_one_work+0x149/0x360
[  245.877899]  worker_thread+0x4d/0x3c0
[  245.882720]  kthread+0x109/0x140
[  245.887051]  ? rescuer_thread+0x380/0x380
[  245.892244]  ? kthread_park+0x60/0x60
[  245.897054]  ret_from_fork+0x25/0x30
[  245.901760] Code: 31 d2 e8 b3 ea ff ff 5b 41 5c 5d c3 0f 1f 40 00 66 
2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 55 48 85 f6 48 89 e5 74 0a 48 
63 07 <48> 8b 04 06 0f 18 08 5d c3 66 0f 1f 44 00 00 0f 1f 44 00 0
[  245.924349] RIP: prefetch_freepointer.isra.65+0x11/0x20 RSP: 
ffffc9000009fcc0
[  245.933145] ---[ end trace 56749c1831388ff9 ]---
[  245.942680] Kernel panic - not syncing: Fatal exception
[  245.950207] Kernel Offset: disabled
[  245.958566] ---[ end Kernel panic - not syncing: Fatal exception
[  245.966082] ------------[ cut here ]------------
[  245.972014] WARNING: CPU: 38 PID: 6 at kernel/sched/core.c:1179 
set_task_cpu+0x191/0x1a0
[  245.981822] Modules linked in: nvme_rdma nvme_fabrics nvme_core 
sch_mqprio ebtable_filter ebtables ip6table_filter ip6_tables 
iptable_filter bridge 8021q garp mrp stp llc rpcrdma ib_isert 
iscsi_target_mod ibd
[  246.066533]  mgag200 i2c_algo_bit drm_kms_helper syscopyarea 
sysfillrect sysimgblt fb_sys_fops ttm drm mlx4_core tg3 ahci libahci ptp 
libata crc32c_intel i2c_core pps_core devlink dm_mirror dm_region_hash dmd
[  246.089836] CPU: 38 PID: 6 Comm: kworker/u368:0 Tainted: G      D 
W       4.14.0-rc1+ #7
[  246.099683] Hardware name: Dell Inc. PowerEdge R430/03XKDV, BIOS 
1.6.2 01/08/2016
[  246.108849] Workqueue: ib_addr process_one_req [ib_core]
[  246.115566] task: ffff88018cb245c0 task.stack: ffffc9000009c000
[  246.122948] RIP: 0010:set_task_cpu+0x191/0x1a0
[  246.128668] RSP: 0018:ffff88103eac3c38 EFLAGS: 00010046
[  246.135255] RAX: 0000000000000100 RBX: ffff88207bf445c0 RCX: 
0000000000000001
[  246.143978] RDX: 0000000000000001 RSI: 0000000000000001 RDI: 
ffff88207bf445c0
[  246.152699] RBP: ffff88103eac3c58 R08: 0000000000000001 R09: 
0000000000000000
[  246.161418] R10: 0000000000000001 R11: 0000000003e236eb R12: 
ffff88207bf4516c
[  246.170137] R13: 0000000000000001 R14: 0000000000000001 R15: 
000000000001b900
[  246.178854] FS:  0000000000000000(0000) GS:ffff88103eac0000(0000) 
knlGS:0000000000000000
[  246.188644] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  246.195812] CR2: 000014940a9b7140 CR3: 00000010119b5000 CR4: 
00000000001606e0
[  246.204540] Call Trace:
[  246.208027]  <IRQ>
[  246.211016]  try_to_wake_up+0x166/0x470
[  246.216036]  default_wake_function+0x12/0x20
[  246.221537]  __wake_up_common+0x8a/0x160
[  246.226641]  __wake_up_locked+0x16/0x20
[  246.231643]  ep_poll_callback+0xd0/0x300
[  246.236727]  __wake_up_common+0x8a/0x160
[  246.241817]  __wake_up_common_lock+0x7e/0xc0
[  246.247291]  __wake_up+0x13/0x20
[  246.251596]  wake_up_klogd_work_func+0x40/0x60
[  246.257265]  irq_work_run_list+0x4d/0x70
[  246.262353]  ? tick_sched_do_timer+0x70/0x70
[  246.267830]  irq_work_tick+0x40/0x50
[  246.272530]  update_process_times+0x42/0x60
[  246.277912]  tick_sched_handle+0x2d/0x60
[  246.282987]  tick_sched_timer+0x39/0x70
[  246.287945]  __hrtimer_run_queues+0xe5/0x230
[  246.293371]  hrtimer_interrupt+0xa8/0x1a0
[  246.298509]  smp_apic_timer_interrupt+0x5f/0x130
[  246.304322]  apic_timer_interrupt+0x9d/0xb0
[  246.309640]  </IRQ>
[  246.312633] RIP: 0010:panic+0x1fd/0x245
[  246.317554] RSP: 0018:ffffc9000009fb18 EFLAGS: 00000246 ORIG_RAX: 
ffffffffffffff10
[  246.326659] RAX: 0000000000000034 RBX: 0000000000000200 RCX: 
0000000000000006
[  246.335268] RDX: 0000000000000000 RSI: 0000000000000086 RDI: 
ffff88103eace030
[  246.343856] RBP: ffffc9000009fb88 R08: 0000000000000000 R09: 
0000000000000877
[  246.352424] R10: 00000000000003ff R11: 0000000000000001 R12: 
ffffffff81a3e1d8
[  246.360975] R13: 0000000000000000 R14: 0000000000000000 R15: 
ffff88018fc07a80
[  246.369508]  ? panic+0x1f6/0x245
[  246.373657]  oops_end+0xb8/0xd0
[  246.377676]  die+0x42/0x50
[  246.381194]  do_general_protection+0xd2/0x160
[  246.386540]  ? nvme_rdma_cm_handler+0x4e0/0x7ab [nvme_rdma]
[  246.393238]  general_protection+0x22/0x30
[  246.398181] RIP: 0010:prefetch_freepointer.isra.65+0x11/0x20
[  246.404964] RSP: 0018:ffffc9000009fcc0 EFLAGS: 00010286
[  246.411258] RAX: 0000000000000000 RBX: cccccccccccccccc RCX: 
0000000000001793
[  246.419692] RDX: 0000000000001792 RSI: cccccccccccccccc RDI: 
ffff88018fc07aa0
[  246.428115] RBP: ffffc9000009fcc0 R08: 000000000001ed40 R09: 
ffff8810098cccc0
[  246.436543] R10: ffffffff818a99e0 R11: 00000000010098cd R12: 
00000000014080c0
[  246.444970] R13: ffffffffa07bd1e0 R14: ffff88018fc07a80 R15: 
ffff88018fc07a80
[  246.453402]  ? nvme_rdma_cm_handler+0x4e0/0x7ab [nvme_rdma]
[  246.460087]  kmem_cache_alloc_trace+0xa0/0x1c0
[  246.465511]  nvme_rdma_cm_handler+0x4e0/0x7ab [nvme_rdma]
[  246.472004]  addr_handler+0xa4/0x1c0 [rdma_cm]
[  246.477424]  process_one_req+0x8d/0x120 [ib_core]
[  246.483128]  process_one_work+0x149/0x360
[  246.488045]  worker_thread+0x4d/0x3c0
[  246.492577]  kthread+0x109/0x140
[  246.496620]  ? rescuer_thread+0x380/0x380
[  246.501540]  ? kthread_park+0x60/0x60
[  246.506070]  ret_from_fork+0x25/0x30
[  246.510496] Code: ff 80 8b ac 08 00 00 04 e9 23 ff ff ff 0f ff e9 bf 
fe ff ff f7 83 84 00 00 00 fd ff ff ff 0f 84 c9 fe ff ff 0f ff e9 c2 fe 
ff ff <0f> ff e9 d1 fe ff ff 0f 1f 84 00 00 00 00 00 0f 1f 44 00 0
[  246.532545] ---[ end trace 56749c1831388ffa ]---

> can you please apply the following patch and report if you see a warning?
> -- 
> diff --git a/drivers/nvme/host/rdma.c b/drivers/nvme/host/rdma.c
> index 92a03ff5fb4d..ef50b58b0bb6 100644
> --- a/drivers/nvme/host/rdma.c
> +++ b/drivers/nvme/host/rdma.c
> @@ -274,7 +274,7 @@ static int nvme_rdma_reinit_request(void *data, 
> struct request *rq)
>         struct nvme_rdma_request *req = blk_mq_rq_to_pdu(rq);
>         int ret = 0;
>
> -       ib_dereg_mr(req->mr);
> +       WARN_ON_ONCE(ib_dereg_mr(req->mr));
>
>         req->mr = ib_alloc_mr(dev->pd, IB_MR_TYPE_MEM_REG,
>                         ctrl->max_fr_pages);
> -- 
>
> _______________________________________________
> Linux-nvme mailing list
> Linux-nvme-IAPFreCvJWM7uuMidbF8XUB+6BGkLq7r@public.gmane.org
> http://lists.infradead.org/mailman/listinfo/linux-nvme

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: nvmeof rdma regression issue on 4.14.0-rc1 (or maybe mlx4?)

[Leon Romanovsky]

[-- Attachment #1: Type: text/plain, Size: 13284 bytes --]

On Sun, Sep 24, 2017 at 05:28:30PM +0800, Yi Zhang wrote:
>
> > Is it possible that ib_dereg_mr failed?
> >
> It seems not, and finally the system get panic, here is the log:

I looked on the issue during the weekend and didn't see any suspicious
commit in the mlx4 alloc/mapping area.

Can I ask you to perform git bisect to find the problematic change?

Added Tariq to the thread.

Thanks

>
> [  104.373784] nvme nvme0: new ctrl: NQN
> "nqn.2014-08.org.nvmexpress.discovery", addr 172.31.0.90:4420
> [  104.564001] nvme nvme0: creating 40 I/O queues.
> [  105.070022] nvme nvme0: new ctrl: NQN "testnqn", addr 172.31.0.90:4420
> [  144.135070] nvme nvme0: rescanning
> [  204.383678] nvme nvme0: Reconnecting in 10 seconds...
> [  214.506489] nvme nvme0: Connect rejected: status 8 (invalid service ID).
> [  214.513996] nvme nvme0: rdma_resolve_addr wait failed (-104).
> [  214.520426] nvme nvme0: Failed reconnect attempt 1
> [  214.525788] nvme nvme0: Reconnecting in 10 seconds...
> [  224.733962] nvme nvme0: Connect rejected: status 8 (invalid service ID).
> [  224.741464] nvme nvme0: rdma_resolve_addr wait failed (-104).
> [  224.747898] nvme nvme0: Failed reconnect attempt 2
> [  224.753301] nvme nvme0: Reconnecting in 10 seconds...
> [  234.973834] nvme nvme0: Connect rejected: status 8 (invalid service ID).
> [  234.981335] nvme nvme0: rdma_resolve_addr wait failed (-104).
> [  234.987768] nvme nvme0: Failed reconnect attempt 3
> [  234.993150] nvme nvme0: Reconnecting in 10 seconds...
> [  245.233395] nvme nvme0: creating 40 I/O queues.
> [  245.238480] DMAR: ERROR: DMA PTE for vPFN 0xe109b already set (to
> 10098cc002 not 103b85e003)
> [  245.247940] ------------[ cut here ]------------
> [  245.253110] WARNING: CPU: 38 PID: 6 at drivers/iommu/intel-iommu.c:2305
> __domain_mapping+0x367/0x380
> [  245.263329] Modules linked in: nvme_rdma nvme_fabrics nvme_core
> sch_mqprio ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter
> bridge 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ibd
> [  245.342493]  mgag200 i2c_algo_bit drm_kms_helper syscopyarea sysfillrect
> sysimgblt fb_sys_fops ttm drm mlx4_core tg3 ahci libahci ptp libata
> crc32c_intel i2c_core pps_core devlink dm_mirror dm_region_hash dmd
> [  245.364191] CPU: 38 PID: 6 Comm: kworker/u368:0 Not tainted 4.14.0-rc1+
> #7
> [  245.371880] Hardware name: Dell Inc. PowerEdge R430/03XKDV, BIOS 1.6.2
> 01/08/2016
> [  245.380265] Workqueue: ib_addr process_one_req [ib_core]
> [  245.386211] task: ffff88018cb245c0 task.stack: ffffc9000009c000
> [  245.392836] RIP: 0010:__domain_mapping+0x367/0x380
> [  245.398194] RSP: 0018:ffffc9000009fa98 EFLAGS: 00010202
> [  245.404039] RAX: 0000000000000004 RBX: 000000103b85e003 RCX:
> 0000000000000000
> [  245.412018] RDX: 0000000000000000 RSI: ffff88103eace038 RDI:
> ffff88103eace038
> [  245.420001] RBP: ffffc9000009faf8 R08: 0000000000000000 R09:
> 0000000000000000
> [  245.427983] R10: 00000000000002f7 R11: 000000000103b85e R12:
> ffff881009bc74d8
> [  245.436711] R13: 0000000000000001 R14: 0000000000000001 R15:
> 00000000000e109b
> [  245.445419] FS:  0000000000000000(0000) GS:ffff88103eac0000(0000)
> knlGS:0000000000000000
> [  245.455199] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> [  245.462357] CR2: 000014940a9b7140 CR3: 00000010119b5000 CR4:
> 00000000001606e0
> [  245.471074] Call Trace:
> [  245.474549]  __intel_map_single+0xeb/0x180
> [  245.479868]  intel_alloc_coherent+0xb5/0x130
> [  245.485388]  mlx4_buf_alloc+0xe5/0x1c0 [mlx4_core]
> [  245.491482]  mlx4_ib_alloc_cq_buf.isra.9+0x38/0xd0 [mlx4_ib]
> [  245.498540]  mlx4_ib_create_cq+0x223/0x450 [mlx4_ib]
> [  245.504822]  ib_alloc_cq+0x49/0x170 [ib_core]
> [  245.510413]  nvme_rdma_cm_handler+0x3a2/0x7ab [nvme_rdma]
> [  245.517179]  ? cma_acquire_dev+0x1e3/0x3b0 [rdma_cm]
> [  245.523456]  addr_handler+0xa4/0x1c0 [rdma_cm]
> [  245.529147]  process_one_req+0x8d/0x120 [ib_core]
> [  245.535132]  process_one_work+0x149/0x360
> [  245.540334]  worker_thread+0x4d/0x3c0
> [  245.545145]  kthread+0x109/0x140
> [  245.549462]  ? rescuer_thread+0x380/0x380
> [  245.554654]  ? kthread_park+0x60/0x60
> [  245.559456]  ret_from_fork+0x25/0x30
> [  245.564153] Code: fe aa 81 4c 89 5d a0 4c 89 4d a8 e8 87 e1 c0 ff 8b 05
> fe 6e 87 00 4c 8b 4d a8 4c 8b 5d a0 85 c0 74 09 83 e8 01 89 05 e9 6e 87 00
> <0f> ff e9 b8 fd ff ff e8 8d c7 ba ff 0f 1f 00 66 2e 0f 1f 8
> [  245.586712] ---[ end trace 56749c1831388ff8 ]---
> [  245.592920] mlx4_core 0000:04:00.0: dma_pool_free mlx4_cmd,
> cccccccccccccccc/ccd80eccccccf203 (bad dma)
> [  245.604179] mlx4_core 0000:04:00.0: dma_pool_free mlx4_cmd,
> cccccccccccccccc/cccccccccccccccc (bad dma)
> [  245.615647] general protection fault: 0000 [#1] SMP
> [  245.621836] Modules linked in: nvme_rdma nvme_fabrics nvme_core
> sch_mqprio ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter
> bridge 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ibd
> [  245.706171]  mgag200 i2c_algo_bit drm_kms_helper syscopyarea sysfillrect
> sysimgblt fb_sys_fops ttm drm mlx4_core tg3 ahci libahci ptp libata
> crc32c_intel i2c_core pps_core devlink dm_mirror dm_region_hash dmd
> [  245.729344] CPU: 38 PID: 6 Comm: kworker/u368:0 Tainted: G W
> 4.14.0-rc1+ #7
> [  245.739128] Hardware name: Dell Inc. PowerEdge R430/03XKDV, BIOS 1.6.2
> 01/08/2016
> [  245.748234] Workqueue: ib_addr process_one_req [ib_core]
> [  245.754905] task: ffff88018cb245c0 task.stack: ffffc9000009c000
> [  245.762256] RIP: 0010:prefetch_freepointer.isra.65+0x11/0x20
> [  245.769313] RSP: 0018:ffffc9000009fcc0 EFLAGS: 00010286
> [  245.775881] RAX: 0000000000000000 RBX: cccccccccccccccc RCX:
> 0000000000001793
> [  245.784591] RDX: 0000000000001792 RSI: cccccccccccccccc RDI:
> ffff88018fc07aa0
> [  245.793294] RBP: ffffc9000009fcc0 R08: 000000000001ed40 R09:
> ffff8810098cccc0
> [  245.802002] R10: ffffffff818a99e0 R11: 00000000010098cd R12:
> 00000000014080c0
> [  245.810706] R13: ffffffffa07bd1e0 R14: ffff88018fc07a80 R15:
> ffff88018fc07a80
> [  245.819409] FS:  0000000000000000(0000) GS:ffff88103eac0000(0000)
> knlGS:0000000000000000
> [  245.829184] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> [  245.836342] CR2: 000014940a9b7140 CR3: 00000010119b5000 CR4:
> 00000000001606e0
> [  245.845056] Call Trace:
> [  245.848524]  kmem_cache_alloc_trace+0xa0/0x1c0
> [  245.854220]  nvme_rdma_cm_handler+0x4e0/0x7ab [nvme_rdma]
> [  245.860990]  addr_handler+0xa4/0x1c0 [rdma_cm]
> [  245.866694]  process_one_req+0x8d/0x120 [ib_core]
> [  245.872687]  process_one_work+0x149/0x360
> [  245.877899]  worker_thread+0x4d/0x3c0
> [  245.882720]  kthread+0x109/0x140
> [  245.887051]  ? rescuer_thread+0x380/0x380
> [  245.892244]  ? kthread_park+0x60/0x60
> [  245.897054]  ret_from_fork+0x25/0x30
> [  245.901760] Code: 31 d2 e8 b3 ea ff ff 5b 41 5c 5d c3 0f 1f 40 00 66 2e
> 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 55 48 85 f6 48 89 e5 74 0a 48 63 07
> <48> 8b 04 06 0f 18 08 5d c3 66 0f 1f 44 00 00 0f 1f 44 00 0
> [  245.924349] RIP: prefetch_freepointer.isra.65+0x11/0x20 RSP:
> ffffc9000009fcc0
> [  245.933145] ---[ end trace 56749c1831388ff9 ]---
> [  245.942680] Kernel panic - not syncing: Fatal exception
> [  245.950207] Kernel Offset: disabled
> [  245.958566] ---[ end Kernel panic - not syncing: Fatal exception
> [  245.966082] ------------[ cut here ]------------
> [  245.972014] WARNING: CPU: 38 PID: 6 at kernel/sched/core.c:1179
> set_task_cpu+0x191/0x1a0
> [  245.981822] Modules linked in: nvme_rdma nvme_fabrics nvme_core
> sch_mqprio ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter
> bridge 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ibd
> [  246.066533]  mgag200 i2c_algo_bit drm_kms_helper syscopyarea sysfillrect
> sysimgblt fb_sys_fops ttm drm mlx4_core tg3 ahci libahci ptp libata
> crc32c_intel i2c_core pps_core devlink dm_mirror dm_region_hash dmd
> [  246.089836] CPU: 38 PID: 6 Comm: kworker/u368:0 Tainted: G      D W
> 4.14.0-rc1+ #7
> [  246.099683] Hardware name: Dell Inc. PowerEdge R430/03XKDV, BIOS 1.6.2
> 01/08/2016
> [  246.108849] Workqueue: ib_addr process_one_req [ib_core]
> [  246.115566] task: ffff88018cb245c0 task.stack: ffffc9000009c000
> [  246.122948] RIP: 0010:set_task_cpu+0x191/0x1a0
> [  246.128668] RSP: 0018:ffff88103eac3c38 EFLAGS: 00010046
> [  246.135255] RAX: 0000000000000100 RBX: ffff88207bf445c0 RCX:
> 0000000000000001
> [  246.143978] RDX: 0000000000000001 RSI: 0000000000000001 RDI:
> ffff88207bf445c0
> [  246.152699] RBP: ffff88103eac3c58 R08: 0000000000000001 R09:
> 0000000000000000
> [  246.161418] R10: 0000000000000001 R11: 0000000003e236eb R12:
> ffff88207bf4516c
> [  246.170137] R13: 0000000000000001 R14: 0000000000000001 R15:
> 000000000001b900
> [  246.178854] FS:  0000000000000000(0000) GS:ffff88103eac0000(0000)
> knlGS:0000000000000000
> [  246.188644] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> [  246.195812] CR2: 000014940a9b7140 CR3: 00000010119b5000 CR4:
> 00000000001606e0
> [  246.204540] Call Trace:
> [  246.208027]  <IRQ>
> [  246.211016]  try_to_wake_up+0x166/0x470
> [  246.216036]  default_wake_function+0x12/0x20
> [  246.221537]  __wake_up_common+0x8a/0x160
> [  246.226641]  __wake_up_locked+0x16/0x20
> [  246.231643]  ep_poll_callback+0xd0/0x300
> [  246.236727]  __wake_up_common+0x8a/0x160
> [  246.241817]  __wake_up_common_lock+0x7e/0xc0
> [  246.247291]  __wake_up+0x13/0x20
> [  246.251596]  wake_up_klogd_work_func+0x40/0x60
> [  246.257265]  irq_work_run_list+0x4d/0x70
> [  246.262353]  ? tick_sched_do_timer+0x70/0x70
> [  246.267830]  irq_work_tick+0x40/0x50
> [  246.272530]  update_process_times+0x42/0x60
> [  246.277912]  tick_sched_handle+0x2d/0x60
> [  246.282987]  tick_sched_timer+0x39/0x70
> [  246.287945]  __hrtimer_run_queues+0xe5/0x230
> [  246.293371]  hrtimer_interrupt+0xa8/0x1a0
> [  246.298509]  smp_apic_timer_interrupt+0x5f/0x130
> [  246.304322]  apic_timer_interrupt+0x9d/0xb0
> [  246.309640]  </IRQ>
> [  246.312633] RIP: 0010:panic+0x1fd/0x245
> [  246.317554] RSP: 0018:ffffc9000009fb18 EFLAGS: 00000246 ORIG_RAX:
> ffffffffffffff10
> [  246.326659] RAX: 0000000000000034 RBX: 0000000000000200 RCX:
> 0000000000000006
> [  246.335268] RDX: 0000000000000000 RSI: 0000000000000086 RDI:
> ffff88103eace030
> [  246.343856] RBP: ffffc9000009fb88 R08: 0000000000000000 R09:
> 0000000000000877
> [  246.352424] R10: 00000000000003ff R11: 0000000000000001 R12:
> ffffffff81a3e1d8
> [  246.360975] R13: 0000000000000000 R14: 0000000000000000 R15:
> ffff88018fc07a80
> [  246.369508]  ? panic+0x1f6/0x245
> [  246.373657]  oops_end+0xb8/0xd0
> [  246.377676]  die+0x42/0x50
> [  246.381194]  do_general_protection+0xd2/0x160
> [  246.386540]  ? nvme_rdma_cm_handler+0x4e0/0x7ab [nvme_rdma]
> [  246.393238]  general_protection+0x22/0x30
> [  246.398181] RIP: 0010:prefetch_freepointer.isra.65+0x11/0x20
> [  246.404964] RSP: 0018:ffffc9000009fcc0 EFLAGS: 00010286
> [  246.411258] RAX: 0000000000000000 RBX: cccccccccccccccc RCX:
> 0000000000001793
> [  246.419692] RDX: 0000000000001792 RSI: cccccccccccccccc RDI:
> ffff88018fc07aa0
> [  246.428115] RBP: ffffc9000009fcc0 R08: 000000000001ed40 R09:
> ffff8810098cccc0
> [  246.436543] R10: ffffffff818a99e0 R11: 00000000010098cd R12:
> 00000000014080c0
> [  246.444970] R13: ffffffffa07bd1e0 R14: ffff88018fc07a80 R15:
> ffff88018fc07a80
> [  246.453402]  ? nvme_rdma_cm_handler+0x4e0/0x7ab [nvme_rdma]
> [  246.460087]  kmem_cache_alloc_trace+0xa0/0x1c0
> [  246.465511]  nvme_rdma_cm_handler+0x4e0/0x7ab [nvme_rdma]
> [  246.472004]  addr_handler+0xa4/0x1c0 [rdma_cm]
> [  246.477424]  process_one_req+0x8d/0x120 [ib_core]
> [  246.483128]  process_one_work+0x149/0x360
> [  246.488045]  worker_thread+0x4d/0x3c0
> [  246.492577]  kthread+0x109/0x140
> [  246.496620]  ? rescuer_thread+0x380/0x380
> [  246.501540]  ? kthread_park+0x60/0x60
> [  246.506070]  ret_from_fork+0x25/0x30
> [  246.510496] Code: ff 80 8b ac 08 00 00 04 e9 23 ff ff ff 0f ff e9 bf fe
> ff ff f7 83 84 00 00 00 fd ff ff ff 0f 84 c9 fe ff ff 0f ff e9 c2 fe ff ff
> <0f> ff e9 d1 fe ff ff 0f 1f 84 00 00 00 00 00 0f 1f 44 00 0
> [  246.532545] ---[ end trace 56749c1831388ffa ]---
>
> > can you please apply the following patch and report if you see a warning?
> > --
> > diff --git a/drivers/nvme/host/rdma.c b/drivers/nvme/host/rdma.c
> > index 92a03ff5fb4d..ef50b58b0bb6 100644
> > --- a/drivers/nvme/host/rdma.c
> > +++ b/drivers/nvme/host/rdma.c
> > @@ -274,7 +274,7 @@ static int nvme_rdma_reinit_request(void *data,
> > struct request *rq)
> >         struct nvme_rdma_request *req = blk_mq_rq_to_pdu(rq);
> >         int ret = 0;
> >
> > -       ib_dereg_mr(req->mr);
> > +       WARN_ON_ONCE(ib_dereg_mr(req->mr));
> >
> >         req->mr = ib_alloc_mr(dev->pd, IB_MR_TYPE_MEM_REG,
> >                         ctrl->max_fr_pages);
> > --
> >
> > _______________________________________________
> > Linux-nvme mailing list
> > Linux-nvme-IAPFreCvJWM7uuMidbF8XUB+6BGkLq7r@public.gmane.org
> > http://lists.infradead.org/mailman/listinfo/linux-nvme
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: nvmeof rdma regression issue on 4.14.0-rc1 (or maybe mlx4?)

[Yi Zhang]

On 09/24/2017 06:34 PM, Leon Romanovsky wrote:
> On Sun, Sep 24, 2017 at 05:28:30PM +0800, Yi Zhang wrote:
>>> Is it possible that ib_dereg_mr failed?
>>>
>> It seems not, and finally the system get panic, here is the log:
> I looked on the issue during the weekend and didn't see any suspicious
> commit in the mlx4 alloc/mapping area.
>
> Can I ask you to perform git bisect to find the problematic change?
Hi Sagi
I did git bisect for this issue, seems it was introduced by your patch 
"Few more patches from the centralization set".
Here is the testing on the patch, let me know if  you need more info.

BAD  148b4e7 nvme-rdma: stop queues instead of simply flipping their state
BAD  a57bd54 nvme-rdma: introduce configure/destroy io queues
Log:
[  127.899255] nvme nvme0: new ctrl: NQN 
"nqn.2014-08.org.nvmexpress.discovery", addr 172.31.0.90:4420
[  128.074263] nvme nvme0: creating 40 I/O queues.
[  128.581822] nvme nvme0: new ctrl: NQN "testnqn", addr 172.31.0.90:4420
[  177.486110] print_req_error: I/O error, dev nvme0n1, sector 0
[  191.256637] nvme nvme0: Reconnecting in 10 seconds...
[  201.855846] nvme nvme0: Connect rejected: status 8 (invalid service ID).
[  201.863353] nvme nvme0: rdma_resolve_addr wait failed (-104).
[  201.869824] nvme nvme0: Failed reconnect attempt 1
[  201.875183] nvme nvme0: Reconnecting in 10 seconds...
[  212.087828] nvme nvme0: Connect rejected: status 8 (invalid service ID).
[  212.095330] nvme nvme0: rdma_resolve_addr wait failed (-104).
[  212.101766] nvme nvme0: Failed reconnect attempt 2
[  212.107129] nvme nvme0: Reconnecting in 10 seconds...
[  222.328398] nvme nvme0: Connect rejected: status 8 (invalid service ID).
[  222.335900] nvme nvme0: rdma_resolve_addr wait failed (-104).
[  222.342335] nvme nvme0: Failed reconnect attempt 3
[  222.347699] nvme nvme0: Reconnecting in 10 seconds...
[  232.567791] nvme nvme0: Connect rejected: status 8 (invalid service ID).
[  232.575292] nvme nvme0: rdma_resolve_addr wait failed (-104).
[  232.581730] nvme nvme0: Failed reconnect attempt 4
[  232.587094] nvme nvme0: Reconnecting in 10 seconds...
[  242.827727] nvme nvme0: creating 40 I/O queues.
[  242.832810] DMAR: ERROR: DMA PTE for vPFN 0xe129b already set (to 
103c692002 not 1000915003)
[  242.842265] ------------[ cut here ]------------
[  242.847437] WARNING: CPU: 0 PID: 783 at 
drivers/iommu/intel-iommu.c:2299 __domain_mapping+0x363/0x370
[  242.857755] Modules linked in: nvme_rdma nvme_fabrics nvme_core 
sch_mqprio ebtable_filter ebtables ip6table_filter ip6_tables 
iptable_filter bridge 8021q garp mrp stp llc rpcrdma ib_isert 
iscsi_target_mod ibd
[  242.936919]  mgag200 i2c_algo_bit drm_kms_helper syscopyarea 
sysfillrect sysimgblt fb_sys_fops ttm drm mlx4_core tg3 ahci libahci 
crc32c_intel ptp libata i2c_core pps_core devlink dm_mirror 
dm_region_hash dmd
[  242.958625] CPU: 0 PID: 783 Comm: kworker/u368:1 Not tainted 
4.13.0-rc7.a57bd54+ #15
[  242.967304] Hardware name: Dell Inc. PowerEdge R430/03XKDV, BIOS 
1.6.2 01/08/2016
[  242.975687] Workqueue: ib_addr process_one_req [ib_core]
[  242.981631] task: ffff881019491740 task.stack: ffffc9000b534000
[  242.989011] RIP: 0010:__domain_mapping+0x363/0x370
[  242.995108] RSP: 0018:ffffc9000b537a50 EFLAGS: 00010202
[  243.001694] RAX: 0000000000000004 RBX: 0000001000915003 RCX: 
0000000000000000
[  243.010433] RDX: 0000000000000000 RSI: ffff88103e60e038 RDI: 
ffff88103e60e038
[  243.019170] RBP: ffffc9000b537ab0 R08: 0000000000000000 R09: 
0000000000000000
[  243.027893] R10: 00000000000002f7 R11: 0000000001000915 R12: 
ffff88201ea5c4d8
[  243.036632] R13: 0000000000000001 R14: 0000000000000001 R15: 
00000000000e129b
[  243.045348] FS:  0000000000000000(0000) GS:ffff88103e600000(0000) 
knlGS:0000000000000000
[  243.055142] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  243.062337] CR2: 00007f9d3b0fab70 CR3: 00000020245b7000 CR4: 
00000000001406f0
[  243.071076] Call Trace:
[  243.074564]  __intel_map_single+0xeb/0x180
[  243.079903]  intel_alloc_coherent+0xb5/0x130
[  243.085445]  mlx4_buf_alloc+0xe5/0x1c0 [mlx4_core]
[  243.091555]  mlx4_ib_alloc_cq_buf.isra.9+0x38/0xd0 [mlx4_ib]
[  243.098621]  mlx4_ib_create_cq+0x223/0x440 [mlx4_ib]
[  243.104901]  ? find_gid.isra.5+0x167/0x1f0 [ib_core]
[  243.111178]  ib_alloc_cq+0x49/0x170 [ib_core]
[  243.116791]  nvme_rdma_cm_handler+0x3e7/0x886 [nvme_rdma]
[  243.123557]  ? cma_attach_to_dev+0x17/0x50 [rdma_cm]
[  243.129838]  ? cma_acquire_dev+0x1e3/0x3b0 [rdma_cm]
[  243.136115]  ? account_entity_dequeue+0xaa/0xe0
[  243.141918]  addr_handler+0xa4/0x1c0 [rdma_cm]
[  243.147604]  process_one_req+0x8d/0x120 [ib_core]
[  243.153585]  process_one_work+0x149/0x360
[  243.158807]  worker_thread+0x4d/0x3c0
[  243.163618]  kthread+0x109/0x140
[  243.167936]  ? rescuer_thread+0x380/0x380
[  243.173131]  ? kthread_park+0x60/0x60
[  243.177930]  ret_from_fork+0x25/0x30
[  243.182641] Code: f1 a9 81 4c 89 5d a0 4c 89 4d a8 e8 0b 58 c1 ff 8b 
05 f2 16 88 00 4c 8b 4d a8 4c 8b 5d a0 85 c0 74 09 83 e8 01 89 05 dd 16 
88 00 <0f> ff e9 bc fd ff ff e8 21 3d bb ff 90 0f 1f 44 00 00 55 4
[  243.205193] ---[ end trace 725c2de52628c061 ]---
[  243.211723] mlx4_core 0000:04:00.0: dma_pool_free mlx4_cmd, 
cccccccccccccccc/cc242fcccccc2801 (bad dma)
[  243.211724] mlx4_core 0000:04:00.0: dma_pool_free mlx4_cmd, 
cccccccccccccccc/cccccccccccccccc (bad dma)
[  243.212312] general protection fault: 0000 [#1] SMP
[  243.212312] Modules linked in: nvme_rdma nvme_fabrics nvme_core 
sch_mqprio ebtable_filter ebtables ip6table_filter ip6_tables 
iptable_filter bridge 8021q garp mrp stp llc rpcrdma ib_isert 
iscsi_target_mod ibd
[  243.212339]  mgag200 i2c_algo_bit drm_kms_helper syscopyarea 
sysfillrect sysimgblt fb_sys_fops ttm drm mlx4_core tg3 ahci libahci 
crc32c_intel ptp libata i2c_core pps_core devlink dm_mirror 
dm_region_hash dmd
[  243.212353] CPU: 36 PID: 783 Comm: kworker/u368:1 Tainted: G        
W       4.13.0-rc7.a57bd54+ #15
[  243.212353] Hardware name: Dell Inc. PowerEdge R430/03XKDV, BIOS 
1.6.2 01/08/2016
[  243.212360] Workqueue: ib_addr process_one_req [ib_core]
[  243.212361] task: ffff881019491740 task.stack: ffffc9000b534000
[  243.212364] RIP: 0010:kmem_cache_alloc_trace+0x7d/0x1b0
[  243.212364] RSP: 0018:ffffc9000b537c88 EFLAGS: 00010282
[  243.212365] RAX: 0000000000000000 RBX: 00000000014080c0 RCX: 
0000000000006b83
[  243.212366] RDX: 0000000000006b82 RSI: 00000000014080c0 RDI: 
ffff88018fc07a80
[  243.212366] RBP: ffffc9000b537cc0 R08: 000000000001ed40 R09: 
0000000000000000
[  243.212373] R10: ffff88018fc07a80 R11: 000000000103c693 R12: 
00000000014080c0
[  243.212374] R13: ffffffffa08b317f R14: ffff88018fc07a80 R15: 
cccccccccccccccc
[  243.212375] FS:  0000000000000000(0000) GS:ffff88103ea80000(0000) 
knlGS:0000000000000000
[  243.212375] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  243.212376] CR2: 00007f736b47c000 CR3: 000000201b770000 CR4: 
00000000001406e0
[  243.212377] Call Trace:
[  243.212379]  nvme_rdma_cm_handler+0x4ef/0x886 [nvme_rdma]
[  243.212382]  ? cma_attach_to_dev+0x17/0x50 [rdma_cm]
[  243.212383]  ? nvme_rdma_memreg_done+0x30/0x30 [nvme_rdma]
[  243.212385]  addr_handler+0xa4/0x1c0 [rdma_cm]
[  243.212390]  process_one_req+0x8d/0x120 [ib_core]
[  243.212398]  process_one_work+0x149/0x360
[  243.212399]  worker_thread+0x4d/0x3c0
[  243.212400]  kthread+0x109/0x140
[  243.212401]  ? rescuer_thread+0x380/0x380
[  243.212403]  ? kthread_park+0x60/0x60
[  243.212404]  ret_from_fork+0x25/0x30
[  243.212405] Code: 4c 03 05 6f 12 dd 7e 4d 8b 38 49 8b 40 10 4d 85 ff 
0f 84 ec 00 00 00 48 85 c0 0f 84 e3 00 00 00 49 63 42 20 48 8d 4a 01 4d 
8b 02 <49> 8b 1c 07 4c 89 f8 65 49 0f c7 08 0f 94 c0 84 c0 74 b9 4
[  243.212420] RIP: kmem_cache_alloc_trace+0x7d/0x1b0 RSP: ffffc9000b537c88
[  243.212519] ---[ end trace 725c2de52628c062 ]---
[  243.216792] Kernel panic - not syncing: Fatal exception
[  243.216878] Kernel Offset: disabled
[  243.583898] ---[ end Kernel panic - not syncing: Fatal exception

Panic after connection with below commits, detailed log here: 
https://pastebin.com/7z0XSGSd
31fdf18     nvme-rdma: reuse configure/destroy_admin_queue
3f02fff       nvme-rdma: don't free tagset on resets
18398af    nvme-rdma: disable the controller on resets
b28a308   nvme-rdma: move tagset allocation to a dedicated routine

good    34b6c23 nvme: Add admin_tagset pointer to nvme_ctrl


Thanks
Yi
> Added Tariq to the thread.
>
> Thanks
>
>>

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: nvmeof rdma regression issue on 4.14.0-rc1 (or maybe mlx4?)

[Sagi Grimberg]

> Hi Sagi
> I did git bisect for this issue, seems it was introduced by your patch 
> "Few more patches from the centralization set".
> Here is the testing on the patch, let me know if  you need more info.

Thanks for reporting.

> BAD  148b4e7 nvme-rdma: stop queues instead of simply flipping their state
> BAD  a57bd54 nvme-rdma: introduce configure/destroy io queues
> Log:
> [  127.899255] nvme nvme0: new ctrl: NQN 
> "nqn.2014-08.org.nvmexpress.discovery", addr 172.31.0.90:4420
> [  128.074263] nvme nvme0: creating 40 I/O queues.
> [  128.581822] nvme nvme0: new ctrl: NQN "testnqn", addr 172.31.0.90:4420
> [  177.486110] print_req_error: I/O error, dev nvme0n1, sector 0
> [  191.256637] nvme nvme0: Reconnecting in 10 seconds...
> [  201.855846] nvme nvme0: Connect rejected: status 8 (invalid service ID).
> [  201.863353] nvme nvme0: rdma_resolve_addr wait failed (-104).
> [  201.869824] nvme nvme0: Failed reconnect attempt 1
> [  201.875183] nvme nvme0: Reconnecting in 10 seconds...
> [  212.087828] nvme nvme0: Connect rejected: status 8 (invalid service ID).
> [  212.095330] nvme nvme0: rdma_resolve_addr wait failed (-104).
> [  212.101766] nvme nvme0: Failed reconnect attempt 2
> [  212.107129] nvme nvme0: Reconnecting in 10 seconds...
> [  222.328398] nvme nvme0: Connect rejected: status 8 (invalid service ID).
> [  222.335900] nvme nvme0: rdma_resolve_addr wait failed (-104).
> [  222.342335] nvme nvme0: Failed reconnect attempt 3
> [  222.347699] nvme nvme0: Reconnecting in 10 seconds...
> [  232.567791] nvme nvme0: Connect rejected: status 8 (invalid service ID).
> [  232.575292] nvme nvme0: rdma_resolve_addr wait failed (-104).
> [  232.581730] nvme nvme0: Failed reconnect attempt 4
> [  232.587094] nvme nvme0: Reconnecting in 10 seconds...
> [  242.827727] nvme nvme0: creating 40 I/O queues.
> [  242.832810] DMAR: ERROR: DMA PTE for vPFN 0xe129b already set (to 
> 103c692002 not 1000915003)
> [  242.842265] ------------[ cut here ]------------
> [  242.847437] WARNING: CPU: 0 PID: 783 at 
> drivers/iommu/intel-iommu.c:2299 __domain_mapping+0x363/0x370
> [  242.857755] Modules linked in: nvme_rdma nvme_fabrics nvme_core 
> sch_mqprio ebtable_filter ebtables ip6table_filter ip6_tables 
> iptable_filter bridge 8021q garp mrp stp llc rpcrdma ib_isert 
> iscsi_target_mod ibd
> [  242.936919]  mgag200 i2c_algo_bit drm_kms_helper syscopyarea 
> sysfillrect sysimgblt fb_sys_fops ttm drm mlx4_core tg3 ahci libahci 
> crc32c_intel ptp libata i2c_core pps_core devlink dm_mirror 
> dm_region_hash dmd
> [  242.958625] CPU: 0 PID: 783 Comm: kworker/u368:1 Not tainted 
> 4.13.0-rc7.a57bd54+ #15
> [  242.967304] Hardware name: Dell Inc. PowerEdge R430/03XKDV, BIOS 
> 1.6.2 01/08/2016
> [  242.975687] Workqueue: ib_addr process_one_req [ib_core]
> [  242.981631] task: ffff881019491740 task.stack: ffffc9000b534000
> [  242.989011] RIP: 0010:__domain_mapping+0x363/0x370
> [  242.995108] RSP: 0018:ffffc9000b537a50 EFLAGS: 00010202
> [  243.001694] RAX: 0000000000000004 RBX: 0000001000915003 RCX: 
> 0000000000000000
> [  243.010433] RDX: 0000000000000000 RSI: ffff88103e60e038 RDI: 
> ffff88103e60e038
> [  243.019170] RBP: ffffc9000b537ab0 R08: 0000000000000000 R09: 
> 0000000000000000
> [  243.027893] R10: 00000000000002f7 R11: 0000000001000915 R12: 
> ffff88201ea5c4d8
> [  243.036632] R13: 0000000000000001 R14: 0000000000000001 R15: 
> 00000000000e129b
> [  243.045348] FS:  0000000000000000(0000) GS:ffff88103e600000(0000) 
> knlGS:0000000000000000
> [  243.055142] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> [  243.062337] CR2: 00007f9d3b0fab70 CR3: 00000020245b7000 CR4: 
> 00000000001406f0
> [  243.071076] Call Trace:
> [  243.074564]  __intel_map_single+0xeb/0x180
> [  243.079903]  intel_alloc_coherent+0xb5/0x130
> [  243.085445]  mlx4_buf_alloc+0xe5/0x1c0 [mlx4_core]
> [  243.091555]  mlx4_ib_alloc_cq_buf.isra.9+0x38/0xd0 [mlx4_ib]
> [  243.098621]  mlx4_ib_create_cq+0x223/0x440 [mlx4_ib]
> [  243.104901]  ? find_gid.isra.5+0x167/0x1f0 [ib_core]
> [  243.111178]  ib_alloc_cq+0x49/0x170 [ib_core]
> [  243.116791]  nvme_rdma_cm_handler+0x3e7/0x886 [nvme_rdma]
> [  243.123557]  ? cma_attach_to_dev+0x17/0x50 [rdma_cm]
> [  243.129838]  ? cma_acquire_dev+0x1e3/0x3b0 [rdma_cm]
> [  243.136115]  ? account_entity_dequeue+0xaa/0xe0
> [  243.141918]  addr_handler+0xa4/0x1c0 [rdma_cm]
> [  243.147604]  process_one_req+0x8d/0x120 [ib_core]
> [  243.153585]  process_one_work+0x149/0x360
> [  243.158807]  worker_thread+0x4d/0x3c0
> [  243.163618]  kthread+0x109/0x140
> [  243.167936]  ? rescuer_thread+0x380/0x380
> [  243.173131]  ? kthread_park+0x60/0x60
> [  243.177930]  ret_from_fork+0x25/0x30

I'm not able to reproduce this.

I still don't understand how we call ib_alloc_cq and get to
a mapped buffer, its calling dma_alloc_coherent.

Can you try the following patch and report what's going on?
--
diff --git a/drivers/nvme/host/rdma.c b/drivers/nvme/host/rdma.c
index 92a03ff5fb4d..3befaa0c53ff 100644
--- a/drivers/nvme/host/rdma.c
+++ b/drivers/nvme/host/rdma.c
@@ -927,10 +927,6 @@ static void nvme_rdma_reconnect_ctrl_work(struct 
work_struct *work)

         ++ctrl->ctrl.nr_reconnects;

-       if (ctrl->ctrl.queue_count > 1)
-               nvme_rdma_destroy_io_queues(ctrl, false);
-
-       nvme_rdma_destroy_admin_queue(ctrl, false);
         ret = nvme_rdma_configure_admin_queue(ctrl, false);
         if (ret)
                 goto requeue;
@@ -938,7 +934,7 @@ static void nvme_rdma_reconnect_ctrl_work(struct 
work_struct *work)
         if (ctrl->ctrl.queue_count > 1) {
                 ret = nvme_rdma_configure_io_queues(ctrl, false);
                 if (ret)
-                       goto requeue;
+                       goto destroy_admin;
         }

         changed = nvme_change_ctrl_state(&ctrl->ctrl, NVME_CTRL_LIVE);
@@ -956,6 +952,8 @@ static void nvme_rdma_reconnect_ctrl_work(struct 
work_struct *work)

         return;

+destroy_admin:
+       nvme_rdma_destroy_admin_queue(ctrl, false);
  requeue:
         dev_info(ctrl->ctrl.device, "Failed reconnect attempt %d\n",
                         ctrl->ctrl.nr_reconnects);
@@ -990,6 +988,10 @@ static void nvme_rdma_error_recovery_work(struct 
work_struct *work)
         blk_mq_unquiesce_queue(ctrl->ctrl.admin_q);
         nvme_start_queues(&ctrl->ctrl);

+       if (ctrl->ctrl.queue_count > 1)
+               nvme_rdma_destroy_io_queues(ctrl, false);
+       nvme_rdma_destroy_admin_queue(ctrl, false);
+
         nvme_rdma_reconnect_or_remove(ctrl);
  }
--

I guess we could see weird phenomenons if we have a resource leak,
but I ran kmemleak and could not get anything in this area...

> Panic after connection with below commits, detailed log here: 
> https://pastebin.com/7z0XSGSd
> 31fdf18     nvme-rdma: reuse configure/destroy_admin_queue
> 3f02fff       nvme-rdma: don't free tagset on resets
> 18398af    nvme-rdma: disable the controller on resets
> b28a308   nvme-rdma: move tagset allocation to a dedicated routine
> 
> good    34b6c23 nvme: Add admin_tagset pointer to nvme_ctrl

Is that a reproducible panic? I'm not seeing this at all.

Can you run gdb on nvme-rdma.ko
$ l *(nvme_rdma_create_ctrl+0x37d)
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: nvmeof rdma regression issue on 4.14.0-rc1 (or maybe mlx4?)

[Yi Zhang]

Hi Sagi
Pls see my comments bellow.
> I'm not able to reproduce this.
>
I've reproduce this on below two environment, and the target side I used 
is nullbk
[root@rdma-virt-00 ~]$ lspci | grep -i mel
04:00.0 Network controller: Mellanox Technologies MT27520 Family 
[ConnectX-3 Pro]
04:00.1 Network controller: Mellanox Technologies MT27500/MT27520 Family 
[ConnectX-3/ConnectX-3 Pro Virtual Function]
04:00.2 Network controller: Mellanox Technologies MT27500/MT27520 Family 
[ConnectX-3/ConnectX-3 Pro Virtual Function]
04:00.3 Network controller: Mellanox Technologies MT27500/MT27520 Family 
[ConnectX-3/ConnectX-3 Pro Virtual Function]
04:00.4 Network controller: Mellanox Technologies MT27500/MT27520 Family 
[ConnectX-3/ConnectX-3 Pro Virtual Function]
04:00.5 Network controller: Mellanox Technologies MT27500/MT27520 Family 
[ConnectX-3/ConnectX-3 Pro Virtual Function]
04:00.6 Network controller: Mellanox Technologies MT27500/MT27520 Family 
[ConnectX-3/ConnectX-3 Pro Virtual Function]
04:00.7 Network controller: Mellanox Technologies MT27500/MT27520 Family 
[ConnectX-3/ConnectX-3 Pro Virtual Function]
06:00.0 Network controller: Mellanox Technologies MT27520 Family 
[ConnectX-3 Pro]
06:00.1 Network controller: Mellanox Technologies MT27500/MT27520 Family 
[ConnectX-3/ConnectX-3 Pro Virtual Function]
06:00.2 Network controller: Mellanox Technologies MT27500/MT27520 Family 
[ConnectX-3/ConnectX-3 Pro Virtual Function]
06:00.3 Network controller: Mellanox Technologies MT27500/MT27520 Family 
[ConnectX-3/ConnectX-3 Pro Virtual Function]
06:00.4 Network controller: Mellanox Technologies MT27500/MT27520 Family 
[ConnectX-3/ConnectX-3 Pro Virtual Function]
06:00.5 Network controller: Mellanox Technologies MT27500/MT27520 Family 
[ConnectX-3/ConnectX-3 Pro Virtual Function]
06:00.6 Network controller: Mellanox Technologies MT27500/MT27520 Family 
[ConnectX-3/ConnectX-3 Pro Virtual Function]
06:00.7 Network controller: Mellanox Technologies MT27500/MT27520 Family 
[ConnectX-3/ConnectX-3 Pro Virtual Function]

[root@rdma-virt-03 ~]$ lspci | grep -i mel
04:00.0 Infiniband controller: Mellanox Technologies MT27700 Family 
[ConnectX-4]
04:00.1 Infiniband controller: Mellanox Technologies MT27700 Family 
[ConnectX-4]
05:00.0 Ethernet controller: Mellanox Technologies MT27710 Family 
[ConnectX-4 Lx]
05:00.1 Ethernet controller: Mellanox Technologies MT27710 Family 
[ConnectX-4 Lx]


> I still don't understand how we call ib_alloc_cq and get to
> a mapped buffer, its calling dma_alloc_coherent.
>
> Can you try the following patch and report what's going on?
> -- 
>
Here is the log, didn't get the error log, but the nvme0n1 device node 
also doesn't exist on host.
[  144.993253] nvme nvme0: new ctrl: NQN 
"nqn.2014-08.org.nvmexpress.discovery", addr 172.31.0.90:4420
[  145.177205] nvme nvme0: creating 40 I/O queues.
[  145.711199] nvme nvme0: new ctrl: NQN "testnqn", addr 172.31.0.90:4420
[  156.482770] nvme nvme0: Reconnecting in 10 seconds...
[  166.881540] nvme nvme0: Connect rejected: status 8 (invalid service ID).
[  166.889043] nvme nvme0: rdma_resolve_addr wait failed (-104).
[  166.895478] nvme nvme0: Failed reconnect attempt 1
[  166.900840] nvme nvme0: Reconnecting in 10 seconds...
[  177.120933] nvme nvme0: Connect rejected: status 8 (invalid service ID).
[  177.128434] nvme nvme0: rdma_resolve_addr wait failed (-104).
[  177.134866] nvme nvme0: Failed reconnect attempt 2
[  177.140227] nvme nvme0: Reconnecting in 10 seconds...
[  187.360819] nvme nvme0: Connect rejected: status 8 (invalid service ID).
[  187.368321] nvme nvme0: rdma_resolve_addr wait failed (-104).
[  187.374752] nvme nvme0: Failed reconnect attempt 3
[  187.380113] nvme nvme0: Reconnecting in 10 seconds...
[  197.620302] nvme nvme0: creating 40 I/O queues.
[  198.227963] nvme nvme0: Successfully reconnected
[  198.228046] nvme nvme0: identifiers changed for nsid 1

[root@rdma-virt-01 ~]$ lsblk
NAME                           MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
sda                              8:0    0 465.8G  0 disk
├─sda2                           8:2    0 464.8G  0 part
│ ├─rhelaa_rdma--virt--01-swap 253:1    0     4G  0 lvm  [SWAP]
│ ├─rhelaa_rdma--virt--01-home 253:2    0 410.8G  0 lvm  /home
│ └─rhelaa_rdma--virt--01-root 253:0    0    50G  0 lvm  /
└─sda1                           8:1    0     1G  0 part /boot

> -- 
>
> I guess we could see weird phenomenons if we have a resource leak,
> but I ran kmemleak and could not get anything in this area...
>
>> Panic after connection with below commits, detailed log here: 
>> https://pastebin.com/7z0XSGSd
>> 31fdf18     nvme-rdma: reuse configure/destroy_admin_queue
>> 3f02fff       nvme-rdma: don't free tagset on resets
>> 18398af    nvme-rdma: disable the controller on resets
>> b28a308   nvme-rdma: move tagset allocation to a dedicated routine
>>
>> good    34b6c23 nvme: Add admin_tagset pointer to nvme_ctrl
>
> Is that a reproducible panic? I'm not seeing this at all.
>

Yes, I can reproduce every time. And the target side kernel version is 
4.14.0-rc1 during the panic occurred.

> Can you run gdb on nvme-rdma.ko
> $ l *(nvme_rdma_create_ctrl+0x37d)
>
[root@rdma-virt-01 linux ((31fdf18...))]$ gdb 
/usr/lib/modules/4.13.0-rc7.31fdf18+/kernel/drivers/nvme/host/nvme-rdma.ko
GNU gdb (GDB) Red Hat Enterprise Linux 7.6.1-100.el7
Copyright (C) 2013 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later 
<http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from 
/usr/lib/modules/4.13.0-rc7.31fdf18+/kernel/drivers/nvme/host/nvme-rdma.ko...done.
(gdb) l *(nvme_rdma_create_ctrl+0x37d)
0x297d is in nvme_rdma_create_ctrl (drivers/nvme/host/rdma.c:656).
651        struct nvme_rdma_ctrl *ctrl = to_rdma_ctrl(nctrl);
652        struct blk_mq_tag_set *set = admin ?
653                &ctrl->admin_tag_set : &ctrl->tag_set;
654
655        blk_mq_free_tag_set(set);
656        nvme_rdma_dev_put(ctrl->device);
657    }
658
659    static struct blk_mq_tag_set *nvme_rdma_alloc_tagset(struct 
nvme_ctrl *nctrl,
660            bool admin)
(gdb)

> _______________________________________________
> Linux-nvme mailing list
> Linux-nvme-IAPFreCvJWM7uuMidbF8XUB+6BGkLq7r@public.gmane.org
> http://lists.infradead.org/mailman/listinfo/linux-nvme

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: nvmeof rdma regression issue on 4.14.0-rc1 (or maybe mlx4?)

[Sagi Grimberg]

> Here is the log, didn't get the error log, but the nvme0n1 device node 
> also doesn't exist on host.

That's because the identifiers changed maybe? did you have nvm0n2 instead?

> Yes, I can reproduce every time. And the target side kernel version is 
> 4.14.0-rc1 during the panic occurred.

What is the host side kernel version? doesn't look like 4.14.0-rc1
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: nvmeof rdma regression issue on 4.14.0-rc1 (or maybe mlx4?)

[Sagi Grimberg]

Hi,

>>> Panic after connection with below commits, detailed log here: 
>>> https://pastebin.com/7z0XSGSd
>>> 31fdf18     nvme-rdma: reuse configure/destroy_admin_queue
>>> 3f02fff       nvme-rdma: don't free tagset on resets
>>> 18398af    nvme-rdma: disable the controller on resets
>>> b28a308   nvme-rdma: move tagset allocation to a dedicated routine
>>>
>>> good    34b6c23 nvme: Add admin_tagset pointer to nvme_ctrl
>>
>> Is that a reproducible panic? I'm not seeing this at all.
>>
> 
> Yes, I can reproduce every time. And the target side kernel version is 
> 4.14.0-rc1 during the panic occurred.
> 
>> Can you run gdb on nvme-rdma.ko
>> $ l *(nvme_rdma_create_ctrl+0x37d)
>>
> [root@rdma-virt-01 linux ((31fdf18...))]$ gdb 
> /usr/lib/modules/4.13.0-rc7.31fdf18+/kernel/drivers/nvme/host/nvme-rdma.ko
> GNU gdb (GDB) Red Hat Enterprise Linux 7.6.1-100.el7
> Copyright (C) 2013 Free Software Foundation, Inc.
> License GPLv3+: GNU GPL version 3 or later 
> <http://gnu.org/licenses/gpl.html>
> This is free software: you are free to change and redistribute it.
> There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
> and "show warranty" for details.
> This GDB was configured as "x86_64-redhat-linux-gnu".
> For bug reporting instructions, please see:
> <http://www.gnu.org/software/gdb/bugs/>...
> Reading symbols from 
> /usr/lib/modules/4.13.0-rc7.31fdf18+/kernel/drivers/nvme/host/nvme-rdma.ko...done. 
> 
> (gdb) l *(nvme_rdma_create_ctrl+0x37d)
> 0x297d is in nvme_rdma_create_ctrl (drivers/nvme/host/rdma.c:656).
> 651        struct nvme_rdma_ctrl *ctrl = to_rdma_ctrl(nctrl);
> 652        struct blk_mq_tag_set *set = admin ?
> 653                &ctrl->admin_tag_set : &ctrl->tag_set;
> 654
> 655        blk_mq_free_tag_set(set);
> 656        nvme_rdma_dev_put(ctrl->device);
> 657    }
> 658
> 659    static struct blk_mq_tag_set *nvme_rdma_alloc_tagset(struct 
> nvme_ctrl *nctrl,
> 660            bool admin)
> (gdb)

Lets take this one step at a time, starting with this issue.

First, there is a reason why a simple create_ctrl fails, can we isolate
exactly which call fails? Was something else going on that might have
made the simple create_ctrl fail?

We don't see any "rdma_resolve_addr failed" or "failed to connect queue"
messages but we do see "creating I/O queues" which means that we either
failed at IO tagset allocation or initializing connect_q.

We have a missing error code assignment so can you try the following patch:
--
diff --git a/drivers/nvme/host/rdma.c b/drivers/nvme/host/rdma.c
index 58983000964b..98dd51e630bd 100644
--- a/drivers/nvme/host/rdma.c
+++ b/drivers/nvme/host/rdma.c
@@ -765,8 +765,10 @@ static int nvme_rdma_configure_admin_queue(struct 
nvme_rdma_ctrl *ctrl,

         if (new) {
                 ctrl->ctrl.admin_tagset = 
nvme_rdma_alloc_tagset(&ctrl->ctrl, true);
-               if (IS_ERR(ctrl->ctrl.admin_tagset))
+               if (IS_ERR(ctrl->ctrl.admin_tagset)) {
+                       error = PTR_ERR(ctrl->ctrl.admin_tagset);
                         goto out_free_queue;
+               }

                 ctrl->ctrl.admin_q = 
blk_mq_init_queue(&ctrl->admin_tag_set);
                 if (IS_ERR(ctrl->ctrl.admin_q)) {
@@ -846,8 +848,10 @@ static int nvme_rdma_configure_io_queues(struct 
nvme_rdma_ctrl *ctrl, bool new)

         if (new) {
                 ctrl->ctrl.tagset = nvme_rdma_alloc_tagset(&ctrl->ctrl, 
false);
-               if (IS_ERR(ctrl->ctrl.tagset))
+               if (IS_ERR(ctrl->ctrl.tagset)) {
+                       ret = PTR_ERR(ctrl->ctrl.tagset);
                         goto out_free_io_queues;
+               }

                 ctrl->ctrl.connect_q = blk_mq_init_queue(&ctrl->tag_set);
                 if (IS_ERR(ctrl->ctrl.connect_q)) {
--

Also, can you add the following debug messages to find out what failed?
--
diff --git a/drivers/nvme/host/rdma.c b/drivers/nvme/host/rdma.c
index 58983000964b..e46475100eea 100644
--- a/drivers/nvme/host/rdma.c
+++ b/drivers/nvme/host/rdma.c
@@ -676,6 +676,12 @@ static void nvme_rdma_free_tagset(struct nvme_ctrl 
*nctrl, bool admin)
         struct blk_mq_tag_set *set = admin ?
                         &ctrl->admin_tag_set : &ctrl->tag_set;

+       if (set == &ctrl->tag_set) {
+               pr_err("%s: freeing IO tagset\n", __func__);
+       } else {
+               pr_err("%s: freeing ADMIN tagset\n", __func__);
+       }
+
         blk_mq_free_tag_set(set);
         nvme_rdma_dev_put(ctrl->device);
  }
--
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: nvmeof rdma regression issue on 4.14.0-rc1 (or maybe mlx4?)

[Yi Zhang]

> Lets take this one step at a time, starting with this issue.
>
> First, there is a reason why a simple create_ctrl fails, can we isolate
> exactly which call fails? Was something else going on that might have
> made the simple create_ctrl fail?
>
> We don't see any "rdma_resolve_addr failed" or "failed to connect queue"
> messages but we do see "creating I/O queues" which means that we either
> failed at IO tagset allocation or initializing connect_q.
>
> We have a missing error code assignment so can you try the following 
> patch:

Hi Sagi
Sorry for the late response, here is the log with your patch, let me 
know if you need more test.

[  236.980239] nvme nvme0: new ctrl: NQN 
"nqn.2014-08.org.nvmexpress.discovery", addr 172.31.0.90:4420
[  237.131416] nvme_rdma: nvme_rdma_free_tagset: freeing ADMIN tagset
[  237.173614] nvme nvme0: creating 40 I/O queues.
[  237.688843] nvme nvme0: new ctrl: NQN "testnqn", addr 172.31.0.90:4420
[  243.384596] nvme nvme0: rescanning
[  243.527922] nvme nvme0: Reconnecting in 10 seconds...
[  254.010547] nvme nvme0: Connect rejected: status 8 (invalid service ID).
[  254.018052] nvme nvme0: rdma_resolve_addr wait failed (-104).
[  254.024482] nvme nvme0: Failed reconnect attempt 1
[  254.029844] nvme nvme0: Reconnecting in 10 seconds...
[  264.238223] nvme nvme0: Connect rejected: status 8 (invalid service ID).
[  264.245727] nvme nvme0: rdma_resolve_addr wait failed (-104).
[  264.252159] nvme nvme0: Failed reconnect attempt 2
[  264.257545] nvme nvme0: Reconnecting in 10 seconds...
[  274.477977] nvme nvme0: Connect rejected: status 8 (invalid service ID).
[  274.485470] nvme nvme0: rdma_resolve_addr wait failed (-104).
[  274.491905] nvme nvme0: Failed reconnect attempt 3
[  274.497287] nvme nvme0: Reconnecting in 10 seconds...
[  284.737691] nvme nvme0: creating 40 I/O queues.
[  284.745163] DMAR: ERROR: DMA PTE for vPFN 0xe0e9d already set (to 
100a1db001 not 1011432003)
[  284.754617] ------------[ cut here ]------------
[  284.759790] WARNING: CPU: 12 PID: 6 at 
drivers/iommu/intel-iommu.c:2305 __domain_mapping+0x367/0x380
[  284.770010] Modules linked in: nvme_rdma nvme_fabrics nvme_core 
sch_mqprio ebtable_filter ebtables ip6table_filter ip6_tables 
iptable_filter bridge 8021q garp mrp stp llc rpcrdma id
[  284.849181]  mgag200 i2c_algo_bit drm_kms_helper syscopyarea 
sysfillrect sysimgblt fb_sys_fops ttm drm tg3 ahci mlx4_core libahci ptp 
libata crc32c_intel i2c_core pps_core devlink d
[  284.870886] CPU: 12 PID: 6 Comm: kworker/u368:0 Not tainted 
4.14.0-rc4.10.02.sagi+ #30
[  284.879746] Hardware name: Dell Inc. PowerEdge R430/03XKDV, BIOS 
1.6.2 01/08/2016
[  284.888133] Workqueue: ib_addr process_one_req [ib_core]
[  284.894074] task: ffff88018cb50000 task.stack: ffffc9000009c000
[  284.900699] RIP: 0010:__domain_mapping+0x367/0x380
[  284.906051] RSP: 0018:ffffc9000009fa98 EFLAGS: 00010202
[  284.911899] RAX: 0000000000000004 RBX: 0000001011432003 RCX: 
0000000000000000
[  284.920643] RDX: 0000000000000000 RSI: ffff88103e78e058 RDI: 
ffff88103e78e058
[  284.929371] RBP: ffffc9000009faf8 R08: 0000000000000000 R09: 
0000000000000000
[  284.938088] R10: 00000000000002f7 R11: 0000000001011432 R12: 
ffff88201ab634e8
[  284.946814] R13: 0000000000000001 R14: 0000000000000001 R15: 
00000000000e0e9d
[  284.955533] FS:  0000000000000000(0000) GS:ffff88103e780000(0000) 
knlGS:0000000000000000
[  284.965318] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  284.972491] CR2: 00007f9348b7b000 CR3: 000000202f7e7005 CR4: 
00000000001606e0
[  284.981227] Call Trace:
[  284.984712]  __intel_map_single+0xeb/0x180
[  284.990059]  intel_alloc_coherent+0xb5/0x130
[  284.995584]  mlx4_buf_alloc+0xe5/0x1c0 [mlx4_core]
[  285.001690]  mlx4_ib_alloc_cq_buf.isra.9+0x38/0xd0 [mlx4_ib]
[  285.008754]  mlx4_ib_create_cq+0x223/0x450 [mlx4_ib]
[  285.015027]  ib_alloc_cq+0x49/0x170 [ib_core]
[  285.020634]  nvme_rdma_cm_handler+0x3a2/0x7ab [nvme_rdma]
[  285.027416]  ? cma_acquire_dev+0x1e3/0x3b0 [rdma_cm]
[  285.033689]  addr_handler+0xa4/0x1c0 [rdma_cm]
[  285.039417]  process_one_req+0x8d/0x120 [ib_core]
[  285.045408]  process_one_work+0x149/0x360
[  285.050620]  worker_thread+0x4d/0x3c0
[  285.055432]  kthread+0x109/0x140
[  285.059780]  ? rescuer_thread+0x380/0x380
[  285.064987]  ? kthread_park+0x60/0x60
[  285.069805]  ret_from_fork+0x25/0x30
[  285.074511] Code: 10 ab 81 4c 89 5d a0 4c 89 4d a8 e8 27 ac c0 ff 8b 
05 fe 39 87 00 4c 8b 4d a8 4c 8b 5d a0 85 c0 74 09 83 e8 01 89 05 e9 39 
87 00 <0f> ff e9 b8 fd ff ff e8 ed 8e b
[  285.097088] ---[ end trace 501470fecce57311 ]---
[  285.103299] DMAR: DRHD: handling fault status reg 2
[  285.103761] DMAR: ERROR: DMA PTE for vPFN 0xe0e9d already set (to 
100a1db001 not 100526b002)
[  285.103766] ------------[ cut here ]------------
[  285.103768] WARNING: CPU: 12 PID: 6 at 
drivers/iommu/intel-iommu.c:2305 __domain_mapping+0x367/0x380
[  285.103768] Modules linked in: nvme_rdma nvme_fabrics nvme_core 
sch_mqprio ebtable_filter ebtables ip6table_filter ip6_tables 
iptable_filter bridge 8021q garp mrp stp llc rpcrdma id
[  285.103785]  mgag200 i2c_algo_bit drm_kms_helper syscopyarea 
sysfillrect sysimgblt fb_sys_fops ttm drm tg3 ahci mlx4_core libahci ptp 
libata crc32c_intel i2c_core pps_core devlink d
[  285.103791] CPU: 12 PID: 6 Comm: kworker/u368:0 Tainted: G W       
4.14.0-rc4.10.02.sagi+ #30
[  285.103792] Hardware name: Dell Inc. PowerEdge R430/03XKDV, BIOS 
1.6.2 01/08/2016
[  285.103797] Workqueue: ib_addr process_one_req [ib_core]
[  285.103798] task: ffff88018cb50000 task.stack: ffffc9000009c000
[  285.103799] RIP: 0010:__domain_mapping+0x367/0x380
[  285.103800] RSP: 0018:ffffc9000009fc48 EFLAGS: 00010206
[  285.103800] RAX: 0000000000000003 RBX: 000000100526b002 RCX: 
0000000000000000
[  285.103801] RDX: 0000000000000000 RSI: 0000000000000002 RDI: 
0000000000000282
[  285.103801] RBP: ffffc9000009fca8 R08: 0000000000000050 R09: 
0000000000000000
[  285.103802] R10: 0000000000000860 R11: 000000000100526b R12: 
ffff88201ab634e8
[  285.103803] R13: 0000000000000001 R14: 0000000000000001 R15: 
00000000000e0e9d
[  285.103803] FS:  0000000000000000(0000) GS:ffff88103e780000(0000) 
knlGS:0000000000000000
[  285.103804] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  285.103805] CR2: 00007f9348b7b000 CR3: 000000202f7e7005 CR4: 
00000000001606e0
[  285.103805] Call Trace:
[  285.103808]  __intel_map_single+0xeb/0x180
[  285.103809]  intel_map_page+0x39/0x40
[  285.103811]  nvme_rdma_cm_handler+0x47b/0x7ab [nvme_rdma]
[  285.103813]  addr_handler+0xa4/0x1c0 [rdma_cm]
[  285.103817]  process_one_req+0x8d/0x120 [ib_core]
[  285.103818]  process_one_work+0x149/0x360
[  285.103819]  worker_thread+0x4d/0x3c0
[  285.103821]  kthread+0x109/0x140
[  285.103822]  ? rescuer_thread+0x380/0x380
[  285.103823]  ? kthread_park+0x60/0x60
[  285.103825]  ret_from_fork+0x25/0x30
[  285.103826] Code: 10 ab 81 4c 89 5d a0 4c 89 4d a8 e8 27 ac c0 ff 8b 
05 fe 39 87 00 4c 8b 4d a8 4c 8b 5d a0 85 c0 74 09 83 e8 01 89 05 e9 39 
87 00 <0f> ff e9 b8 fd ff ff e8 ed 8e b
[  285.103839] ---[ end trace 501470fecce57312 ]---
[  285.190580] DMAR: ERROR: DMA PTE for vPFN 0xe0e9d already set (to 
100a1db001 not 100d51b001)
[  285.190586] ------------[ cut here ]------------
[  285.190590] WARNING: CPU: 30 PID: 757 at 
drivers/iommu/intel-iommu.c:2305 __domain_mapping+0x367/0x380
[  285.190590] Modules linked in: nvme_rdma nvme_fabrics nvme_core 
sch_mqprio ebtable_filter ebtables ip6table_filter ip6_tables 
iptable_filter bridge 8021q garp mrp stp llc rpcrdma id
[  285.190609]  mgag200 i2c_algo_bit drm_kms_helper syscopyarea 
sysfillrect sysimgblt fb_sys_fops ttm drm tg3 ahci mlx4_core libahci ptp 
libata crc32c_intel i2c_core pps_core devlink d
[  285.190616] CPU: 30 PID: 757 Comm: kworker/u369:7 Tainted: G        
W       4.14.0-rc4.10.02.sagi+ #30
[  285.190617] Hardware name: Dell Inc. PowerEdge R430/03XKDV, BIOS 
1.6.2 01/08/2016
[  285.190619] Workqueue: nvme-wq nvme_rdma_reconnect_ctrl_work [nvme_rdma]
[  285.190620] task: ffff881032bd8000 task.stack: ffffc9000851c000
[  285.190622] RIP: 0010:__domain_mapping+0x367/0x380
[  285.190622] RSP: 0018:ffffc9000851fc60 EFLAGS: 00010202
[  285.190623] RAX: 0000000000000002 RBX: 000000100d51b001 RCX: 
0000000000000000
[  285.190624] RDX: 0000000000000000 RSI: 0000000000000002 RDI: 
0000000000000292
[  285.190624] RBP: ffffc9000851fcc0 R08: 0000000000000050 R09: 
0000000000000000
[  285.190625] R10: 0000000000000881 R11: 000000000100d51b R12: 
ffff88201ab634e8
[  285.190625] R13: 0000000000000001 R14: 0000000000000001 R15: 
00000000000e0e9d
[  285.190626] FS:  0000000000000000(0000) GS:ffff88103e9c0000(0000) 
knlGS:0000000000000000
[  285.190626] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  285.190627] CR2: 00007f4a9e61fb70 CR3: 0000001011bdd001 CR4: 
00000000001606e0
[  285.190628] Call Trace:
[  285.190630]  __intel_map_single+0xeb/0x180
[  285.190632]  intel_map_page+0x39/0x40
[  285.190636]  mlx4_ib_alloc_mr+0x141/0x220 [mlx4_ib]
[  285.190642]  ib_alloc_mr+0x26/0x50 [ib_core]
[  285.190644]  nvme_rdma_reinit_request+0x3a/0x70 [nvme_rdma]
[  285.190645]  ? nvme_rdma_free_ctrl+0xb0/0xb0 [nvme_rdma]
[  285.190647]  blk_mq_reinit_tagset+0x5c/0x90
[  285.190649]  nvme_rdma_configure_io_queues+0x204/0x280 [nvme_rdma]
[  285.190650]  nvme_rdma_reconnect_ctrl_work+0x5b/0xe0 [nvme_rdma]
[  285.190652]  process_one_work+0x149/0x360
[  285.190653]  worker_thread+0x4d/0x3c0
[  285.190655]  kthread+0x109/0x140
[  285.190656]  ? rescuer_thread+0x380/0x380
[  285.190657]  ? kthread_park+0x60/0x60
[  285.190660]  ret_from_fork+0x25/0x30
[  285.190660] Code: 10 ab 81 4c 89 5d a0 4c 89 4d a8 e8 27 ac c0 ff 8b 
05 fe 39 87 00 4c 8b 4d a8 4c 8b 5d a0 85 c0 74 09 83 e8 01 89 05 e9 39 
87 00 <0f> ff e9 b8 fd ff ff e8 ed 8e b
[  285.190673] ---[ end trace 501470fecce57313 ]---
[  285.819126] DMAR: [DMA Write] Request device [04:00.0] fault addr 
e0e9d000 [fault reason 05] PTE Write access is not set
[  286.035086] nvme nvme0: Successfully reconnected
[  286.035143] nvme nvme0: MEMREG for CQE 0xffff88100a1e6f80 failed with 
status memory management operation error (6)
[  286.187508] nvme nvme0: Reconnecting in 10 seconds...
[  296.270043] nvme nvme0: creating 40 I/O queues.
[  296.336505] DMAR: ERROR: DMA PTE for vPFN 0xe0e9d already set (to 
1012082002 not 1012082002)
[  296.346704] ------------[ cut here ]------------
[  296.352618] WARNING: CPU: 32 PID: 815 at 
drivers/iommu/intel-iommu.c:2305 __domain_mapping+0x367/0x380
[  296.363755] Modules linked in: nvme_rdma nvme_fabrics nvme_core 
sch_mqprio ebtable_filter ebtables ip6table_filter ip6_tables 
iptable_filter bridge 8021q garp mrp stp llc rpcrdma id
[  296.447983]  mgag200 i2c_algo_bit drm_kms_helper syscopyarea 
sysfillrect sysimgblt fb_sys_fops ttm drm tg3 ahci mlx4_core libahci ptp 
libata crc32c_intel i2c_core pps_core devlink d
[  296.471170] CPU: 32 PID: 815 Comm: kworker/u368:1 Tainted: G        
W       4.14.0-rc4.10.02.sagi+ #30
[  296.482344] Hardware name: Dell Inc. PowerEdge R430/03XKDV, BIOS 
1.6.2 01/08/2016
[  296.491493] Workqueue: ib_addr process_one_req [ib_core]
[  296.498221] task: ffff8810373d2e80 task.stack: ffffc9000849c000
[  296.505613] RIP: 0010:__domain_mapping+0x367/0x380
[  296.511751] RSP: 0018:ffffc9000849fc48 EFLAGS: 00010202
[  296.518380] RAX: 0000000000000001 RBX: 0000001012082002 RCX: 
0000000000000000
[  296.527146] RDX: 0000000000000000 RSI: ffff88103ea0e058 RDI: 
ffff88103ea0e058
[  296.535918] RBP: ffffc9000849fca8 R08: 0000000000000000 R09: 
0000000000000000
[  296.544691] R10: 00000000000003ff R11: 0000000001012082 R12: 
ffff88201ab634e8
[  296.553462] R13: 0000000000000001 R14: 0000000000000001 R15: 
00000000000e0e9d
[  296.562240] FS:  0000000000000000(0000) GS:ffff88103ea00000(0000) 
knlGS:0000000000000000
[  296.572078] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  296.579292] CR2: 00007fdc7ae90000 CR3: 0000001034df0002 CR4: 
00000000001606e0
[  296.588070] Call Trace:
[  296.591620]  __intel_map_single+0xeb/0x180
[  296.597012]  intel_map_page+0x39/0x40
[  296.601912]  nvme_rdma_cm_handler+0x47b/0x7ab [nvme_rdma]
[  296.608760]  addr_handler+0xa4/0x1c0 [rdma_cm]
[  296.614540]  process_one_req+0x8d/0x120 [ib_core]
[  296.620602]  process_one_work+0x149/0x360
[  296.625883]  worker_thread+0x4d/0x3c0
[  296.630779]  kthread+0x109/0x140
[  296.635177]  ? rescuer_thread+0x380/0x380
[  296.640453]  ? kthread_park+0x60/0x60
[  296.645341]  ret_from_fork+0x25/0x30
[  296.650132] Code: 10 ab 81 4c 89 5d a0 4c 89 4d a8 e8 27 ac c0 ff 8b 
05 fe 39 87 00 4c 8b 4d a8 4c 8b 5d a0 85 c0 74 09 83 e8 01 89 05 e9 39 
87 00 <0f> ff e9 b8 fd ff ff e8 ed 8e b
[  296.672876] ---[ end trace 501470fecce57314 ]---
[  297.231475] nvme nvme0: Successfully reconnected
[  297.231523] nvme nvme0: MEMREG for CQE 0xffff88100a1e6f80 failed with 
status memory management operation error (6)
[  297.378026] nvme nvme0: Reconnecting in 10 seconds...
[  307.765849] ------------[ cut here ]------------
[  307.771793] kernel BUG at drivers/iommu/iova.c:836!
[  307.778007] invalid opcode: 0000 [#1] SMP
[  307.783257] Modules linked in: nvme_rdma nvme_fabrics nvme_core 
sch_mqprio ebtable_filter ebtables ip6table_filter ip6_tables 
iptable_filter bridge 8021q garp mrp stp llc rpcrdma id
[  307.867936]  mgag200 i2c_algo_bit drm_kms_helper syscopyarea 
sysfillrect sysimgblt fb_sys_fops ttm drm tg3 ahci mlx4_core libahci ptp 
libata crc32c_intel i2c_core pps_core devlink d
[  307.891229] CPU: 30 PID: 757 Comm: kworker/u369:7 Tainted: G        
W       4.14.0-rc4.10.02.sagi+ #30
[  307.902427] Hardware name: Dell Inc. PowerEdge R430/03XKDV, BIOS 
1.6.2 01/08/2016
[  307.911579] Workqueue: nvme-wq nvme_rdma_reconnect_ctrl_work [nvme_rdma]
[  307.919847] task: ffff881032bd8000 task.stack: ffffc9000851c000
[  307.927232] RIP: 0010:iova_magazine_free_pfns+0x76/0x80
[  307.933827] RSP: 0018:ffffc9000851fc58 EFLAGS: 00010046
[  307.940415] RAX: 0000000000000000 RBX: 0000000000000017 RCX: 
0000000100400038
[  307.949132] RDX: 00000000000e0e9c RSI: 00000000000e0e9d RDI: 
ffff8810372c4e58
[  307.957857] RBP: ffffc9000851fc78 R08: ffff88202fd41a40 R09: 
0000000100400038
[  307.966586] R10: 000000002fd41c01 R11: ffff88202fd41a40 R12: 
ffff880ffda45000
[  307.975308] R13: ffff8810372c4e58 R14: 0000000000000046 R15: 
ffff880ffda45000
[  307.984032] FS:  0000000000000000(0000) GS:ffff88103e9c0000(0000) 
knlGS:0000000000000000
[  307.993834] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  308.001012] CR2: 00007f4a9e61fb70 CR3: 0000001011bdd001 CR4: 
00000000001606e0
[  308.009749] Call Trace:
[  308.013243]  free_iova_fast+0xc9/0x1d0
[  308.018198]  fq_ring_free+0x78/0xc0
[  308.022845]  queue_iova+0x111/0x130
[  308.027478]  intel_unmap+0xa5/0x130
[  308.032118]  intel_unmap_page+0xe/0x10
[  308.037044]  nvme_rdma_free_ring.constprop.61+0x4a/0x90 [nvme_rdma]
[  308.044789]  nvme_rdma_destroy_queue_ib+0x3a/0x50 [nvme_rdma]
[  308.051951]  nvme_rdma_free_queue+0x1a/0x30 [nvme_rdma]
[  308.058529]  nvme_rdma_destroy_io_queues+0x80/0xb0 [nvme_rdma]
[  308.065786]  nvme_rdma_reconnect_ctrl_work+0x30/0xe0 [nvme_rdma]
[  308.073240]  process_one_work+0x149/0x360
[  308.078470]  worker_thread+0x4d/0x3c0
[  308.083299]  kthread+0x109/0x140
[  308.087652]  ? rescuer_thread+0x380/0x380
[  308.092878]  ? kthread_park+0x60/0x60
[  308.097699]  ret_from_fork+0x25/0x30
[  308.102407] Code: ef e8 ef fe ff ff 48 63 d3 49 3b 14 24 72 d7 4c 89 
f6 4c 89 ef e8 3b 3d 2b 00 49 c7 04 24 00 00 00 00 5b 41 5c 41 5d 41 5e 
5d c3 <0f> 0b 0f 1f 84 00 00 00 00 00 0
[  308.124980] RIP: iova_magazine_free_pfns+0x76/0x80 RSP: ffffc9000851fc58
[  308.133265] ---[ end trace 501470fecce57315 ]---
[  308.142811] Kernel panic - not syncing: Fatal exception
[  308.149425] Kernel Offset: disabled
[  308.158679] ---[ end Kernel panic - not syncing: Fatal exception
[  308.166135] ------------[ cut here ]------------
[  308.172010] WARNING: CPU: 30 PID: 757 at kernel/sched/core.c:1179 
set_task_cpu+0x191/0x1a0
[  308.181959] Modules linked in: nvme_rdma nvme_fabrics nvme_core 
sch_mqprio ebtable_filter ebtables ip6table_filter ip6_tables 
iptable_filter bridge 8021q garp mrp stp llc rpcrdma id
[  308.266001]  mgag200 i2c_algo_bit drm_kms_helper syscopyarea 
sysfillrect sysimgblt fb_sys_fops ttm drm tg3 ahci mlx4_core libahci ptp 
libata crc32c_intel i2c_core pps_core devlink d
[  308.289112] CPU: 30 PID: 757 Comm: kworker/u369:7 Tainted: G D 
W       4.14.0-rc4.10.02.sagi+ #30
[  308.300236] Hardware name: Dell Inc. PowerEdge R430/03XKDV, BIOS 
1.6.2 01/08/2016
[  308.309330] Workqueue: nvme-wq nvme_rdma_reconnect_ctrl_work [nvme_rdma]
[  308.317552] task: ffff881032bd8000 task.stack: ffffc9000851c000
[  308.324905] RIP: 0010:set_task_cpu+0x191/0x1a0
[  308.330621] RSP: 0018:ffff88103e9c3c38 EFLAGS: 00010046
[  308.337191] RAX: 0000000000000200 RBX: ffff881034cc5d00 RCX: 
0000000000000001
[  308.345896] RDX: 0000000000000001 RSI: 0000000000000000 RDI: 
ffff881034cc5d00
[  308.354604] RBP: ffff88103e9c3c58 R08: 0000000000000000 R09: 
0000000000000000
[  308.363314] R10: 00000000000003ff R11: 0000000000000001 R12: 
ffff881034cc68ac
[  308.372018] R13: 0000000000000000 R14: 0000000000000000 R15: 
000000000001b900
[  308.380720] FS:  0000000000000000(0000) GS:ffff88103e9c0000(0000) 
knlGS:0000000000000000
[  308.390495] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  308.397632] CR2: 00007f4a9e61fb70 CR3: 0000001011bdd001 CR4: 
00000000001606e0
[  308.406347] Call Trace:
[  308.409809]  <IRQ>
[  308.412789]  try_to_wake_up+0x166/0x470
[  308.417816]  default_wake_function+0x12/0x20
[  308.423325]  __wake_up_common+0x8a/0x160
[  308.428438]  __wake_up_locked+0x16/0x20
[  308.433450]  ep_poll_callback+0xd0/0x300
[  308.438562]  __wake_up_common+0x8a/0x160
[  308.443664]  __wake_up_common_lock+0x7e/0xc0
[  308.449150]  __wake_up+0x13/0x20
[  308.453455]  wake_up_klogd_work_func+0x40/0x60
[  308.459124]  irq_work_run_list+0x4d/0x70
[  308.464217]  ? tick_sched_do_timer+0x70/0x70
[  308.469701]  irq_work_tick+0x40/0x50
[  308.474409]  update_process_times+0x42/0x60
[  308.479800]  tick_sched_handle+0x2d/0x60
[  308.484881]  tick_sched_timer+0x39/0x70
[  308.489852]  __hrtimer_run_queues+0xe5/0x230
[  308.495291]  hrtimer_interrupt+0xa8/0x1a0
[  308.500466]  smp_apic_timer_interrupt+0x5f/0x130
[  308.506320]  apic_timer_interrupt+0x9d/0xb0
[  308.511661]  </IRQ>
[  308.514658] RIP: 0010:panic+0x1fd/0x245
[  308.519603] RSP: 0018:ffffc9000851f9a8 EFLAGS: 00000246 ORIG_RAX: 
ffffffffffffff10
[  308.528740] RAX: 0000000000000034 RBX: 0000000000000000 RCX: 
0000000000000006
[  308.537352] RDX: 0000000000000000 RSI: 0000000000000086 RDI: 
ffff88103e9ce050
[  308.545943] RBP: ffffc9000851fa18 R08: 0000000000000000 R09: 
00000000000008f9
[  308.554511] R10: 00000000000003ff R11: 0000000000000001 R12: 
ffffffff81a3f340
[  308.563067] R13: 0000000000000000 R14: 0000000000000000 R15: 
0000000000000004
[  308.571619]  ? panic+0x1f6/0x245
[  308.575773]  oops_end+0xb8/0xd0
[  308.579822]  die+0x42/0x50
[  308.583348]  do_trap+0x16a/0x180
[  308.587442]  do_error_trap+0x89/0x110
[  308.592007]  ? iova_magazine_free_pfns+0x76/0x80
[  308.597624]  ? rem_res_range+0x1ae/0x200 [mlx4_core]
[  308.603625]  do_invalid_op+0x20/0x30
[  308.608063]  invalid_op+0x18/0x20
[  308.612198] RIP: 0010:iova_magazine_free_pfns+0x76/0x80
[  308.618473] RSP: 0018:ffffc9000851fc58 EFLAGS: 00010046
[  308.624735] RAX: 0000000000000000 RBX: 0000000000000017 RCX: 
0000000100400038
[  308.633146] RDX: 00000000000e0e9c RSI: 00000000000e0e9d RDI: 
ffff8810372c4e58
[  308.641555] RBP: ffffc9000851fc78 R08: ffff88202fd41a40 R09: 
0000000100400038
[  308.649969] R10: 000000002fd41c01 R11: ffff88202fd41a40 R12: 
ffff880ffda45000
[  308.658387] R13: ffff8810372c4e58 R14: 0000000000000046 R15: 
ffff880ffda45000
[  308.666799]  ? iova_magazine_free_pfns+0x3e/0x80
[  308.672399]  free_iova_fast+0xc9/0x1d0
[  308.677046]  fq_ring_free+0x78/0xc0
[  308.681370]  queue_iova+0x111/0x130
[  308.685691]  intel_unmap+0xa5/0x130
[  308.690011]  intel_unmap_page+0xe/0x10
[  308.694626]  nvme_rdma_free_ring.constprop.61+0x4a/0x90 [nvme_rdma]
[  308.702056]  nvme_rdma_destroy_queue_ib+0x3a/0x50 [nvme_rdma]
[  308.708909]  nvme_rdma_free_queue+0x1a/0x30 [nvme_rdma]
[  308.715182]  nvme_rdma_destroy_io_queues+0x80/0xb0 [nvme_rdma]
[  308.722148]  nvme_rdma_reconnect_ctrl_work+0x30/0xe0 [nvme_rdma]
[  308.729311]  process_one_work+0x149/0x360
[  308.734242]  worker_thread+0x4d/0x3c0
[  308.738783]  kthread+0x109/0x140
[  308.742840]  ? rescuer_thread+0x380/0x380
[  308.747774]  ? kthread_park+0x60/0x60
[  308.752309]  ret_from_fork+0x25/0x30
[  308.756747] Code: ff 80 8b ac 08 00 00 04 e9 23 ff ff ff 0f ff e9 bf 
fe ff ff f7 83 84 00 00 00 fd ff ff ff 0f 84 c9 fe ff ff 0f ff e9 c2 fe 
ff ff <0f> ff e9 d1 fe ff ff 0f 1f 84 0
[  308.778800] ---[ end trace 501470fecce57316 ]---
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: nvmeof rdma regression issue on 4.14.0-rc1 (or maybe mlx4?)

[Sagi Grimberg]

>> Lets take this one step at a time, starting with this issue.
>>
>> First, there is a reason why a simple create_ctrl fails, can we isolate
>> exactly which call fails? Was something else going on that might have
>> made the simple create_ctrl fail?
>>
>> We don't see any "rdma_resolve_addr failed" or "failed to connect queue"
>> messages but we do see "creating I/O queues" which means that we either
>> failed at IO tagset allocation or initializing connect_q.
>>
>> We have a missing error code assignment so can you try the following 
>> patch:
> 
> Hi Sagi
> Sorry for the late response, here is the log with your patch, let me 
> know if you need more test.

Hi Yi,

I was referring to the bug you reported on a simple create_ctrl failed:
https://pastebin.com/7z0XSGSd

Does it reproduce?
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: nvmeof rdma regression issue on 4.14.0-rc1 (or maybe mlx4?)

[Yi Zhang]

On 10/16/2017 06:21 PM, Sagi Grimberg wrote:
>>> Lets take this one step at a time, starting with this issue.
>>>
>>> First, there is a reason why a simple create_ctrl fails, can we isolate
>>> exactly which call fails? Was something else going on that might have
>>> made the simple create_ctrl fail?
>>>
>>> We don't see any "rdma_resolve_addr failed" or "failed to connect 
>>> queue"
>>> messages but we do see "creating I/O queues" which means that we either
>>> failed at IO tagset allocation or initializing connect_q.
>>>
>>> We have a missing error code assignment so can you try the following 
>>> patch:
>>
>> Hi Sagi
>> Sorry for the late response, here is the log with your patch, let me 
>> know if you need more test.
>
> Hi Yi,
>
> I was referring to the bug you reported on a simple create_ctrl failed:
> https://pastebin.com/7z0XSGSd
>
> Does it reproduce?
>
yes, this issue was reproduced during "git bisect" with below patch

commit 31fdf18401703df246e20fc0382ed3594e6cf8d8
Author: Sagi Grimberg <sagi-NQWnxTmZq1alnMjI0IkVqw@public.gmane.org>
Date:   Mon Aug 28 21:40:06 2017 +0200

     nvme-rdma: reuse configure/destroy_admin_queue

     No need to open-code it.

     Signed-off-by: Sagi Grimberg <sagi-NQWnxTmZq1alnMjI0IkVqw@public.gmane.org>
     Signed-off-by: Christoph Hellwig <hch-jcswGhMUV9g@public.gmane.org>

> _______________________________________________
> Linux-nvme mailing list
> Linux-nvme-IAPFreCvJWM7uuMidbF8XUB+6BGkLq7r@public.gmane.org
> http://lists.infradead.org/mailman/listinfo/linux-nvme

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: nvmeof rdma regression issue on 4.14.0-rc1 (or maybe mlx4?)

[Sagi Grimberg]

>> Hi Yi,
>>
>> I was referring to the bug you reported on a simple create_ctrl failed:
>> https://pastebin.com/7z0XSGSd
>>
>> Does it reproduce?
>>
> yes, this issue was reproduced during "git bisect" with below patch

OK, if this does not reproduce with the latest code, lets put it aside
for now.

So as for the error you see, can you please try the following patch?
--
diff --git a/drivers/nvme/host/rdma.c b/drivers/nvme/host/rdma.c
index 405895b1dff2..916658e010ff 100644
--- a/drivers/nvme/host/rdma.c
+++ b/drivers/nvme/host/rdma.c
@@ -572,6 +572,11 @@ static void nvme_rdma_free_queue(struct 
nvme_rdma_queue *queue)
         if (!test_and_clear_bit(NVME_RDMA_Q_ALLOCATED, &queue->flags))
                 return;

+       if(nvme_rdma_queue_idx(queue) == 0)
+               nvme_rdma_free_qe(queue->device->dev,
+                       &queue->ctrl->async_event_sqe,
+                       sizeof(struct nvme_command), DMA_TO_DEVICE);
+
         nvme_rdma_destroy_queue_ib(queue);
         rdma_destroy_id(queue->cm_id);
  }
@@ -739,8 +744,6 @@ static struct blk_mq_tag_set 
*nvme_rdma_alloc_tagset(struct nvme_ctrl *nctrl,
  static void nvme_rdma_destroy_admin_queue(struct nvme_rdma_ctrl *ctrl,
                 bool remove)
  {
-       nvme_rdma_free_qe(ctrl->queues[0].device->dev, 
&ctrl->async_event_sqe,
-                       sizeof(struct nvme_command), DMA_TO_DEVICE);
         nvme_rdma_stop_queue(&ctrl->queues[0]);
         if (remove) {
                 blk_cleanup_queue(ctrl->ctrl.admin_q);
--
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: nvmeof rdma regression issue on 4.14.0-rc1 (or maybe mlx4?)

[Yi Zhang]

On 10/19/2017 02:55 PM, Sagi Grimberg wrote:
>
>>> Hi Yi,
>>>
>>> I was referring to the bug you reported on a simple create_ctrl failed:
>>> https://pastebin.com/7z0XSGSd
>>>
>>> Does it reproduce?
>>>
>> yes, this issue was reproduced during "git bisect" with below patch
>
> OK, if this does not reproduce with the latest code, lets put it aside
> for now.
>
> So as for the error you see, can you please try the following patch?
Hi Sagi
With this patch, no such error log found on host side, but I found there 
is no nvme0n1 device node even get "nvme nvme0: Successfully 
reconnected" on host.

Host side:
[   98.181089] nvme nvme0: new ctrl: NQN 
"nqn.2014-08.org.nvmexpress.discovery", addr 172.31.0.90:4420
[   98.329464] nvme nvme0: creating 40 I/O queues.
[   98.835409] nvme nvme0: new ctrl: NQN "testnqn", addr 172.31.0.90:4420
[  107.873586] nvme nvme0: Reconnecting in 10 seconds...
[  118.505937] nvme nvme0: Connect rejected: status 8 (invalid service ID).
[  118.513443] nvme nvme0: rdma_resolve_addr wait failed (-104).
[  118.519875] nvme nvme0: Failed reconnect attempt 1
[  118.525241] nvme nvme0: Reconnecting in 10 seconds...
[  128.733311] nvme nvme0: Connect rejected: status 8 (invalid service ID).
[  128.740812] nvme nvme0: rdma_resolve_addr wait failed (-104).
[  128.747247] nvme nvme0: Failed reconnect attempt 2
[  128.752609] nvme nvme0: Reconnecting in 10 seconds...
[  138.973404] nvme nvme0: Connect rejected: status 8 (invalid service ID).
[  138.980904] nvme nvme0: rdma_resolve_addr wait failed (-104).
[  138.987329] nvme nvme0: Failed reconnect attempt 3
[  138.992691] nvme nvme0: Reconnecting in 10 seconds...
[  149.232610] nvme nvme0: creating 40 I/O queues.
[  149.831443] nvme nvme0: Successfully reconnected
[  149.831519] nvme nvme0: identifiers changed for nsid 1
[root@rdma-virt-01 linux ((dafb1b2...))]$ lsblk
NAME                           MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
sda                              8:0    0 465.8G  0 disk
├─sda2                           8:2    0 464.8G  0 part
│ ├─rhelaa_rdma--virt--01-swap 253:1    0     4G  0 lvm  [SWAP]
│ ├─rhelaa_rdma--virt--01-home 253:2    0 410.8G  0 lvm  /home
│ └─rhelaa_rdma--virt--01-root 253:0    0    50G  0 lvm  /
└─sda1                           8:1    0     1G  0 part /boot

> -- 
> diff --git a/drivers/nvme/host/rdma.c b/drivers/nvme/host/rdma.c
> index 405895b1dff2..916658e010ff 100644
> --- a/drivers/nvme/host/rdma.c
> +++ b/drivers/nvme/host/rdma.c
> @@ -572,6 +572,11 @@ static void nvme_rdma_free_queue(struct 
> nvme_rdma_queue *queue)
>         if (!test_and_clear_bit(NVME_RDMA_Q_ALLOCATED, &queue->flags))
>                 return;
>
> +       if(nvme_rdma_queue_idx(queue) == 0)
> +               nvme_rdma_free_qe(queue->device->dev,
> +                       &queue->ctrl->async_event_sqe,
> +                       sizeof(struct nvme_command), DMA_TO_DEVICE);
> +
>         nvme_rdma_destroy_queue_ib(queue);
>         rdma_destroy_id(queue->cm_id);
>  }
> @@ -739,8 +744,6 @@ static struct blk_mq_tag_set 
> *nvme_rdma_alloc_tagset(struct nvme_ctrl *nctrl,
>  static void nvme_rdma_destroy_admin_queue(struct nvme_rdma_ctrl *ctrl,
>                 bool remove)
>  {
> -       nvme_rdma_free_qe(ctrl->queues[0].device->dev, 
> &ctrl->async_event_sqe,
> -                       sizeof(struct nvme_command), DMA_TO_DEVICE);
>         nvme_rdma_stop_queue(&ctrl->queues[0]);
>         if (remove) {
>                 blk_cleanup_queue(ctrl->ctrl.admin_q);
> -- 
>
> _______________________________________________
> Linux-nvme mailing list
> Linux-nvme-IAPFreCvJWM7uuMidbF8XUB+6BGkLq7r@public.gmane.org
> http://lists.infradead.org/mailman/listinfo/linux-nvme

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: nvmeof rdma regression issue on 4.14.0-rc1 (or maybe mlx4?)

[Sagi Grimberg]

> Hi Sagi
> With this patch, no such error log found on host side,


Awsome. that's the culprit...

> but I found there 
> is no nvme0n1 device node even get "nvme nvme0: Successfully 
> reconnected" on host.

That is expected because you did not persist a namespace UUID
which caused the kernel to generate a random one. That confused
the host as it got the same namespace ID with a different UUID.

Can you please set a uuid when you rerun the test?
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: nvmeof rdma regression issue on 4.14.0-rc1 (or maybe mlx4?)

[Yi Zhang]

On 10/19/2017 05:44 PM, Sagi Grimberg wrote:
>
>> Hi Sagi
>> With this patch, no such error log found on host side,
>
>
> Awsome. that's the culprit...
>
>> but I found there is no nvme0n1 device node even get "nvme nvme0: 
>> Successfully reconnected" on host.
>
> That is expected because you did not persist a namespace UUID
> which caused the kernel to generate a random one. That confused
> the host as it got the same namespace ID with a different UUID.
>
> Can you please set a uuid when you rerun the test?
>
I tried add uuid filed on rdma.json, it works well now.

[root@rdma-virt-00 ~]$ cat /etc/rdma.json
{
   "hosts": [
     {
       "nqn": "hostnqn"
     }
   ],
   "ports": [
     {
       "addr": {
         "adrfam": "ipv4",
         "traddr": "172.31.0.90",
         "treq": "not specified",
         "trsvcid": "4420",
         "trtype": "rdma"
       },
       "portid": 2,
       "referrals": [],
       "subsystems": [
         "testnqn"
       ]
     }
   ],
   "subsystems": [
     {
       "allowed_hosts": [],
       "attr": {
         "allow_any_host": "1"
       },
       "namespaces": [
         {
           "device": {
             "nguid": "ef90689c-6c46-d44c-89c1-4067801309a8",
             "path": "/dev/nullb0",
             "uuid": "00000000-0000-0000-0000-000000000001"
           },
           "enable": 1,
           "nsid": 1
         }
       ],
       "nqn": "testnqn"
     }
   ]
}

> _______________________________________________
> Linux-nvme mailing list
> Linux-nvme-IAPFreCvJWM7uuMidbF8XUB+6BGkLq7r@public.gmane.org
> http://lists.infradead.org/mailman/listinfo/linux-nvme

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html