Re: Unexpected issues with 2 NVME initiators using the same target

Re: Unexpected issues with 2 NVME initiators using the same target

[Sagi Grimberg]

> I am using 2 initiators + 1 target using nvmet with 1 subsystem and 4
> backend
> devices. Kernel is 4.9.6, NVME/rdma drivers are all from the vanilla
> kernel, I
> had a probelm connecting the NVME using the OFED drivers, so I removed
> all the
> mlx_compat and everything which depends on it.

Would it be possible to test with latest upstream kernel?

>
> When I perform simultaneous writes (non direct fio) from both of the
> initiators
> to the same device (overlapping areas), I get NVMEf disconnect followed
> by "dump
> error cqe", successful reconnect, and then on one of the servers I get a
> WARN_ON. After this the server gets stuck and I have to power cycle it
> to get it
> back up...

The error cqes seem to indicate that a memory registration operation
failed which escalated to something worse.

I noticed some issues before with CX4 having problems with memory
registration in the presence of network retransmissions (due to
network congestion).

I notified Mellanox folks on that too, CC'ing Linux-rdma for some
more attention.

After that, I see that ib_modify_qp failed which I've never seen
before (might indicate the the device is in bad shape), and the WARN_ON
is really weird given that nvme-rdma never uses IB_POLL_DIRECT.

> Here are the printouts from the server that got stuck:
>
> Feb  6 09:20:13 kblock01-knode02 kernel: [59976.204216]
> mlx5_0:dump_cqe:262:(pid 0): dump error cqe
> Feb  6 09:20:13 kblock01-knode02 kernel: [59976.204219] 00000000
> 00000000 00000000 00000000
> Feb  6 09:20:13 kblock01-knode02 kernel: [59976.204220] 00000000
> 00000000 00000000 00000000
> Feb  6 09:20:13 kblock01-knode02 kernel: [59976.204220] 00000000
> 00000000 00000000 00000000
> Feb  6 09:20:13 kblock01-knode02 kernel: [59976.204221] 00000000
> 08007806 25000129 015557d0
> Feb  6 09:20:13 kblock01-knode02 kernel: [59976.204234] nvme nvme0:
> MEMREG for CQE 0xffff96ddd747a638 failed with status
> memory management operation error (6)
> Feb  6 09:20:13 kblock01-knode02 kernel: [59976.204375] nvme nvme0:
> reconnecting in 10 seconds
> Feb  6 09:20:13 kblock01-knode02 kernel: [59976.205512]
> mlx5_0:dump_cqe:262:(pid 0): dump error cqe
> Feb  6 09:20:13 kblock01-knode02 kernel: [59976.205514] 00000000
> 00000000 00000000 00000000
> Feb  6 09:20:13 kblock01-knode02 kernel: [59976.205515] 00000000
> 00000000 00000000 00000000
> Feb  6 09:20:13 kblock01-knode02 kernel: [59976.205515] 00000000
> 00000000 00000000 00000000
> Feb  6 09:20:13 kblock01-knode02 kernel: [59976.205516] 00000000
> 08007806 25000126 00692bd0
> Feb  6 09:20:23 kblock01-knode02 kernel: [59986.452887] nvme nvme0:
> Successfully reconnected
> Feb  6 09:20:24 kblock01-knode02 kernel: [59986.682887]
> mlx5_0:dump_cqe:262:(pid 0): dump error cqe
> Feb  6 09:20:24 kblock01-knode02 kernel: [59986.682890] 00000000
> 00000000 00000000 00000000
> Feb  6 09:20:24 kblock01-knode02 kernel: [59986.682891] 00000000
> 00000000 00000000 00000000
> Feb  6 09:20:24 kblock01-knode02 kernel: [59986.682892] 00000000
> 00000000 00000000 00000000
> Feb  6 09:20:24 kblock01-knode02 kernel: [59986.682892] 00000000
> 08007806 25000158 04cdd7d0
> ...
> Feb  6 09:20:24 kblock01-knode02 kernel: [59986.687737]
> mlx5_0:dump_cqe:262:(pid 0): dump error cqe
> Feb  6 09:20:24 kblock01-knode02 kernel: [59986.687739] 00000000
> 00000000 00000000 00000000
> Feb  6 09:20:24 kblock01-knode02 kernel: [59986.687740] 00000000
> 00000000 00000000 00000000
> Feb  6 09:20:24 kblock01-knode02 kernel: [59986.687740] 00000000
> 00000000 00000000 00000000
> Feb  6 09:20:24 kblock01-knode02 kernel: [59986.687741] 00000000
> 93005204 00000155 00a385e0
> Feb  6 09:20:34 kblock01-knode02 kernel: [59997.389290] nvme nvme0:
> Successfully reconnected
> Feb  6 09:21:19 kblock01-knode02 rsyslogd: -- MARK --
> Feb  6 09:21:38 kblock01-knode02 kernel: [60060.927832]
> mlx5_0:dump_cqe:262:(pid 0): dump error cqe
> Feb  6 09:21:38 kblock01-knode02 kernel: [60060.927835] 00000000
> 00000000 00000000 00000000
> Feb  6 09:21:38 kblock01-knode02 kernel: [60060.927836] 00000000
> 00000000 00000000 00000000
> Feb  6 09:21:38 kblock01-knode02 kernel: [60060.927837] 00000000
> 00000000 00000000 00000000
> Feb  6 09:21:38 kblock01-knode02 kernel: [60060.927837] 00000000
> 93005204 00000167 b44e76e0
> Feb  6 09:21:38 kblock01-knode02 kernel: [60060.927846] nvme nvme0: RECV
> for CQE 0xffff96fe64f18750 failed with status local protection error (4)
> Feb  6 09:21:38 kblock01-knode02 kernel: [60060.928200] nvme nvme0:
> reconnecting in 10 seconds
> ...
> Feb  6 09:23:54 kblock01-knode02 kernel: [60196.736182] mlx5_core
> 0000:04:00.0: wait_func:879:(pid 22709): 2ERR_QP(0x507) timeout. Will
> cause a leak of a command resource
> Feb  6 09:23:54 kblock01-knode02 kernel: [60196.736190] ------------[
> cut here ]------------
> Feb  6 09:23:54 kblock01-knode02 kernel: [60196.736211] WARNING: CPU: 18
> PID: 22709 at drivers/infiniband/core/verbs.c:1963
> __ib_drain_sq+0x135/0x1d0 [ib_core]
> Feb  6 09:23:54 kblock01-knode02 kernel: [60196.736212] failed to drain
> send queue: -110
> Feb  6 09:23:54 kblock01-knode02 kernel: [60196.736213] Modules linked
> in: nvme_rdma rdma_cm ib_cm iw_cm nvme_fabrics nvme_core
> ocs_fc_scst(POE) scst(OE) mptctl mptbase qla2xxx_scst(OE)
> scsi_transport_fc dm_multipath drbd lru_cache netconsole mst_pciconf(OE)
> nfsd nfs_acl auth_rpcgss ipmi_devintf lockd sunrpc grace ipt_MASQUERADE
> nf_nat_masquerade_ipv4 xt_nat iptable_nat nf_nat_ipv4 nf_nat
> nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack fuse binfmt_misc
> iTCO_wdt iTCO_vendor_support pcspkr serio_raw joydev igb i2c_i801
> i2c_smbus lpc_ich mei_me mei ioatdma dca ses enclosure ipmi_ssif ipmi_si
> ipmi_msghandler bnx2x libcrc32c mdio mlx5_ib ib_core mlx5_core devlink
> ptp pps_core tpm_tis tpm_tis_core tpm ext4(E) mbcache(E) jbd2(E) isci(E)
> libsas(E) mpt3sas(E) scsi_transport_sas(E) raid_class(E) megaraid_sas(E)
> wmi(E) mgag200(E) ttm(E) drm_kms_helper(E)
> Feb  6 09:23:54 kblock01-knode02 kernel: [60196.736293]  drm(E)
> i2c_algo_bit(E) [last unloaded: nvme_core]
> Feb  6 09:23:54 kblock01-knode02 kernel: [60196.736301] CPU: 18 PID:
> 22709 Comm: kworker/18:4 Tainted: P           OE   4.9.6-KM1 #0
> Feb  6 09:23:54 kblock01-knode02 kernel: [60196.736303] Hardware name:
> Supermicro SYS-1027R-72BRFTP5-EI007/X9DRW-7/iTPF, BIOS 3.0a 01/22/2014
> Feb  6 09:23:54 kblock01-knode02 kernel: [60196.736312] Workqueue:
> nvme_rdma_wq nvme_rdma_reconnect_ctrl_work [nvme_rdma]
> Feb  6 09:23:54 kblock01-knode02 kernel: [60196.736315] ffff96de27e6f9b8
> ffffffffb537f3ff ffffffffc06a00e5 ffff96de27e6fa18
> Feb  6 09:23:54 kblock01-knode02 kernel: [60196.736320] ffff96de27e6fa18
> 0000000000000000 ffff96de27e6fa08 ffffffffb5091a7d
> Feb  6 09:23:54 kblock01-knode02 kernel: [60196.736324] 0000000300000000
> 000007ab00000006 0507000000000000 ffff96de27e6fad8
> Feb  6 09:23:54 kblock01-knode02 kernel: [60196.736328] Call Trace:
> Feb  6 09:23:54 kblock01-knode02 kernel: [60196.736341]
> [<ffffffffb537f3ff>] dump_stack+0x67/0x98
> Feb  6 09:23:54 kblock01-knode02 kernel: [60196.736352]
> [<ffffffffc06a00e5>] ? __ib_drain_sq+0x135/0x1d0 [ib_core]
> Feb  6 09:23:54 kblock01-knode02 kernel: [60196.736364]
> [<ffffffffb5091a7d>] __warn+0xfd/0x120
> Feb  6 09:23:54 kblock01-knode02 kernel: [60196.736368]
> [<ffffffffb5091b59>] warn_slowpath_fmt+0x49/0x50
> Feb  6 09:23:54 kblock01-knode02 kernel: [60196.736378]
> [<ffffffffc069fdb5>] ? ib_modify_qp+0x45/0x50 [ib_core]
> Feb  6 09:23:54 kblock01-knode02 kernel: [60196.736388]
> [<ffffffffc06a00e5>] __ib_drain_sq+0x135/0x1d0 [ib_core]
> Feb  6 09:23:54 kblock01-knode02 kernel: [60196.736398]
> [<ffffffffc069f5a0>] ? ib_create_srq+0xa0/0xa0 [ib_core]
> Feb  6 09:23:54 kblock01-knode02 kernel: [60196.736408]
> [<ffffffffc06a01a5>] ib_drain_sq+0x25/0x30 [ib_core]
> Feb  6 09:23:54 kblock01-knode02 kernel: [60196.736418]
> [<ffffffffc06a01c6>] ib_drain_qp+0x16/0x40 [ib_core]
> Feb  6 09:23:54 kblock01-knode02 kernel: [60196.736422]
> [<ffffffffc0c4d25b>] nvme_rdma_stop_and_free_queue+0x2b/0x50 [nvme_rdma]
> Feb  6 09:23:54 kblock01-knode02 kernel: [60196.736426]
> [<ffffffffc0c4d2ad>] nvme_rdma_free_io_queues+0x2d/0x40 [nvme_rdma]
> Feb  6 09:23:54 kblock01-knode02 kernel: [60196.736429]
> [<ffffffffc0c4d884>] nvme_rdma_reconnect_ctrl_work+0x34/0x1e0 [nvme_rdma]
> Feb  6 09:23:54 kblock01-knode02 kernel: [60196.736434]
> [<ffffffffb50ac7ce>] process_one_work+0x17e/0x4f0
> Feb  6 09:23:54 kblock01-knode02 kernel: [60196.736444]
> [<ffffffffb50cefc5>] ? dequeue_task_fair+0x85/0x870
> Feb  6 09:23:54 kblock01-knode02 kernel: [60196.736454]
> [<ffffffffb5799c6a>] ? schedule+0x3a/0xa0
> Feb  6 09:23:54 kblock01-knode02 kernel: [60196.736456]
> [<ffffffffb50ad653>] worker_thread+0x153/0x660
> Feb  6 09:23:54 kblock01-knode02 kernel: [60196.736464]
> [<ffffffffb5026b4c>] ? __switch_to+0x1dc/0x670
> Feb  6 09:23:54 kblock01-knode02 kernel: [60196.736468]
> [<ffffffffb5799706>] ? __schedule+0x226/0x6a0
> Feb  6 09:23:54 kblock01-knode02 kernel: [60196.736471]
> [<ffffffffb50be7c2>] ? default_wake_function+0x12/0x20
> Feb  6 09:23:54 kblock01-knode02 kernel: [60196.736474]
> [<ffffffffb50d7636>] ? __wake_up_common+0x56/0x90
> Feb  6 09:23:54 kblock01-knode02 kernel: [60196.736477]
> [<ffffffffb50ad500>] ? workqueue_prepare_cpu+0x80/0x80
> Feb  6 09:23:54 kblock01-knode02 kernel: [60196.736480]
> [<ffffffffb5799c6a>] ? schedule+0x3a/0xa0
> Feb  6 09:23:54 kblock01-knode02 kernel: [60196.736483]
> [<ffffffffb50ad500>] ? workqueue_prepare_cpu+0x80/0x80
> Feb  6 09:23:54 kblock01-knode02 kernel: [60196.736487]
> [<ffffffffb50b237d>] kthread+0xcd/0xf0
> Feb  6 09:23:54 kblock01-knode02 kernel: [60196.736492]
> [<ffffffffb50bc40e>] ? schedule_tail+0x1e/0xc0
> Feb  6 09:23:54 kblock01-knode02 kernel: [60196.736495]
> [<ffffffffb50b22b0>] ? __kthread_init_worker+0x40/0x40
> Feb  6 09:23:54 kblock01-knode02 kernel: [60196.736499]
> [<ffffffffb579ded5>] ret_from_fork+0x25/0x30
> Feb  6 09:23:54 kblock01-knode02 kernel: [60196.736502] ---[ end trace
> eb0e5ba7dc81a687 ]---
> Feb  6 09:24:55 kblock01-knode02 kernel: [60258.176054] mlx5_core
> 0000:04:00.0: wait_func:879:(pid 22709): 2ERR_QP(0x507) timeout. Will
> cause a leak of a command resource
> Feb  6 09:24:55 kblock01-knode02 kernel: [60258.176059] ------------[
> cut here ]------------
> Feb  6 09:24:55 kblock01-knode02 kernel: [60258.176073] WARNING: CPU: 18
> PID: 22709 at drivers/infiniband/core/verbs.c:1998
> __ib_drain_rq+0x12a/0x1c0 [ib_core]
> Feb  6 09:24:55 kblock01-knode02 kernel: [60258.176075] failed to drain
> recv queue: -110
> Feb  6 09:24:55 kblock01-knode02 kernel: [60258.176076] Modules linked
> in: nvme_rdma rdma_cm ib_cm iw_cm nvme_fabrics nvme_core
> ocs_fc_scst(POE) scst(OE) mptctl mptbase qla2xxx_scst(OE)
> scsi_transport_fc dm_multipath drbd lru_cache netconsole mst_pciconf(OE)
> nfsd nfs_acl auth_rpcgss ipmi_devintf lockd sunrpc grace ipt_MASQUERADE
> nf_nat_masquerade_ipv4 xt_nat iptable_nat nf_nat_ipv4 nf_nat
> nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack fuse binfmt_misc
> iTCO_wdt iTCO_vendor_support pcspkr serio_raw joydev igb i2c_i801
> i2c_smbus lpc_ich mei_me mei ioatdma dca ses enclosure ipmi_ssif ipmi_si
> ipmi_msghandler bnx2x libcrc32c mdio mlx5_ib ib_core mlx5_core devlink
> ptp pps_core tpm_tis tpm_tis_core tpm ext4(E) mbcache(E) jbd2(E) isci(E)
> libsas(E) mpt3sas(E) scsi_transport_sas(E) raid_class(E) megaraid_sas(E)
> wmi(E) mgag200(E) ttm(E) drm_kms_helper(E)
> Feb  6 09:24:55 kblock01-knode02 kernel: [60258.176134]  drm(E)
> i2c_algo_bit(E) [last unloaded: nvme_core]
> Feb  6 09:24:55 kblock01-knode02 kernel: [60258.176140] CPU: 18 PID:
> 22709 Comm: kworker/18:4 Tainted: P        W  OE   4.9.6-KM1 #0
> Feb  6 09:24:55 kblock01-knode02 kernel: [60258.176141] Hardware name:
> Supermicro SYS-1027R-72BRFTP5-EI007/X9DRW-7/iTPF, BIOS 3.0a 01/22/2014
> Feb  6 09:24:55 kblock01-knode02 kernel: [60258.176147] Workqueue:
> nvme_rdma_wq nvme_rdma_reconnect_ctrl_work [nvme_rdma]
> Feb  6 09:24:55 kblock01-knode02 kernel: [60258.176150] ffff96de27e6f9b8
> ffffffffb537f3ff ffffffffc069feea ffff96de27e6fa18
> Feb  6 09:24:55 kblock01-knode02 kernel: [60258.176155] ffff96de27e6fa18
> 0000000000000000 ffff96de27e6fa08 ffffffffb5091a7d
> Feb  6 09:24:55 kblock01-knode02 kernel: [60258.176159] 0000000300000000
> 000007ce00000006 0507000000000000 ffff96de27e6fad8
> Feb  6 09:24:55 kblock01-knode02 kernel: [60258.176163] Call Trace:
> Feb  6 09:24:55 kblock01-knode02 kernel: [60258.176169]
> [<ffffffffb537f3ff>] dump_stack+0x67/0x98
> Feb  6 09:24:55 kblock01-knode02 kernel: [60258.176180]
> [<ffffffffc069feea>] ? __ib_drain_rq+0x12a/0x1c0 [ib_core]
> Feb  6 09:24:55 kblock01-knode02 kernel: [60258.176185]
> [<ffffffffb5091a7d>] __warn+0xfd/0x120
> Feb  6 09:24:55 kblock01-knode02 kernel: [60258.176189]
> [<ffffffffb5091b59>] warn_slowpath_fmt+0x49/0x50
> Feb  6 09:24:55 kblock01-knode02 kernel: [60258.176199]
> [<ffffffffc069fdb5>] ? ib_modify_qp+0x45/0x50 [ib_core]
> Feb  6 09:24:55 kblock01-knode02 kernel: [60258.176216]
> [<ffffffffc06de770>] ? mlx5_ib_modify_qp+0x980/0xec0 [mlx5_ib]
> Feb  6 09:24:55 kblock01-knode02 kernel: [60258.176225]
> [<ffffffffc069feea>] __ib_drain_rq+0x12a/0x1c0 [ib_core]
> Feb  6 09:24:55 kblock01-knode02 kernel: [60258.176235]
> [<ffffffffc069f5a0>] ? ib_create_srq+0xa0/0xa0 [ib_core]
> Feb  6 09:24:55 kblock01-knode02 kernel: [60258.176246]
> [<ffffffffc069ffa5>] ib_drain_rq+0x25/0x30 [ib_core]
> Feb  6 09:24:55 kblock01-knode02 kernel: [60258.176255]
> [<ffffffffc06a01dc>] ib_drain_qp+0x2c/0x40 [ib_core]
> Feb  6 09:24:55 kblock01-knode02 kernel: [60258.176259]
> [<ffffffffc0c4d25b>] nvme_rdma_stop_and_free_queue+0x2b/0x50 [nvme_rdma]
> Feb  6 09:24:55 kblock01-knode02 kernel: [60258.176263]
> [<ffffffffc0c4d2ad>] nvme_rdma_free_io_queues+0x2d/0x40 [nvme_rdma]
> Feb  6 09:24:55 kblock01-knode02 kernel: [60258.176267]
> [<ffffffffc0c4d884>] nvme_rdma_reconnect_ctrl_work+0x34/0x1e0 [nvme_rdma]
> Feb  6 09:24:55 kblock01-knode02 kernel: [60258.176270]
> [<ffffffffb50ac7ce>] process_one_work+0x17e/0x4f0
> Feb  6 09:24:55 kblock01-knode02 kernel: [60258.176275]
> [<ffffffffb50cefc5>] ? dequeue_task_fair+0x85/0x870
> Feb  6 09:24:55 kblock01-knode02 kernel: [60258.176278]
> [<ffffffffb5799c6a>] ? schedule+0x3a/0xa0
> Feb  6 09:24:55 kblock01-knode02 kernel: [60258.176281]
> [<ffffffffb50ad653>] worker_thread+0x153/0x660
> Feb  6 09:24:55 kblock01-knode02 kernel: [60258.176285]
> [<ffffffffb5026b4c>] ? __switch_to+0x1dc/0x670
> Feb  6 09:24:55 kblock01-knode02 kernel: [60258.176289]
> [<ffffffffb5799706>] ? __schedule+0x226/0x6a0
> Feb  6 09:24:55 kblock01-knode02 kernel: [60258.176291]
> [<ffffffffb50be7c2>] ? default_wake_function+0x12/0x20
> Feb  6 09:24:55 kblock01-knode02 kernel: [60258.176294]
> [<ffffffffb50d7636>] ? __wake_up_common+0x56/0x90
> Feb  6 09:24:55 kblock01-knode02 kernel: [60258.176297]
> [<ffffffffb50ad500>] ? workqueue_prepare_cpu+0x80/0x80
> Feb  6 09:24:55 kblock01-knode02 kernel: [60258.176300]
> [<ffffffffb5799c6a>] ? schedule+0x3a/0xa0
> Feb  6 09:24:55 kblock01-knode02 kernel: [60258.176302]
> [<ffffffffb50ad500>] ? workqueue_prepare_cpu+0x80/0x80
> Feb  6 09:24:55 kblock01-knode02 kernel: [60258.176306]
> [<ffffffffb50b237d>] kthread+0xcd/0xf0
> Feb  6 09:24:55 kblock01-knode02 kernel: [60258.176310]
> [<ffffffffb50bc40e>] ? schedule_tail+0x1e/0xc0
> Feb  6 09:24:55 kblock01-knode02 kernel: [60258.176313]
> [<ffffffffb50b22b0>] ? __kthread_init_worker+0x40/0x40
> Feb  6 09:24:55 kblock01-knode02 kernel: [60258.176316]
> [<ffffffffb579ded5>] ret_from_fork+0x25/0x30
> Feb  6 09:24:55 kblock01-knode02 kernel: [60258.176318] ---[ end trace
> eb0e5ba7dc81a688 ]---
> Feb  6 09:25:26 kblock01-knode02 udevd[9344]: worker [21322]
> unexpectedly returned with status 0x0100
> Feb  6 09:25:26 kblock01-knode02 udevd[9344]: worker [21322] failed
> while handling '/devices/virtual/nvme-fabrics/ctl/nvme0/nvme0n1'
> Feb  6 09:25:26 kblock01-knode02 udevd[9344]: worker [21323]
> unexpectedly returned with status 0x0100
> Feb  6 09:25:26 kblock01-knode02 udevd[9344]: worker [21323] failed
> while handling '/devices/virtual/nvme-fabrics/ctl/nvme0/nvme0n2'
> Feb  6 09:25:26 kblock01-knode02 udevd[9344]: worker [21741]
> unexpectedly returned with status 0x0100
> Feb  6 09:25:26 kblock01-knode02 udevd[9344]: worker [21741] failed
> while handling '/devices/virtual/nvme-fabrics/ctl/nvme0/nvme0n3'
> Feb  6 09:25:57 kblock01-knode02 kernel: [60319.615916] mlx5_core
> 0000:04:00.0: wait_func:879:(pid 22709): 2RST_QP(0x50a) timeout. Will
> cause a leak of a command resource
> Feb  6 09:25:57 kblock01-knode02 kernel: [60319.615922]
> mlx5_0:destroy_qp_common:1936:(pid 22709): mlx5_ib: modify QP 0x00015e
> to RESET failed
> Feb  6 09:26:19 kblock01-knode02 udevd[9344]: worker [21740]
> unexpectedly returned with status 0x0100
> Feb  6 09:26:19 kblock01-knode02 udevd[9344]: worker [21740] failed
> while handling '/devices/virtual/nvme-fabrics/ctl/nvme0/nvme0n4'
>
>
> _______________________________________________
> Linux-nvme mailing list
> Linux-nvme-IAPFreCvJWM7uuMidbF8XUB+6BGkLq7r@public.gmane.org
> http://lists.infradead.org/mailman/listinfo/linux-nvme
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Unexpected issues with 2 NVME initiators using the same target

[Laurence Oberman]

----- Original Message -----
> From: "Sagi Grimberg" <sagi-NQWnxTmZq1alnMjI0IkVqw@public.gmane.org>
> To: "shahar.salzman" <shahar.salzman-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>, linux-nvme-IAPFreCvJWM7uuMidbF8XUB+6BGkLq7r@public.gmane.org, linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> Sent: Tuesday, February 21, 2017 5:50:39 PM
> Subject: Re: Unexpected issues with 2 NVME initiators using the same target
> 
> 
> > I am using 2 initiators + 1 target using nvmet with 1 subsystem and 4
> > backend
> > devices. Kernel is 4.9.6, NVME/rdma drivers are all from the vanilla
> > kernel, I
> > had a probelm connecting the NVME using the OFED drivers, so I removed
> > all the
> > mlx_compat and everything which depends on it.
> 
> Would it be possible to test with latest upstream kernel?
> 
> >
> > When I perform simultaneous writes (non direct fio) from both of the
> > initiators
> > to the same device (overlapping areas), I get NVMEf disconnect followed
> > by "dump
> > error cqe", successful reconnect, and then on one of the servers I get a
> > WARN_ON. After this the server gets stuck and I have to power cycle it
> > to get it
> > back up...
> 
> The error cqes seem to indicate that a memory registration operation
> failed which escalated to something worse.
> 
> I noticed some issues before with CX4 having problems with memory
> registration in the presence of network retransmissions (due to
> network congestion).
> 
> I notified Mellanox folks on that too, CC'ing Linux-rdma for some
> more attention.
> 
> After that, I see that ib_modify_qp failed which I've never seen
> before (might indicate the the device is in bad shape), and the WARN_ON
> is really weird given that nvme-rdma never uses IB_POLL_DIRECT.
> 
> > Here are the printouts from the server that got stuck:
> >
> > Feb  6 09:20:13 kblock01-knode02 kernel: [59976.204216]
> > mlx5_0:dump_cqe:262:(pid 0): dump error cqe
> > Feb  6 09:20:13 kblock01-knode02 kernel: [59976.204219] 00000000
> > 00000000 00000000 00000000
> > Feb  6 09:20:13 kblock01-knode02 kernel: [59976.204220] 00000000
> > 00000000 00000000 00000000
> > Feb  6 09:20:13 kblock01-knode02 kernel: [59976.204220] 00000000
> > 00000000 00000000 00000000
> > Feb  6 09:20:13 kblock01-knode02 kernel: [59976.204221] 00000000
> > 08007806 25000129 015557d0
> > Feb  6 09:20:13 kblock01-knode02 kernel: [59976.204234] nvme nvme0:
> > MEMREG for CQE 0xffff96ddd747a638 failed with status
> > memory management operation error (6)
> > Feb  6 09:20:13 kblock01-knode02 kernel: [59976.204375] nvme nvme0:
> > reconnecting in 10 seconds
> > Feb  6 09:20:13 kblock01-knode02 kernel: [59976.205512]
> > mlx5_0:dump_cqe:262:(pid 0): dump error cqe
> > Feb  6 09:20:13 kblock01-knode02 kernel: [59976.205514] 00000000
> > 00000000 00000000 00000000
> > Feb  6 09:20:13 kblock01-knode02 kernel: [59976.205515] 00000000
> > 00000000 00000000 00000000
> > Feb  6 09:20:13 kblock01-knode02 kernel: [59976.205515] 00000000
> > 00000000 00000000 00000000
> > Feb  6 09:20:13 kblock01-knode02 kernel: [59976.205516] 00000000
> > 08007806 25000126 00692bd0
> > Feb  6 09:20:23 kblock01-knode02 kernel: [59986.452887] nvme nvme0:
> > Successfully reconnected
> > Feb  6 09:20:24 kblock01-knode02 kernel: [59986.682887]
> > mlx5_0:dump_cqe:262:(pid 0): dump error cqe
> > Feb  6 09:20:24 kblock01-knode02 kernel: [59986.682890] 00000000
> > 00000000 00000000 00000000
> > Feb  6 09:20:24 kblock01-knode02 kernel: [59986.682891] 00000000
> > 00000000 00000000 00000000
> > Feb  6 09:20:24 kblock01-knode02 kernel: [59986.682892] 00000000
> > 00000000 00000000 00000000
> > Feb  6 09:20:24 kblock01-knode02 kernel: [59986.682892] 00000000
> > 08007806 25000158 04cdd7d0
> > ...
> > Feb  6 09:20:24 kblock01-knode02 kernel: [59986.687737]
> > mlx5_0:dump_cqe:262:(pid 0): dump error cqe
> > Feb  6 09:20:24 kblock01-knode02 kernel: [59986.687739] 00000000
> > 00000000 00000000 00000000
> > Feb  6 09:20:24 kblock01-knode02 kernel: [59986.687740] 00000000
> > 00000000 00000000 00000000
> > Feb  6 09:20:24 kblock01-knode02 kernel: [59986.687740] 00000000
> > 00000000 00000000 00000000
> > Feb  6 09:20:24 kblock01-knode02 kernel: [59986.687741] 00000000
> > 93005204 00000155 00a385e0
> > Feb  6 09:20:34 kblock01-knode02 kernel: [59997.389290] nvme nvme0:
> > Successfully reconnected
> > Feb  6 09:21:19 kblock01-knode02 rsyslogd: -- MARK --
> > Feb  6 09:21:38 kblock01-knode02 kernel: [60060.927832]
> > mlx5_0:dump_cqe:262:(pid 0): dump error cqe
> > Feb  6 09:21:38 kblock01-knode02 kernel: [60060.927835] 00000000
> > 00000000 00000000 00000000
> > Feb  6 09:21:38 kblock01-knode02 kernel: [60060.927836] 00000000
> > 00000000 00000000 00000000
> > Feb  6 09:21:38 kblock01-knode02 kernel: [60060.927837] 00000000
> > 00000000 00000000 00000000
> > Feb  6 09:21:38 kblock01-knode02 kernel: [60060.927837] 00000000
> > 93005204 00000167 b44e76e0
> > Feb  6 09:21:38 kblock01-knode02 kernel: [60060.927846] nvme nvme0: RECV
> > for CQE 0xffff96fe64f18750 failed with status local protection error (4)
> > Feb  6 09:21:38 kblock01-knode02 kernel: [60060.928200] nvme nvme0:
> > reconnecting in 10 seconds
> > ...
> > Feb  6 09:23:54 kblock01-knode02 kernel: [60196.736182] mlx5_core
> > 0000:04:00.0: wait_func:879:(pid 22709): 2ERR_QP(0x507) timeout. Will
> > cause a leak of a command resource
> > Feb  6 09:23:54 kblock01-knode02 kernel: [60196.736190] ------------[
> > cut here ]------------
> > Feb  6 09:23:54 kblock01-knode02 kernel: [60196.736211] WARNING: CPU: 18
> > PID: 22709 at drivers/infiniband/core/verbs.c:1963
> > __ib_drain_sq+0x135/0x1d0 [ib_core]
> > Feb  6 09:23:54 kblock01-knode02 kernel: [60196.736212] failed to drain
> > send queue: -110
> > Feb  6 09:23:54 kblock01-knode02 kernel: [60196.736213] Modules linked
> > in: nvme_rdma rdma_cm ib_cm iw_cm nvme_fabrics nvme_core
> > ocs_fc_scst(POE) scst(OE) mptctl mptbase qla2xxx_scst(OE)
> > scsi_transport_fc dm_multipath drbd lru_cache netconsole mst_pciconf(OE)
> > nfsd nfs_acl auth_rpcgss ipmi_devintf lockd sunrpc grace ipt_MASQUERADE
> > nf_nat_masquerade_ipv4 xt_nat iptable_nat nf_nat_ipv4 nf_nat
> > nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack fuse binfmt_misc
> > iTCO_wdt iTCO_vendor_support pcspkr serio_raw joydev igb i2c_i801
> > i2c_smbus lpc_ich mei_me mei ioatdma dca ses enclosure ipmi_ssif ipmi_si
> > ipmi_msghandler bnx2x libcrc32c mdio mlx5_ib ib_core mlx5_core devlink
> > ptp pps_core tpm_tis tpm_tis_core tpm ext4(E) mbcache(E) jbd2(E) isci(E)
> > libsas(E) mpt3sas(E) scsi_transport_sas(E) raid_class(E) megaraid_sas(E)
> > wmi(E) mgag200(E) ttm(E) drm_kms_helper(E)
> > Feb  6 09:23:54 kblock01-knode02 kernel: [60196.736293]  drm(E)
> > i2c_algo_bit(E) [last unloaded: nvme_core]
> > Feb  6 09:23:54 kblock01-knode02 kernel: [60196.736301] CPU: 18 PID:
> > 22709 Comm: kworker/18:4 Tainted: P           OE   4.9.6-KM1 #0
> > Feb  6 09:23:54 kblock01-knode02 kernel: [60196.736303] Hardware name:
> > Supermicro SYS-1027R-72BRFTP5-EI007/X9DRW-7/iTPF, BIOS 3.0a 01/22/2014
> > Feb  6 09:23:54 kblock01-knode02 kernel: [60196.736312] Workqueue:
> > nvme_rdma_wq nvme_rdma_reconnect_ctrl_work [nvme_rdma]
> > Feb  6 09:23:54 kblock01-knode02 kernel: [60196.736315] ffff96de27e6f9b8
> > ffffffffb537f3ff ffffffffc06a00e5 ffff96de27e6fa18
> > Feb  6 09:23:54 kblock01-knode02 kernel: [60196.736320] ffff96de27e6fa18
> > 0000000000000000 ffff96de27e6fa08 ffffffffb5091a7d
> > Feb  6 09:23:54 kblock01-knode02 kernel: [60196.736324] 0000000300000000
> > 000007ab00000006 0507000000000000 ffff96de27e6fad8
> > Feb  6 09:23:54 kblock01-knode02 kernel: [60196.736328] Call Trace:
> > Feb  6 09:23:54 kblock01-knode02 kernel: [60196.736341]
> > [<ffffffffb537f3ff>] dump_stack+0x67/0x98
> > Feb  6 09:23:54 kblock01-knode02 kernel: [60196.736352]
> > [<ffffffffc06a00e5>] ? __ib_drain_sq+0x135/0x1d0 [ib_core]
> > Feb  6 09:23:54 kblock01-knode02 kernel: [60196.736364]
> > [<ffffffffb5091a7d>] __warn+0xfd/0x120
> > Feb  6 09:23:54 kblock01-knode02 kernel: [60196.736368]
> > [<ffffffffb5091b59>] warn_slowpath_fmt+0x49/0x50
> > Feb  6 09:23:54 kblock01-knode02 kernel: [60196.736378]
> > [<ffffffffc069fdb5>] ? ib_modify_qp+0x45/0x50 [ib_core]
> > Feb  6 09:23:54 kblock01-knode02 kernel: [60196.736388]
> > [<ffffffffc06a00e5>] __ib_drain_sq+0x135/0x1d0 [ib_core]
> > Feb  6 09:23:54 kblock01-knode02 kernel: [60196.736398]
> > [<ffffffffc069f5a0>] ? ib_create_srq+0xa0/0xa0 [ib_core]
> > Feb  6 09:23:54 kblock01-knode02 kernel: [60196.736408]
> > [<ffffffffc06a01a5>] ib_drain_sq+0x25/0x30 [ib_core]
> > Feb  6 09:23:54 kblock01-knode02 kernel: [60196.736418]
> > [<ffffffffc06a01c6>] ib_drain_qp+0x16/0x40 [ib_core]
> > Feb  6 09:23:54 kblock01-knode02 kernel: [60196.736422]
> > [<ffffffffc0c4d25b>] nvme_rdma_stop_and_free_queue+0x2b/0x50 [nvme_rdma]
> > Feb  6 09:23:54 kblock01-knode02 kernel: [60196.736426]
> > [<ffffffffc0c4d2ad>] nvme_rdma_free_io_queues+0x2d/0x40 [nvme_rdma]
> > Feb  6 09:23:54 kblock01-knode02 kernel: [60196.736429]
> > [<ffffffffc0c4d884>] nvme_rdma_reconnect_ctrl_work+0x34/0x1e0 [nvme_rdma]
> > Feb  6 09:23:54 kblock01-knode02 kernel: [60196.736434]
> > [<ffffffffb50ac7ce>] process_one_work+0x17e/0x4f0
> > Feb  6 09:23:54 kblock01-knode02 kernel: [60196.736444]
> > [<ffffffffb50cefc5>] ? dequeue_task_fair+0x85/0x870
> > Feb  6 09:23:54 kblock01-knode02 kernel: [60196.736454]
> > [<ffffffffb5799c6a>] ? schedule+0x3a/0xa0
> > Feb  6 09:23:54 kblock01-knode02 kernel: [60196.736456]
> > [<ffffffffb50ad653>] worker_thread+0x153/0x660
> > Feb  6 09:23:54 kblock01-knode02 kernel: [60196.736464]
> > [<ffffffffb5026b4c>] ? __switch_to+0x1dc/0x670
> > Feb  6 09:23:54 kblock01-knode02 kernel: [60196.736468]
> > [<ffffffffb5799706>] ? __schedule+0x226/0x6a0
> > Feb  6 09:23:54 kblock01-knode02 kernel: [60196.736471]
> > [<ffffffffb50be7c2>] ? default_wake_function+0x12/0x20
> > Feb  6 09:23:54 kblock01-knode02 kernel: [60196.736474]
> > [<ffffffffb50d7636>] ? __wake_up_common+0x56/0x90
> > Feb  6 09:23:54 kblock01-knode02 kernel: [60196.736477]
> > [<ffffffffb50ad500>] ? workqueue_prepare_cpu+0x80/0x80
> > Feb  6 09:23:54 kblock01-knode02 kernel: [60196.736480]
> > [<ffffffffb5799c6a>] ? schedule+0x3a/0xa0
> > Feb  6 09:23:54 kblock01-knode02 kernel: [60196.736483]
> > [<ffffffffb50ad500>] ? workqueue_prepare_cpu+0x80/0x80
> > Feb  6 09:23:54 kblock01-knode02 kernel: [60196.736487]
> > [<ffffffffb50b237d>] kthread+0xcd/0xf0
> > Feb  6 09:23:54 kblock01-knode02 kernel: [60196.736492]
> > [<ffffffffb50bc40e>] ? schedule_tail+0x1e/0xc0
> > Feb  6 09:23:54 kblock01-knode02 kernel: [60196.736495]
> > [<ffffffffb50b22b0>] ? __kthread_init_worker+0x40/0x40
> > Feb  6 09:23:54 kblock01-knode02 kernel: [60196.736499]
> > [<ffffffffb579ded5>] ret_from_fork+0x25/0x30
> > Feb  6 09:23:54 kblock01-knode02 kernel: [60196.736502] ---[ end trace
> > eb0e5ba7dc81a687 ]---
> > Feb  6 09:24:55 kblock01-knode02 kernel: [60258.176054] mlx5_core
> > 0000:04:00.0: wait_func:879:(pid 22709): 2ERR_QP(0x507) timeout. Will
> > cause a leak of a command resource
> > Feb  6 09:24:55 kblock01-knode02 kernel: [60258.176059] ------------[
> > cut here ]------------
> > Feb  6 09:24:55 kblock01-knode02 kernel: [60258.176073] WARNING: CPU: 18
> > PID: 22709 at drivers/infiniband/core/verbs.c:1998
> > __ib_drain_rq+0x12a/0x1c0 [ib_core]
> > Feb  6 09:24:55 kblock01-knode02 kernel: [60258.176075] failed to drain
> > recv queue: -110
> > Feb  6 09:24:55 kblock01-knode02 kernel: [60258.176076] Modules linked
> > in: nvme_rdma rdma_cm ib_cm iw_cm nvme_fabrics nvme_core
> > ocs_fc_scst(POE) scst(OE) mptctl mptbase qla2xxx_scst(OE)
> > scsi_transport_fc dm_multipath drbd lru_cache netconsole mst_pciconf(OE)
> > nfsd nfs_acl auth_rpcgss ipmi_devintf lockd sunrpc grace ipt_MASQUERADE
> > nf_nat_masquerade_ipv4 xt_nat iptable_nat nf_nat_ipv4 nf_nat
> > nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack fuse binfmt_misc
> > iTCO_wdt iTCO_vendor_support pcspkr serio_raw joydev igb i2c_i801
> > i2c_smbus lpc_ich mei_me mei ioatdma dca ses enclosure ipmi_ssif ipmi_si
> > ipmi_msghandler bnx2x libcrc32c mdio mlx5_ib ib_core mlx5_core devlink
> > ptp pps_core tpm_tis tpm_tis_core tpm ext4(E) mbcache(E) jbd2(E) isci(E)
> > libsas(E) mpt3sas(E) scsi_transport_sas(E) raid_class(E) megaraid_sas(E)
> > wmi(E) mgag200(E) ttm(E) drm_kms_helper(E)
> > Feb  6 09:24:55 kblock01-knode02 kernel: [60258.176134]  drm(E)
> > i2c_algo_bit(E) [last unloaded: nvme_core]
> > Feb  6 09:24:55 kblock01-knode02 kernel: [60258.176140] CPU: 18 PID:
> > 22709 Comm: kworker/18:4 Tainted: P        W  OE   4.9.6-KM1 #0
> > Feb  6 09:24:55 kblock01-knode02 kernel: [60258.176141] Hardware name:
> > Supermicro SYS-1027R-72BRFTP5-EI007/X9DRW-7/iTPF, BIOS 3.0a 01/22/2014
> > Feb  6 09:24:55 kblock01-knode02 kernel: [60258.176147] Workqueue:
> > nvme_rdma_wq nvme_rdma_reconnect_ctrl_work [nvme_rdma]
> > Feb  6 09:24:55 kblock01-knode02 kernel: [60258.176150] ffff96de27e6f9b8
> > ffffffffb537f3ff ffffffffc069feea ffff96de27e6fa18
> > Feb  6 09:24:55 kblock01-knode02 kernel: [60258.176155] ffff96de27e6fa18
> > 0000000000000000 ffff96de27e6fa08 ffffffffb5091a7d
> > Feb  6 09:24:55 kblock01-knode02 kernel: [60258.176159] 0000000300000000
> > 000007ce00000006 0507000000000000 ffff96de27e6fad8
> > Feb  6 09:24:55 kblock01-knode02 kernel: [60258.176163] Call Trace:
> > Feb  6 09:24:55 kblock01-knode02 kernel: [60258.176169]
> > [<ffffffffb537f3ff>] dump_stack+0x67/0x98
> > Feb  6 09:24:55 kblock01-knode02 kernel: [60258.176180]
> > [<ffffffffc069feea>] ? __ib_drain_rq+0x12a/0x1c0 [ib_core]
> > Feb  6 09:24:55 kblock01-knode02 kernel: [60258.176185]
> > [<ffffffffb5091a7d>] __warn+0xfd/0x120
> > Feb  6 09:24:55 kblock01-knode02 kernel: [60258.176189]
> > [<ffffffffb5091b59>] warn_slowpath_fmt+0x49/0x50
> > Feb  6 09:24:55 kblock01-knode02 kernel: [60258.176199]
> > [<ffffffffc069fdb5>] ? ib_modify_qp+0x45/0x50 [ib_core]
> > Feb  6 09:24:55 kblock01-knode02 kernel: [60258.176216]
> > [<ffffffffc06de770>] ? mlx5_ib_modify_qp+0x980/0xec0 [mlx5_ib]
> > Feb  6 09:24:55 kblock01-knode02 kernel: [60258.176225]
> > [<ffffffffc069feea>] __ib_drain_rq+0x12a/0x1c0 [ib_core]
> > Feb  6 09:24:55 kblock01-knode02 kernel: [60258.176235]
> > [<ffffffffc069f5a0>] ? ib_create_srq+0xa0/0xa0 [ib_core]
> > Feb  6 09:24:55 kblock01-knode02 kernel: [60258.176246]
> > [<ffffffffc069ffa5>] ib_drain_rq+0x25/0x30 [ib_core]
> > Feb  6 09:24:55 kblock01-knode02 kernel: [60258.176255]
> > [<ffffffffc06a01dc>] ib_drain_qp+0x2c/0x40 [ib_core]
> > Feb  6 09:24:55 kblock01-knode02 kernel: [60258.176259]
> > [<ffffffffc0c4d25b>] nvme_rdma_stop_and_free_queue+0x2b/0x50 [nvme_rdma]
> > Feb  6 09:24:55 kblock01-knode02 kernel: [60258.176263]
> > [<ffffffffc0c4d2ad>] nvme_rdma_free_io_queues+0x2d/0x40 [nvme_rdma]
> > Feb  6 09:24:55 kblock01-knode02 kernel: [60258.176267]
> > [<ffffffffc0c4d884>] nvme_rdma_reconnect_ctrl_work+0x34/0x1e0 [nvme_rdma]
> > Feb  6 09:24:55 kblock01-knode02 kernel: [60258.176270]
> > [<ffffffffb50ac7ce>] process_one_work+0x17e/0x4f0
> > Feb  6 09:24:55 kblock01-knode02 kernel: [60258.176275]
> > [<ffffffffb50cefc5>] ? dequeue_task_fair+0x85/0x870
> > Feb  6 09:24:55 kblock01-knode02 kernel: [60258.176278]
> > [<ffffffffb5799c6a>] ? schedule+0x3a/0xa0
> > Feb  6 09:24:55 kblock01-knode02 kernel: [60258.176281]
> > [<ffffffffb50ad653>] worker_thread+0x153/0x660
> > Feb  6 09:24:55 kblock01-knode02 kernel: [60258.176285]
> > [<ffffffffb5026b4c>] ? __switch_to+0x1dc/0x670
> > Feb  6 09:24:55 kblock01-knode02 kernel: [60258.176289]
> > [<ffffffffb5799706>] ? __schedule+0x226/0x6a0
> > Feb  6 09:24:55 kblock01-knode02 kernel: [60258.176291]
> > [<ffffffffb50be7c2>] ? default_wake_function+0x12/0x20
> > Feb  6 09:24:55 kblock01-knode02 kernel: [60258.176294]
> > [<ffffffffb50d7636>] ? __wake_up_common+0x56/0x90
> > Feb  6 09:24:55 kblock01-knode02 kernel: [60258.176297]
> > [<ffffffffb50ad500>] ? workqueue_prepare_cpu+0x80/0x80
> > Feb  6 09:24:55 kblock01-knode02 kernel: [60258.176300]
> > [<ffffffffb5799c6a>] ? schedule+0x3a/0xa0
> > Feb  6 09:24:55 kblock01-knode02 kernel: [60258.176302]
> > [<ffffffffb50ad500>] ? workqueue_prepare_cpu+0x80/0x80
> > Feb  6 09:24:55 kblock01-knode02 kernel: [60258.176306]
> > [<ffffffffb50b237d>] kthread+0xcd/0xf0
> > Feb  6 09:24:55 kblock01-knode02 kernel: [60258.176310]
> > [<ffffffffb50bc40e>] ? schedule_tail+0x1e/0xc0
> > Feb  6 09:24:55 kblock01-knode02 kernel: [60258.176313]
> > [<ffffffffb50b22b0>] ? __kthread_init_worker+0x40/0x40
> > Feb  6 09:24:55 kblock01-knode02 kernel: [60258.176316]
> > [<ffffffffb579ded5>] ret_from_fork+0x25/0x30
> > Feb  6 09:24:55 kblock01-knode02 kernel: [60258.176318] ---[ end trace
> > eb0e5ba7dc81a688 ]---
> > Feb  6 09:25:26 kblock01-knode02 udevd[9344]: worker [21322]
> > unexpectedly returned with status 0x0100
> > Feb  6 09:25:26 kblock01-knode02 udevd[9344]: worker [21322] failed
> > while handling '/devices/virtual/nvme-fabrics/ctl/nvme0/nvme0n1'
> > Feb  6 09:25:26 kblock01-knode02 udevd[9344]: worker [21323]
> > unexpectedly returned with status 0x0100
> > Feb  6 09:25:26 kblock01-knode02 udevd[9344]: worker [21323] failed
> > while handling '/devices/virtual/nvme-fabrics/ctl/nvme0/nvme0n2'
> > Feb  6 09:25:26 kblock01-knode02 udevd[9344]: worker [21741]
> > unexpectedly returned with status 0x0100
> > Feb  6 09:25:26 kblock01-knode02 udevd[9344]: worker [21741] failed
> > while handling '/devices/virtual/nvme-fabrics/ctl/nvme0/nvme0n3'
> > Feb  6 09:25:57 kblock01-knode02 kernel: [60319.615916] mlx5_core
> > 0000:04:00.0: wait_func:879:(pid 22709): 2RST_QP(0x50a) timeout. Will
> > cause a leak of a command resource
> > Feb  6 09:25:57 kblock01-knode02 kernel: [60319.615922]
> > mlx5_0:destroy_qp_common:1936:(pid 22709): mlx5_ib: modify QP 0x00015e
> > to RESET failed
> > Feb  6 09:26:19 kblock01-knode02 udevd[9344]: worker [21740]
> > unexpectedly returned with status 0x0100
> > Feb  6 09:26:19 kblock01-knode02 udevd[9344]: worker [21740] failed
> > while handling '/devices/virtual/nvme-fabrics/ctl/nvme0/nvme0n4'
> >
> >
> > _______________________________________________
> > Linux-nvme mailing list
> > Linux-nvme-IAPFreCvJWM7uuMidbF8XUB+6BGkLq7r@public.gmane.org
> > http://lists.infradead.org/mailman/listinfo/linux-nvme
> --
> To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 

Hi Sagi,

Have not looked in depth here but is this maybe the GAPS issue again I bumped into where we had to revert the patch for the SRP transport.
Would we have to do a similar revert in the NVME space or modify mlx5 where this issue exists.

Thanks
Laurnce



--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Unexpected issues with 2 NVME initiators using the same target

[Sagi Grimberg]

> Hi Sagi,
>
> Have not looked in depth here but is this maybe the GAPS issue again I bumped into where we had to revert the patch for the SRP transport.
> Would we have to do a similar revert in the NVME space or modify mlx5 where this issue exists.

We didn't add gaps support to nvme, the block layer splits gappy
io for us.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Unexpected issues with 2 NVME initiators using the same target

[shahar.salzman]

Hi,

I will try with kernel 4.10 when I get my hands on the machine.

In addition, I found that the machine did not have the kernel-firmware 
package installed. Could this cause the "strange" CX4 behavior? I will 
obviously re-test with the kernel-firmware package.

Shahar


On 02/22/2017 06:52 PM, Laurence Oberman wrote:
>
> ----- Original Message -----
>> From: "Sagi Grimberg" <sagi-NQWnxTmZq1alnMjI0IkVqw@public.gmane.org>
>> To: "shahar.salzman" <shahar.salzman-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>, linux-nvme-IAPFreCvJWM7uuMidbF8XUB+6BGkLq7r@public.gmane.org, linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
>> Sent: Tuesday, February 21, 2017 5:50:39 PM
>> Subject: Re: Unexpected issues with 2 NVME initiators using the same target
>>
>>
>>> I am using 2 initiators + 1 target using nvmet with 1 subsystem and 4
>>> backend
>>> devices. Kernel is 4.9.6, NVME/rdma drivers are all from the vanilla
>>> kernel, I
>>> had a probelm connecting the NVME using the OFED drivers, so I removed
>>> all the
>>> mlx_compat and everything which depends on it.
>> Would it be possible to test with latest upstream kernel?
>>
>>> When I perform simultaneous writes (non direct fio) from both of the
>>> initiators
>>> to the same device (overlapping areas), I get NVMEf disconnect followed
>>> by "dump
>>> error cqe", successful reconnect, and then on one of the servers I get a
>>> WARN_ON. After this the server gets stuck and I have to power cycle it
>>> to get it
>>> back up...
>> The error cqes seem to indicate that a memory registration operation
>> failed which escalated to something worse.
>>
>> I noticed some issues before with CX4 having problems with memory
>> registration in the presence of network retransmissions (due to
>> network congestion).
>>
>> I notified Mellanox folks on that too, CC'ing Linux-rdma for some
>> more attention.
>>
>> After that, I see that ib_modify_qp failed which I've never seen
>> before (might indicate the the device is in bad shape), and the WARN_ON
>> is really weird given that nvme-rdma never uses IB_POLL_DIRECT.
>>
>>> Here are the printouts from the server that got stuck:
>>>
>>> Feb  6 09:20:13 kblock01-knode02 kernel: [59976.204216]
>>> mlx5_0:dump_cqe:262:(pid 0): dump error cqe
>>> Feb  6 09:20:13 kblock01-knode02 kernel: [59976.204219] 00000000
>>> 00000000 00000000 00000000
>>> Feb  6 09:20:13 kblock01-knode02 kernel: [59976.204220] 00000000
>>> 00000000 00000000 00000000
>>> Feb  6 09:20:13 kblock01-knode02 kernel: [59976.204220] 00000000
>>> 00000000 00000000 00000000
>>> Feb  6 09:20:13 kblock01-knode02 kernel: [59976.204221] 00000000
>>> 08007806 25000129 015557d0
>>> Feb  6 09:20:13 kblock01-knode02 kernel: [59976.204234] nvme nvme0:
>>> MEMREG for CQE 0xffff96ddd747a638 failed with status
>>> memory management operation error (6)
>>> Feb  6 09:20:13 kblock01-knode02 kernel: [59976.204375] nvme nvme0:
>>> reconnecting in 10 seconds
>>> Feb  6 09:20:13 kblock01-knode02 kernel: [59976.205512]
>>> mlx5_0:dump_cqe:262:(pid 0): dump error cqe
>>> Feb  6 09:20:13 kblock01-knode02 kernel: [59976.205514] 00000000
>>> 00000000 00000000 00000000
>>> Feb  6 09:20:13 kblock01-knode02 kernel: [59976.205515] 00000000
>>> 00000000 00000000 00000000
>>> Feb  6 09:20:13 kblock01-knode02 kernel: [59976.205515] 00000000
>>> 00000000 00000000 00000000
>>> Feb  6 09:20:13 kblock01-knode02 kernel: [59976.205516] 00000000
>>> 08007806 25000126 00692bd0
>>> Feb  6 09:20:23 kblock01-knode02 kernel: [59986.452887] nvme nvme0:
>>> Successfully reconnected
>>> Feb  6 09:20:24 kblock01-knode02 kernel: [59986.682887]
>>> mlx5_0:dump_cqe:262:(pid 0): dump error cqe
>>> Feb  6 09:20:24 kblock01-knode02 kernel: [59986.682890] 00000000
>>> 00000000 00000000 00000000
>>> Feb  6 09:20:24 kblock01-knode02 kernel: [59986.682891] 00000000
>>> 00000000 00000000 00000000
>>> Feb  6 09:20:24 kblock01-knode02 kernel: [59986.682892] 00000000
>>> 00000000 00000000 00000000
>>> Feb  6 09:20:24 kblock01-knode02 kernel: [59986.682892] 00000000
>>> 08007806 25000158 04cdd7d0
>>> ...
>>> Feb  6 09:20:24 kblock01-knode02 kernel: [59986.687737]
>>> mlx5_0:dump_cqe:262:(pid 0): dump error cqe
>>> Feb  6 09:20:24 kblock01-knode02 kernel: [59986.687739] 00000000
>>> 00000000 00000000 00000000
>>> Feb  6 09:20:24 kblock01-knode02 kernel: [59986.687740] 00000000
>>> 00000000 00000000 00000000
>>> Feb  6 09:20:24 kblock01-knode02 kernel: [59986.687740] 00000000
>>> 00000000 00000000 00000000
>>> Feb  6 09:20:24 kblock01-knode02 kernel: [59986.687741] 00000000
>>> 93005204 00000155 00a385e0
>>> Feb  6 09:20:34 kblock01-knode02 kernel: [59997.389290] nvme nvme0:
>>> Successfully reconnected
>>> Feb  6 09:21:19 kblock01-knode02 rsyslogd: -- MARK --
>>> Feb  6 09:21:38 kblock01-knode02 kernel: [60060.927832]
>>> mlx5_0:dump_cqe:262:(pid 0): dump error cqe
>>> Feb  6 09:21:38 kblock01-knode02 kernel: [60060.927835] 00000000
>>> 00000000 00000000 00000000
>>> Feb  6 09:21:38 kblock01-knode02 kernel: [60060.927836] 00000000
>>> 00000000 00000000 00000000
>>> Feb  6 09:21:38 kblock01-knode02 kernel: [60060.927837] 00000000
>>> 00000000 00000000 00000000
>>> Feb  6 09:21:38 kblock01-knode02 kernel: [60060.927837] 00000000
>>> 93005204 00000167 b44e76e0
>>> Feb  6 09:21:38 kblock01-knode02 kernel: [60060.927846] nvme nvme0: RECV
>>> for CQE 0xffff96fe64f18750 failed with status local protection error (4)
>>> Feb  6 09:21:38 kblock01-knode02 kernel: [60060.928200] nvme nvme0:
>>> reconnecting in 10 seconds
>>> ...
>>> Feb  6 09:23:54 kblock01-knode02 kernel: [60196.736182] mlx5_core
>>> 0000:04:00.0: wait_func:879:(pid 22709): 2ERR_QP(0x507) timeout. Will
>>> cause a leak of a command resource
>>> Feb  6 09:23:54 kblock01-knode02 kernel: [60196.736190] ------------[
>>> cut here ]------------
>>> Feb  6 09:23:54 kblock01-knode02 kernel: [60196.736211] WARNING: CPU: 18
>>> PID: 22709 at drivers/infiniband/core/verbs.c:1963
>>> __ib_drain_sq+0x135/0x1d0 [ib_core]
>>> Feb  6 09:23:54 kblock01-knode02 kernel: [60196.736212] failed to drain
>>> send queue: -110
>>> Feb  6 09:23:54 kblock01-knode02 kernel: [60196.736213] Modules linked
>>> in: nvme_rdma rdma_cm ib_cm iw_cm nvme_fabrics nvme_core
>>> ocs_fc_scst(POE) scst(OE) mptctl mptbase qla2xxx_scst(OE)
>>> scsi_transport_fc dm_multipath drbd lru_cache netconsole mst_pciconf(OE)
>>> nfsd nfs_acl auth_rpcgss ipmi_devintf lockd sunrpc grace ipt_MASQUERADE
>>> nf_nat_masquerade_ipv4 xt_nat iptable_nat nf_nat_ipv4 nf_nat
>>> nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack fuse binfmt_misc
>>> iTCO_wdt iTCO_vendor_support pcspkr serio_raw joydev igb i2c_i801
>>> i2c_smbus lpc_ich mei_me mei ioatdma dca ses enclosure ipmi_ssif ipmi_si
>>> ipmi_msghandler bnx2x libcrc32c mdio mlx5_ib ib_core mlx5_core devlink
>>> ptp pps_core tpm_tis tpm_tis_core tpm ext4(E) mbcache(E) jbd2(E) isci(E)
>>> libsas(E) mpt3sas(E) scsi_transport_sas(E) raid_class(E) megaraid_sas(E)
>>> wmi(E) mgag200(E) ttm(E) drm_kms_helper(E)
>>> Feb  6 09:23:54 kblock01-knode02 kernel: [60196.736293]  drm(E)
>>> i2c_algo_bit(E) [last unloaded: nvme_core]
>>> Feb  6 09:23:54 kblock01-knode02 kernel: [60196.736301] CPU: 18 PID:
>>> 22709 Comm: kworker/18:4 Tainted: P           OE   4.9.6-KM1 #0
>>> Feb  6 09:23:54 kblock01-knode02 kernel: [60196.736303] Hardware name:
>>> Supermicro SYS-1027R-72BRFTP5-EI007/X9DRW-7/iTPF, BIOS 3.0a 01/22/2014
>>> Feb  6 09:23:54 kblock01-knode02 kernel: [60196.736312] Workqueue:
>>> nvme_rdma_wq nvme_rdma_reconnect_ctrl_work [nvme_rdma]
>>> Feb  6 09:23:54 kblock01-knode02 kernel: [60196.736315] ffff96de27e6f9b8
>>> ffffffffb537f3ff ffffffffc06a00e5 ffff96de27e6fa18
>>> Feb  6 09:23:54 kblock01-knode02 kernel: [60196.736320] ffff96de27e6fa18
>>> 0000000000000000 ffff96de27e6fa08 ffffffffb5091a7d
>>> Feb  6 09:23:54 kblock01-knode02 kernel: [60196.736324] 0000000300000000
>>> 000007ab00000006 0507000000000000 ffff96de27e6fad8
>>> Feb  6 09:23:54 kblock01-knode02 kernel: [60196.736328] Call Trace:
>>> Feb  6 09:23:54 kblock01-knode02 kernel: [60196.736341]
>>> [<ffffffffb537f3ff>] dump_stack+0x67/0x98
>>> Feb  6 09:23:54 kblock01-knode02 kernel: [60196.736352]
>>> [<ffffffffc06a00e5>] ? __ib_drain_sq+0x135/0x1d0 [ib_core]
>>> Feb  6 09:23:54 kblock01-knode02 kernel: [60196.736364]
>>> [<ffffffffb5091a7d>] __warn+0xfd/0x120
>>> Feb  6 09:23:54 kblock01-knode02 kernel: [60196.736368]
>>> [<ffffffffb5091b59>] warn_slowpath_fmt+0x49/0x50
>>> Feb  6 09:23:54 kblock01-knode02 kernel: [60196.736378]
>>> [<ffffffffc069fdb5>] ? ib_modify_qp+0x45/0x50 [ib_core]
>>> Feb  6 09:23:54 kblock01-knode02 kernel: [60196.736388]
>>> [<ffffffffc06a00e5>] __ib_drain_sq+0x135/0x1d0 [ib_core]
>>> Feb  6 09:23:54 kblock01-knode02 kernel: [60196.736398]
>>> [<ffffffffc069f5a0>] ? ib_create_srq+0xa0/0xa0 [ib_core]
>>> Feb  6 09:23:54 kblock01-knode02 kernel: [60196.736408]
>>> [<ffffffffc06a01a5>] ib_drain_sq+0x25/0x30 [ib_core]
>>> Feb  6 09:23:54 kblock01-knode02 kernel: [60196.736418]
>>> [<ffffffffc06a01c6>] ib_drain_qp+0x16/0x40 [ib_core]
>>> Feb  6 09:23:54 kblock01-knode02 kernel: [60196.736422]
>>> [<ffffffffc0c4d25b>] nvme_rdma_stop_and_free_queue+0x2b/0x50 [nvme_rdma]
>>> Feb  6 09:23:54 kblock01-knode02 kernel: [60196.736426]
>>> [<ffffffffc0c4d2ad>] nvme_rdma_free_io_queues+0x2d/0x40 [nvme_rdma]
>>> Feb  6 09:23:54 kblock01-knode02 kernel: [60196.736429]
>>> [<ffffffffc0c4d884>] nvme_rdma_reconnect_ctrl_work+0x34/0x1e0 [nvme_rdma]
>>> Feb  6 09:23:54 kblock01-knode02 kernel: [60196.736434]
>>> [<ffffffffb50ac7ce>] process_one_work+0x17e/0x4f0
>>> Feb  6 09:23:54 kblock01-knode02 kernel: [60196.736444]
>>> [<ffffffffb50cefc5>] ? dequeue_task_fair+0x85/0x870
>>> Feb  6 09:23:54 kblock01-knode02 kernel: [60196.736454]
>>> [<ffffffffb5799c6a>] ? schedule+0x3a/0xa0
>>> Feb  6 09:23:54 kblock01-knode02 kernel: [60196.736456]
>>> [<ffffffffb50ad653>] worker_thread+0x153/0x660
>>> Feb  6 09:23:54 kblock01-knode02 kernel: [60196.736464]
>>> [<ffffffffb5026b4c>] ? __switch_to+0x1dc/0x670
>>> Feb  6 09:23:54 kblock01-knode02 kernel: [60196.736468]
>>> [<ffffffffb5799706>] ? __schedule+0x226/0x6a0
>>> Feb  6 09:23:54 kblock01-knode02 kernel: [60196.736471]
>>> [<ffffffffb50be7c2>] ? default_wake_function+0x12/0x20
>>> Feb  6 09:23:54 kblock01-knode02 kernel: [60196.736474]
>>> [<ffffffffb50d7636>] ? __wake_up_common+0x56/0x90
>>> Feb  6 09:23:54 kblock01-knode02 kernel: [60196.736477]
>>> [<ffffffffb50ad500>] ? workqueue_prepare_cpu+0x80/0x80
>>> Feb  6 09:23:54 kblock01-knode02 kernel: [60196.736480]
>>> [<ffffffffb5799c6a>] ? schedule+0x3a/0xa0
>>> Feb  6 09:23:54 kblock01-knode02 kernel: [60196.736483]
>>> [<ffffffffb50ad500>] ? workqueue_prepare_cpu+0x80/0x80
>>> Feb  6 09:23:54 kblock01-knode02 kernel: [60196.736487]
>>> [<ffffffffb50b237d>] kthread+0xcd/0xf0
>>> Feb  6 09:23:54 kblock01-knode02 kernel: [60196.736492]
>>> [<ffffffffb50bc40e>] ? schedule_tail+0x1e/0xc0
>>> Feb  6 09:23:54 kblock01-knode02 kernel: [60196.736495]
>>> [<ffffffffb50b22b0>] ? __kthread_init_worker+0x40/0x40
>>> Feb  6 09:23:54 kblock01-knode02 kernel: [60196.736499]
>>> [<ffffffffb579ded5>] ret_from_fork+0x25/0x30
>>> Feb  6 09:23:54 kblock01-knode02 kernel: [60196.736502] ---[ end trace
>>> eb0e5ba7dc81a687 ]---
>>> Feb  6 09:24:55 kblock01-knode02 kernel: [60258.176054] mlx5_core
>>> 0000:04:00.0: wait_func:879:(pid 22709): 2ERR_QP(0x507) timeout. Will
>>> cause a leak of a command resource
>>> Feb  6 09:24:55 kblock01-knode02 kernel: [60258.176059] ------------[
>>> cut here ]------------
>>> Feb  6 09:24:55 kblock01-knode02 kernel: [60258.176073] WARNING: CPU: 18
>>> PID: 22709 at drivers/infiniband/core/verbs.c:1998
>>> __ib_drain_rq+0x12a/0x1c0 [ib_core]
>>> Feb  6 09:24:55 kblock01-knode02 kernel: [60258.176075] failed to drain
>>> recv queue: -110
>>> Feb  6 09:24:55 kblock01-knode02 kernel: [60258.176076] Modules linked
>>> in: nvme_rdma rdma_cm ib_cm iw_cm nvme_fabrics nvme_core
>>> ocs_fc_scst(POE) scst(OE) mptctl mptbase qla2xxx_scst(OE)
>>> scsi_transport_fc dm_multipath drbd lru_cache netconsole mst_pciconf(OE)
>>> nfsd nfs_acl auth_rpcgss ipmi_devintf lockd sunrpc grace ipt_MASQUERADE
>>> nf_nat_masquerade_ipv4 xt_nat iptable_nat nf_nat_ipv4 nf_nat
>>> nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack fuse binfmt_misc
>>> iTCO_wdt iTCO_vendor_support pcspkr serio_raw joydev igb i2c_i801
>>> i2c_smbus lpc_ich mei_me mei ioatdma dca ses enclosure ipmi_ssif ipmi_si
>>> ipmi_msghandler bnx2x libcrc32c mdio mlx5_ib ib_core mlx5_core devlink
>>> ptp pps_core tpm_tis tpm_tis_core tpm ext4(E) mbcache(E) jbd2(E) isci(E)
>>> libsas(E) mpt3sas(E) scsi_transport_sas(E) raid_class(E) megaraid_sas(E)
>>> wmi(E) mgag200(E) ttm(E) drm_kms_helper(E)
>>> Feb  6 09:24:55 kblock01-knode02 kernel: [60258.176134]  drm(E)
>>> i2c_algo_bit(E) [last unloaded: nvme_core]
>>> Feb  6 09:24:55 kblock01-knode02 kernel: [60258.176140] CPU: 18 PID:
>>> 22709 Comm: kworker/18:4 Tainted: P        W  OE   4.9.6-KM1 #0
>>> Feb  6 09:24:55 kblock01-knode02 kernel: [60258.176141] Hardware name:
>>> Supermicro SYS-1027R-72BRFTP5-EI007/X9DRW-7/iTPF, BIOS 3.0a 01/22/2014
>>> Feb  6 09:24:55 kblock01-knode02 kernel: [60258.176147] Workqueue:
>>> nvme_rdma_wq nvme_rdma_reconnect_ctrl_work [nvme_rdma]
>>> Feb  6 09:24:55 kblock01-knode02 kernel: [60258.176150] ffff96de27e6f9b8
>>> ffffffffb537f3ff ffffffffc069feea ffff96de27e6fa18
>>> Feb  6 09:24:55 kblock01-knode02 kernel: [60258.176155] ffff96de27e6fa18
>>> 0000000000000000 ffff96de27e6fa08 ffffffffb5091a7d
>>> Feb  6 09:24:55 kblock01-knode02 kernel: [60258.176159] 0000000300000000
>>> 000007ce00000006 0507000000000000 ffff96de27e6fad8
>>> Feb  6 09:24:55 kblock01-knode02 kernel: [60258.176163] Call Trace:
>>> Feb  6 09:24:55 kblock01-knode02 kernel: [60258.176169]
>>> [<ffffffffb537f3ff>] dump_stack+0x67/0x98
>>> Feb  6 09:24:55 kblock01-knode02 kernel: [60258.176180]
>>> [<ffffffffc069feea>] ? __ib_drain_rq+0x12a/0x1c0 [ib_core]
>>> Feb  6 09:24:55 kblock01-knode02 kernel: [60258.176185]
>>> [<ffffffffb5091a7d>] __warn+0xfd/0x120
>>> Feb  6 09:24:55 kblock01-knode02 kernel: [60258.176189]
>>> [<ffffffffb5091b59>] warn_slowpath_fmt+0x49/0x50
>>> Feb  6 09:24:55 kblock01-knode02 kernel: [60258.176199]
>>> [<ffffffffc069fdb5>] ? ib_modify_qp+0x45/0x50 [ib_core]
>>> Feb  6 09:24:55 kblock01-knode02 kernel: [60258.176216]
>>> [<ffffffffc06de770>] ? mlx5_ib_modify_qp+0x980/0xec0 [mlx5_ib]
>>> Feb  6 09:24:55 kblock01-knode02 kernel: [60258.176225]
>>> [<ffffffffc069feea>] __ib_drain_rq+0x12a/0x1c0 [ib_core]
>>> Feb  6 09:24:55 kblock01-knode02 kernel: [60258.176235]
>>> [<ffffffffc069f5a0>] ? ib_create_srq+0xa0/0xa0 [ib_core]
>>> Feb  6 09:24:55 kblock01-knode02 kernel: [60258.176246]
>>> [<ffffffffc069ffa5>] ib_drain_rq+0x25/0x30 [ib_core]
>>> Feb  6 09:24:55 kblock01-knode02 kernel: [60258.176255]
>>> [<ffffffffc06a01dc>] ib_drain_qp+0x2c/0x40 [ib_core]
>>> Feb  6 09:24:55 kblock01-knode02 kernel: [60258.176259]
>>> [<ffffffffc0c4d25b>] nvme_rdma_stop_and_free_queue+0x2b/0x50 [nvme_rdma]
>>> Feb  6 09:24:55 kblock01-knode02 kernel: [60258.176263]
>>> [<ffffffffc0c4d2ad>] nvme_rdma_free_io_queues+0x2d/0x40 [nvme_rdma]
>>> Feb  6 09:24:55 kblock01-knode02 kernel: [60258.176267]
>>> [<ffffffffc0c4d884>] nvme_rdma_reconnect_ctrl_work+0x34/0x1e0 [nvme_rdma]
>>> Feb  6 09:24:55 kblock01-knode02 kernel: [60258.176270]
>>> [<ffffffffb50ac7ce>] process_one_work+0x17e/0x4f0
>>> Feb  6 09:24:55 kblock01-knode02 kernel: [60258.176275]
>>> [<ffffffffb50cefc5>] ? dequeue_task_fair+0x85/0x870
>>> Feb  6 09:24:55 kblock01-knode02 kernel: [60258.176278]
>>> [<ffffffffb5799c6a>] ? schedule+0x3a/0xa0
>>> Feb  6 09:24:55 kblock01-knode02 kernel: [60258.176281]
>>> [<ffffffffb50ad653>] worker_thread+0x153/0x660
>>> Feb  6 09:24:55 kblock01-knode02 kernel: [60258.176285]
>>> [<ffffffffb5026b4c>] ? __switch_to+0x1dc/0x670
>>> Feb  6 09:24:55 kblock01-knode02 kernel: [60258.176289]
>>> [<ffffffffb5799706>] ? __schedule+0x226/0x6a0
>>> Feb  6 09:24:55 kblock01-knode02 kernel: [60258.176291]
>>> [<ffffffffb50be7c2>] ? default_wake_function+0x12/0x20
>>> Feb  6 09:24:55 kblock01-knode02 kernel: [60258.176294]
>>> [<ffffffffb50d7636>] ? __wake_up_common+0x56/0x90
>>> Feb  6 09:24:55 kblock01-knode02 kernel: [60258.176297]
>>> [<ffffffffb50ad500>] ? workqueue_prepare_cpu+0x80/0x80
>>> Feb  6 09:24:55 kblock01-knode02 kernel: [60258.176300]
>>> [<ffffffffb5799c6a>] ? schedule+0x3a/0xa0
>>> Feb  6 09:24:55 kblock01-knode02 kernel: [60258.176302]
>>> [<ffffffffb50ad500>] ? workqueue_prepare_cpu+0x80/0x80
>>> Feb  6 09:24:55 kblock01-knode02 kernel: [60258.176306]
>>> [<ffffffffb50b237d>] kthread+0xcd/0xf0
>>> Feb  6 09:24:55 kblock01-knode02 kernel: [60258.176310]
>>> [<ffffffffb50bc40e>] ? schedule_tail+0x1e/0xc0
>>> Feb  6 09:24:55 kblock01-knode02 kernel: [60258.176313]
>>> [<ffffffffb50b22b0>] ? __kthread_init_worker+0x40/0x40
>>> Feb  6 09:24:55 kblock01-knode02 kernel: [60258.176316]
>>> [<ffffffffb579ded5>] ret_from_fork+0x25/0x30
>>> Feb  6 09:24:55 kblock01-knode02 kernel: [60258.176318] ---[ end trace
>>> eb0e5ba7dc81a688 ]---
>>> Feb  6 09:25:26 kblock01-knode02 udevd[9344]: worker [21322]
>>> unexpectedly returned with status 0x0100
>>> Feb  6 09:25:26 kblock01-knode02 udevd[9344]: worker [21322] failed
>>> while handling '/devices/virtual/nvme-fabrics/ctl/nvme0/nvme0n1'
>>> Feb  6 09:25:26 kblock01-knode02 udevd[9344]: worker [21323]
>>> unexpectedly returned with status 0x0100
>>> Feb  6 09:25:26 kblock01-knode02 udevd[9344]: worker [21323] failed
>>> while handling '/devices/virtual/nvme-fabrics/ctl/nvme0/nvme0n2'
>>> Feb  6 09:25:26 kblock01-knode02 udevd[9344]: worker [21741]
>>> unexpectedly returned with status 0x0100
>>> Feb  6 09:25:26 kblock01-knode02 udevd[9344]: worker [21741] failed
>>> while handling '/devices/virtual/nvme-fabrics/ctl/nvme0/nvme0n3'
>>> Feb  6 09:25:57 kblock01-knode02 kernel: [60319.615916] mlx5_core
>>> 0000:04:00.0: wait_func:879:(pid 22709): 2RST_QP(0x50a) timeout. Will
>>> cause a leak of a command resource
>>> Feb  6 09:25:57 kblock01-knode02 kernel: [60319.615922]
>>> mlx5_0:destroy_qp_common:1936:(pid 22709): mlx5_ib: modify QP 0x00015e
>>> to RESET failed
>>> Feb  6 09:26:19 kblock01-knode02 udevd[9344]: worker [21740]
>>> unexpectedly returned with status 0x0100
>>> Feb  6 09:26:19 kblock01-knode02 udevd[9344]: worker [21740] failed
>>> while handling '/devices/virtual/nvme-fabrics/ctl/nvme0/nvme0n4'
>>>
>>>
>>> _______________________________________________
>>> Linux-nvme mailing list
>>> Linux-nvme-IAPFreCvJWM7uuMidbF8XUB+6BGkLq7r@public.gmane.org
>>> http://lists.infradead.org/mailman/listinfo/linux-nvme
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
>> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>
> Hi Sagi,
>
> Have not looked in depth here but is this maybe the GAPS issue again I bumped into where we had to revert the patch for the SRP transport.
> Would we have to do a similar revert in the NVME space or modify mlx5 where this issue exists.
>
> Thanks
> Laurnce
>
>
>

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Unexpected issues with 2 NVME initiators using the same target

[Sagi Grimberg]

> Hi,
>
> I will try with kernel 4.10 when I get my hands on the machine.
>
> In addition, I found that the machine did not have the kernel-firmware
> package installed. Could this cause the "strange" CX4 behavior? I will
> obviously re-test with the kernel-firmware package.

Not really, Mellanox FW is obtained by mellanox provided tools.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Unexpected issues with 2 NVME initiators using the same target

[Sagi Grimberg]

Hey Joseph,

> In our lab we are dealing with an issue which has some of the same symptoms.  Wanted to add to the thread in case it is useful here.  We have a target system with 16 Intel P3520 disks and a Mellanox CX4 50Gb NIC directly connected (no switch) to a single initiator system with a matching Mellanox CX4 50Gb NIC.  We are running Ubuntu 16.10 with 4.10-RC8 mainline kernel.  All drivers are kernel default drivers.  I've attached our nvmetcli json, and FIO workload, and dmesg from both systems.
>
> We are able to provoke this problem with a variety of workloads but a high bandwidth read operation seems to cause it the most reliably, harder to produce with smaller block sizes.  For some reason the problem seems produced when we stop and restart IO - I can run the FIO workload on the initiator system for 1-2 hours without any new events in dmesg, pushing about 5500MB/sec the whole time, then kill it and wait 10 seconds and restart it, and the errors and reconnect events happen reliably at that point.  Working to characterize further this week and also to see if we can produce on a smaller configuration.  Happy to provide any additional details that would be useful or try any fixes!
>
> On the initiator we see events like this:
>
> [51390.065641] mlx5_0:dump_cqe:262:(pid 0): dump error cqe
> [51390.065644] 00000000 00000000 00000000 00000000
> [51390.065645] 00000000 00000000 00000000 00000000
> [51390.065646] 00000000 00000000 00000000 00000000
> [51390.065648] 00000000 08007806 250003ab 02b9dcd2
> [51390.065666] nvme nvme3: MEMREG for CQE 0xffff9fc845039410 failed with status memory management operation error (6)
> [51390.079156] nvme nvme3: reconnecting in 10 seconds
> [51400.432782] nvme nvme3: Successfully reconnected

Seems to me this is a CX4 FW issue. Mellanox can elaborate on these
vendor specific syndromes on this output.

> On the target we see events like this:
>
> [51370.394694] mlx5_0:dump_cqe:262:(pid 6623): dump error cqe
> [51370.394696] 00000000 00000000 00000000 00000000
> [51370.394697] 00000000 00000000 00000000 00000000
> [51370.394699] 00000000 00000000 00000000 00000000
> [51370.394701] 00000000 00008813 080003ea 00c3b1d2

If the host is failing on memory mapping while the target is initiating
rdma access it makes sense that it will see errors.

>
> Sometimes, but less frequently, we also will see events on the target like this as part of the problem:
>
> [21322.678571] nvmet: ctrl 1 fatal error occurred!

Again, also makes sense because for nvmet this is a fatal error and we
need to teardown the controller.

You can try out this patch to see if it makes the memreg issues to go
away:
--
diff --git a/drivers/infiniband/hw/mlx5/qp.c 
b/drivers/infiniband/hw/mlx5/qp.c
index ad8a2638e339..0f9a12570262 100644
--- a/drivers/infiniband/hw/mlx5/qp.c
+++ b/drivers/infiniband/hw/mlx5/qp.c
@@ -3893,7 +3893,7 @@ int mlx5_ib_post_send(struct ib_qp *ibqp, struct 
ib_send_wr *wr,
                                 goto out;

                         case IB_WR_LOCAL_INV:
-                               next_fence = 
MLX5_FENCE_MODE_INITIATOR_SMALL;
+                               next_fence = 
MLX5_FENCE_MODE_STRONG_ORDERING;
                                 qp->sq.wr_data[idx] = IB_WR_LOCAL_INV;
                                 ctrl->imm = 
cpu_to_be32(wr->ex.invalidate_rkey);
                                 set_linv_wr(qp, &seg, &size);
@@ -3901,7 +3901,7 @@ int mlx5_ib_post_send(struct ib_qp *ibqp, struct 
ib_send_wr *wr,
                                 break;

                         case IB_WR_REG_MR:
-                               next_fence = 
MLX5_FENCE_MODE_INITIATOR_SMALL;
+                               next_fence = 
MLX5_FENCE_MODE_STRONG_ORDERING;
                                 qp->sq.wr_data[idx] = IB_WR_REG_MR;
                                 ctrl->imm = cpu_to_be32(reg_wr(wr)->key);
                                 err = set_reg_wr(qp, reg_wr(wr), &seg, 
&size);
--

Note that this will have a big performance (negative) impact on small
read workloads.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Unexpected issues with 2 NVME initiators using the same target

[Sagi Grimberg]

> Hi Sagi,
>
> I think we need to add fence to the UMR wqe.
>
> so lets try this one:
>
> diff --git a/drivers/infiniband/hw/mlx5/qp.c
> b/drivers/infiniband/hw/mlx5/qp.c
> index ad8a263..c38c4fa 100644
> --- a/drivers/infiniband/hw/mlx5/qp.c
> +++ b/drivers/infiniband/hw/mlx5/qp.c
> @@ -3737,8 +3737,7 @@ static void dump_wqe(struct mlx5_ib_qp *qp, int
> idx, int size_16)
>
>  static u8 get_fence(u8 fence, struct ib_send_wr *wr)
>  {
> -       if (unlikely(wr->opcode == IB_WR_LOCAL_INV &&
> -                    wr->send_flags & IB_SEND_FENCE))
> +       if (wr->opcode == IB_WR_LOCAL_INV || wr->opcode == IB_WR_REG_MR)
>                 return MLX5_FENCE_MODE_STRONG_ORDERING;
>
>         if (unlikely(fence)) {

This will kill performance, isn't there another fix that can
be applied just for retransmission flow?

> Couldn't repro that case but I run some initial tests in my Lab (with my
> patch above) - not performace servers:
>
> Initiator with 24 CPUs (2 threads/core, 6 cores/socket, 2 sockets),
> Connect IB (same driver mlx5_ib), kernel 4.10.0, fio test with 24 jobs
> and 128 iodepth.
> register_always=N
>
> Target - 1 subsystem with 1 ns (null_blk)
>
> bs   read (without/with patch)   write (without/with patch)
> --- --------------------------  ---------------------------
> 512     1019k / 1008k                 1004k / 992k
> 1k      1021k / 1013k                 1002k / 991k
> 4k      1030k / 1022k                 978k  / 969k
>
> CPU usage is 100% for both cases in the initiator side.
> haven't seen difference with bs = 16k.
> No so big drop like we would expect,

Obviously you won't see a drop without registering memory
for small IO (register_always=N), this would bypass registration
altogether... Please retest with register_always=Y.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Unexpected issues with 2 NVME initiators using the same target

[Sagi Grimberg]

> Hi,
>
> Shahar/Joseph, what is your link layer conf (IB/Eth) ?
> In eth case, have you configured some PFC ? if not, can you try it ?
> I suspect that this is the root cause

The root cause is that the device fails frwr in retransmission
flow, if PFC is not on, it will happen almost immediately, if not
it will happen at some point...

> and it might help you avoiding
> this case, meanwhile we're looking for for the best solution.
>
> Adding Vladimir that will run iSER on his performance setup with the new
> fencing patch (not an NVMEoF related issue).
> We can run also NVMEoF later on if needed.

Thanks.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Unexpected issues with 2 NVME initiators using the same target

[shahar.salzman]

Hi,

Sorry for the delay, I have been OOO for the past few days.

Indeed the underlying transport is Ethernet, and we have found that flow 
control is disabled on the switch side.
I still do not have access to the system for re-tests with the flow 
control, and 4.10. But I am working on assembling another system.

A little off-topic, but something that may help me get the other system 
up faster, can I use a dual CX4 card as initiator and target (i.e. one 
port going up to the switch, and the other coming back into the system) 
without the kernel looping back? I have a spare CX4, so if possible, I 
will use it to build a mini system for recreates of this sort.

Thanks,
Shahar


On 03/07/2017 03:41 PM, Sagi Grimberg wrote:
>
>> Hi,
>>
>> Shahar/Joseph, what is your link layer conf (IB/Eth) ?
>> In eth case, have you configured some PFC ? if not, can you try it ?
>> I suspect that this is the root cause
>
> The root cause is that the device fails frwr in retransmission
> flow, if PFC is not on, it will happen almost immediately, if not
> it will happen at some point...
>
>> and it might help you avoiding
>> this case, meanwhile we're looking for for the best solution.
>>
>> Adding Vladimir that will run iSER on his performance setup with the new
>> fencing patch (not an NVMEoF related issue).
>> We can run also NVMEoF later on if needed.
>
> Thanks.

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Unexpected issues with 2 NVME initiators using the same target

[Sagi Grimberg]

> Patched  by patch of Max (block 4K):
>
> allways reg          Y                N
> write              1902K            1923.3K
> read               1315K            2009K
>
> Original  OFED code (block 4K)
>
> allways reg          Y                N
> write              1947K           1982K
> read               1273K           1978K												

First, the write comparison is redundant because
we send immediate data without memory registration.

And, I'd compare against upstream code and not OFED.

So it seems that strong fencing does not effect performance
from the ULP point of view, surprising...
I'd suggest comparing on nvmf and srp as well.

If this is the case, and it indeed resolves the issue, we
should move forward with it as is.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Unexpected issues with 2 NVME initiators using the same target

[Sagi Grimberg]

Hey Joseph,

> Hey folks.  Apologies if this message comes through twice, but when I originally sent it the list flagged it as too large due to the dmesg log attachments, and then a coworker just told me they never saw it, so I don't think it made it through on the first attempt.
>
> Please see last note above and dmesg example attached - after more extensive testing with Max's patch we are still able to produce cqe dump errors (at a much lower frequency) as well as a new failure mode involving a crash dump.
>

This is a different issue AFAICT,

Looks like nvmet_sq_destroy() is stuck waiting for
the final reference to drop (which seems to never happen).

I'm trying to look for a code path where this may happen.
Can jyou tell if the backend block device completed all of
its I/O when this happens (can check for active tags in debugfs).
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Unexpected issues with 2 NVME initiators using the same target

[Marta Rybczynska]

> On 3/17/2017 8:37 PM, Gruher, Joseph R wrote:
>>
>>
>>> -----Original Message-----
>>> From: Max Gurtovoy [mailto:maxg-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org]
>>>
>>> I think we need to add fence to the UMR wqe.
>>>
>>> diff --git a/drivers/infiniband/hw/mlx5/qp.c b/drivers/infiniband/hw/mlx5/qp.c
>>> index ad8a263..c38c4fa 100644
>>> --- a/drivers/infiniband/hw/mlx5/qp.c
>>> +++ b/drivers/infiniband/hw/mlx5/qp.c
>>> @@ -3737,8 +3737,7 @@ static void dump_wqe(struct mlx5_ib_qp *qp, int idx,
>>> int size_16)
>>>
>>>   static u8 get_fence(u8 fence, struct ib_send_wr *wr)
>>>   {
>>> -       if (unlikely(wr->opcode == IB_WR_LOCAL_INV &&
>>> -                    wr->send_flags & IB_SEND_FENCE))
>>> +       if (wr->opcode == IB_WR_LOCAL_INV || wr->opcode == IB_WR_REG_MR)
>>>                  return MLX5_FENCE_MODE_STRONG_ORDERING;
>>>
>>>          if (unlikely(fence)) {
>>>
>>> 
>>
>> You mention these patches are only for testing.  How do we get to something
>> which can be submitted to upstream?
> 
> Yes, we need to be careful and not put the strong_fence if it's not a must.
> I'll be out for the upcoming week, but I'll ask our mlx5 maintainers to
> prepare a suitable patch and check some other applications performance
> numbers.
> Thanks for the testing, you can use this patch meanwhile till we push
> the formal solution.
> 

Hello Max,
We're seeing the same issue in our setup and we're running this patch
on our system for some time already. It seems to have fixed the issue.
When there is a final patch available, we can test it too.

Thanks,
Marta
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Unexpected issues with 2 NVME initiators using the same target

[Marta Rybczynska]

----- Mail original -----
> On 4/10/2017 2:40 PM, Marta Rybczynska wrote:
>>> On 3/17/2017 8:37 PM, Gruher, Joseph R wrote:
>>>>
>>>>
>>>>> -----Original Message-----
>>>>> From: Max Gurtovoy [mailto:maxg-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org]
>>>>>
>>>>> I think we need to add fence to the UMR wqe.
>>>>>
>>>>> diff --git a/drivers/infiniband/hw/mlx5/qp.c b/drivers/infiniband/hw/mlx5/qp.c
>>>>> index ad8a263..c38c4fa 100644
>>>>> --- a/drivers/infiniband/hw/mlx5/qp.c
>>>>> +++ b/drivers/infiniband/hw/mlx5/qp.c
>>>>> @@ -3737,8 +3737,7 @@ static void dump_wqe(struct mlx5_ib_qp *qp, int idx,
>>>>> int size_16)
>>>>>
>>>>>   static u8 get_fence(u8 fence, struct ib_send_wr *wr)
>>>>>   {
>>>>> -       if (unlikely(wr->opcode == IB_WR_LOCAL_INV &&
>>>>> -                    wr->send_flags & IB_SEND_FENCE))
>>>>> +       if (wr->opcode == IB_WR_LOCAL_INV || wr->opcode == IB_WR_REG_MR)
>>>>>                  return MLX5_FENCE_MODE_STRONG_ORDERING;
>>>>>
>>>>>          if (unlikely(fence)) {
>>>>>
>>>>>
>>>>
>>>> You mention these patches are only for testing.  How do we get to something
>>>> which can be submitted to upstream?
>>>
>>> Yes, we need to be careful and not put the strong_fence if it's not a must.
>>> I'll be out for the upcoming week, but I'll ask our mlx5 maintainers to
>>> prepare a suitable patch and check some other applications performance
>>> numbers.
>>> Thanks for the testing, you can use this patch meanwhile till we push
>>> the formal solution.
>>>
>>
>> Hello Max,
>> We're seeing the same issue in our setup and we're running this patch
>> on our system for some time already. It seems to have fixed the issue.
>> When there is a final patch available, we can test it too.
>>
>> Thanks,
>> Marta
>>
> 
> Hi Marta,
> thanks for testing my patch. I'll send it early next week (holiday's in
> our country) so it will be available in 4.11 kernel hopefully.
> if you can share on which NIC's you tested it and the perf numbers you
> get (with and without the patch), it will be great.
> 

Hello Max,
That's great news. We're ready to start testing as the final version
of the patch is out. We're using ConnectX 4. Unfortunately in our use
case the patch is not a question of performance: the workload doesn't
work without it. We may think about running other workloads for the
tests.

Marta
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Unexpected issues with 2 NVME initiators using the same target

[Sagi Grimberg]

Max,

> Hi Marta,
> thanks for testing my patch. I'll send it early next week (holiday's in
> our country) so it will be available in 4.11 kernel hopefully.
> if you can share on which NIC's you tested it and the perf numbers you
> get (with and without the patch), it will be great.

Can we get it to 4.12 please (we need you to send it out today).
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Unexpected issues with 2 NVME initiators using the same target

[Sagi Grimberg]

>> Can we get it to 4.12 please (we need you to send it out today).
>
> Sagi,
> we are preparing a cap bit for it.
> I'll send it as soon as it will be ready (otherwise we'll add fence for
> devices that can handle without it - such as ConnectX5).

Why? can't you just get what you have as a fix and incrementally add the
optimization for CX5?
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Unexpected issues with 2 NVME initiators using the same target

[Sagi Grimberg]

> Any update here?  Would love to be able to load up the new kernels without patching them every time. :)

I would like that too,

Max, can you send a patch? or should I?
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Unexpected issues with 2 NVME initiators using the same target

[Leon Romanovsky]

[-- Attachment #1: Type: text/plain, Size: 637 bytes --]

On Mon, May 15, 2017 at 03:00:07PM +0300, Sagi Grimberg wrote:
> > Any update here?  Would love to be able to load up the new kernels without patching them every time. :)
>
> I would like that too,
>
> Max, can you send a patch? or should I?

Sagi,

Max is doing his best to provide a patch, unfortunately he is limited
by various architecture implications which he needs to resolve before
sending it.

Thanks

> --
> To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: Unexpected issues with 2 NVME initiators using the same target

[Sagi Grimberg]

>>> Any update here?  Would love to be able to load up the new kernels without patching them every time. :)
>>
>> I would like that too,
>>
>> Max, can you send a patch? or should I?
>
> Sagi,
>
> Max is doing his best to provide a patch, unfortunately he is limited
> by various architecture implications which he needs to resolve before
> sending it.

Well, he already sent a patch that fixes the issue.
He said that he needs additional optimization for CX5 (which I assume
doesn't need the strong fence), but that still does not change the
fact that CX4 is broken. I asked to include his patch, fix the
existing bug and incrementally optimize CX5.

What do you mean by architecture implications? it's broken and there is
a request from the community to fix it. Are you suggesting that it
doesn't solve the issue?

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Unexpected issues with 2 NVME initiators using the same target

[Leon Romanovsky]

[-- Attachment #1: Type: text/plain, Size: 1255 bytes --]

On Mon, May 15, 2017 at 04:43:18PM +0300, Sagi Grimberg wrote:
> > > > Any update here?  Would love to be able to load up the new kernels without patching them every time. :)
> > >
> > > I would like that too,
> > >
> > > Max, can you send a patch? or should I?
> >
> > Sagi,
> >
> > Max is doing his best to provide a patch, unfortunately he is limited
> > by various architecture implications which he needs to resolve before
> > sending it.
>
> Well, he already sent a patch that fixes the issue.
> He said that he needs additional optimization for CX5 (which I assume
> doesn't need the strong fence), but that still does not change the
> fact that CX4 is broken. I asked to include his patch, fix the
> existing bug and incrementally optimize CX5.
>
> What do you mean by architecture implications? it's broken and there is
> a request from the community to fix it. Are you suggesting that it
> doesn't solve the issue?

I understand you and both Max and me are feeling the same as you. For more
than 2 months, we constantly (almost on daily basis) asked for a solution from
architecture group, but received different answers. The proposals were
extremely broad from need for strong fence for all cards to no need for
strong fence at all.

Thanks

>

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: Unexpected issues with 2 NVME initiators using the same target

[Christoph Hellwig]

On Mon, May 15, 2017 at 05:36:32PM +0300, Leon Romanovsky wrote:
> I understand you and both Max and me are feeling the same as you. For more
> than 2 months, we constantly (almost on daily basis) asked for a solution from
> architecture group, but received different answers. The proposals were
> extremely broad from need for strong fence for all cards to no need for
> strong fence at all.

So let's get the patch to do a strong fence everywhere now, and relax
it later where possible.

Correntness before speed..
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Unexpected issues with 2 NVME initiators using the same target

[Leon Romanovsky]

[-- Attachment #1: Type: text/plain, Size: 786 bytes --]

On Mon, May 15, 2017 at 07:59:52AM -0700, Christoph Hellwig wrote:
> On Mon, May 15, 2017 at 05:36:32PM +0300, Leon Romanovsky wrote:
> > I understand you and both Max and me are feeling the same as you. For more
> > than 2 months, we constantly (almost on daily basis) asked for a solution from
> > architecture group, but received different answers. The proposals were
> > extremely broad from need for strong fence for all cards to no need for
> > strong fence at all.
>
> So let's get the patch to do a strong fence everywhere now, and relax
> it later where possible.
>
> Correntness before speed..

OK, please give me and Max till EOW to stop this saga. One of the two
options will be: Max will resend original patch, or Max will send patch
blessed by architecture group.

Thanks

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: Unexpected issues with 2 NVME initiators using the same target

[Marta Rybczynska]

> On Mon, May 15, 2017 at 07:59:52AM -0700, Christoph Hellwig wrote:
>> On Mon, May 15, 2017 at 05:36:32PM +0300, Leon Romanovsky wrote:
>> > I understand you and both Max and me are feeling the same as you. For more
>> > than 2 months, we constantly (almost on daily basis) asked for a solution from
>> > architecture group, but received different answers. The proposals were
>> > extremely broad from need for strong fence for all cards to no need for
>> > strong fence at all.
>>
>> So let's get the patch to do a strong fence everywhere now, and relax
>> it later where possible.
>>
>> Correntness before speed..
> 
> OK, please give me and Max till EOW to stop this saga. One of the two
> options will be: Max will resend original patch, or Max will send patch
> blessed by architecture group.
> 

Good luck with this Max & Leon! It seems to be a complicated problem.
Just an idea: in our case it *seems* that the problem started appearing
after a firmware upgrade, older ones do not seem to have the same
behaviour. Maybe it's a hint for you.

Thanks!
Marta
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Unexpected issues with 2 NVME initiators using the same target

[Leon Romanovsky]

[-- Attachment #1: Type: text/plain, Size: 1870 bytes --]

On Wed, May 17, 2017 at 02:56:36PM +0200, Marta Rybczynska wrote:
> > On Mon, May 15, 2017 at 07:59:52AM -0700, Christoph Hellwig wrote:
> >> On Mon, May 15, 2017 at 05:36:32PM +0300, Leon Romanovsky wrote:
> >> > I understand you and both Max and me are feeling the same as you. For more
> >> > than 2 months, we constantly (almost on daily basis) asked for a solution from
> >> > architecture group, but received different answers. The proposals were
> >> > extremely broad from need for strong fence for all cards to no need for
> >> > strong fence at all.
> >>
> >> So let's get the patch to do a strong fence everywhere now, and relax
> >> it later where possible.
> >>
> >> Correntness before speed..
> >
> > OK, please give me and Max till EOW to stop this saga. One of the two
> > options will be: Max will resend original patch, or Max will send patch
> > blessed by architecture group.
> >
>
> Good luck with this Max & Leon! It seems to be a complicated problem.
> Just an idea: in our case it *seems* that the problem started appearing
> after a firmware upgrade, older ones do not seem to have the same
> behaviour. Maybe it's a hint for you.

OK, we came to the agreement which capability bits we should add. Max
will return to the office at the middle of the next week and we will
proceed with the submission of proper patch once our shared code will
be accepted.

In the meantime, i put the original patch to be part of our regression.
https://git.kernel.org/pub/scm/linux/kernel/git/leon/linux-rdma.git/commit/?h=testing/queue-next&id=a40ac569f243db552661e6efad70080bb406823c

Thank you for your patience.

>
> Thanks!
> Marta
> --
> To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: Unexpected issues with 2 NVME initiators using the same target

[Robert LeBlanc]

I ran into this with 4.9.32 when I rebooted the target. I tested
4.12-rc6 and this particular error seems to have been resolved, but I
now get a new one on the initiator. This one doesn't seem as
impactful.

[Mon Jun 19 11:17:20 2017] mlx5_0:dump_cqe:275:(pid 0): dump error cqe
[Mon Jun 19 11:17:20 2017] 00000000 00000000 00000000 00000000
[Mon Jun 19 11:17:20 2017] 00000000 00000000 00000000 00000000
[Mon Jun 19 11:17:20 2017] 00000000 00000000 00000000 00000000
[Mon Jun 19 11:17:20 2017] 00000000 93005204 0a0001bd 45c8e0d2
[Mon Jun 19 11:17:20 2017] iser: iser_err_comp: command failure: local
protection error (4) vend_err 52
[Mon Jun 19 11:17:20 2017]  connection3:0: detected conn error (1011)
[Mon Jun 19 11:17:31 2017] mlx5_0:dump_cqe:275:(pid 0): dump error cqe
[Mon Jun 19 11:17:31 2017] mlx5_0:dump_cqe:275:(pid 0): dump error cqe
[Mon Jun 19 11:17:31 2017] 00000000 00000000 00000000 00000000
[Mon Jun 19 11:17:31 2017] 00000000 00000000 00000000 00000000
[Mon Jun 19 11:17:31 2017] 00000000 00000000 00000000 00000000
[Mon Jun 19 11:17:31 2017] 00000000 93005204 0a0001e7 45dd82d2
[Mon Jun 19 11:17:31 2017] iser: iser_err_comp: command failure: local
protection error (4) vend_err 52
[Mon Jun 19 11:17:31 2017]  connection4:0: detected conn error (1011)
[Mon Jun 19 11:17:31 2017] 00000000 00000000 00000000 00000000
[Mon Jun 19 11:17:31 2017] 00000000 00000000 00000000 00000000
[Mon Jun 19 11:17:31 2017] 00000000 00000000 00000000 00000000
[Mon Jun 19 11:17:31 2017] 00000000 93005204 0a0001f4 004915d2
[Mon Jun 19 11:17:31 2017] iser: iser_err_comp: command failure: local
protection error (4) vend_err 52
[Mon Jun 19 11:17:31 2017]  connection3:0: detected conn error (1011)
[Mon Jun 19 11:17:44 2017] mlx5_0:dump_cqe:275:(pid 0): dump error cqe
[Mon Jun 19 11:17:44 2017] 00000000 00000000 00000000 00000000
[Mon Jun 19 11:17:44 2017] 00000000 00000000 00000000 00000000
[Mon Jun 19 11:17:44 2017] 00000000 00000000 00000000 00000000
[Mon Jun 19 11:17:44 2017] 00000000 93005204 0a0001f6 004519d2
[Mon Jun 19 11:17:44 2017] iser: iser_err_comp: command failure: local
protection error (4) vend_err 52
[Mon Jun 19 11:17:44 2017]  connection3:0: detected conn error (1011)
[Mon Jun 19 11:18:55 2017] mlx5_0:dump_cqe:275:(pid 0): dump error cqe
[Mon Jun 19 11:18:55 2017] 00000000 00000000 00000000 00000000
[Mon Jun 19 11:18:55 2017] 00000000 00000000 00000000 00000000
[Mon Jun 19 11:18:55 2017] 00000000 00000000 00000000 00000000
[Mon Jun 19 11:18:55 2017] 00000000 93005204 0a0001f7 01934fd2
[Mon Jun 19 11:18:55 2017] iser: iser_err_comp: command failure: local
protection error (4) vend_err 52
[Mon Jun 19 11:18:55 2017]  connection3:0: detected conn error (1011)
[Mon Jun 19 11:20:25 2017] mlx5_0:dump_cqe:275:(pid 0): dump error cqe
[Mon Jun 19 11:20:25 2017] 00000000 00000000 00000000 00000000
[Mon Jun 19 11:20:25 2017] 00000000 00000000 00000000 00000000
[Mon Jun 19 11:20:25 2017] 00000000 00000000 00000000 00000000
[Mon Jun 19 11:20:25 2017] 00000000 93005204 0a0001f8 0274edd2
[Mon Jun 19 11:20:25 2017] iser: iser_err_comp: command failure: local
protection error (4) vend_err 52
[Mon Jun 19 11:20:25 2017]  connection3:0: detected conn error (1011)

I'm going to try to cherry-pick the fix to 4.9.x and do some testing there.

Thanks,

----------------
Robert LeBlanc
PGP Fingerprint 79A2 9CA4 6CC4 45DD A904  C70E E654 3BB2 FA62 B9F1


On Thu, May 18, 2017 at 7:34 AM, Leon Romanovsky <leon-DgEjT+Ai2ygdnm+yROfE0A@public.gmane.org> wrote:
> On Wed, May 17, 2017 at 02:56:36PM +0200, Marta Rybczynska wrote:
>> > On Mon, May 15, 2017 at 07:59:52AM -0700, Christoph Hellwig wrote:
>> >> On Mon, May 15, 2017 at 05:36:32PM +0300, Leon Romanovsky wrote:
>> >> > I understand you and both Max and me are feeling the same as you. For more
>> >> > than 2 months, we constantly (almost on daily basis) asked for a solution from
>> >> > architecture group, but received different answers. The proposals were
>> >> > extremely broad from need for strong fence for all cards to no need for
>> >> > strong fence at all.
>> >>
>> >> So let's get the patch to do a strong fence everywhere now, and relax
>> >> it later where possible.
>> >>
>> >> Correntness before speed..
>> >
>> > OK, please give me and Max till EOW to stop this saga. One of the two
>> > options will be: Max will resend original patch, or Max will send patch
>> > blessed by architecture group.
>> >
>>
>> Good luck with this Max & Leon! It seems to be a complicated problem.
>> Just an idea: in our case it *seems* that the problem started appearing
>> after a firmware upgrade, older ones do not seem to have the same
>> behaviour. Maybe it's a hint for you.
>
> OK, we came to the agreement which capability bits we should add. Max
> will return to the office at the middle of the next week and we will
> proceed with the submission of proper patch once our shared code will
> be accepted.
>
> In the meantime, i put the original patch to be part of our regression.
> https://git.kernel.org/pub/scm/linux/kernel/git/leon/linux-rdma.git/commit/?h=testing/queue-next&id=a40ac569f243db552661e6efad70080bb406823c
>
> Thank you for your patience.
>
>>
>> Thanks!
>> Marta
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
>> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Unexpected issues with 2 NVME initiators using the same target

[Sagi Grimberg]

Hi Robert,

> I ran into this with 4.9.32 when I rebooted the target. I tested
> 4.12-rc6 and this particular error seems to have been resolved, but I
> now get a new one on the initiator. This one doesn't seem as
> impactful.
> 
> [Mon Jun 19 11:17:20 2017] mlx5_0:dump_cqe:275:(pid 0): dump error cqe
> [Mon Jun 19 11:17:20 2017] 00000000 00000000 00000000 00000000
> [Mon Jun 19 11:17:20 2017] 00000000 00000000 00000000 00000000
> [Mon Jun 19 11:17:20 2017] 00000000 00000000 00000000 00000000
> [Mon Jun 19 11:17:20 2017] 00000000 93005204 0a0001bd 45c8e0d2

Max, Leon,

Care to parse this syndrome for us? ;)
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Unexpected issues with 2 NVME initiators using the same target

[Leon Romanovsky]

[-- Attachment #1: Type: text/plain, Size: 1444 bytes --]

On Tue, Jun 20, 2017 at 09:39:36AM +0300, Sagi Grimberg wrote:
> Hi Robert,
>
> > I ran into this with 4.9.32 when I rebooted the target. I tested
> > 4.12-rc6 and this particular error seems to have been resolved, but I
> > now get a new one on the initiator. This one doesn't seem as
> > impactful.
> >
> > [Mon Jun 19 11:17:20 2017] mlx5_0:dump_cqe:275:(pid 0): dump error cqe
> > [Mon Jun 19 11:17:20 2017] 00000000 00000000 00000000 00000000
> > [Mon Jun 19 11:17:20 2017] 00000000 00000000 00000000 00000000
> > [Mon Jun 19 11:17:20 2017] 00000000 00000000 00000000 00000000
> > [Mon Jun 19 11:17:20 2017] 00000000 93005204 0a0001bd 45c8e0d2
>
> Max, Leon,
>
> Care to parse this syndrome for us? ;)

Here the parsed output, it says that it was access to mkey which is
free.

======== cqe_with_error ========
wqe_id                           : 0x0
srqn_usr_index                   : 0x0
byte_cnt                         : 0x0
hw_error_syndrome                : 0x93
hw_syndrome_type                 : 0x0
vendor_error_syndrome            : 0x52
syndrome                         : LOCAL_PROTECTION_ERROR (0x4)
s_wqe_opcode                     : SEND (0xa)
qpn_dctn_flow_tag                : 0x1bd
wqe_counter                      : 0x45c8
signature                        : 0xe0
opcode                           : REQUESTOR_ERROR (0xd)
cqe_format                       : NO_INLINE_DATA (0x0)
owner                            : 0x0

Thanks

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: Unexpected issues with 2 NVME initiators using the same target

[Sagi Grimberg]

>> Hi Robert,
>>
>>> I ran into this with 4.9.32 when I rebooted the target. I tested
>>> 4.12-rc6 and this particular error seems to have been resolved, but I
>>> now get a new one on the initiator. This one doesn't seem as
>>> impactful.
>>>
>>> [Mon Jun 19 11:17:20 2017] mlx5_0:dump_cqe:275:(pid 0): dump error cqe
>>> [Mon Jun 19 11:17:20 2017] 00000000 00000000 00000000 00000000
>>> [Mon Jun 19 11:17:20 2017] 00000000 00000000 00000000 00000000
>>> [Mon Jun 19 11:17:20 2017] 00000000 00000000 00000000 00000000
>>> [Mon Jun 19 11:17:20 2017] 00000000 93005204 0a0001bd 45c8e0d2
>>
>> Max, Leon,
>>
>> Care to parse this syndrome for us? ;)
> 
> Here the parsed output, it says that it was access to mkey which is
> free.
> 
> ======== cqe_with_error ========
> wqe_id                           : 0x0
> srqn_usr_index                   : 0x0
> byte_cnt                         : 0x0
> hw_error_syndrome                : 0x93
> hw_syndrome_type                 : 0x0
> vendor_error_syndrome            : 0x52

Can you share the check that correlates to the vendor+hw syndrome?

> syndrome                         : LOCAL_PROTECTION_ERROR (0x4)
> s_wqe_opcode                     : SEND (0xa)

That's interesting, the opcode is a send operation. I'm assuming
that this is immediate-data write? Robert, did this happen when
you issued >4k writes to the target?
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Unexpected issues with 2 NVME initiators using the same target

[Leon Romanovsky]

[-- Attachment #1: Type: text/plain, Size: 1541 bytes --]

On Tue, Jun 20, 2017 at 10:58:47AM +0300, Sagi Grimberg wrote:
>
> > > Hi Robert,
> > >
> > > > I ran into this with 4.9.32 when I rebooted the target. I tested
> > > > 4.12-rc6 and this particular error seems to have been resolved, but I
> > > > now get a new one on the initiator. This one doesn't seem as
> > > > impactful.
> > > >
> > > > [Mon Jun 19 11:17:20 2017] mlx5_0:dump_cqe:275:(pid 0): dump error cqe
> > > > [Mon Jun 19 11:17:20 2017] 00000000 00000000 00000000 00000000
> > > > [Mon Jun 19 11:17:20 2017] 00000000 00000000 00000000 00000000
> > > > [Mon Jun 19 11:17:20 2017] 00000000 00000000 00000000 00000000
> > > > [Mon Jun 19 11:17:20 2017] 00000000 93005204 0a0001bd 45c8e0d2
> > >
> > > Max, Leon,
> > >
> > > Care to parse this syndrome for us? ;)
> >
> > Here the parsed output, it says that it was access to mkey which is
> > free.
> >
> > ======== cqe_with_error ========
> > wqe_id                           : 0x0
> > srqn_usr_index                   : 0x0
> > byte_cnt                         : 0x0
> > hw_error_syndrome                : 0x93
> > hw_syndrome_type                 : 0x0
> > vendor_error_syndrome            : 0x52
>
> Can you share the check that correlates to the vendor+hw syndrome?

mkey.free == 1

>
> > syndrome                         : LOCAL_PROTECTION_ERROR (0x4)
> > s_wqe_opcode                     : SEND (0xa)
>
> That's interesting, the opcode is a send operation. I'm assuming
> that this is immediate-data write? Robert, did this happen when
> you issued >4k writes to the target?

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: Unexpected issues with 2 NVME initiators using the same target

[Sagi Grimberg]

>>> Here the parsed output, it says that it was access to mkey which is
>>> free.

Missed that :)

>>> ======== cqe_with_error ========
>>> wqe_id                           : 0x0
>>> srqn_usr_index                   : 0x0
>>> byte_cnt                         : 0x0
>>> hw_error_syndrome                : 0x93
>>> hw_syndrome_type                 : 0x0
>>> vendor_error_syndrome            : 0x52
>>
>> Can you share the check that correlates to the vendor+hw syndrome?
> 
> mkey.free == 1

Hmm, the way I understand it is that the HW is trying to access
(locally via send) a MR which was already invalidated.

Thinking of this further, this can happen in a case where the target
already completed the transaction, sent SEND_WITH_INVALIDATE but the
original send ack was lost somewhere causing the device to retransmit
from the MR (which was already invalidated). This is highly unlikely
though.

Shouldn't this be protected somehow by the device?
Can someone explain why the above cannot happen? Jason? Liran? Anyone?

Say host register MR (a) and send (1) from that MR to a target,
send (1) ack got lost, and the target issues SEND_WITH_INVALIDATE
on MR (a) and the host HCA process it, then host HCA timeout on send (1)
so it retries, but ehh, its already invalidated.

Or, we can also have a race where we destroy all our MRs when I/O
is still running (but from the code we should be safe here).

Robert, when you rebooted the target, I assume iscsi ping
timeout expired and the connection teardown started correct?
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Unexpected issues with 2 NVME initiators using the same target

[Sagi Grimberg]

>>> Can you share the check that correlates to the vendor+hw syndrome?
>>
>> mkey.free == 1
> 
> Hmm, the way I understand it is that the HW is trying to access
> (locally via send) a MR which was already invalidated.
> 
> Thinking of this further, this can happen in a case where the target
> already completed the transaction, sent SEND_WITH_INVALIDATE but the
> original send ack was lost somewhere causing the device to retransmit
> from the MR (which was already invalidated). This is highly unlikely
> though.
> 
> Shouldn't this be protected somehow by the device?
> Can someone explain why the above cannot happen? Jason? Liran? Anyone?
> 
> Say host register MR (a) and send (1) from that MR to a target,
> send (1) ack got lost, and the target issues SEND_WITH_INVALIDATE
> on MR (a) and the host HCA process it, then host HCA timeout on send (1)
> so it retries, but ehh, its already invalidated.

Well, this entire flow is broken, why should the host send the MR rkey
to the target if it is not using it for remote access, the target
should never have a chance to remote invalidate something it did not
access.

I think we have a bug in iSER code, as we should not send the key
for remote invalidation if we do inline data send...

Robert, can you try the following:
--
diff --git a/drivers/infiniband/ulp/iser/iser_initiator.c 
b/drivers/infiniband/ulp/iser/iser_initiator.c
index 12ed62ce9ff7..2a07692007bd 100644
--- a/drivers/infiniband/ulp/iser/iser_initiator.c
+++ b/drivers/infiniband/ulp/iser/iser_initiator.c
@@ -137,8 +137,10 @@ iser_prepare_write_cmd(struct iscsi_task *task,

         if (unsol_sz < edtl) {
                 hdr->flags     |= ISER_WSV;
-               hdr->write_stag = cpu_to_be32(mem_reg->rkey);
-               hdr->write_va   = cpu_to_be64(mem_reg->sge.addr + unsol_sz);
+               if (buf_out->data_len > imm_sz) {
+                       hdr->write_stag = cpu_to_be32(mem_reg->rkey);
+                       hdr->write_va = cpu_to_be64(mem_reg->sge.addr + 
unsol_sz);
+               }

                 iser_dbg("Cmd itt:%d, WRITE tags, RKEY:%#.4X "
                          "VA:%#llX + unsol:%d\n",
--

Although, I still don't think its enough. We need to delay the
local invalidate till we received a send completion (guarantees
that ack was received)...

If this indeed the case, _all_ ULP initiator drivers share it because we
never condition on a send completion in order to complete an I/O, and
in the case of lost ack on send, looks like we need to... It *will* hurt
performance.

What do other folks think?

CC'ing Bart, Chuck, Christoph.

Guys, for summary, I think we might have a broken behavior in the
initiator mode drivers. We never condition send completions (for
requests) before we complete an I/O. The issue is that the ack for those
sends might get lost, which means that the HCA will retry them (dropped
by the peer HCA) but if we happen to complete the I/O before, either we
can unmap the request area, or for inline data, we invalidate it (so the
HCA will try to access a MR which was invalidated).

Signalling all send completions and also finishing I/Os only after we
got them will add latency, and that sucks...
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Unexpected issues with 2 NVME initiators using the same target

[Chuck Lever]

> On Jun 20, 2017, at 8:02 AM, Sagi Grimberg <sagi-NQWnxTmZq1alnMjI0IkVqw@public.gmane.org> wrote:
> 
> 
>>>> Can you share the check that correlates to the vendor+hw syndrome?
>>> 
>>> mkey.free == 1
>> Hmm, the way I understand it is that the HW is trying to access
>> (locally via send) a MR which was already invalidated.
>> Thinking of this further, this can happen in a case where the target
>> already completed the transaction, sent SEND_WITH_INVALIDATE but the
>> original send ack was lost somewhere causing the device to retransmit
>> from the MR (which was already invalidated). This is highly unlikely
>> though.
>> Shouldn't this be protected somehow by the device?
>> Can someone explain why the above cannot happen? Jason? Liran? Anyone?
>> Say host register MR (a) and send (1) from that MR to a target,
>> send (1) ack got lost, and the target issues SEND_WITH_INVALIDATE
>> on MR (a) and the host HCA process it, then host HCA timeout on send (1)
>> so it retries, but ehh, its already invalidated.
> 
> Well, this entire flow is broken, why should the host send the MR rkey
> to the target if it is not using it for remote access, the target
> should never have a chance to remote invalidate something it did not
> access.
> 
> I think we have a bug in iSER code, as we should not send the key
> for remote invalidation if we do inline data send...
> 
> Robert, can you try the following:
> --
> diff --git a/drivers/infiniband/ulp/iser/iser_initiator.c b/drivers/infiniband/ulp/iser/iser_initiator.c
> index 12ed62ce9ff7..2a07692007bd 100644
> --- a/drivers/infiniband/ulp/iser/iser_initiator.c
> +++ b/drivers/infiniband/ulp/iser/iser_initiator.c
> @@ -137,8 +137,10 @@ iser_prepare_write_cmd(struct iscsi_task *task,
> 
>        if (unsol_sz < edtl) {
>                hdr->flags     |= ISER_WSV;
> -               hdr->write_stag = cpu_to_be32(mem_reg->rkey);
> -               hdr->write_va   = cpu_to_be64(mem_reg->sge.addr + unsol_sz);
> +               if (buf_out->data_len > imm_sz) {
> +                       hdr->write_stag = cpu_to_be32(mem_reg->rkey);
> +                       hdr->write_va = cpu_to_be64(mem_reg->sge.addr + unsol_sz);
> +               }
> 
>                iser_dbg("Cmd itt:%d, WRITE tags, RKEY:%#.4X "
>                         "VA:%#llX + unsol:%d\n",
> --
> 
> Although, I still don't think its enough. We need to delay the
> local invalidate till we received a send completion (guarantees
> that ack was received)...
> 
> If this indeed the case, _all_ ULP initiator drivers share it because we
> never condition on a send completion in order to complete an I/O, and
> in the case of lost ack on send, looks like we need to... It *will* hurt
> performance.
> 
> What do other folks think?
> 
> CC'ing Bart, Chuck, Christoph.
> 
> Guys, for summary, I think we might have a broken behavior in the
> initiator mode drivers. We never condition send completions (for
> requests) before we complete an I/O. The issue is that the ack for those
> sends might get lost, which means that the HCA will retry them (dropped
> by the peer HCA) but if we happen to complete the I/O before, either we
> can unmap the request area, or for inline data, we invalidate it (so the
> HCA will try to access a MR which was invalidated).

So on occasion there is a Remote Access Error. That would
trigger connection loss, and the retransmitted Send request
is discarded (if there was externally exposed memory involved
with the original transaction that is now invalid).

NFS has a duplicate replay cache. If it sees a repeated RPC
XID it will send a cached reply. I guess the trick there is
to squelch remote invalidation for such retransmits to avoid
spurious Remote Access Errors. Should be rare, though.

RPC-over-RDMA uses persistent registration for its inline
buffers. The problem there is avoiding buffer reuse to soon.
Otherwise a garbled inline message is presented on retransmit.
Those would probably not be caught by the DRC.

But the real problem is preventing retransmitted Sends from
causing a ULP request to be executed multiple times.


> Signalling all send completions and also finishing I/Os only after we
> got them will add latency, and that sucks...

Typically, Sends will complete before the response arrives.
The additional cost will be handling extra interrupts, IMO.

With FRWR, won't subsequent WRs be delayed until the HCA is
done with the Send? I don't think a signal is necessary in
every case. Send Queue accounting currently relies on that.

RPC-over-RDMA relies on the completion of Local Invalidation
to ensure that the initial Send WR is complete. For Remote
Invalidation and pure inline, there is nothing to fence that
Send.

The question I have is: how often do these Send retransmissions
occur? Is it enough to have a robust recovery mechanism, or
do we have to wire in assumptions about retransmission to
every Send operation?


--
Chuck Lever



--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Unexpected issues with 2 NVME initiators using the same target

[Sagi Grimberg]

> So on occasion there is a Remote Access Error. That would
> trigger connection loss, and the retransmitted Send request
> is discarded (if there was externally exposed memory involved
> with the original transaction that is now invalid).

I'm actually not concerned about the remote invalidation, that
is good that its discarded or failed. Its the inline sends that
are a bug here.

> But the real problem is preventing retransmitted Sends from
> causing a ULP request to be executed multiple times.

Exactly.

>> Signalling all send completions and also finishing I/Os only after we
>> got them will add latency, and that sucks...
> 
> Typically, Sends will complete before the response arrives.
> The additional cost will be handling extra interrupts, IMO.

Not quite, heavy traffic _can_ results in dropped acks, my gut
feeling is that it can happen more than we suspect.

and yea, extra interrupt, extra cachelines, extra state,
but I do not see any other way around it.

> With FRWR, won't subsequent WRs be delayed until the HCA is
> done with the Send? I don't think a signal is necessary in
> every case. Send Queue accounting currently relies on that.

Not really, the Send after the FRWR might have a fence (not strong
ordered one) and CX3/CX4 strong order FRWR so for them that is a
non-issue. The problem is that ULPs can't rely on it.

> RPC-over-RDMA relies on the completion of Local Invalidation
> to ensure that the initial Send WR is complete.

Wait, is that guaranteed?

> For Remote
> Invalidation and pure inline, there is nothing to fence that
> Send.
> 
> The question I have is: how often do these Send retransmissions
> occur? Is it enough to have a robust recovery mechanism, or
> do we have to wire in assumptions about retransmission to
> every Send operation?

Even if its rare, we don't have any way to protect against devices
retrying the send operation.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Unexpected issues with 2 NVME initiators using the same target

[Jason Gunthorpe]

On Tue, Jun 20, 2017 at 01:01:39PM -0400, Chuck Lever wrote:

> >> Shouldn't this be protected somehow by the device?
> >> Can someone explain why the above cannot happen? Jason? Liran? Anyone?
> >> Say host register MR (a) and send (1) from that MR to a target,
> >> send (1) ack got lost, and the target issues SEND_WITH_INVALIDATE
> >> on MR (a) and the host HCA process it, then host HCA timeout on send (1)
> >> so it retries, but ehh, its already invalidated.

I'm not sure I understand the example.. but...

If you pass a MR key to a send, then that MR must remain valid until
the send completion is implied by an observation on the CQ. The HCA is
free to re-execute the SEND against the MR at any time up until the
completion reaches the CQ.

As I've explained before, a ULP must not use 'implied completion', eg
a receive that could only have happened if the far side got the
send. In particular this means it cannot use an incoming SEND_INV/etc
to invalidate an MR associated with a local SEND, as that is a form
of 'implied completion'

For sanity a MR associated with a local send should not be remote
accessible at all, and shouldn't even have a 'rkey', just a 'lkey'.

Similarly, you cannot use a MR with SEND and remote access sanely, as
the far end could corrupt or invalidate the MR while the local HCA is
still using it.

> So on occasion there is a Remote Access Error. That would
> trigger connection loss, and the retransmitted Send request
> is discarded (if there was externally exposed memory involved
> with the original transaction that is now invalid).

Once you get a connection loss I would think the state of all the MRs
need to be resync'd. Running through the CQ should indicate which ones
are invalidate and which ones are still good.

> NFS has a duplicate replay cache. If it sees a repeated RPC
> XID it will send a cached reply. I guess the trick there is
> to squelch remote invalidation for such retransmits to avoid
> spurious Remote Access Errors. Should be rare, though.

.. and because of the above if a RPC is re-issued it must be re-issued
with corrected, now-valid rkeys, and the sender must somehow detect
that the far side dropped it for replay and tear down the MRs.

> RPC-over-RDMA uses persistent registration for its inline
> buffers. The problem there is avoiding buffer reuse to soon.
> Otherwise a garbled inline message is presented on retransmit.
> Those would probably not be caught by the DRC.

We've had this discussion on the list before. You can *never* re-use a
SEND, or RDMA WRITE buffer until you observe the HCA is done with it
via a CQ poll.

> But the real problem is preventing retransmitted Sends from
> causing a ULP request to be executed multiple times.

IB RC guarentees single delivery for SEND, so that doesn't seem
possible unless the ULP re-transmits the SEND on a new QP.

> > Signalling all send completions and also finishing I/Os only after
> > we got them will add latency, and that sucks...

There is no choice, you *MUST* see the send completion before
reclamining any resources associated with the send. Only the
completion guarentees that the HCA will not resend the packet or
otherwise continue to use the resources.

> With FRWR, won't subsequent WRs be delayed until the HCA is
> done with the Send? I don't think a signal is necessary in
> every case. Send Queue accounting currently relies on that.

No. The SQ side is asynchronous to the CQ side, the HCA will pipeline
send packets on the wire up to some internal limit.

Only the local state changed by FRWR related op codes happens
sequentially with other SQ work.

Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Unexpected issues with 2 NVME initiators using the same target

[Chuck Lever]

> On Jun 20, 2017, at 1:35 PM, Jason Gunthorpe <jgunthorpe-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org> wrote:
> 
> On Tue, Jun 20, 2017 at 01:01:39PM -0400, Chuck Lever wrote:
> 
>>>> Shouldn't this be protected somehow by the device?
>>>> Can someone explain why the above cannot happen? Jason? Liran? Anyone?
>>>> Say host register MR (a) and send (1) from that MR to a target,
>>>> send (1) ack got lost, and the target issues SEND_WITH_INVALIDATE
>>>> on MR (a) and the host HCA process it, then host HCA timeout on send (1)
>>>> so it retries, but ehh, its already invalidated.
> 
> I'm not sure I understand the example.. but...
> 
> If you pass a MR key to a send, then that MR must remain valid until
> the send completion is implied by an observation on the CQ. The HCA is
> free to re-execute the SEND against the MR at any time up until the
> completion reaches the CQ.
> 
> As I've explained before, a ULP must not use 'implied completion', eg
> a receive that could only have happened if the far side got the
> send. In particular this means it cannot use an incoming SEND_INV/etc
> to invalidate an MR associated with a local SEND, as that is a form
> of 'implied completion'
> 
> For sanity a MR associated with a local send should not be remote
> accessible at all, and shouldn't even have a 'rkey', just a 'lkey'.
> 
> Similarly, you cannot use a MR with SEND and remote access sanely, as
> the far end could corrupt or invalidate the MR while the local HCA is
> still using it.
> 
>> So on occasion there is a Remote Access Error. That would
>> trigger connection loss, and the retransmitted Send request
>> is discarded (if there was externally exposed memory involved
>> with the original transaction that is now invalid).
> 
> Once you get a connection loss I would think the state of all the MRs
> need to be resync'd. Running through the CQ should indicate which ones
> are invalidate and which ones are still good.
> 
>> NFS has a duplicate replay cache. If it sees a repeated RPC
>> XID it will send a cached reply. I guess the trick there is
>> to squelch remote invalidation for such retransmits to avoid
>> spurious Remote Access Errors. Should be rare, though.
> 
> .. and because of the above if a RPC is re-issued it must be re-issued
> with corrected, now-valid rkeys, and the sender must somehow detect
> that the far side dropped it for replay and tear down the MRs.

Yes, if RPC-over-RDMA ULP is involved, any externally accessible
memory will be re-registered before an RPC retransmission.

The concern is whether a retransmitted Send will be exposed
to the receiving ULP. Below you imply that it will not be, so
perhaps this is not a concern after all.


>> RPC-over-RDMA uses persistent registration for its inline
>> buffers. The problem there is avoiding buffer reuse to soon.
>> Otherwise a garbled inline message is presented on retransmit.
>> Those would probably not be caught by the DRC.
> 
> We've had this discussion on the list before. You can *never* re-use a
> SEND, or RDMA WRITE buffer until you observe the HCA is done with it
> via a CQ poll.

RPC-over-RDMA is careful to invalidate buffers that are the
target of RDMA Write before RPC completion, as we have
discussed before.

Sends are assumed to be complete when a LocalInv completes.

When we had this discussion before, you explained the problem
with retransmitted Sends, but it appears that all the ULPs we
have operate without Send completion. Others whom I trust have
suggested that operating without that extra interrupt is
preferred. The client has operated this way since it was added
to the kernel almost 10 years ago.

So I took it as a "in a perfect world" kind of admonition.
You are making a stronger and more normative assertion here.


>> But the real problem is preventing retransmitted Sends from
>> causing a ULP request to be executed multiple times.
> 
> IB RC guarentees single delivery for SEND, so that doesn't seem
> possible unless the ULP re-transmits the SEND on a new QP.
> 
>>> Signalling all send completions and also finishing I/Os only after
>>> we got them will add latency, and that sucks...
> 
> There is no choice, you *MUST* see the send completion before
> reclamining any resources associated with the send. Only the
> completion guarentees that the HCA will not resend the packet or
> otherwise continue to use the resources.

On the NFS server side, I believe every Send is signaled.

On the NFS client side, we assume LocalInv completion is
good enough.


>> With FRWR, won't subsequent WRs be delayed until the HCA is
>> done with the Send? I don't think a signal is necessary in
>> every case. Send Queue accounting currently relies on that.
> 
> No. The SQ side is asynchronous to the CQ side, the HCA will pipeline
> send packets on the wire up to some internal limit.

So if my ULP issues FastReg followed by Send followed by
LocalInv (signaled), I can't rely on the LocalInv completion
to imply that the Send is also complete?


> Only the local state changed by FRWR related op codes happens
> sequentially with other SQ work.


--
Chuck Lever



--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Unexpected issues with 2 NVME initiators using the same target

[Jason Gunthorpe]

On Tue, Jun 20, 2017 at 02:17:39PM -0400, Chuck Lever wrote:

> The concern is whether a retransmitted Send will be exposed
> to the receiving ULP. Below you imply that it will not be, so
> perhaps this is not a concern after all.

A retransmitted SEND will never be exposed to the Reciever ULP for
Reliable Connected. That is part of the guarantee.

> > We've had this discussion on the list before. You can *never* re-use a
> > SEND, or RDMA WRITE buffer until you observe the HCA is done with it
> > via a CQ poll.
> 
> RPC-over-RDMA is careful to invalidate buffers that are the
> target of RDMA Write before RPC completion, as we have
> discussed before.
> 
> Sends are assumed to be complete when a LocalInv completes.
> 
> When we had this discussion before, you explained the problem
> with retransmitted Sends, but it appears that all the ULPs we
> have operate without Send completion. Others whom I trust have
> suggested that operating without that extra interrupt is

Operating without the interrupt is of course preferred, but that means
you have to defer the invalidate for MR's refered to by SEND until a
CQ observation as well.

> preferred. The client has operated this way since it was added
> to the kernel almost 10 years ago.

I thought the use of MR's with SEND was a new invention? If you use
the local rdma lkey with send, it is never invalidated, and this is
not an issue, which IIRC, was the historical configuration for NFS.

> So I took it as a "in a perfect world" kind of admonition.
> You are making a stronger and more normative assertion here.

All ULPs must have periodic (related to SQ depth) signaled completions
or some of our supported hardware will explode.

All ULPs must flow control additions to the SQ based on CQ feedback,
or they will fail under load with SQ overflows, if this is done, then
the above happens correctly for free.

All ULPs must ensure SEND/RDMA Write resources remain stable until the
CQ indicates that work is completed. 'In a perfect world' this
includes not changing the source memory as that would cause
retransmitted packets to be different.

All ULPs must ensure the lkey remains valid until the CQ confirms
the work is done. This is not important if the lkey is always the
local rdma lkey, which is always valid.

> > No. The SQ side is asynchronous to the CQ side, the HCA will pipeline
> > send packets on the wire up to some internal limit.
> 
> So if my ULP issues FastReg followed by Send followed by
> LocalInv (signaled), I can't rely on the LocalInv completion
> to imply that the Send is also complete?

Correct.

This is explicitly defined in Table 79 of the IBA.

It describes the ordering requirements, if you order Send followed by
LocalInv the ordering is 'L' which means they are not ordered unless
the WR has the Local Invalidate Fence bit set.

LIF is an optional feature, I do not know if any of our hardware
supports it, but it is defined to cause the local invalidate to wait
until all ongoing references to the MR are completed.

No idea on the relative performance of LIF vs doing it manually, but
the need for one or the other is unambigously clear in the spec.

Why are you invaliding lkeys anyhow, that doesn't seem like something
that needs to happen synchronously.

Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Unexpected issues with 2 NVME initiators using the same target

[Chuck Lever]

> On Jun 20, 2017, at 3:27 PM, Jason Gunthorpe <jgunthorpe-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org> wrote:
> 
> On Tue, Jun 20, 2017 at 02:17:39PM -0400, Chuck Lever wrote:
> 
>> The concern is whether a retransmitted Send will be exposed
>> to the receiving ULP. Below you imply that it will not be, so
>> perhaps this is not a concern after all.
> 
> A retransmitted SEND will never be exposed to the Reciever ULP for
> Reliable Connected. That is part of the guarantee.
> 
>>> We've had this discussion on the list before. You can *never* re-use a
>>> SEND, or RDMA WRITE buffer until you observe the HCA is done with it
>>> via a CQ poll.
>> 
>> RPC-over-RDMA is careful to invalidate buffers that are the
>> target of RDMA Write before RPC completion, as we have
>> discussed before.
>> 
>> Sends are assumed to be complete when a LocalInv completes.
>> 
>> When we had this discussion before, you explained the problem
>> with retransmitted Sends, but it appears that all the ULPs we
>> have operate without Send completion. Others whom I trust have
>> suggested that operating without that extra interrupt is
> 
> Operating without the interrupt is of course preferred, but that means
> you have to defer the invalidate for MR's refered to by SEND until a
> CQ observation as well.
> 
>> preferred. The client has operated this way since it was added
>> to the kernel almost 10 years ago.
> 
> I thought the use of MR's with SEND was a new invention? If you use
> the local rdma lkey with send, it is never invalidated, and this is
> not an issue, which IIRC, was the historical configuration for NFS.

We may be conflating things a bit.

RPC-over-RDMA client uses persistently registered buffers, using
the lkey, for inline data. The use of MRs is reserved for NFS READ
and WRITE payloads. The inline buffers are never explicitly
invalidated by RPC-over-RDMA.


>> So I took it as a "in a perfect world" kind of admonition.
>> You are making a stronger and more normative assertion here.
> 
> All ULPs must have periodic (related to SQ depth) signaled completions
> or some of our supported hardware will explode.

RPC-over-RDMA client does that.


> All ULPs must flow control additions to the SQ based on CQ feedback,
> or they will fail under load with SQ overflows, if this is done, then
> the above happens correctly for free.

RPC-over-RDMA client does that.


> All ULPs must ensure SEND/RDMA Write resources remain stable until the
> CQ indicates that work is completed. 'In a perfect world' this
> includes not changing the source memory as that would cause
> retransmitted packets to be different.

I assume you mean the sending side (the server) for RDMA
Write. I believe rdma_rw uses the local rdma lkey by default
for RDMA Write source buffers.


> All ULPs must ensure the lkey remains valid until the CQ confirms
> the work is done. This is not important if the lkey is always the
> local rdma lkey, which is always valid.

As above, Send buffers use the local rdma lkey.


>>> No. The SQ side is asynchronous to the CQ side, the HCA will pipeline
>>> send packets on the wire up to some internal limit.
>> 
>> So if my ULP issues FastReg followed by Send followed by
>> LocalInv (signaled), I can't rely on the LocalInv completion
>> to imply that the Send is also complete?
> 
> Correct.
> 
> This is explicitly defined in Table 79 of the IBA.
> 
> It describes the ordering requirements, if you order Send followed by
> LocalInv the ordering is 'L' which means they are not ordered unless
> the WR has the Local Invalidate Fence bit set.
> 
> LIF is an optional feature, I do not know if any of our hardware
> supports it, but it is defined to cause the local invalidate to wait
> until all ongoing references to the MR are completed.

Now, since there was confusion about using an MR for a
Send operation, let me clarify. If the client does:

FastReg(payload buffer)
Send(inline buffer)
...
Recv
LocalInv(payload buffer)
wait for LI completion

Is setting IB_SEND_FENCE on the LocalInv enough to ensure
that the Send is complete?

cscope seems to suggest all our devices support IB_SEND_FENCE.
Sagi mentioned some devices do this fencing automatically.


> No idea on the relative performance of LIF vs doing it manually, but
> the need for one or the other is unambigously clear in the spec.

It seems to me that the guarantee that the server sees
only one copy of the Send payload is good enough. That
means that by the time Recv completion occurs on the
client, even if the client HCA still thinks it needs to
retransmit the Send containing the RPC Call, the server
ULP has already seen and processed that Send payload,
and the HCA on the server won't deliver that payload a
second time.

The RPC Reply is evidence that the server saw the correct
RPC Call message payload, and the client always preserves
the Send's inline buffer until the reply has been received.

If the only concern about preserving that inline buffer is
guaranteeing that retransmits contain the same content, I
don't think we have a problem. All HCA retransmits of an
RPC Call, until the matching RPC Reply is received on the
client, will contain the same content.

The issue about the HCA not being able to access the inline
buffer during a retransmit is also not an issue for RPC-
over-RDMA because these buffers are always registered with
the local rdma lkey.


> Why are you invaliding lkeys anyhow, that doesn't seem like something
> that needs to happen synchronously.


--
Chuck Lever



--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Unexpected issues with 2 NVME initiators using the same target

[Jason Gunthorpe]

On Tue, Jun 20, 2017 at 04:56:39PM -0400, Chuck Lever wrote:

> > I thought the use of MR's with SEND was a new invention? If you use
> > the local rdma lkey with send, it is never invalidated, and this is
> > not an issue, which IIRC, was the historical configuration for NFS.
> 
> We may be conflating things a bit.
> 
> RPC-over-RDMA client uses persistently registered buffers, using
> the lkey, for inline data. The use of MRs is reserved for NFS READ
> and WRITE payloads. The inline buffers are never explicitly
> invalidated by RPC-over-RDMA.

That makes much more sense, but is that the original question in this
thread? Why are we even talking about invalidate ordering then?

> > All ULPs must ensure SEND/RDMA Write resources remain stable until the
> > CQ indicates that work is completed. 'In a perfect world' this
> > includes not changing the source memory as that would cause
> > retransmitted packets to be different.
> 
> I assume you mean the sending side (the server) for RDMA
> Write. I believe rdma_rw uses the local rdma lkey by default
> for RDMA Write source buffers.

RDMA Write or SEND

> >>> No. The SQ side is asynchronous to the CQ side, the HCA will pipeline
> >>> send packets on the wire up to some internal limit.
> >> 
> >> So if my ULP issues FastReg followed by Send followed by
> >> LocalInv (signaled), I can't rely on the LocalInv completion
> >> to imply that the Send is also complete?
> > 
> > Correct.
> > 
> > This is explicitly defined in Table 79 of the IBA.
> > 
> > It describes the ordering requirements, if you order Send followed by
> > LocalInv the ordering is 'L' which means they are not ordered unless
> > the WR has the Local Invalidate Fence bit set.
> > 
> > LIF is an optional feature, I do not know if any of our hardware
> > supports it, but it is defined to cause the local invalidate to wait
> > until all ongoing references to the MR are completed.
> 
> Now, since there was confusion about using an MR for a
> Send operation, let me clarify. If the client does:

> FastReg(payload buffer)
> Send(inline buffer)
> ...
> Recv
> LocalInv(payload buffer)
> wait for LI completion

Not sure what you are describing?

Is Recv landing memory for a SEND? In that case it is using a lkey,
lkeys are not remotely usable, so it does not need synchronous
invalidation. In all cases the LocalInv must only be posted once a CQE
for the Recv is observed.

If Recv is RDMA WRITE target memory, then it using the rkey and it
does does need synchronous invalidation. This must be done once a recv
CQE is observed, or optimized by having the other send via one of the
_INV operations.

In no case can you pipeline a LocalInv into the SQ that would impact
RQ activity, even with any of the fences.

> Is setting IB_SEND_FENCE on the LocalInv enough to ensure
> that the Send is complete?

No.

There are two fences in the spec, IB_SEND_FENCE is the mandatory one,
and it only interacts with RDMA READ and ATOMIC entries.

Local Invalidate Fence (the optinal one) also will not order the two
because LIF is only defined to order against SQE's that use the
MR. Since Send is using the global dma lkey it does not interact with
the LocalInv and LIF will not order them.

> > No idea on the relative performance of LIF vs doing it manually, but
> > the need for one or the other is unambigously clear in the spec.
> 
> It seems to me that the guarantee that the server sees
> only one copy of the Send payload is good enough. That
> means that by the time Recv completion occurs on the
> client, even if the client HCA still thinks it needs to
> retransmit the Send containing the RPC Call, the server
> ULP has already seen and processed that Send payload,
> and the HCA on the server won't deliver that payload a
> second time.

Yes, that is OK reasoning.

> If the only concern about preserving that inline buffer is
> guaranteeing that retransmits contain the same content, I
> don't think we have a problem. All HCA retransmits of an
> RPC Call, until the matching RPC Reply is received on the
> client, will contain the same content.

Right.

> The issue about the HCA not being able to access the inline
> buffer during a retransmit is also not an issue for RPC-
> over-RDMA because these buffers are always registered with
> the local rdma lkey.

Exactly.

Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Unexpected issues with 2 NVME initiators using the same target

[Sagi Grimberg]

Jason,

>> The issue about the HCA not being able to access the inline
>> buffer during a retransmit is also not an issue for RPC-
>> over-RDMA because these buffers are always registered with
>> the local rdma lkey.
> 
> Exactly.

Lost track of the thread...


Indeed you raised this issue lots of times before, and I failed to see
why its important or why its error prone, but now I do...

My apologies for not listening :(

We should fix _all_ initiators for it, nvme-rdma, iser, srp
and xprtrdma (and probably some more ULPs out there)...

It also means that we cannot really suppress any send completions as
that would result in an unpredictable latency (which is not acceptable).

I wish we could somehow tell the HCA that it can ignore access fail to a
specific address when retransmitting.. but maybe its too much to ask...
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Unexpected issues with 2 NVME initiators using the same target

[Chuck Lever]

> On Jun 27, 2017, at 3:37 AM, Sagi Grimberg <sagi-NQWnxTmZq1alnMjI0IkVqw@public.gmane.org> wrote:
> 
> Jason,
> 
>>> The issue about the HCA not being able to access the inline
>>> buffer during a retransmit is also not an issue for RPC-
>>> over-RDMA because these buffers are always registered with
>>> the local rdma lkey.
>> Exactly.
> 
> Lost track of the thread...
> 
> 
> Indeed you raised this issue lots of times before, and I failed to see
> why its important or why its error prone, but now I do...
> 
> My apologies for not listening :(
> 
> We should fix _all_ initiators for it, nvme-rdma, iser, srp
> and xprtrdma (and probably some more ULPs out there)...

Go back and browse the end of the thread: there's no need to change
xprtrdma, and maybe no need to change the others either.


> It also means that we cannot really suppress any send completions as
> that would result in an unpredictable latency (which is not acceptable).
> 
> I wish we could somehow tell the HCA that it can ignore access fail to a
> specific address when retransmitting.. but maybe its too much to ask...

--
Chuck Lever



--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Unexpected issues with 2 NVME initiators using the same target

[Sagi Grimberg]

> Go back and browse the end of the thread: there's no need to change
> xprtrdma, and maybe no need to change the others either.

I think there is, even with inline, xprtrdma dma maps the immediate
buffers, also the message head and tail so unmapping these buffers
without waiting for the send completion would trigger a IOMMU access
error (the HCA (re)tries to access an already unmapped buffer).
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Unexpected issues with 2 NVME initiators using the same target

[Jason Gunthorpe]

On Tue, Jun 27, 2017 at 07:07:08PM +0300, Sagi Grimberg wrote:
> 
> >Go back and browse the end of the thread: there's no need to change
> >xprtrdma, and maybe no need to change the others either.
> 
> I think there is, even with inline, xprtrdma dma maps the immediate
> buffers, also the message head and tail so unmapping these buffers
> without waiting for the send completion would trigger a IOMMU access
> error (the HCA (re)tries to access an already unmapped buffer).

Yes, that is an excellent observation. When using the local rdma lkey
you still need to ensure the linux API DMA map remains until
completion.

send completion mitigation is still possible, if it is OK for the
backing pages to remain, but I think a more sophisticated strategy is
needed - eg maybe push some kind of NOP through the send q after a
timer or only complete when the last available work is stuffed or
something.

Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Unexpected issues with 2 NVME initiators using the same target

[Sagi Grimberg]

> send completion mitigation is still possible, if it is OK for the
> backing pages to remain, but I think a more sophisticated strategy is
> needed - eg maybe push some kind of NOP through the send q after a
> timer or only complete when the last available work is stuffed or
> something.

I'm not smart enough to come up with something like that.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Unexpected issues with 2 NVME initiators using the same target

[Chuck Lever]

> On Jun 27, 2017, at 12:07 PM, Sagi Grimberg <sagi-NQWnxTmZq1alnMjI0IkVqw@public.gmane.org> wrote:
> 
> 
>> Go back and browse the end of the thread: there's no need to change
>> xprtrdma, and maybe no need to change the others either.
> 
> I think there is, even with inline, xprtrdma dma maps the immediate
> buffers, also the message head and tail so unmapping these buffers
> without waiting for the send completion would trigger a IOMMU access
> error (the HCA (re)tries to access an already unmapped buffer).

Thinking out loud:

IIRC the message head and tail reside in the persistently registered
and DMA mapped buffers with few exceptions.

However, when page cache pages are involved, xprtrdma will do a DMA
unmap as you say.

So:
- we don't have a problem transmitting a garbled request thanks to
  exactly-once receive semantics
- we don't have a problem with the timing of registration and
  invalidation on the initiator because the PD's DMA lkey is used
- we do have a problem with DMA unmap

Using only persistently registered and DMA mapped Send buffers
should avoid the need to signal all Sends. However, when page
cache pages are involved, then the Send needs to be signaled,
and the pages unmapped only after Send completion, to be completely
safe.

Or, the initiator should just use RDMA Read and Write, and stick
with small inline sizes.


--
Chuck Lever



--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Unexpected issues with 2 NVME initiators using the same target

[Sagi Grimberg]

> Thinking out loud:
> 
> IIRC the message head and tail reside in the persistently registered
> and DMA mapped buffers with few exceptions.
> 
> However, when page cache pages are involved, xprtrdma will do a DMA
> unmap as you say.
> 
> So:
> - we don't have a problem transmitting a garbled request thanks to
>    exactly-once receive semantics
> - we don't have a problem with the timing of registration and
>    invalidation on the initiator because the PD's DMA lkey is used
> - we do have a problem with DMA unmap
> 
> Using only persistently registered and DMA mapped Send buffers
> should avoid the need to signal all Sends. However, when page
> cache pages are involved, then the Send needs to be signaled,
> and the pages unmapped only after Send completion, to be completely
> safe.

How do you know when that happens?
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Unexpected issues with 2 NVME initiators using the same target

[Chuck Lever]

> On Jun 28, 2017, at 3:08 AM, Sagi Grimberg <sagi-NQWnxTmZq1alnMjI0IkVqw@public.gmane.org> wrote:
> 
> 
>> Thinking out loud:
>> IIRC the message head and tail reside in the persistently registered
>> and DMA mapped buffers with few exceptions.
>> However, when page cache pages are involved, xprtrdma will do a DMA
>> unmap as you say.
>> So:
>> - we don't have a problem transmitting a garbled request thanks to
>>   exactly-once receive semantics
>> - we don't have a problem with the timing of registration and
>>   invalidation on the initiator because the PD's DMA lkey is used
>> - we do have a problem with DMA unmap
>> Using only persistently registered and DMA mapped Send buffers
>> should avoid the need to signal all Sends. However, when page
>> cache pages are involved, then the Send needs to be signaled,
>> and the pages unmapped only after Send completion, to be completely
>> safe.
> 
> How do you know when that happens?

The RPC Call send path sets up the Send SGE array. If it includes
page cache pages, it can set IB_SEND_SIGNALED.

The SGE array and the ib_cqe for the send are in the same data
structure, so the Send completion handler can find the SGE array
and figure out what needs to be unmapped.

The only problem is if a POSIX signal fires. In that case the
data structure can be released before the Send completion fires,
and we get touch-after-free in the completion handler.

I'm thinking that it just isn't going to be practical to handle
unmapping this way, and I should just revert back to using RDMA
Read instead of adding page cache pages to the Send SGE.


--
Chuck Lever



--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Unexpected issues with 2 NVME initiators using the same target

[Sagi Grimberg]

>> How do you know when that happens?
> 
> The RPC Call send path sets up the Send SGE array. If it includes
> page cache pages, it can set IB_SEND_SIGNALED.
> 
> The SGE array and the ib_cqe for the send are in the same data
> structure, so the Send completion handler can find the SGE array
> and figure out what needs to be unmapped.
> 
> The only problem is if a POSIX signal fires. In that case the
> data structure can be released before the Send completion fires,
> and we get touch-after-free in the completion handler.
> 
> I'm thinking that it just isn't going to be practical to handle
> unmapping this way, and I should just revert back to using RDMA
> Read instead of adding page cache pages to the Send SGE.

Or wait for the send completion before completing the I/O?
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Unexpected issues with 2 NVME initiators using the same target

[Chuck Lever]

> On Jun 29, 2017, at 1:35 AM, Sagi Grimberg <sagi-NQWnxTmZq1alnMjI0IkVqw@public.gmane.org> wrote:
> 
> 
>>> How do you know when that happens?
>> The RPC Call send path sets up the Send SGE array. If it includes
>> page cache pages, it can set IB_SEND_SIGNALED.
>> The SGE array and the ib_cqe for the send are in the same data
>> structure, so the Send completion handler can find the SGE array
>> and figure out what needs to be unmapped.
>> The only problem is if a POSIX signal fires. In that case the
>> data structure can be released before the Send completion fires,
>> and we get touch-after-free in the completion handler.
>> I'm thinking that it just isn't going to be practical to handle
>> unmapping this way, and I should just revert back to using RDMA
>> Read instead of adding page cache pages to the Send SGE.
> 
> Or wait for the send completion before completing the I/O?

In the normal case, that works.

If a POSIX signal occurs (^C, RPC timeout), the RPC exits immediately
and recovers all resources. The Send can still be running at that
point, and it can't be stopped (without transitioning the QP to
error state, I guess).

The alternative is reference-counting the data structure that has
the ib_cqe and the SGE array. That adds one or more atomic_t
operations per I/O that I'd like to avoid.


--
Chuck Lever



--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Unexpected issues with 2 NVME initiators using the same target

[Sagi Grimberg]

>> Or wait for the send completion before completing the I/O?
> 
> In the normal case, that works.
> 
> If a POSIX signal occurs (^C, RPC timeout), the RPC exits immediately
> and recovers all resources. The Send can still be running at that
> point, and it can't be stopped (without transitioning the QP to
> error state, I guess).

In that case we can't complete the I/O either (or move the
QP into error state), we need to defer/sleep on send completion.


> The alternative is reference-counting the data structure that has
> the ib_cqe and the SGE array. That adds one or more atomic_t
> operations per I/O that I'd like to avoid.

Why atomics?
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Unexpected issues with 2 NVME initiators using the same target

[Chuck Lever]

> On Jul 2, 2017, at 5:45 AM, Sagi Grimberg <sagi-NQWnxTmZq1alnMjI0IkVqw@public.gmane.org> wrote:
> 
> 
>>> Or wait for the send completion before completing the I/O?
>> In the normal case, that works.
>> If a POSIX signal occurs (^C, RPC timeout), the RPC exits immediately
>> and recovers all resources. The Send can still be running at that
>> point, and it can't be stopped (without transitioning the QP to
>> error state, I guess).
> 
> In that case we can't complete the I/O either (or move the
> QP into error state), we need to defer/sleep on send completion.

Unfortunately the RPC client finite state machine mustn't
sleep when a POSIX signal fires. xprtrdma has to unblock the
waiting application process but clean up the resources
asynchronously.

The RPC completion doesn't have to wait on DMA unmapping the
send buffer. What would have to wait is cleaning up the
resources -- in particular, allowing the rpcrdma_req
structure, where the send SGEs are kept, to be re-used. In
the current design, both happen at the same time.


>> The alternative is reference-counting the data structure that has
>> the ib_cqe and the SGE array. That adds one or more atomic_t
>> operations per I/O that I'd like to avoid.
> 
> Why atomics?

Either an atomic reference count or a spin lock is necessary
because there are two different ways an RPC can exit:

1. The common way, which is through receipt of an RPC reply,
handled by rpcrdma_reply_handler.

2. POSIX signal, where the RPC reply races with the wake-up
of the application process (in other words, the reply can
still arrive while the RPC is terminating).

In both cases, the RPC client has to invalidate any
registered memory, and it has to be done no more and no less
than once.

I deal with some of this in my for-13 patches:

http://marc.info/?l=linux-nfs&m=149693711119727&w=2

The first seven patches handle the race condition and the
need for exactly-once invalidation.

But the issue with unmapping the Send buffers has to do
with how the Send SGEs are managed. The data structure
containing the SGEs goes away once the RPC is complete.

So there are two "users": one is the RPC completion, and
one is the Send completion. Once both are done, the data
structure can be released. But RPC completion can't wait
if the Send completion hasn't yet fired.

I could kmalloc the SGE array instead, signal each Send,
and then in the Send completion handler, unmap the SGEs
and then kfree the SGE array. That's a lot of overhead.

Or I could revert all the "map page cache pages" logic and
just use memcpy for small NFS WRITEs, and RDMA the rest of
the time. That keeps everything simple, but means large
inline thresholds can't use send-in-place.

I'm still open to suggestion. for-4.14 will deal with other
problems, unless an obvious and easy fix arises.


--
Chuck Lever



--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Unexpected issues with 2 NVME initiators using the same target

[Jason Gunthorpe]

On Sun, Jul 02, 2017 at 02:17:52PM -0400, Chuck Lever wrote:

> I could kmalloc the SGE array instead, signal each Send,
> and then in the Send completion handler, unmap the SGEs
> and then kfree the SGE array. That's a lot of overhead.

Usually after allocating the send queue you'd pre-allocate all the
tracking memory needed for each SQE - eg enough information to do the
dma unmaps/etc?

> Or I could revert all the "map page cache pages" logic and
> just use memcpy for small NFS WRITEs, and RDMA the rest of
> the time. That keeps everything simple, but means large
> inline thresholds can't use send-in-place.

Don't you have the same problem with RDMA WRITE?

Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Unexpected issues with 2 NVME initiators using the same target

[Chuck Lever]

> On Jul 9, 2017, at 12:47 PM, Jason Gunthorpe <jgunthorpe-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org> wrote:
> 
> On Sun, Jul 02, 2017 at 02:17:52PM -0400, Chuck Lever wrote:
> 
>> I could kmalloc the SGE array instead, signal each Send,
>> and then in the Send completion handler, unmap the SGEs
>> and then kfree the SGE array. That's a lot of overhead.
> 
> Usually after allocating the send queue you'd pre-allocate all the
> tracking memory needed for each SQE - eg enough information to do the
> dma unmaps/etc?

Right. In xprtrdma, the QP resources are allocated in rpcrdma_ep_create.
For every RPC-over-RDMA credit, rpcrdma_buffer_create allocates an
rpcrdma_req structure, which contains an ib_cqe and an array of SGEs for
the Send, and a number of other resources used to maintain registration
state during an RPC-over-RDMA call. Both of these functions are invoked
during transport instance set-up.

The problem is the lifetime for the rpcrdma_req structure. Currently it
is acquired when an RPC is started, and it is released when the RPC
terminates.

Inline send buffers are never unmapped until transport tear-down, but
since:

commit 655fec6987be05964e70c2e2efcbb253710e282f
Author:     Chuck Lever <chuck.lever-QHcLZuEGTsvQT0dZR+AlfA@public.gmane.org>
AuthorDate: Thu Sep 15 10:57:24 2016 -0400
Commit:     Anna Schumaker <Anna.Schumaker-ZwjVKphTwtPQT0dZR+AlfA@public.gmane.org>
CommitDate: Mon Sep 19 13:08:38 2016 -0400

    xprtrdma: Use gathered Send for large inline messages

Part of the Send payload can come from page cache pages for NFS WRITE
and NFS SYMLINK operations. Send buffers that are page cache pages are
DMA unmapped when rpcrdma_req is released.

IIUC what Sagi found is that Send WRs can continue running even after an
RPC completes in certain pathological cases. Therefore the Send WR can
complete after the rpcrdma_req is released and page cache-related Send
buffers have been unmapped.

It's not an issue to make the RPC reply handler wait for Send completion.
In most cases, this is not going to add any additional latency because
the Send will complete long before the RPC reply arrives. By far the
common case, and that's an extra completion interrupt for nothing.

The problem arises if the RPC is terminated locally before the reply
arrives. Suppose, for example, user hits ^C, or a timer fires. Then the
rpcrdma_req can be released and re-used before the Send completes.
There's no way to make RPC completion wait for Send completion.

One option is to somehow split the Send-related data structures from
rpcrdma_req, and manage them independently. I've already done that for
MRs: MR state is now located in rpcrdma_mw.

If instead I just never DMA map page cache pages, then all Send buffers
are always left DMA mapped while the transport is active. There's no
problem there with Send retransmits. The overhead is that I have to
either copy data into the Send buffers, or force the server to use RDMA
Read, which has a palpable overhead.


>> Or I could revert all the "map page cache pages" logic and
>> just use memcpy for small NFS WRITEs, and RDMA the rest of
>> the time. That keeps everything simple, but means large
>> inline thresholds can't use send-in-place.
> 
> Don't you have the same problem with RDMA WRITE?

The server side initiates RDMA Writes. The final RDMA Write in a WR
chain is signaled, but a subsequent Send completion is used to
determine when the server may release resources used for the Writes.
We're already doing it the slow way there, and there's no ^C hazard
on the server.


--
Chuck Lever



--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Unexpected issues with 2 NVME initiators using the same target

[Jason Gunthorpe]

On Mon, Jul 10, 2017 at 03:03:18PM -0400, Chuck Lever wrote:

> One option is to somehow split the Send-related data structures from
> rpcrdma_req, and manage them independently. I've already done that for
> MRs: MR state is now located in rpcrdma_mw.

Yes, this is is what I was implying.. Track the SQE related stuff
seperately in memory allocated during SQ setup - MR, dma maps, etc.

No need for an atomic/lock then, right? The required memory is bounded
since the inline send depth is bounded.

Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Unexpected issues with 2 NVME initiators using the same target

[Chuck Lever]

> On Jul 10, 2017, at 4:05 PM, Jason Gunthorpe <jgunthorpe-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org> wrote:
> 
> On Mon, Jul 10, 2017 at 03:03:18PM -0400, Chuck Lever wrote:
> 
>> One option is to somehow split the Send-related data structures from
>> rpcrdma_req, and manage them independently. I've already done that for
>> MRs: MR state is now located in rpcrdma_mw.
> 
> Yes, this is is what I was implying.. Track the SQE related stuff
> seperately in memory allocated during SQ setup - MR, dma maps, etc.

> No need for an atomic/lock then, right? The required memory is bounded
> since the inline send depth is bounded.

Perhaps I lack some imagination, but I don't see how I can manage
these small objects without a serialized free list or circular
array that would be accessed in the forward path and also in a
Send completion handler.

And I would still have to signal all Sends, which is extra
interrupts and context switches.

This seems like a lot of overhead to deal with a very uncommon
case. I can reduce this overhead by signaling only Sends that
need to unmap page cache pages, but still.


But I also realized that Send Queue accounting can be broken by a
delayed Send completion.

As we previously discussed, xprtrdma does SQ accounting using RPC
completion as the gate. Basically xprtrdma will send another RPC
as soon as a previous one is terminated. If the Send WR is still
running when the RPC terminates, I can potentially overrun the
Send Queue.


--
Chuck Lever



--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Unexpected issues with 2 NVME initiators using the same target

[Jason Gunthorpe]

On Mon, Jul 10, 2017 at 04:51:20PM -0400, Chuck Lever wrote:
> 
> > On Jul 10, 2017, at 4:05 PM, Jason Gunthorpe <jgunthorpe-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org> wrote:
> > 
> > On Mon, Jul 10, 2017 at 03:03:18PM -0400, Chuck Lever wrote:
> > 
> >> One option is to somehow split the Send-related data structures from
> >> rpcrdma_req, and manage them independently. I've already done that for
> >> MRs: MR state is now located in rpcrdma_mw.
> > 
> > Yes, this is is what I was implying.. Track the SQE related stuff
> > seperately in memory allocated during SQ setup - MR, dma maps, etc.
> 
> > No need for an atomic/lock then, right? The required memory is bounded
> > since the inline send depth is bounded.
> 
> Perhaps I lack some imagination, but I don't see how I can manage
> these small objects without a serialized free list or circular
> array that would be accessed in the forward path and also in a
> Send completion handler.

I don't get it, dma unmap can only ever happen in the send completion
handler, it can never happen in the forward path. (this is the whole
point of this thread)

Since you are not using send completion today you can just use the
wr_id to point to the pre-allocated memory containing the pages to
invalidate. Completely remove dma unmap from the forward path.

Usually I work things out so that the meta-data array is a ring and
every SQE post consumes a meta-data entry. Then occasionally I signal
completion and provide a wr_id of the latest ring index and the
completion handler runs through all the accumulated meta-data and acts
on it (eg unmaps/etc). This approach still allows batching
completions.

Since ring entries are bounded size we just preallocate the largest
size at QP creation. In this case it is some multiple of the number of
inline send pages * number of SQE entries.

> This seems like a lot of overhead to deal with a very uncommon
> case. I can reduce this overhead by signaling only Sends that
> need to unmap page cache pages, but still.

Yes, but it is not avoidable..

> As we previously discussed, xprtrdma does SQ accounting using RPC
> completion as the gate. Basically xprtrdma will send another RPC
> as soon as a previous one is terminated. If the Send WR is still
> running when the RPC terminates, I can potentially overrun the
> Send Queue.

Makes sense. The SQ accounting must be precise.

Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Unexpected issues with 2 NVME initiators using the same target

[Jason Gunthorpe]

On Mon, Jul 10, 2017 at 03:03:18PM -0400, Chuck Lever wrote:

> >> Or I could revert all the "map page cache pages" logic and
> >> just use memcpy for small NFS WRITEs, and RDMA the rest of
> >> the time. That keeps everything simple, but means large
> >> inline thresholds can't use send-in-place.
> > 
> > Don't you have the same problem with RDMA WRITE?
> 
> The server side initiates RDMA Writes. The final RDMA Write in a WR
> chain is signaled, but a subsequent Send completion is used to
> determine when the server may release resources used for the Writes.
> We're already doing it the slow way there, and there's no ^C hazard
> on the server.

Wait, I guess I meant RDMA READ path.

The same contraints apply to RKeys as inline send - you cannot DMA
unmap rkey memory until the rkey is invalidated at the HCA.

So posting an invalidate SQE and then immediately unmapping the DMA
pages is bad too..

No matter how the data is transfered the unmapping must follow the
same HCA synchronous model.. DMA unmap must only be done from the send
completion handler (inline send or invalidate rkey), from the recv
completion handler (send with invalidate), or from QP error state teardown.

Anything that does DMA memory unmap from another thread is very, very
suspect, eg async from a ctrl-c trigger event.

Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Unexpected issues with 2 NVME initiators using the same target

[Chuck Lever]

> On Jul 10, 2017, at 5:24 PM, Jason Gunthorpe <jgunthorpe-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org> wrote:
> 
> On Mon, Jul 10, 2017 at 03:03:18PM -0400, Chuck Lever wrote:
> 
>>>> Or I could revert all the "map page cache pages" logic and
>>>> just use memcpy for small NFS WRITEs, and RDMA the rest of
>>>> the time. That keeps everything simple, but means large
>>>> inline thresholds can't use send-in-place.
>>> 
>>> Don't you have the same problem with RDMA WRITE?
>> 
>> The server side initiates RDMA Writes. The final RDMA Write in a WR
>> chain is signaled, but a subsequent Send completion is used to
>> determine when the server may release resources used for the Writes.
>> We're already doing it the slow way there, and there's no ^C hazard
>> on the server.
> 
> Wait, I guess I meant RDMA READ path.
> 
> The same contraints apply to RKeys as inline send - you cannot DMA
> unmap rkey memory until the rkey is invalidated at the HCA.
> 
> So posting an invalidate SQE and then immediately unmapping the DMA
> pages is bad too..
> 
> No matter how the data is transfered the unmapping must follow the
> same HCA synchronous model.. DMA unmap must only be done from the send
> completion handler (inline send or invalidate rkey), from the recv
> completion handler (send with invalidate), or from QP error state teardown.
> 
> Anything that does DMA memory unmap from another thread is very, very
> suspect, eg async from a ctrl-c trigger event.

4.13 server side is converted to use the rdma_rw API for
handling RDMA Read. For non-iWARP cases, it's using the
local DMA key for Read sink buffers. For iWARP it should
be using Read-with-invalidate (IIRC).

--
Chuck Lever



--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Unexpected issues with 2 NVME initiators using the same target

[Jason Gunthorpe]

On Mon, Jul 10, 2017 at 05:29:53PM -0400, Chuck Lever wrote:
> 
> > On Jul 10, 2017, at 5:24 PM, Jason Gunthorpe <jgunthorpe-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org> wrote:
> > 
> > On Mon, Jul 10, 2017 at 03:03:18PM -0400, Chuck Lever wrote:
> > 
> >>>> Or I could revert all the "map page cache pages" logic and
> >>>> just use memcpy for small NFS WRITEs, and RDMA the rest of
> >>>> the time. That keeps everything simple, but means large
> >>>> inline thresholds can't use send-in-place.
> >>> 
> >>> Don't you have the same problem with RDMA WRITE?
> >> 
> >> The server side initiates RDMA Writes. The final RDMA Write in a WR
> >> chain is signaled, but a subsequent Send completion is used to
> >> determine when the server may release resources used for the Writes.
> >> We're already doing it the slow way there, and there's no ^C hazard
> >> on the server.
> > 
> > Wait, I guess I meant RDMA READ path.
> > 
> > The same contraints apply to RKeys as inline send - you cannot DMA
> > unmap rkey memory until the rkey is invalidated at the HCA.
> > 
> > So posting an invalidate SQE and then immediately unmapping the DMA
> > pages is bad too..
> > 
> > No matter how the data is transfered the unmapping must follow the
> > same HCA synchronous model.. DMA unmap must only be done from the send
> > completion handler (inline send or invalidate rkey), from the recv
> > completion handler (send with invalidate), or from QP error state teardown.
> > 
> > Anything that does DMA memory unmap from another thread is very, very
> > suspect, eg async from a ctrl-c trigger event.
> 
> 4.13 server side is converted to use the rdma_rw API for
> handling RDMA Read. For non-iWARP cases, it's using the
> local DMA key for Read sink buffers. For iWARP it should
> be using Read-with-invalidate (IIRC).

The server sounds fine, how does the client work?

Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Unexpected issues with 2 NVME initiators using the same target

[Chuck Lever]

> On Jul 10, 2017, at 5:32 PM, Jason Gunthorpe <jgunthorpe-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org> wrote:
> 
> On Mon, Jul 10, 2017 at 05:29:53PM -0400, Chuck Lever wrote:
>> 
>>> On Jul 10, 2017, at 5:24 PM, Jason Gunthorpe <jgunthorpe-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org> wrote:
>>> 
>>> On Mon, Jul 10, 2017 at 03:03:18PM -0400, Chuck Lever wrote:
>>> 
>>>>>> Or I could revert all the "map page cache pages" logic and
>>>>>> just use memcpy for small NFS WRITEs, and RDMA the rest of
>>>>>> the time. That keeps everything simple, but means large
>>>>>> inline thresholds can't use send-in-place.
>>>>> 
>>>>> Don't you have the same problem with RDMA WRITE?
>>>> 
>>>> The server side initiates RDMA Writes. The final RDMA Write in a WR
>>>> chain is signaled, but a subsequent Send completion is used to
>>>> determine when the server may release resources used for the Writes.
>>>> We're already doing it the slow way there, and there's no ^C hazard
>>>> on the server.
>>> 
>>> Wait, I guess I meant RDMA READ path.
>>> 
>>> The same contraints apply to RKeys as inline send - you cannot DMA
>>> unmap rkey memory until the rkey is invalidated at the HCA.
>>> 
>>> So posting an invalidate SQE and then immediately unmapping the DMA
>>> pages is bad too..
>>> 
>>> No matter how the data is transfered the unmapping must follow the
>>> same HCA synchronous model.. DMA unmap must only be done from the send
>>> completion handler (inline send or invalidate rkey), from the recv
>>> completion handler (send with invalidate), or from QP error state teardown.
>>> 
>>> Anything that does DMA memory unmap from another thread is very, very
>>> suspect, eg async from a ctrl-c trigger event.
>> 
>> 4.13 server side is converted to use the rdma_rw API for
>> handling RDMA Read. For non-iWARP cases, it's using the
>> local DMA key for Read sink buffers. For iWARP it should
>> be using Read-with-invalidate (IIRC).
> 
> The server sounds fine, how does the client work?

The client does not initiate RDMA Read or Write today.


--
Chuck Lever



--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Unexpected issues with 2 NVME initiators using the same target

[Jason Gunthorpe]

On Mon, Jul 10, 2017 at 06:04:18PM -0400, Chuck Lever wrote:

> > The server sounds fine, how does the client work?
> 
> The client does not initiate RDMA Read or Write today.

Right, but it provides an rkey that the server uses for READ or WRITE.

The invalidate of that rkey at the client must follow the same rules
as inline send.

Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Unexpected issues with 2 NVME initiators using the same target

[Chuck Lever]

> On Jul 10, 2017, at 6:09 PM, Jason Gunthorpe <jgunthorpe-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org> wrote:
> 
> On Mon, Jul 10, 2017 at 06:04:18PM -0400, Chuck Lever wrote:
> 
>>> The server sounds fine, how does the client work?
>> 
>> The client does not initiate RDMA Read or Write today.
> 
> Right, but it provides an rkey that the server uses for READ or WRITE.
> 
> The invalidate of that rkey at the client must follow the same rules
> as inline send.

Ah, I see.

The RPC reply handler calls frwr_op_unmap_sync to invalidate
any MRs associated with the RPC.

frwr_op_unmap_sync has to sort the rkeys that are remotely
invalidated, and those that have not been.

The first step is to ensure all the rkeys for an RPC are
invalid. The rkey that was remotely invalidated is skipped
here, and a chain of LocalInv WRs is posted to invalidate
any remaining rkeys. The last WR in the chain is signaled.

If one or more LocalInv WRs are posted, this function waits
for LocalInv completion.

The last step is always DMA unmapping. Note that we can't
get a completion for a remotely invalidated rkey, and we
have to wait for LocalInv to complete anyway. So the DMA
unmapping is always handled here instead of in a
completion handler.

When frwr_op_unmap_sync returns to the RPC reply handler,
the handler calls xprt_complete_rqst, and the RPC is
terminated. This guarantees that the MRs are invalid before
control is returned to the RPC consumer.


In the ^C case, frwr_op_unmap_safe is invoked during RPC
termination. The MRs are passed to the background recovery
task, which invokes frwr_op_recover_mr.

frwr_op_recover_mr destroys the fr_mr and DMA unmaps the
memory. (It's also used when registration or invalidation
flushes, which is why it uses a hammer).

So here, we're a little fast/loose: the ordering of
invalidation and unmapping is correct, but the MRs can be
invalidated after the RPC completes. Since RPC termination
can't wait, this is the best I can do for now.


--
Chuck Lever



--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Unexpected issues with 2 NVME initiators using the same target

[Tom Talpey]

On 7/10/2017 11:57 PM, Chuck Lever wrote:
> 
>> On Jul 10, 2017, at 6:09 PM, Jason Gunthorpe <jgunthorpe-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org> wrote:
>>
>> On Mon, Jul 10, 2017 at 06:04:18PM -0400, Chuck Lever wrote:
>>
>>>> The server sounds fine, how does the client work?
>>>
>>> The client does not initiate RDMA Read or Write today.
>>
>> Right, but it provides an rkey that the server uses for READ or WRITE.
>>
>> The invalidate of that rkey at the client must follow the same rules
>> as inline send.
> 
> Ah, I see.
> 
> The RPC reply handler calls frwr_op_unmap_sync to invalidate
> any MRs associated with the RPC.
> 
> frwr_op_unmap_sync has to sort the rkeys that are remotely
> invalidated, and those that have not been.

Does the reply handler consider the possibility that the reply is
being signaled before the send WRs? There are some really interesting
races on shared or multiple CQs when the completion upcalls start
to back up under heavy load that we've seen in Windows SMB Direct.

In the end, we had to put explicit reference counts on each and
every object, and added rundown references to everything before
completing an operation and signaling the upper layer (SMB3, in
our case). This found a surprising number of double completions,
and missing completions from drivers as well.

> The first step is to ensure all the rkeys for an RPC are
> invalid. The rkey that was remotely invalidated is skipped
> here, and a chain of LocalInv WRs is posted to invalidate
> any remaining rkeys. The last WR in the chain is signaled.
> 
> If one or more LocalInv WRs are posted, this function waits
> for LocalInv completion.
> 
> The last step is always DMA unmapping. Note that we can't
> get a completion for a remotely invalidated rkey, and we
> have to wait for LocalInv to complete anyway. So the DMA
> unmapping is always handled here instead of in a
> completion handler.
> 
> When frwr_op_unmap_sync returns to the RPC reply handler,
> the handler calls xprt_complete_rqst, and the RPC is
> terminated. This guarantees that the MRs are invalid before
> control is returned to the RPC consumer.
> 
> 
> In the ^C case, frwr_op_unmap_safe is invoked during RPC
> termination. The MRs are passed to the background recovery
> task, which invokes frwr_op_recover_mr.

That worries me. How do you know it's going in sequence, and
that it will result in an invalidated MR?

> frwr_op_recover_mr destroys the fr_mr and DMA unmaps the
> memory. (It's also used when registration or invalidation
> flushes, which is why it uses a hammer).
> 
> So here, we're a little fast/loose: the ordering of
> invalidation and unmapping is correct, but the MRs can be
> invalidated after the RPC completes. Since RPC termination
> can't wait, this is the best I can do for now.

That would worry me even more. "fast/loose" isn't a good
situation when storage is concerned. Shouldn't you just be
closing the connection?

Tom.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Unexpected issues with 2 NVME initiators using the same target

[Chuck Lever]

Hi Tom-


> On Jul 11, 2017, at 9:23 AM, Tom Talpey <tom-CLs1Zie5N5HQT0dZR+AlfA@public.gmane.org> wrote:
> 
> On 7/10/2017 11:57 PM, Chuck Lever wrote:
>>> On Jul 10, 2017, at 6:09 PM, Jason Gunthorpe <jgunthorpe-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org> wrote:
>>> 
>>> On Mon, Jul 10, 2017 at 06:04:18PM -0400, Chuck Lever wrote:
>>> 
>>>>> The server sounds fine, how does the client work?
>>>> 
>>>> The client does not initiate RDMA Read or Write today.
>>> 
>>> Right, but it provides an rkey that the server uses for READ or WRITE.
>>> 
>>> The invalidate of that rkey at the client must follow the same rules
>>> as inline send.
>> Ah, I see.
>> The RPC reply handler calls frwr_op_unmap_sync to invalidate
>> any MRs associated with the RPC.
>> frwr_op_unmap_sync has to sort the rkeys that are remotely
>> invalidated, and those that have not been.
> 
> Does the reply handler consider the possibility that the reply is
> being signaled before the send WRs? There are some really interesting
> races on shared or multiple CQs when the completion upcalls start
> to back up under heavy load that we've seen in Windows SMB Direct.

If I understand you correctly, that's exactly what we're discussing.
The Send WR that transmitted the RPC Call can outlive the RPC
transaction in rare cases.

A partial solution is to move the Send SGE array into a data structure
whose lifetime is independent of rpcrdma_req. That allows the client
to guarantee that appropriate DMA unmaps are done only after the Send
is complete.

The other part of this issue is that delayed Send completion also
needs to prevent initiation of another RPC, otherwise the Send Queue
can overflow or the client might exceed the credit grant on this
connection. This part worries me because it could mean that some
serialization (eg. a completion and context switch) is needed for
every Send operation on the client.


> In the end, we had to put explicit reference counts on each and
> every object, and added rundown references to everything before
> completing an operation and signaling the upper layer (SMB3, in
> our case). This found a surprising number of double completions,
> and missing completions from drivers as well.
> 
>> The first step is to ensure all the rkeys for an RPC are
>> invalid. The rkey that was remotely invalidated is skipped
>> here, and a chain of LocalInv WRs is posted to invalidate
>> any remaining rkeys. The last WR in the chain is signaled.
>> If one or more LocalInv WRs are posted, this function waits
>> for LocalInv completion.
>> The last step is always DMA unmapping. Note that we can't
>> get a completion for a remotely invalidated rkey, and we
>> have to wait for LocalInv to complete anyway. So the DMA
>> unmapping is always handled here instead of in a
>> completion handler.
>> When frwr_op_unmap_sync returns to the RPC reply handler,
>> the handler calls xprt_complete_rqst, and the RPC is
>> terminated. This guarantees that the MRs are invalid before
>> control is returned to the RPC consumer.
>> In the ^C case, frwr_op_unmap_safe is invoked during RPC
>> termination. The MRs are passed to the background recovery
>> task, which invokes frwr_op_recover_mr.
> 
> That worries me. How do you know it's going in sequence,
> and that it will result in an invalidated MR?

The MR is destroyed synchronously. Isn't that enough to guarantee
that the rkey is invalid and DMA unmapping is safe to proceed?

With FRWR, if a LocalInv flushes with "memory management error"
there's no way to invalidate that MR other than:

- destroy the MR, or

- disconnect

The latter is a mighty sledgehammer that affects all the live MRs
and RPCs on that connection. That's why recover_mr destroys the
MR in all cases.


>> frwr_op_recover_mr destroys the fr_mr and DMA unmaps the
>> memory. (It's also used when registration or invalidation
>> flushes, which is why it uses a hammer).
>> So here, we're a little fast/loose: the ordering of
>> invalidation and unmapping is correct, but the MRs can be
>> invalidated after the RPC completes. Since RPC termination
>> can't wait, this is the best I can do for now.
> 
> That would worry me even more. "fast/loose" isn't a good
> situation when storage is concerned.

I agree!


> Shouldn't you just be closing the connection?

We'd have to wait for the connection close to complete in a
function that is not allowed to sleep, so it would have to be
done in the background as well. This is no better than handing
the MR to a background process.

And if possible we'd like to avoid connection loss (see above).

I don't like this situation, but it's the best I can do with
the current architecture of the RPC client. I've been staring
at this for a couple of years now wondering how to fix it.
Suggestions happily accepted!


--
Chuck Lever



--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Unexpected issues with 2 NVME initiators using the same target

[Bart Van Assche]

On Tue, 2017-06-27 at 10:37 +0300, Sagi Grimberg wrote:
> Jason,
> 
> > > The issue about the HCA not being able to access the inline
> > > buffer during a retransmit is also not an issue for RPC-
> > > over-RDMA because these buffers are always registered with
> > > the local rdma lkey.
> > 
> > Exactly.
> 
> Lost track of the thread...
> 
> 
> Indeed you raised this issue lots of times before, and I failed to see
> why its important or why its error prone, but now I do...
> 
> My apologies for not listening :(
> 
> We should fix _all_ initiators for it, nvme-rdma, iser, srp
> and xprtrdma (and probably some more ULPs out there)...
> 
> It also means that we cannot really suppress any send completions as
> that would result in an unpredictable latency (which is not acceptable).
> 
> I wish we could somehow tell the HCA that it can ignore access fail to a
> specific address when retransmitting.. but maybe its too much to ask...

Hello Sagi,

Can you clarify why you think that the SRP initiator needs to be changed?
The SRP initiator submits the local invalidate work request after the RDMA
write request. According to table 79 "Work Request Operation Ordering" the
order of these work requests must be maintained by the HCA. I think if a HCA
would start with invalidating the MR before the remote HCA has acknowledged
the written data that that's a firmware bug.

The upstream SRP initiator does not use inline data.

Bart.--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Unexpected issues with 2 NVME initiators using the same target

[Jason Gunthorpe]

On Tue, Jun 27, 2017 at 06:08:57PM +0000, Bart Van Assche wrote:

> Can you clarify why you think that the SRP initiator needs to be changed?
> The SRP initiator submits the local invalidate work request after the RDMA
> write request. According to table 79 "Work Request Operation
> Ordering" the

That table has a 'L' for invalidate that follows RDMA Write.

L means they are not ordered unless the optional Local Invalidate
Fence mode is used.

Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Unexpected issues with 2 NVME initiators using the same target

[Sagi Grimberg]

> Hello Sagi,

Hi Bart,

> Can you clarify why you think that the SRP initiator needs to be changed?
> The SRP initiator submits the local invalidate work request after the RDMA
> write request. According to table 79 "Work Request Operation Ordering" the
> order of these work requests must be maintained by the HCA. I think if a HCA
> would start with invalidating the MR before the remote HCA has acknowledged
> the written data that that's a firmware bug.

That flow is fine, we were discussing immediate data sends.

SRP only needs fixing by waiting for the all local invalidates to
complete before unmapping the user buffers and completing the I/O.

BTW, did the efforts on standardizing remote invalidate in SRP ever
evolved to something? Would it be acceptable to add a non-standard
ib_srp and ib_srpt? We can default to off and allow the user to opt
it in if it knows the two sides comply...

We need to fix that in nvmf and iser too btw (luckily we have remote
invalidate so its not a big issue).

> The upstream SRP initiator does not use inline data.

Yet :)

IIRC you have a branch with immediate-data support against scst so
it might be relevant there...
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Unexpected issues with 2 NVME initiators using the same target

[Bart Van Assche]

On Wed, 2017-06-28 at 10:16 +0300, Sagi Grimberg wrote:
> BTW, did the efforts on standardizing remote invalidate in SRP ever
> evolved to something? Would it be acceptable to add a non-standard
> ib_srp and ib_srpt? We can default to off and allow the user to opt
> it in if it knows the two sides comply...

I'd like to hear Doug's opinion about whether it's OK to add this
feature to the upstream drivers before support for remote invalidate
is standardized.

> We need to fix that in nvmf and iser too btw (luckily we have remote
> invalidate so its not a big issue).
> 
> > The upstream SRP initiator does not use inline data.
> 
> Yet :)
> 
> IIRC you have a branch with immediate-data support against scst so
> it might be relevant there...

Support for immediate data is being standardized. My plan is to add
support for immediate data to the upstream drivers once the T10 committee
agrees about the implementation. See also
http://www.t10.org/cgi-bin/ac.pl?t=f&f=srp2r02a.pdf

Bart.--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Unexpected issues with 2 NVME initiators using the same target

[Robert LeBlanc]

On Tue, Jun 20, 2017 at 6:02 AM, Sagi Grimberg <sagi-NQWnxTmZq1alnMjI0IkVqw@public.gmane.org> wrote:
>
>>>> Can you share the check that correlates to the vendor+hw syndrome?
>>>
>>>
>>> mkey.free == 1
>>
>>
>> Hmm, the way I understand it is that the HW is trying to access
>> (locally via send) a MR which was already invalidated.
>>
>> Thinking of this further, this can happen in a case where the target
>> already completed the transaction, sent SEND_WITH_INVALIDATE but the
>> original send ack was lost somewhere causing the device to retransmit
>> from the MR (which was already invalidated). This is highly unlikely
>> though.
>>
>> Shouldn't this be protected somehow by the device?
>> Can someone explain why the above cannot happen? Jason? Liran? Anyone?
>>
>> Say host register MR (a) and send (1) from that MR to a target,
>> send (1) ack got lost, and the target issues SEND_WITH_INVALIDATE
>> on MR (a) and the host HCA process it, then host HCA timeout on send (1)
>> so it retries, but ehh, its already invalidated.
>
>
> Well, this entire flow is broken, why should the host send the MR rkey
> to the target if it is not using it for remote access, the target
> should never have a chance to remote invalidate something it did not
> access.
>
> I think we have a bug in iSER code, as we should not send the key
> for remote invalidation if we do inline data send...
>
> Robert, can you try the following:
> --
> diff --git a/drivers/infiniband/ulp/iser/iser_initiator.c
> b/drivers/infiniband/ulp/iser/iser_initiator.c
> index 12ed62ce9ff7..2a07692007bd 100644
> --- a/drivers/infiniband/ulp/iser/iser_initiator.c
> +++ b/drivers/infiniband/ulp/iser/iser_initiator.c
> @@ -137,8 +137,10 @@ iser_prepare_write_cmd(struct iscsi_task *task,
>
>         if (unsol_sz < edtl) {
>                 hdr->flags     |= ISER_WSV;
> -               hdr->write_stag = cpu_to_be32(mem_reg->rkey);
> -               hdr->write_va   = cpu_to_be64(mem_reg->sge.addr + unsol_sz);
> +               if (buf_out->data_len > imm_sz) {
> +                       hdr->write_stag = cpu_to_be32(mem_reg->rkey);
> +                       hdr->write_va = cpu_to_be64(mem_reg->sge.addr +
> unsol_sz);
> +               }
>
>                 iser_dbg("Cmd itt:%d, WRITE tags, RKEY:%#.4X "
>                          "VA:%#llX + unsol:%d\n",
> --
>
> Although, I still don't think its enough. We need to delay the
> local invalidate till we received a send completion (guarantees
> that ack was received)...
>
> If this indeed the case, _all_ ULP initiator drivers share it because we
> never condition on a send completion in order to complete an I/O, and
> in the case of lost ack on send, looks like we need to... It *will* hurt
> performance.
>
> What do other folks think?
>
> CC'ing Bart, Chuck, Christoph.
>
> Guys, for summary, I think we might have a broken behavior in the
> initiator mode drivers. We never condition send completions (for
> requests) before we complete an I/O. The issue is that the ack for those
> sends might get lost, which means that the HCA will retry them (dropped
> by the peer HCA) but if we happen to complete the I/O before, either we
> can unmap the request area, or for inline data, we invalidate it (so the
> HCA will try to access a MR which was invalidated).
>
> Signalling all send completions and also finishing I/Os only after we
> got them will add latency, and that sucks...

Testing this patch I didn't see these new messages even when rebooting
the targets multiple times. It also resolved some performance problems
I was seeing (I think our switches are having bugs with IPv6 and
routing) and I was receiving expected performance. At one point in the
test, one target (4.9.33) showed:
[Tue Jun 20 10:11:20 2017] iSCSI Login timeout on Network Portal [::]:3260
[Tue Jun 20 10:11:39 2017] iSCSI Login timeout on Network Portal [::]:3260
[Tue Jun 20 10:11:58 2017] isert: isert_print_wc: login send failure:
transport retry counter exceeded (12) vend_err 81

After this and a reboot of the target, the initiator would drop the
connection after 1.5-2 minutes then faster and faster until it was
every 5 seconds. It is almost like it set up the connection then lose
the first ping, or the ping wasn't set-up right. I tried rebooting the
target multiple times.

I tried to logout the "bad" session and got a back trace.

[Tue Jun 20 10:30:08 2017] ------------[ cut here ]------------
[Tue Jun 20 10:30:08 2017] WARNING: CPU: 20 PID: 783 at
fs/sysfs/group.c:237 sysfs_remove_group+0x82/0x90
[Tue Jun 20 10:30:08 2017] Modules linked in: ib_iser rdma_ucm ib_ucm
ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx5_ib mlx4_ib ib_core 8021q
garp mrp sb_edac x86_pkg_temp_thermal intel_powerclamp coretemp
kvm_intel kvm iTCO_wdt iTCO_vendor_support mei_me irqbypass
crct10dif_pclmul mei crc32_pclmul ioatdma ghash_clmulni_intel lpc_ich
i2c_i801 pcbc mfd_core aesni_intel crypto_simd glue_helper cryptd
joydevsg pcspkr shpchp wmi ipmi_si ipmi_devintf ipmi_msghandler
acpi_power_meter ipt_REJECT nf_reject_ipv4 nf_conntrack_ipv4 acpi_pad
nf_defrag_ipv4 xt_conntrack nf_conntrack iptable_filter nfsd
auth_rpcgss nfs_acl lockd grace sunrpc ip_tables xfs libcrc32c raid1
dm_service_time sd_mod mlx4_en be2iscsi bnx2i cnic uio qla4xxx
iscsi_boot_sysfs ast drm_kms_helper mlx5_core syscopyarea sysfillrect
sysimgblt fb_sys_fops ttm
[Tue Jun 20 10:30:08 2017]  drm mlx4_core igb ahci libahci ptp libata
pps_core dca i2c_algo_bit dm_multipath dm_mirror dm_region_hash dm_log
dm_mod dax
[Tue Jun 20 10:30:08 2017] CPU: 20 PID: 783 Comm: kworker/u64:2 Not
tainted 4.12.0-rc6+ #5
[Tue Jun 20 10:30:08 2017] Hardware name: Supermicro
SYS-6028TP-HTFR/X10DRT-PIBF, BIOS 1.1 08/03/2015
[Tue Jun 20 10:30:08 2017] Workqueue: scsi_wq_12 __iscsi_unbind_session
[Tue Jun 20 10:30:08 2017] task: ffff887f5c45cb00 task.stack: ffffc90032ef4000
[Tue Jun 20 10:30:08 2017] RIP: 0010:sysfs_remove_group+0x82/0x90
[Tue Jun 20 10:30:08 2017] RSP: 0018:ffffc90032ef7d18 EFLAGS: 00010246
[Tue Jun 20 10:30:08 2017] RAX: 0000000000000038 RBX: 0000000000000000
RCX: 0000000000000000
[Tue Jun 20 10:30:08 2017] RDX: 0000000000000000 RSI: ffff883f7fd0e068
RDI: ffff883f7fd0e068
[Tue Jun 20 10:30:08 2017] RBP: ffffc90032ef7d30 R08: 0000000000000000
R09: 0000000000000676
[Tue Jun 20 10:30:08 2017] R10: 00000000000003ff R11: 0000000000000001
R12: ffffffff81da8a40
[Tue Jun 20 10:30:08 2017] R13: ffff887f52ec0838 R14: ffff887f52ec08d8
R15: ffff883f4c611000
[Tue Jun 20 10:30:08 2017] FS:  0000000000000000(0000)
GS:ffff883f7fd00000(0000) knlGS:0000000000000000
[Tue Jun 20 10:30:08 2017] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[Tue Jun 20 10:30:08 2017] CR2: 000055eef886b398 CR3: 0000000001c09000
CR4: 00000000001406e0
[Tue Jun 20 10:30:08 2017] Call Trace:
[Tue Jun 20 10:30:08 2017]  dpm_sysfs_remove+0x57/0x60
[Tue Jun 20 10:30:08 2017]  device_del+0x107/0x330
[Tue Jun 20 10:30:08 2017]  scsi_target_reap_ref_release+0x2d/0x40
[Tue Jun 20 10:30:08 2017]  scsi_target_reap+0x2e/0x40
[Tue Jun 20 10:30:08 2017]  scsi_remove_target+0x197/0x1b0
[Tue Jun 20 10:30:08 2017]  __iscsi_unbind_session+0xbe/0x170
[Tue Jun 20 10:30:08 2017]  process_one_work+0x149/0x360
[Tue Jun 20 10:30:08 2017]  worker_thread+0x4d/0x3c0
[Tue Jun 20 10:30:08 2017]  kthread+0x109/0x140
[Tue Jun 20 10:30:08 2017]  ? rescuer_thread+0x380/0x380
[Tue Jun 20 10:30:08 2017]  ? kthread_park+0x60/0x60
[Tue Jun 20 10:30:08 2017]  ret_from_fork+0x25/0x30
[Tue Jun 20 10:30:08 2017] Code: d5 c0 ff ff 5b 41 5c 41 5d 5d c3 48
89 df e8 66 bd ff ff eb c6 49 8b 55 00 49 8b 34 24 48 c7 c7 d0 3ca7 81
31 c0 e8 3c 01 ee ff <0f> ff eb d5 66 2e 0f 1f 84 00 00 00 00 00 0f 1f
44 00 00 55 48
[Tue Jun 20 10:30:08 2017] ---[ end trace 6161f21139b6a1ea ]---

Logging back into that target didn't help stabilize the connection. I
rebooted both initiator and targets to clear things up and after the
initiator went down, the target showed the timeout message again. It
seems something got out of whack and never recovered and "poisoned"
the other node in the process.

I'll test Max's patch now and report back.

----------------
Robert LeBlanc
PGP Fingerprint 79A2 9CA4 6CC4 45DD A904  C70E E654 3BB2 FA62 B9F1
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Unexpected issues with 2 NVME initiators using the same target

[Sagi Grimberg]

> Testing this patch I didn't see these new messages even when rebooting
> the targets multiple times. It also resolved some performance problems
> I was seeing (I think our switches are having bugs with IPv6 and
> routing) and I was receiving expected performance. At one point in the
> test, one target (4.9.33) showed:
> [Tue Jun 20 10:11:20 2017] iSCSI Login timeout on Network Portal [::]:3260
> [Tue Jun 20 10:11:39 2017] iSCSI Login timeout on Network Portal [::]:3260
> [Tue Jun 20 10:11:58 2017] isert: isert_print_wc: login send failure:
> transport retry counter exceeded (12) vend_err 81

I don't understand, is this new with the patch applied?

> After this and a reboot of the target, the initiator would drop the
> connection after 1.5-2 minutes then faster and faster until it was
> every 5 seconds. It is almost like it set up the connection then lose
> the first ping, or the ping wasn't set-up right. I tried rebooting the
> target multiple times.

So the initiator could not recover even after the target as available
again?
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Unexpected issues with 2 NVME initiators using the same target

[Robert LeBlanc]

On Tue, Jun 20, 2017 at 11:19 AM, Sagi Grimberg <sagi-NQWnxTmZq1alnMjI0IkVqw@public.gmane.org> wrote:
>
>> Testing this patch I didn't see these new messages even when rebooting
>> the targets multiple times. It also resolved some performance problems
>> I was seeing (I think our switches are having bugs with IPv6 and
>> routing) and I was receiving expected performance. At one point in the
>> test, one target (4.9.33) showed:
>> [Tue Jun 20 10:11:20 2017] iSCSI Login timeout on Network Portal [::]:3260
>> [Tue Jun 20 10:11:39 2017] iSCSI Login timeout on Network Portal [::]:3260
>> [Tue Jun 20 10:11:58 2017] isert: isert_print_wc: login send failure:
>> transport retry counter exceeded (12) vend_err 81
>
>
> I don't understand, is this new with the patch applied?

I applied your patch to 4.12-rc6 on the initiator, but my targets are
still 4.9.33 since it looked like the patch only affected the
initiator. I did not see this before your patch, but I also didn't try
rebooting the targets multiple times before because of the previous
messages.

>> After this and a reboot of the target, the initiator would drop the
>> connection after 1.5-2 minutes then faster and faster until it was
>> every 5 seconds. It is almost like it set up the connection then lose
>> the first ping, or the ping wasn't set-up right. I tried rebooting the
>> target multiple times.
>
>
> So the initiator could not recover even after the target as available
> again?

The initiator recovered the connection when the target came back, but
the connection was not stable. I/O would happen on the connection,
then it would get shaky and then finally disconnect. Then it would
reconnect, pass more I/O, then get shaky and go down again. With the 5
second disconnects, it would pass traffic for 5 seconds, then as soon
as I saw the ping timeout, the I/O would stop until it reconnected. At
that point it seems that the lack of pings would kill the I/O unlike
earlier where there was a stall in I/O and then the connection would
be torn down. I can try to see if I can get it to happen again.

----------------
Robert LeBlanc
PGP Fingerprint 79A2 9CA4 6CC4 45DD A904  C70E E654 3BB2 FA62 B9F1
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Unexpected issues with 2 NVME initiators using the same target

[Sagi Grimberg]

>> I don't understand, is this new with the patch applied?
> 
> I applied your patch to 4.12-rc6 on the initiator, but my targets are
> still 4.9.33 since it looked like the patch only affected the
> initiator. I did not see this before your patch, but I also didn't try
> rebooting the targets multiple times before because of the previous
> messages.

That sounds like a separate issue. Should we move forward with the
suggested patch?

>>> After this and a reboot of the target, the initiator would drop the
>>> connection after 1.5-2 minutes then faster and faster until it was
>>> every 5 seconds. It is almost like it set up the connection then lose
>>> the first ping, or the ping wasn't set-up right. I tried rebooting the
>>> target multiple times.
>>
>>
>> So the initiator could not recover even after the target as available
>> again?
> 
> The initiator recovered the connection when the target came back, but
> the connection was not stable. I/O would happen on the connection,
> then it would get shaky and then finally disconnect. Then it would
> reconnect, pass more I/O, then get shaky and go down again. With the 5
> second disconnects, it would pass traffic for 5 seconds, then as soon
> as I saw the ping timeout, the I/O would stop until it reconnected. At
> that point it seems that the lack of pings would kill the I/O unlike
> earlier where there was a stall in I/O and then the connection would
> be torn down. I can try to see if I can get it to happen again.

So looks like the target is not responding to NOOP_OUTs (or traffic
at all for that matter).

The messages:
[Tue Jun 20 10:11:20 2017] iSCSI Login timeout on Network Portal [::]:3260

Are indicating that something is stuck in the login thread, not sure
where though. Did you see a watchdog popping on a hang?

And massage:
[Tue Jun 20 10:11:58 2017] isert: isert_print_wc: login send failure:
transport retry counter exceeded (12) vend_err 81

Is an indication that the rdma fabric is in some error state.

On which reboot attempt all this happened? the first one?

Again, CCing target-devel.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Unexpected issues with 2 NVME initiators using the same target

[Robert LeBlanc]

On Tue, Jun 20, 2017 at 3:33 AM, Sagi Grimberg <sagi-NQWnxTmZq1alnMjI0IkVqw@public.gmane.org> wrote:
>
>>>> Here the parsed output, it says that it was access to mkey which is
>>>> free.
>
>
> Missed that :)
>
>>>> ======== cqe_with_error ========
>>>> wqe_id                           : 0x0
>>>> srqn_usr_index                   : 0x0
>>>> byte_cnt                         : 0x0
>>>> hw_error_syndrome                : 0x93
>>>> hw_syndrome_type                 : 0x0
>>>> vendor_error_syndrome            : 0x52
>>>
>>>
>>> Can you share the check that correlates to the vendor+hw syndrome?
>>
>>
>> mkey.free == 1
>
>
> Hmm, the way I understand it is that the HW is trying to access
> (locally via send) a MR which was already invalidated.
>
> Thinking of this further, this can happen in a case where the target
> already completed the transaction, sent SEND_WITH_INVALIDATE but the
> original send ack was lost somewhere causing the device to retransmit
> from the MR (which was already invalidated). This is highly unlikely
> though.
>
> Shouldn't this be protected somehow by the device?
> Can someone explain why the above cannot happen? Jason? Liran? Anyone?
>
> Say host register MR (a) and send (1) from that MR to a target,
> send (1) ack got lost, and the target issues SEND_WITH_INVALIDATE
> on MR (a) and the host HCA process it, then host HCA timeout on send (1)
> so it retries, but ehh, its already invalidated.
>
> Or, we can also have a race where we destroy all our MRs when I/O
> is still running (but from the code we should be safe here).
>
> Robert, when you rebooted the target, I assume iscsi ping
> timeout expired and the connection teardown started correct?

I do remember that the ping timed out and the connection was torn down
according to the messages.

----------------
Robert LeBlanc
PGP Fingerprint 79A2 9CA4 6CC4 45DD A904  C70E E654 3BB2 FA62 B9F1
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Unexpected issues with 2 NVME initiators using the same target

[Robert LeBlanc]

On Tue, Jun 20, 2017 at 4:31 AM, Max Gurtovoy <maxg-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org> wrote:
>
>
> On 6/20/2017 12:33 PM, Sagi Grimberg wrote:
>>
>>
>>>>> Here the parsed output, it says that it was access to mkey which is
>>>>> free.
>>
>>
>> Missed that :)
>>
>>>>> ======== cqe_with_error ========
>>>>> wqe_id                           : 0x0
>>>>> srqn_usr_index                   : 0x0
>>>>> byte_cnt                         : 0x0
>>>>> hw_error_syndrome                : 0x93
>>>>> hw_syndrome_type                 : 0x0
>>>>> vendor_error_syndrome            : 0x52
>>>>
>>>>
>>>> Can you share the check that correlates to the vendor+hw syndrome?
>>>
>>>
>>> mkey.free == 1
>>
>>
>> Hmm, the way I understand it is that the HW is trying to access
>> (locally via send) a MR which was already invalidated.
>>
>> Thinking of this further, this can happen in a case where the target
>> already completed the transaction, sent SEND_WITH_INVALIDATE but the
>> original send ack was lost somewhere causing the device to retransmit
>> from the MR (which was already invalidated). This is highly unlikely
>> though.
>>
>> Shouldn't this be protected somehow by the device?
>> Can someone explain why the above cannot happen? Jason? Liran? Anyone?
>>
>> Say host register MR (a) and send (1) from that MR to a target,
>> send (1) ack got lost, and the target issues SEND_WITH_INVALIDATE
>> on MR (a) and the host HCA process it, then host HCA timeout on send (1)
>> so it retries, but ehh, its already invalidated.
>
>
> This might happen IMO.
> Robert, can you test this untested patch (this is not the full solution,
> just something to think about):
>
> diff --git a/drivers/infiniband/ulp/iser/iser_verbs.c
> b/drivers/infiniband/ulp/iser/iser_verbs.c
> index c538a38..e93bd40 100644
> --- a/drivers/infiniband/ulp/iser/iser_verbs.c
> +++ b/drivers/infiniband/ulp/iser/iser_verbs.c
> @@ -1079,7 +1079,7 @@ int iser_post_send(struct ib_conn *ib_conn, struct
> iser_tx_desc *tx_desc,
>         wr->sg_list = tx_desc->tx_sg;
>         wr->num_sge = tx_desc->num_sge;
>         wr->opcode = IB_WR_SEND;
> -       wr->send_flags = signal ? IB_SEND_SIGNALED : 0;
> +       wr->send_flags = IB_SEND_SIGNALED;
>
>         ib_ret = ib_post_send(ib_conn->qp, &tx_desc->wrs[0].send, &bad_wr);
>         if (ib_ret)
>
>
>>
>> Or, we can also have a race where we destroy all our MRs when I/O
>> is still running (but from the code we should be safe here).
>>
>> Robert, when you rebooted the target, I assume iscsi ping
>> timeout expired and the connection teardown started correct?

I still get the local protection errors with this patch. I am seeing a
ping timeout on the initiator when I reboot the target.
[Tue Jun 20 16:41:21 2017]  connection7:0: detected conn error (1011)
[Tue Jun 20 16:41:26 2017]  session7: session recovery timed out after 5 secs

Since I'm gracefully shutting down the targets in this case, shouldn't
the connection be closed gracefully by the target instead of the
initiator having to wait for ping to fail?

----------------
Robert LeBlanc
PGP Fingerprint 79A2 9CA4 6CC4 45DD A904  C70E E654 3BB2 FA62 B9F1
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Unexpected issues with 2 NVME initiators using the same target

[Sagi Grimberg]

> I still get the local protection errors with this patch. I am seeing a
> ping timeout on the initiator when I reboot the target.
> [Tue Jun 20 16:41:21 2017]  connection7:0: detected conn error (1011)
> [Tue Jun 20 16:41:26 2017]  session7: session recovery timed out after 5 secs
> 

Not a big surprise as its not really addressing the issue...

> Since I'm gracefully shutting down the targets in this case, shouldn't
> the connection be closed gracefully by the target instead of the
> initiator having to wait for ping to fail?

Not really, even in orderly shutdown, the device driver (mlx5 in
this case) shutdown sequence is triggered before ib_isert and
fires DEVICE_REMOVAL events to all its upper layer users (ib_isert
being one of them), which forces resource teardown (no disconnect).

We could resgister a shutdown handler in ib_isert, but its not really
its responsibility as a transport driver..

It would be nice if we had targetcli daemon'ised as a service
and register a shutdown notification, orderly remove and save the
existing configuration before the kernel even sees it. But its
a different scope really...

CC'ing target-devel

Re: Unexpected issues with 2 NVME initiators using the same target

[Robert LeBlanc]

On Tue, Jun 20, 2017 at 1:58 AM, Sagi Grimberg <sagi-NQWnxTmZq1alnMjI0IkVqw@public.gmane.org> wrote:
>
>>> Hi Robert,
>>>
>>>> I ran into this with 4.9.32 when I rebooted the target. I tested
>>>> 4.12-rc6 and this particular error seems to have been resolved, but I
>>>> now get a new one on the initiator. This one doesn't seem as
>>>> impactful.
>>>>
>>>> [Mon Jun 19 11:17:20 2017] mlx5_0:dump_cqe:275:(pid 0): dump error cqe
>>>> [Mon Jun 19 11:17:20 2017] 00000000 00000000 00000000 00000000
>>>> [Mon Jun 19 11:17:20 2017] 00000000 00000000 00000000 00000000
>>>> [Mon Jun 19 11:17:20 2017] 00000000 00000000 00000000 00000000
>>>> [Mon Jun 19 11:17:20 2017] 00000000 93005204 0a0001bd 45c8e0d2
>>>
>>>
>>> Max, Leon,
>>>
>>> Care to parse this syndrome for us? ;)
>>
>>
>> Here the parsed output, it says that it was access to mkey which is
>> free.
>>
>> ======== cqe_with_error ========
>> wqe_id                           : 0x0
>> srqn_usr_index                   : 0x0
>> byte_cnt                         : 0x0
>> hw_error_syndrome                : 0x93
>> hw_syndrome_type                 : 0x0
>> vendor_error_syndrome            : 0x52
>
>
> Can you share the check that correlates to the vendor+hw syndrome?
>
>> syndrome                         : LOCAL_PROTECTION_ERROR (0x4)
>> s_wqe_opcode                     : SEND (0xa)
>
>
> That's interesting, the opcode is a send operation. I'm assuming
> that this is immediate-data write? Robert, did this happen when
> you issued >4k writes to the target?

I was running dd with oflag=direct, so yes.

----------------
Robert LeBlanc
PGP Fingerprint 79A2 9CA4 6CC4 45DD A904  C70E E654 3BB2 FA62 B9F1
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html