All of lore.kernel.org
 help / color / mirror / Atom feed
From: "Steve Wise" <swise-7bPotxP6k4+P2YhJcF5u+vpXobYPEAuW@public.gmane.org>
To: 'Sagi Grimberg' <sagi-NQWnxTmZq1alnMjI0IkVqw@public.gmane.org>,
	'target-devel'
	<target-devel-u79uwXL29TY76Z2rM5mHXA@public.gmane.org>
Cc: linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
Subject: RE: stuck iscsi/iser target with linux-4.15.0-rc1
Date: Wed, 13 Dec 2017 15:40:12 -0600	[thread overview]
Message-ID: <01bd01d3745a$f4bc82f0$de3588d0$@opengridcomputing.com> (raw)
In-Reply-To: <1dee9f68-a81b-b7b8-9e70-e0ef5c63c520-NQWnxTmZq1alnMjI0IkVqw@public.gmane.org>

> > [239800.115739] target_wait_for_sess_cmds: Waiting for se_cmd:
> ffff88034082c998 t_state: 6, fabric state: 12
> 
> Hmm, this means that the command was delegated to isert to send
> data+response... Which means we lose a reference put somewhere here.
> 
> I'm assuming that this happens before your changes to ib_drain_qp
> correct? If this does not happen without your changes it might indicate
> that drain_qp is missing an error (or successful?) completion which
> would prevent a final reference drop (isert_completion_put).

Hey Sagi, I'm trying to reproduce this on CX4 cards with mlx5.  I have the two nodes setup via RoCEv2 and rping works over mlx5 fine, but when I try to discover the iSER targets, the initiator fails with:

[root@potato1 ~]# iscsiadm -m discovery -t sendtargets -p 172.16.99.239:3260 -I iser
iscsiadm: recv's end state machine bug?
iscsiadm: Could not perform SendTargets discovery: iSCSI PDU timed out
[root@potato1 ~]# uname -r
4.15.0-rc3+

And the target logs this:

[  873.240460] mlx5_0:dump_cqe:277:(pid 494): dump error cqe
[  873.246665] 00000000 00000000 00000000 00000000
[  873.251942] 00000000 00000000 00000000 00000000
[  873.257214] 00000000 00000000 00000000 00000000
[  873.262472] 00000000 00008a12 0a0000f6 00014bd2
[  873.267711] isert: isert_print_wc: send failure: invalid request error (9) vend_err 8a

Any ideas?  I'm using straight 4.15.0-rc3 + a workaround to avoid crashing my x86 systems at bootup from here:

https://www.mail-archive.com/netdev-u79uwXL29TY76Z2rM5mHXA@public.gmane.org/msg203210.html'

Steve.
Thanks,

Steve.



---
This email has been checked for viruses by AVG.
http://www.avg.com

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

WARNING: multiple messages have this Message-ID (diff)
From: "Steve Wise" <swise@opengridcomputing.com>
To: 'Sagi Grimberg' <sagi-NQWnxTmZq1alnMjI0IkVqw@public.gmane.org>,
	'target-devel'
	<target-devel-u79uwXL29TY76Z2rM5mHXA@public.gmane.org>
Cc: linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
Subject: RE: stuck iscsi/iser target with linux-4.15.0-rc1
Date: Wed, 13 Dec 2017 21:40:12 +0000	[thread overview]
Message-ID: <01bd01d3745a$f4bc82f0$de3588d0$@opengridcomputing.com> (raw)
In-Reply-To: <1dee9f68-a81b-b7b8-9e70-e0ef5c63c520-NQWnxTmZq1alnMjI0IkVqw@public.gmane.org>

> > [239800.115739] target_wait_for_sess_cmds: Waiting for se_cmd:
> ffff88034082c998 t_state: 6, fabric state: 12
> 
> Hmm, this means that the command was delegated to isert to send
> data+response... Which means we lose a reference put somewhere here.
> 
> I'm assuming that this happens before your changes to ib_drain_qp
> correct? If this does not happen without your changes it might indicate
> that drain_qp is missing an error (or successful?) completion which
> would prevent a final reference drop (isert_completion_put).

Hey Sagi, I'm trying to reproduce this on CX4 cards with mlx5.  I have the two nodes setup via RoCEv2 and rping works over mlx5 fine, but when I try to discover the iSER targets, the initiator fails with:

[root@potato1 ~]# iscsiadm -m discovery -t sendtargets -p 172.16.99.239:3260 -I iser
iscsiadm: recv's end state machine bug?
iscsiadm: Could not perform SendTargets discovery: iSCSI PDU timed out
[root@potato1 ~]# uname -r
4.15.0-rc3+

And the target logs this:

[  873.240460] mlx5_0:dump_cqe:277:(pid 494): dump error cqe
[  873.246665] 00000000 00000000 00000000 00000000
[  873.251942] 00000000 00000000 00000000 00000000
[  873.257214] 00000000 00000000 00000000 00000000
[  873.262472] 00000000 00008a12 0a0000f6 00014bd2
[  873.267711] isert: isert_print_wc: send failure: invalid request error (9) vend_err 8a

Any ideas?  I'm using straight 4.15.0-rc3 + a workaround to avoid crashing my x86 systems at bootup from here:

https://www.mail-archive.com/netdev@vger.kernel.org/msg203210.html'

Steve.
Thanks,

Steve.



---
This email has been checked for viruses by AVG.
http://www.avg.com


  parent reply	other threads:[~2017-12-13 21:40 UTC|newest]

Thread overview: 28+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2017-12-01 17:05 stuck iscsi/iser target with linux-4.15.0-rc1 Steve Wise
2017-12-01 17:05 ` Steve Wise
2017-12-03 13:29 ` Sagi Grimberg
2017-12-03 13:29   ` Sagi Grimberg
     [not found]   ` <0ba7e891-f020-26fb-9945-9e824332593c-NQWnxTmZq1alnMjI0IkVqw@public.gmane.org>
2017-12-04 15:49     ` Steve Wise
2017-12-04 15:49       ` Steve Wise
2017-12-04 18:40       ` Sagi Grimberg
2017-12-04 18:40         ` Sagi Grimberg
     [not found]         ` <1dee9f68-a81b-b7b8-9e70-e0ef5c63c520-NQWnxTmZq1alnMjI0IkVqw@public.gmane.org>
2017-12-13 21:40           ` Steve Wise [this message]
2017-12-13 21:40             ` Steve Wise
2017-12-13 22:34             ` Steve Wise
2017-12-13 22:34               ` Steve Wise
2017-12-13 22:35               ` Steve Wise
2017-12-13 22:35                 ` Steve Wise
2017-12-18 16:32                 ` Max Gurtovoy
2017-12-18 16:32                   ` Max Gurtovoy
     [not found]                   ` <cbe1b731-8e37-884a-1406-6a7ca1f6a0be-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
2017-12-18 17:12                     ` Steve Wise
2017-12-18 17:12                       ` Steve Wise
2017-12-18 21:43         ` Steve Wise
2017-12-18 21:43           ` Steve Wise
2017-12-19 21:53         ` Steve Wise
2017-12-19 21:53           ` Steve Wise
2017-12-19 22:21         ` Steve Wise
2017-12-19 22:21           ` Steve Wise
2017-12-21  8:08           ` Sagi Grimberg
2017-12-21  8:08             ` Sagi Grimberg
2017-12-21 15:21             ` Steve Wise
2017-12-21 15:21               ` Steve Wise

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to='01bd01d3745a$f4bc82f0$de3588d0$@opengridcomputing.com' \
    --to=swise-7bpotxp6k4+p2yhjcf5u+vpxobypeauw@public.gmane.org \
    --cc=linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org \
    --cc=sagi-NQWnxTmZq1alnMjI0IkVqw@public.gmane.org \
    --cc=target-devel-u79uwXL29TY76Z2rM5mHXA@public.gmane.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.