From mboxrd@z Thu Jan 1 00:00:00 1970 From: Mike Christie Date: Thu, 14 Dec 2017 05:56:29 +0000 Subject: iscsi target regression due to "tcp: remove prequeue support" patch Message-Id: <5A32128D.4050207@redhat.com> MIME-Version: 1 Content-Type: multipart/mixed; boundary="------------070703050208000100010308" List-Id: To: target-devel@vger.kernel.org This is a multi-part message in MIME format. --------------070703050208000100010308 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit Hey Nick and Florian, Starting in 4.14 iscsi logins will fail around 50% of the time. I git bisected the issue down to this commit: commit e7942d0633c47c791ece6afa038be9cf977226de Author: Florian Westphal Date: Sun Jul 30 03:57:18 2017 +0200 tcp: remove prequeue support Nick, attached is the iscsi target log info when the login fails. You can see at: Dec 13 17:55:01 rhel73n1 kernel: Got Login Command, Flags 0x81, ITT: 0x00000000, CmdSN: 0x00000000, ExpStatSN: 0xf86dc69b, CID: 0, Length: 65 we have got a login command and we seem to then go into iscsit_do_rx_data -> sock_recvmsg We seem to get stuck in there though, because we stay blocked until: Dec 13 17:55:01 rhel73n1 kernel: Entering iscsi_target_sk_data_ready: conn: ffff88b35cbb3000 Dec 13 17:55:01 rhel73n1 kernel: Got LOGIN_FLAGS_READ_ACTIVE=1, conn: ffff88b35cbb3000 >>>> where initiator side timeout fires 15 seconds later and it disconnects the tcp connection, and we eventually break out of the recvmsg call: Dec 13 17:55:16 rhel73n1 kernel: Entering iscsi_target_sk_state_change Dec 13 17:55:16 rhel73n1 kernel: __iscsi_target_sk_check_close: TCP_CLOSE_WAIT|TCP_CLOSE,returning FALSE .... Dec 13 17:55:16 rhel73n1 kernel: rx_loop: 68, total_rx: 68, data: 68 Dec 13 17:55:16 rhel73n1 kernel: iscsi_target_do_login_rx after rx_login_io, ffff88b35cbb3000, kworker/2:2:1829 Is the iscsi target doing something incorrect in its use of sk_data_ready and sock_recvmsg or is the tcp patch at fault? --------------070703050208000100010308 Content-Type: text/plain; charset=UTF-8; name="iscsi-chap-login-failure.txt" Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="iscsi-chap-login-failure.txt" CkRlYyAxMyAxNzo1NTowMSByaGVsNzNuMSBrZXJuZWw6IEFkZGVkIHRpbWVvdXQgdGltZXIg dG8gaVNDU0kgbG9naW4gcmVxdWVzdCBmb3IgMTUgc2Vjb25kcy4KRGVjIDEzIDE3OjU1OjAx IHJoZWw3M24xIGtlcm5lbDogTW92aW5nIHRvIFRBUkdfQ09OTl9TVEFURV9YUFRfVVAuCkRl YyAxMyAxNzo1NTowMSByaGVsNzNuMSBrZXJuZWw6IHJ4X2xvb3A6IDQ4LCB0b3RhbF9yeDog NDgsIGRhdGE6IDQ4CkRlYyAxMyAxNzo1NTowMSByaGVsNzNuMSBrZXJuZWw6IEdvdCBMb2dp biBDb21tYW5kLCBGbGFncyAweDgxLCBJVFQ6IDB4MDAwMDAwMDAsIENtZFNOOiAweDAwMDAw MDAwLCBFeHBTdGF0U046IDB4MDAwMDAwMDAsIENJRDogMCwgTGVuZ3RoOiAxNDQKRGVjIDEz IDE3OjU1OjAxIHJoZWw3M24xIGtlcm5lbDogcnhfbG9vcDogMTQ0LCB0b3RhbF9yeDogMTQ0 LCBkYXRhOiAxNDQKRGVjIDEzIDE3OjU1OjAxIHJoZWw3M24xIGtlcm5lbDogUmVjZWl2ZWQg aVNDU0kgbG9naW4gcmVxdWVzdCBmcm9tIDIwLjE1LjAuMjAyOjYwMzcyIG9uIGlTQ1NJL1RD UCBOZXR3b3JrIFBvcnRhbCAyMC4xNS4wLjEzMTozMjYwCkRlYyAxMyAxNzo1NTowMSByaGVs NzNuMSBrZXJuZWw6IE1vdmluZyB0byBUQVJHX0NPTk5fU1RBVEVfSU5fTE9HSU4uCkRlYyAx MyAxNzo1NTowMSByaGVsNzNuMSBrZXJuZWw6IEVudGVyaW5nIGlzY3NpX3RhcmdldF9zZXRf c29ja19jYWxsYmFja3M6IGNvbm46IGZmZmY4OGIzNWNiYjMwMDAKRGVjIDEzIDE3OjU1OjAx IHJoZWw3M24xIGtlcm5lbDogTG9jYXRlZCBTdG9yYWdlIE9iamVjdDogaXFuLjIwMDMtMDEu Y29tLnJlZGhhdDpjZXBoLWlzY3NpCkRlYyAxMyAxNzo1NTowMSByaGVsNzNuMSBrZXJuZWw6 IExvY2F0ZWQgUG9ydGFsIEdyb3VwIE9iamVjdDogMQpEZWMgMTMgMTc6NTU6MDEgcmhlbDcz bjEga2VybmVsOiBpU0NTSSBQYXJhbWV0ZXIgdXBkYXRlZCB0byBUYXJnZXRQb3J0YWxHcm91 cFRhZz0xCkRlYyAxMyAxNzo1NTowMSByaGVsNzNuMSBrZXJuZWw6IGlTQ1NJIFBhcmFtZXRl ciB1cGRhdGVkIHRvIEVycm9yUmVjb3ZlcnlMZXZlbD0wCkRlYyAxMyAxNzo1NTowMSByaGVs NzNuMSBrZXJuZWw6IEdvdCBrZXk6IEluaXRpYXRvck5hbWU9aXFuLjIwMDUtMDMuY29tLmNl cGg6aW5pMQpEZWMgMTMgMTc6NTU6MDEgcmhlbDczbjEga2VybmVsOiBpU0NTSSBQYXJhbWV0 ZXIgdXBkYXRlZCB0byBJbml0aWF0b3JOYW1lPWlxbi4yMDA1LTAzLmNvbS5jZXBoOmluaTEK RGVjIDEzIDE3OjU1OjAxIHJoZWw3M24xIGtlcm5lbDogR290IGtleTogSW5pdGlhdG9yQWxp YXM9cmgyCkRlYyAxMyAxNzo1NTowMSByaGVsNzNuMSBrZXJuZWw6IGlTQ1NJIFBhcmFtZXRl ciB1cGRhdGVkIHRvIEluaXRpYXRvckFsaWFzPXJoMgpEZWMgMTMgMTc6NTU6MDEgcmhlbDcz bjEga2VybmVsOiBHb3Qga2V5OiBUYXJnZXROYW1lPWlxbi4yMDAzLTAxLmNvbS5yZWRoYXQ6 Y2VwaC1pc2NzaQpEZWMgMTMgMTc6NTU6MDEgcmhlbDczbjEga2VybmVsOiBpU0NTSSBQYXJh bWV0ZXIgdXBkYXRlZCB0byBUYXJnZXROYW1lPWlxbi4yMDAzLTAxLmNvbS5yZWRoYXQ6Y2Vw aC1pc2NzaQpEZWMgMTMgMTc6NTU6MDEgcmhlbDczbjEga2VybmVsOiBHb3Qga2V5OiBTZXNz aW9uVHlwZT1Ob3JtYWwKRGVjIDEzIDE3OjU1OjAxIHJoZWw3M24xIGtlcm5lbDogaVNDU0kg UGFyYW1ldGVyIHVwZGF0ZWQgdG8gU2Vzc2lvblR5cGU9Tm9ybWFsCkRlYyAxMyAxNzo1NTow MSByaGVsNzNuMSBrZXJuZWw6IEdvdCBrZXk6IEF1dGhNZXRob2Q9Q0hBUCxOb25lCkRlYyAx MyAxNzo1NTowMSByaGVsNzNuMSBrZXJuZWw6IGlTQ1NJIFBhcmFtZXRlciB1cGRhdGVkIHRv IEF1dGhNZXRob2Q9Q0hBUApEZWMgMTMgMTc6NTU6MDEgcmhlbDczbjEga2VybmVsOiBTZW5k aW5nIGtleTogQXV0aE1ldGhvZD1DSEFQCkRlYyAxMyAxNzo1NTowMSByaGVsNzNuMSBrZXJu ZWw6IFNlbmRpbmcga2V5OiBUYXJnZXRBbGlhcz1MSU8gVGFyZ2V0CkRlYyAxMyAxNzo1NTow MSByaGVsNzNuMSBrZXJuZWw6IFNlbmRpbmcga2V5OiBUYXJnZXRQb3J0YWxHcm91cFRhZz0x CkRlYyAxMyAxNzo1NTowMSByaGVsNzNuMSBrZXJuZWw6IFNlbmRpbmcgTG9naW4gUmVzcG9u c2UsIEZsYWdzOiAweDAwLCBJVFQ6IDB4MDAwMDAwMDAsIEV4cENtZFNOOyAweDAwMDAwMDAw LCBNYXhDbWRTTjogMHgwMDAwMDAwMCwgU3RhdFNOOiAweDliYzY2ZGY2LCBMZW5ndGg6IDYy CkRlYyAxMyAxNzo1NTowMSByaGVsNzNuMSBrZXJuZWw6IHR4X2xvb3A6IDExMiwgdG90YWxf dHg6IDExMiwgZGF0YTogMTEyCkRlYyAxMyAxNzo1NTowMSByaGVsNzNuMSBrZXJuZWw6IE1v dmluZyB0byBUQVJHX0NPTk5fU1RBVEVfRlJFRS4KRGVjIDEzIDE3OjU1OjAxIHJoZWw3M24x IGtlcm5lbDogRW50ZXJpbmcgaXNjc2lfdGFyZ2V0X3NrX2RhdGFfcmVhZHk6IGNvbm46IGZm ZmY4OGIzNWNiYjMwMDAKRGVjIDEzIDE3OjU1OjAxIHJoZWw3M24xIGtlcm5lbDogRW50ZXJp bmcgaXNjc2lfdGFyZ2V0X3NrX2RhdGFfcmVhZHk6IGNvbm46IGZmZmY4OGIzNWNiYjMwMDAK RGVjIDEzIDE3OjU1OjAxIHJoZWw3M24xIGtlcm5lbDogR290IExPR0lOX0ZMQUdTX1JFQURf QUNUSVZFPTEsIGNvbm46IGZmZmY4OGIzNWNiYjMwMDAgPj4+PgpEZWMgMTMgMTc6NTU6MDEg cmhlbDczbjEga2VybmVsOiBlbnRlcmluZyBpc2NzaV90YXJnZXRfZG9fbG9naW5fcngsIGNv bm46IGZmZmY4OGIzNWNiYjMwMDAsIGt3b3JrZXIvMjoyOjE4MjkKRGVjIDEzIDE3OjU1OjAx IHJoZWw3M24xIGtlcm5lbDogU3RhcnRpbmcgbG9naW5fdGltZXIgZm9yIGt3b3JrZXIvMjoy LzE4MjkKRGVjIDEzIDE3OjU1OjAxIHJoZWw3M24xIGtlcm5lbDogcnhfbG9vcDogNDgsIHRv dGFsX3J4OiA0OCwgZGF0YTogNDgKRGVjIDEzIDE3OjU1OjAxIHJoZWw3M24xIGtlcm5lbDog R290IExvZ2luIENvbW1hbmQsIEZsYWdzIDB4MDAsIElUVDogMHgwMDAwMDAwMCwgQ21kU046 IDB4MDAwMDAwMDAsIEV4cFN0YXRTTjogMHhmNzZkYzY5YiwgQ0lEOiAwLCBMZW5ndGg6IDkK RGVjIDEzIDE3OjU1OjAxIHJoZWw3M24xIGtlcm5lbDogcnhfbG9vcDogMTIsIHRvdGFsX3J4 OiAxMiwgZGF0YTogMTIKRGVjIDEzIDE3OjU1OjAxIHJoZWw3M24xIGtlcm5lbDogaXNjc2lf dGFyZ2V0X2RvX2xvZ2luX3J4IGFmdGVyIHJ4X2xvZ2luX2lvLCBmZmZmODhiMzVjYmIzMDAw LCBrd29ya2VyLzI6MjoxODI5CkRlYyAxMyAxNzo1NTowMSByaGVsNzNuMSBrZXJuZWw6IEdv dCBrZXk6IENIQVBfQT01CkRlYyAxMyAxNzo1NTowMSByaGVsNzNuMSBrZXJuZWw6IFNlbGVj dGVkIE1ENSBBbGdvcml0aG0KRGVjIDEzIDE3OjU1OjAxIHJoZWw3M24xIGtlcm5lbDogW3Nl cnZlcl0gR290IENIQVBfQT01CkRlYyAxMyAxNzo1NTowMSByaGVsNzNuMSBrZXJuZWw6IFtz ZXJ2ZXJdIFNlbmRpbmcgQ0hBUF9BPTUKRGVjIDEzIDE3OjU1OjAxIHJoZWw3M24xIGtlcm5l bDogW3NlcnZlcl0gU2VuZGluZyBDSEFQX0k9MgpEZWMgMTMgMTc6NTU6MDEgcmhlbDczbjEg a2VybmVsOiBbc2VydmVyXSBTZW5kaW5nIENIQVBfQz0weDNlYjYzOWQzNzBmYjdjODQxYTQw Mzg0NzQ1MzFjYmFhCkRlYyAxMyAxNzo1NTowMSByaGVsNzNuMSBrZXJuZWw6IFJlY2VpdmVk IE9LIHJlc3BvbnNlIGZyb20gTElPIEF1dGhlbnRpY2F0aW9uLCBjb250aW51aW5nLgpEZWMg MTMgMTc6NTU6MDEgcmhlbDczbjEga2VybmVsOiBTZW5kaW5nIExvZ2luIFJlc3BvbnNlLCBG bGFnczogMHgwMCwgSVRUOiAweDAwMDAwMDAwLCBFeHBDbWRTTjsgMHgwMDAwMDAwMCwgTWF4 Q21kU046IDB4MDAwMDAwMDAsIFN0YXRTTjogMHg5YmM2NmRmNywgTGVuZ3RoOiA2MApEZWMg MTMgMTc6NTU6MDEgcmhlbDczbjEga2VybmVsOiB0eF9sb29wOiAxMDgsIHRvdGFsX3R4OiAx MDgsIGRhdGE6IDEwOApEZWMgMTMgMTc6NTU6MDEgcmhlbDczbjEga2VybmVsOiBFbnRlcmlu ZyBpc2NzaV90YXJnZXRfc2tfZGF0YV9yZWFkeTogY29ubjogZmZmZjg4YjM1Y2JiMzAwMApE ZWMgMTMgMTc6NTU6MDEgcmhlbDczbjEga2VybmVsOiBlbnRlcmluZyBpc2NzaV90YXJnZXRf ZG9fbG9naW5fcngsIGNvbm46IGZmZmY4OGIzNWNiYjMwMDAsIGt3b3JrZXIvMjoyOjE4MjkK RGVjIDEzIDE3OjU1OjAxIHJoZWw3M24xIGtlcm5lbDogU3RhcnRpbmcgbG9naW5fdGltZXIg Zm9yIGt3b3JrZXIvMjoyLzE4MjkKRGVjIDEzIDE3OjU1OjAxIHJoZWw3M24xIGtlcm5lbDog cnhfbG9vcDogNDgsIHRvdGFsX3J4OiA0OCwgZGF0YTogNDgKRGVjIDEzIDE3OjU1OjAxIHJo ZWw3M24xIGtlcm5lbDogR290IExvZ2luIENvbW1hbmQsIEZsYWdzIDB4ODEsIElUVDogMHgw MDAwMDAwMCwgQ21kU046IDB4MDAwMDAwMDAsIEV4cFN0YXRTTjogMHhmODZkYzY5YiwgQ0lE OiAwLCBMZW5ndGg6IDY1CkRlYyAxMyAxNzo1NTowMSByaGVsNzNuMSBrZXJuZWw6IEVudGVy aW5nIGlzY3NpX3RhcmdldF9za19kYXRhX3JlYWR5OiBjb25uOiBmZmZmODhiMzVjYmIzMDAw CkRlYyAxMyAxNzo1NTowMSByaGVsNzNuMSBrZXJuZWw6IEdvdCBMT0dJTl9GTEFHU19SRUFE X0FDVElWRT0xLCBjb25uOiBmZmZmODhiMzVjYmIzMDAwID4+Pj4KRGVjIDEzIDE3OjU1OjAx IHJoZWw3M24xIHN5c3RlbWRbMV06IENyZWF0ZWQgc2xpY2UgVXNlciBTbGljZSBvZiBwY3Au CkRlYyAxMyAxNzo1NTowMSByaGVsNzNuMSBzeXN0ZW1kWzFdOiBTdGFydGluZyBVc2VyIFNs aWNlIG9mIHBjcC4KRGVjIDEzIDE3OjU1OjAxIHJoZWw3M24xIHN5c3RlbWRbMV06IFN0YXJ0 ZWQgU2Vzc2lvbiA4IG9mIHVzZXIgcGNwLgpEZWMgMTMgMTc6NTU6MDEgcmhlbDczbjEgc3lz dGVtZFsxXTogU3RhcnRpbmcgU2Vzc2lvbiA4IG9mIHVzZXIgcGNwLgpEZWMgMTMgMTc6NTU6 MDEgcmhlbDczbjEgQ1JPTkRbMTg0OF06IChwY3ApIENNRCAoIC91c3IvbGliZXhlYy9wY3Av YmluL3BtbG9nZ2VyX2NoZWNrIC1DKQpEZWMgMTMgMTc6NTU6MDEgcmhlbDczbjEgc3lzdGVt ZFsxXTogUmVtb3ZlZCBzbGljZSBVc2VyIFNsaWNlIG9mIHBjcC4KRGVjIDEzIDE3OjU1OjAx IHJoZWw3M24xIHN5c3RlbWRbMV06IFN0b3BwaW5nIFVzZXIgU2xpY2Ugb2YgcGNwLgpEZWMg MTMgMTc6NTU6MTYgcmhlbDczbjEga2VybmVsOiBFbnRlcmluZyBpc2NzaV90YXJnZXRfc2tf c3RhdGVfY2hhbmdlCkRlYyAxMyAxNzo1NToxNiByaGVsNzNuMSBrZXJuZWw6IF9faXNjc2lf dGFyZ2V0X3NrX2NoZWNrX2Nsb3NlOiBUQ1BfQ0xPU0VfV0FJVHxUQ1BfQ0xPU0UscmV0dXJu aW5nIEZBTFNFCkRlYyAxMyAxNzo1NToxNiByaGVsNzNuMSBrZXJuZWw6IF9faXNjc2lfdGFy Z2V0X3NrX2Nsb3NlX2NoYW5nZTogc3RhdGU6IDEKRGVjIDEzIDE3OjU1OjE2IHJoZWw3M24x IGtlcm5lbDogR290IExPR0lOX0ZMQUdTX1JFQURfQUNUSVZFPTEgc2tfc3RhdGVfY2hhbmdl IGNvbm46IGZmZmY4OGIzNWNiYjMwMDAKRGVjIDEzIDE3OjU1OjE2IHJoZWw3M24xIGtlcm5l bDogcnhfbG9vcDogNjgsIHRvdGFsX3J4OiA2OCwgZGF0YTogNjgKRGVjIDEzIDE3OjU1OjE2 IHJoZWw3M24xIGtlcm5lbDogaXNjc2lfdGFyZ2V0X2RvX2xvZ2luX3J4IGFmdGVyIHJ4X2xv Z2luX2lvLCBmZmZmODhiMzVjYmIzMDAwLCBrd29ya2VyLzI6MjoxODI5CkRlYyAxMyAxNzo1 NToxNiByaGVsNzNuMSBrZXJuZWw6IEdvdCBrZXk6IENIQVBfTj1teWlzY3NpdXNlcm5hbWUK RGVjIDEzIDE3OjU1OjE2IHJoZWw3M24xIGtlcm5lbDogW3NlcnZlcl0gR290IENIQVBfTj1t eWlzY3NpdXNlcm5hbWUKRGVjIDEzIDE3OjU1OjE2IHJoZWw3M24xIGtlcm5lbDogW3NlcnZl cl0gR290IENIQVBfUj1hMzNiNTY0NTJmMWYwNmQzMzFjOWVmOTkwYThhMmUyNApEZWMgMTMg MTc6NTU6MTYgcmhlbDczbjEga2VybmVsOiBbc2VydmVyXSBNRDUgU2VydmVyIERpZ2VzdDog YTMzYjU2NDUyZjFmMDZkMzMxYzllZjk5MGE4YTJlMjQKRGVjIDEzIDE3OjU1OjE2IHJoZWw3 M24xIGtlcm5lbDogW3NlcnZlcl0gTUQ1IERpZ2VzdHMgbWF0Y2gsIENIQVAgY29ubmVjdGlv biBzdWNjZXNzZnVsLgpEZWMgMTMgMTc6NTU6MTYgcmhlbDczbjEga2VybmVsOiBpU0NTSSBz ZWN1cml0eSBuZWdvdGlhdGlvbiBjb21wbGV0ZWQgc3VjY2Vzc2Z1bGx5LgpEZWMgMTMgMTc6 NTU6MTYgcmhlbDczbjEga2VybmVsOiBTZW5kaW5nIExvZ2luIFJlc3BvbnNlLCBGbGFnczog MHg4MSwgSVRUOiAweDAwMDAwMDAwLCBFeHBDbWRTTjsgMHgwMDAwMDAwMCwgTWF4Q21kU046 IDB4MDAwMDAwMDAsIFN0YXRTTjogMHg5YmM2NmRmOCwgTGVuZ3RoOiAwCkRlYyAxMyAxNzo1 NToxNiByaGVsNzNuMSBrZXJuZWw6IHR4X2xvb3A6IC0xMDQgdG90YWxfdHggMApEZWMgMTMg MTc6NTU6MTYgcmhlbDczbjEga2VybmVsOiB0eF9kYXRhIHJldHVybmVkIC0xMDQsIGV4cGVj dGluZyA0OC4KRGVjIDEzIDE3OjU1OjE2IHJoZWw3M24xIGtlcm5lbDogRW50ZXJpbmcgaXNj c2lfdGFyZ2V0X3Jlc3RvcmVfc29ja19jYWxsYmFja3M6IGNvbm46IGZmZmY4OGIzNWNiYjMw MDAKRGVjIDEzIDE3OjU1OjE2IHJoZWw3M24xIGtlcm5lbDogaVNDU0kgTG9naW4gbmVnb3Rp YXRpb24gZmFpbGVkLgoKCgoKCgoKCgoKCgoKCgo= --------------070703050208000100010308--