From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 020F4C04AA5 for ; Thu, 25 Aug 2022 12:07:59 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231438AbiHYMH7 (ORCPT ); Thu, 25 Aug 2022 08:07:59 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:49342 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231283AbiHYMH6 (ORCPT ); Thu, 25 Aug 2022 08:07:58 -0400 Received: from mail-qk1-x72a.google.com (mail-qk1-x72a.google.com [IPv6:2607:f8b0:4864:20::72a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 34586AA4C7 for ; Thu, 25 Aug 2022 05:07:56 -0700 (PDT) Received: by mail-qk1-x72a.google.com with SMTP id b2so14937298qkh.12 for ; Thu, 25 Aug 2022 05:07:56 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=broadcom.com; s=google; h=cc:to:subject:message-id:date:thread-index:mime-version:in-reply-to :references:from:from:to:cc; bh=GtJwD94B81UeJw4SrTkkDVYA89Bhgibqw30sewM4dog=; b=MENgpmx/E7Azil0V4vhu9VsYT0Bd30HOLPXmI6DXbrZ9UnQk+KoRsQQP+c9f9yhw6X s/JaxcCaqVGHRw7nEVrEz8cke9BDUAYgYFOQYyink14FvaIUPJ7lsnnlcRI8EHzzDddX THnnoEME4+naRNNt5a5MXUVr9/tK+L/XjlIyU= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:subject:message-id:date:thread-index:mime-version:in-reply-to :references:from:x-gm-message-state:from:to:cc; bh=GtJwD94B81UeJw4SrTkkDVYA89Bhgibqw30sewM4dog=; b=ttNzGOELTiJ6cxI7flRN7aCBuaCoRqsSmMH9TweBAFEl9GZllMQPBY8VnOZ88f6+ko 3+iPRHxIV7Ufh/jXof/y9U4MLXWY4q/66LP7fX4xVI1sMW1lT8pAmi1jndcvo0TgIyI9 cQQqANLz8Qa5akAVKzfCz1VWiDsl0J3Q63GVqv153M4mCvEAMm4+k3L0V5gCeWmU4+/o BEjfKtfS8YfuxzKna3e41JE/RvTtwMFFc0s1mn69LlAJ/UTnxJ+pL6BltwNw64+7kN9n VJefKiOAnQMuwcQhtek9BsejPLzwb31KVruOvFv15dOcZ9v4OeCZ9pwDDjnMtnlUtGLl ZjzA== X-Gm-Message-State: ACgBeo0+6kW1Oj6BDoGXrnUaApfoHj6MXmw6HaJK5nnKAw/n6gSjfpDU 2IW5c5lAY3PMf6cJH/HJLL9NeywNkBkQodgekeUrSA== X-Google-Smtp-Source: AA6agR5/dv2iUWmzBvyrgFnYQNRAjndntTRg+lBcb0bGHIPQFNrDtoiZQj8EhfnQUWIASDWsWbG4LrUHt1dAdlyuXPY= X-Received: by 2002:a37:4c2:0:b0:6bb:f708:5878 with SMTP id 185-20020a3704c2000000b006bbf7085878mr2639442qke.772.1661429275314; Thu, 25 Aug 2022 05:07:55 -0700 (PDT) From: Kashyap Desai References: <20220824124236.812395-1-kashyap.desai@broadcom.com> <8288a63a-d708-4e41-78fb-e3ed7d6fd9c2@nvidia.com> In-Reply-To: <8288a63a-d708-4e41-78fb-e3ed7d6fd9c2@nvidia.com> MIME-Version: 1.0 X-Mailer: Microsoft Outlook 15.0 Thread-Index: AQEQ25uoTbK0MBSJOrKF/AGGatZ0AgH8+4TBrz9EebA= Date: Thu, 25 Aug 2022 17:37:52 +0530 Message-ID: Subject: RE: [PATCH RFC] IB/iser: add task reference counter for tx commands To: Max Gurtovoy , linux-rdma@vger.kernel.org Cc: Selvin Xavier , Andrew Gospodarek , Jason Gunthorpe , Sagi Grimberg Content-Type: multipart/signed; protocol="application/pkcs7-signature"; micalg=sha-256; boundary="000000000000ca4a0f05e70fa54b" Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org --000000000000ca4a0f05e70fa54b Content-Type: text/plain; charset="UTF-8" > > Hi Kashyap Desai, > > Unfortunately, there is a wider problem in iser that we do the local > invalidation if any, after we complete the iscsi task. > So the right solution is to do the logic we have in NVMe/RDMA that checks > if > remote invalidation happened and if not, it does local invalidation. > And the task is released after getting the send_completion and > local_invalidation/remote invalidation. Hi Max , I noticed the same. Agree that this RFC is not close to actual solution. Final solution require major lift in iSER instead of just adding refcount in couple of places. In case of NVME every ib_post_send is posted with IB_SEND_SIGNALED and nvme_rdma_send_done does the better job of handling out of order TX and RX completion. nvme_rdma_end_request() can be called either nvme_rdma_process_nvme_rsp, nvme_rdma_send_done OR nvme_rdma_inv_rkey_done. Similar concept is what we need in iSER side as well. Right ? Currently iSER stack is freeing scsi request only from iser_task_rsp context. > > There is some infrastructure work needed to be done there. I would like to contribute if you want me to test, review and/or co develop the changes. > > Sagi, > > Please remind me few things regarding the iSCSI bidir cmds. > > In case of bidir IO cmd, if we need to use a reg_mr for it - I see a > potential > problem there. > Is this scenario possible ? > > I'm asking because if it is possible, so what happens in remote > invalidation ? > what key is invalidated ? the read_key or the write_key ? > in ib_isert there is a priority to the read_key (is it by the spec ?). > We don't consider this in the iser recv completion and the remote > invalidation > checker logic. > If it's not possible we should re-design few components in the code and > fix the > issue that we do local_invalidation of cmd N during the send of cmd N + 1. > > Cheers, > -Max. > --000000000000ca4a0f05e70fa54b Content-Type: application/pkcs7-signature; name="smime.p7s" Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="smime.p7s" Content-Description: S/MIME Cryptographic Signature MIIQcAYJKoZIhvcNAQcCoIIQYTCCEF0CAQExDzANBglghkgBZQMEAgEFADALBgkqhkiG9w0BBwGg gg3HMIIFDTCCA/WgAwIBAgIQeEqpED+lv77edQixNJMdADANBgkqhkiG9w0BAQsFADBMMSAwHgYD VQQLExdHbG9iYWxTaWduIFJvb3QgQ0EgLSBSMzETMBEGA1UEChMKR2xvYmFsU2lnbjETMBEGA1UE AxMKR2xvYmFsU2lnbjAeFw0yMDA5MTYwMDAwMDBaFw0yODA5MTYwMDAwMDBaMFsxCzAJBgNVBAYT AkJFMRkwFwYDVQQKExBHbG9iYWxTaWduIG52LXNhMTEwLwYDVQQDEyhHbG9iYWxTaWduIEdDQyBS MyBQZXJzb25hbFNpZ24gMiBDQSAyMDIwMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA vbCmXCcsbZ/a0fRIQMBxp4gJnnyeneFYpEtNydrZZ+GeKSMdHiDgXD1UnRSIudKo+moQ6YlCOu4t rVWO/EiXfYnK7zeop26ry1RpKtogB7/O115zultAz64ydQYLe+a1e/czkALg3sgTcOOcFZTXk38e aqsXsipoX1vsNurqPtnC27TWsA7pk4uKXscFjkeUE8JZu9BDKaswZygxBOPBQBwrA5+20Wxlk6k1 e6EKaaNaNZUy30q3ArEf30ZDpXyfCtiXnupjSK8WU2cK4qsEtj09JS4+mhi0CTCrCnXAzum3tgcH cHRg0prcSzzEUDQWoFxyuqwiwhHu3sPQNmFOMwIDAQABo4IB2jCCAdYwDgYDVR0PAQH/BAQDAgGG MGAGA1UdJQRZMFcGCCsGAQUFBwMCBggrBgEFBQcDBAYKKwYBBAGCNxQCAgYKKwYBBAGCNwoDBAYJ KwYBBAGCNxUGBgorBgEEAYI3CgMMBggrBgEFBQcDBwYIKwYBBQUHAxEwEgYDVR0TAQH/BAgwBgEB /wIBADAdBgNVHQ4EFgQUljPR5lgXWzR1ioFWZNW+SN6hj88wHwYDVR0jBBgwFoAUj/BLf6guRSSu TVD6Y5qL3uLdG7wwegYIKwYBBQUHAQEEbjBsMC0GCCsGAQUFBzABhiFodHRwOi8vb2NzcC5nbG9i YWxzaWduLmNvbS9yb290cjMwOwYIKwYBBQUHMAKGL2h0dHA6Ly9zZWN1cmUuZ2xvYmFsc2lnbi5j b20vY2FjZXJ0L3Jvb3QtcjMuY3J0MDYGA1UdHwQvMC0wK6ApoCeGJWh0dHA6Ly9jcmwuZ2xvYmFs c2lnbi5jb20vcm9vdC1yMy5jcmwwWgYDVR0gBFMwUTALBgkrBgEEAaAyASgwQgYKKwYBBAGgMgEo CjA0MDIGCCsGAQUFBwIBFiZodHRwczovL3d3dy5nbG9iYWxzaWduLmNvbS9yZXBvc2l0b3J5LzAN BgkqhkiG9w0BAQsFAAOCAQEAdAXk/XCnDeAOd9nNEUvWPxblOQ/5o/q6OIeTYvoEvUUi2qHUOtbf jBGdTptFsXXe4RgjVF9b6DuizgYfy+cILmvi5hfk3Iq8MAZsgtW+A/otQsJvK2wRatLE61RbzkX8 9/OXEZ1zT7t/q2RiJqzpvV8NChxIj+P7WTtepPm9AIj0Keue+gS2qvzAZAY34ZZeRHgA7g5O4TPJ /oTd+4rgiU++wLDlcZYd/slFkaT3xg4qWDepEMjT4T1qFOQIL+ijUArYS4owpPg9NISTKa1qqKWJ jFoyms0d0GwOniIIbBvhI2MJ7BSY9MYtWVT5jJO3tsVHwj4cp92CSFuGwunFMzCCA18wggJHoAMC AQICCwQAAAAAASFYUwiiMA0GCSqGSIb3DQEBCwUAMEwxIDAeBgNVBAsTF0dsb2JhbFNpZ24gUm9v dCBDQSAtIFIzMRMwEQYDVQQKEwpHbG9iYWxTaWduMRMwEQYDVQQDEwpHbG9iYWxTaWduMB4XDTA5 MDMxODEwMDAwMFoXDTI5MDMxODEwMDAwMFowTDEgMB4GA1UECxMXR2xvYmFsU2lnbiBSb290IENB IC0gUjMxEzARBgNVBAoTCkdsb2JhbFNpZ24xEzARBgNVBAMTCkdsb2JhbFNpZ24wggEiMA0GCSqG SIb3DQEBAQUAA4IBDwAwggEKAoIBAQDMJXaQeQZ4Ihb1wIO2hMoonv0FdhHFrYhy/EYCQ8eyip0E XyTLLkvhYIJG4VKrDIFHcGzdZNHr9SyjD4I9DCuul9e2FIYQebs7E4B3jAjhSdJqYi8fXvqWaN+J J5U4nwbXPsnLJlkNc96wyOkmDoMVxu9bi9IEYMpJpij2aTv2y8gokeWdimFXN6x0FNx04Druci8u nPvQu7/1PQDhBjPogiuuU6Y6FnOM3UEOIDrAtKeh6bJPkC4yYOlXy7kEkmho5TgmYHWyn3f/kRTv riBJ/K1AFUjRAjFhGV64l++td7dkmnq/X8ET75ti+w1s4FRpFqkD2m7pg5NxdsZphYIXAgMBAAGj QjBAMA4GA1UdDwEB/wQEAwIBBjAPBgNVHRMBAf8EBTADAQH/MB0GA1UdDgQWBBSP8Et/qC5FJK5N UPpjmove4t0bvDANBgkqhkiG9w0BAQsFAAOCAQEAS0DbwFCq/sgM7/eWVEVJu5YACUGssxOGhigH M8pr5nS5ugAtrqQK0/Xx8Q+Kv3NnSoPHRHt44K9ubG8DKY4zOUXDjuS5V2yq/BKW7FPGLeQkbLmU Y/vcU2hnVj6DuM81IcPJaP7O2sJTqsyQiunwXUaMld16WCgaLx3ezQA3QY/tRG3XUyiXfvNnBB4V 14qWtNPeTCekTBtzc3b0F5nCH3oO4y0IrQocLP88q1UOD5F+NuvDV0m+4S4tfGCLw0FREyOdzvcy a5QBqJnnLDMfOjsl0oZAzjsshnjJYS8Uuu7bVW/fhO4FCU29KNhyztNiUGUe65KXgzHZs7XKR1g/ XzCCBU8wggQ3oAMCAQICDHA7TgNc55htm2viYDANBgkqhkiG9w0BAQsFADBbMQswCQYDVQQGEwJC RTEZMBcGA1UEChMQR2xvYmFsU2lnbiBudi1zYTExMC8GA1UEAxMoR2xvYmFsU2lnbiBHQ0MgUjMg UGVyc29uYWxTaWduIDIgQ0EgMjAyMDAeFw0yMTAyMjIxMjU2MDJaFw0yMjA5MTUxMTQ1MTZaMIGQ MQswCQYDVQQGEwJJTjESMBAGA1UECBMJS2FybmF0YWthMRIwEAYDVQQHEwlCYW5nYWxvcmUxFjAU BgNVBAoTDUJyb2FkY29tIEluYy4xFjAUBgNVBAMTDUthc2h5YXAgRGVzYWkxKTAnBgkqhkiG9w0B CQEWGmthc2h5YXAuZGVzYWlAYnJvYWRjb20uY29tMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIB CgKCAQEAzPAzyHBqFL/1u7ttl86wZrWK3vYcqFH+GBe0laKvAGOuEkaHijHa8iH+9GA8FUv1cdWF WY3c3BGA+omJGYc4eHLEyKowuLRWvjV3MEjGBG7NIVoIaTkH4R+6Xs1P4/9EmUA0WI881B3pTv5W nHG54/aqGUDSRDyWVhK7TLqJQkkiYKB0kH0GkB/UfmU/pmCaV68w5J6l4vz/TG23hWJmTg1lW5mu P3lSxcw4Cg90iKHqfpwLnGNc9AGXHMxUCukpnAHRlivljilKHMx1ymb180BLmtF+ZLm6KrFLQWzB 4KeiUOMtKM13wJrQubqTeZgB1XA+89jeLYlxagVsMyksdwIDAQABo4IB2zCCAdcwDgYDVR0PAQH/ BAQDAgWgMIGjBggrBgEFBQcBAQSBljCBkzBOBggrBgEFBQcwAoZCaHR0cDovL3NlY3VyZS5nbG9i YWxzaWduLmNvbS9jYWNlcnQvZ3NnY2NyM3BlcnNvbmFsc2lnbjJjYTIwMjAuY3J0MEEGCCsGAQUF BzABhjVodHRwOi8vb2NzcC5nbG9iYWxzaWduLmNvbS9nc2djY3IzcGVyc29uYWxzaWduMmNhMjAy MDBNBgNVHSAERjBEMEIGCisGAQQBoDIBKAowNDAyBggrBgEFBQcCARYmaHR0cHM6Ly93d3cuZ2xv YmFsc2lnbi5jb20vcmVwb3NpdG9yeS8wCQYDVR0TBAIwADBJBgNVHR8EQjBAMD6gPKA6hjhodHRw Oi8vY3JsLmdsb2JhbHNpZ24uY29tL2dzZ2NjcjNwZXJzb25hbHNpZ24yY2EyMDIwLmNybDAlBgNV HREEHjAcgRprYXNoeWFwLmRlc2FpQGJyb2FkY29tLmNvbTATBgNVHSUEDDAKBggrBgEFBQcDBDAf BgNVHSMEGDAWgBSWM9HmWBdbNHWKgVZk1b5I3qGPzzAdBgNVHQ4EFgQUkTOZp9jXE3yPj4ieKeDT OiNyCtswDQYJKoZIhvcNAQELBQADggEBABG1KCh7cLjStywh4S37nKE1eE8KPyAxDzQCkhxYLBVj gnnhaLmEOayEucPAsM1hCRAm/vR3RQ27lMXBGveCHaq9RZkzTjGSbzr8adOGK3CluPrasNf5StX3 GSk4HwCapA39BDUrhnc/qG5vHwLrgA1jwAvSy8e/vn4F4h+KPrPoFNd1OnCafedbuiEXTqTkn5Rk vZ2AOTcSbxvmyKBMb/iu1vn7AAoui0d8GYCPoz8shf2iWMSUXVYJAMrtRHVJr47J5jlopF5F2ghC MzNfx6QsmJhYiRByd8L9sUOjp/DMgkC6H93PyYpYMiBGapgNf6UMsLg/1kx5DATNwhPAJbkxggJt MIICaQIBATBrMFsxCzAJBgNVBAYTAkJFMRkwFwYDVQQKExBHbG9iYWxTaWduIG52LXNhMTEwLwYD VQQDEyhHbG9iYWxTaWduIEdDQyBSMyBQZXJzb25hbFNpZ24gMiBDQSAyMDIwAgxwO04DXOeYbZtr 4mAwDQYJYIZIAWUDBAIBBQCggdQwLwYJKoZIhvcNAQkEMSIEILnBXR/oCGhlEW+zTSV+uFVTUjxS xxH7NrKJTWQYrVtJMBgGCSqGSIb3DQEJAzELBgkqhkiG9w0BBwEwHAYJKoZIhvcNAQkFMQ8XDTIy MDgyNTEyMDc1NVowaQYJKoZIhvcNAQkPMVwwWjALBglghkgBZQMEASowCwYJYIZIAWUDBAEWMAsG CWCGSAFlAwQBAjAKBggqhkiG9w0DBzALBgkqhkiG9w0BAQowCwYJKoZIhvcNAQEHMAsGCWCGSAFl AwQCATANBgkqhkiG9w0BAQEFAASCAQA4ps0uylDgpWAYqdfi3FERPGWRqKleDql+0BnJez2bvGiK z62aNkaybKt4dgz3aczz/0vJ+RJoGMoSMxJOXl1KeyrZKoTm1s9KbjtEUFUU9eJQZfEYg9j+Z18U bHZjqFoSAT26VtQI/3X3gBqZSj/liRTKt3g8n8xjZHeaVWcdFaspFw+gK2hNA3fdcNfQT/AkamJj 4WkEHExBCoEqycoc/rQq8J3LRoNrlTKndJWAzPZKegdELShk2Pg9CYsrOzjWV35TjqKJe9K8ISlc NqRP+TZwZQkM6pl2DJebDAt8wH/oYKDMmbh0K41V+nnAoqVqSeKg6cemlZJtGzTvbDjF --000000000000ca4a0f05e70fa54b--