From mboxrd@z Thu Jan  1 00:00:00 1970
Return-Path: <netdev-owner@vger.kernel.org>
X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on
	aws-us-west-2-korg-lkml-1.web.codeaurora.org
Received: from vger.kernel.org (vger.kernel.org [23.128.96.18])
	by smtp.lore.kernel.org (Postfix) with ESMTP id D7114C54EBE
	for <netdev@archiver.kernel.org>; Mon, 16 Jan 2023 13:07:00 +0000 (UTC)
Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand
        id S231152AbjAPNG5 (ORCPT <rfc822;netdev@archiver.kernel.org>);
        Mon, 16 Jan 2023 08:06:57 -0500
Received: from lindbergh.monkeyblade.net ([23.128.96.19]:55128 "EHLO
        lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org
        with ESMTP id S231209AbjAPNGi (ORCPT
        <rfc822;netdev@vger.kernel.org>); Mon, 16 Jan 2023 08:06:38 -0500
Received: from ams.source.kernel.org (ams.source.kernel.org [IPv6:2604:1380:4601:e00::1])
        by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 819283ABB;
        Mon, 16 Jan 2023 05:06:08 -0800 (PST)
Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140])
        (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
        (No client certificate requested)
        by ams.source.kernel.org (Postfix) with ESMTPS id 320BBB80E37;
        Mon, 16 Jan 2023 13:06:07 +0000 (UTC)
Received: by smtp.kernel.org (Postfix) with ESMTPSA id 1E9ECC433EF;
        Mon, 16 Jan 2023 13:06:05 +0000 (UTC)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org;
        s=k20201202; t=1673874365;
        bh=rNdp8EhsRlR5Ld3e2m1Ss4Ma8PuD1081i4b85L/XLws=;
        h=From:To:Cc:Subject:Date:From;
        b=IEbq92PfyqWg1QDwxKyyWT4dZQl4cCMHxgcSytzpU6C+EQ8kHuwbL/i4b4s4lSHe9
         6AWMyv617QjAgpBTPasIqxYjGgeR9wh0T5T3rerp3l4jTTm8vgaIT8iznUz38gMp9j
         ifQqOFD4SdOOTenk2PqaNWRPr6HRr6t9scWWpkCFDaA/vIH11jBcQPDGoKFY3dQTOy
         L+2Ok+ApFxP6iVtXTZcfVNYq+DWhcWBN9fHzlLD79cN3fGbOFxslD8InNf+KGMTKgU
         2BqHGnr9Z1r1MmJyR4JIH8tWicS/dLAKXcPBA2SykiC/XFWCR0jjNiqHAGDKTHfJq1
         9Ggv8K4bc352Q==
From:   Leon Romanovsky <leon@kernel.org>
To:     Jason Gunthorpe <jgg@nvidia.com>
Cc:     Bryan Tan <bryantan@vmware.com>, Christoph Hellwig <hch@lst.de>,
        Eric Dumazet <edumazet@google.com>,
        Israel Rukshin <israelr@nvidia.com>,
        Jakub Kicinski <kuba@kernel.org>, Jens Axboe <axboe@fb.com>,
        Keith Busch <kbusch@kernel.org>, linux-kernel@vger.kernel.org,
        linux-nvme@lists.infradead.org, linux-rdma@vger.kernel.org,
        linux-trace-kernel@vger.kernel.org,
        Masami Hiramatsu <mhiramat@kernel.org>,
        Max Gurtovoy <mgurtovoy@nvidia.com>, netdev@vger.kernel.org,
        Paolo Abeni <pabeni@redhat.com>,
        Saeed Mahameed <saeedm@nvidia.com>,
        Sagi Grimberg <sagi@grimberg.me>,
        Selvin Xavier <selvin.xavier@broadcom.com>,
        Steven Rostedt <rostedt@goodmis.org>,
        Vishnu Dasa <vdasa@vmware.com>,
        Yishai Hadas <yishaih@nvidia.com>
Subject: [PATCH rdma-next 00/13] Add RDMA inline crypto support
Date:   Mon, 16 Jan 2023 15:05:47 +0200
Message-Id: <cover.1673873422.git.leon@kernel.org>
X-Mailer: git-send-email 2.39.0
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit
Precedence: bulk
List-ID: <netdev.vger.kernel.org>
X-Mailing-List: netdev@vger.kernel.org

>From Israel,

The purpose of this patchset is to add support for inline
encryption/decryption of the data at storage protocols like nvmf over
RDMA (at a similar way like integrity is used via unique mkey).

This patchset adds support for plaintext keys. The patches were tested
on BF-3 HW with fscrypt tool to test this feature, which showed reduce
in CPU utilization when comparing at 64k or more IO size. The CPU utilization
was improved by more than 50% comparing to the SW only solution at this case.

How to configure fscrypt to enable plaintext keys:
 # mkfs.ext4 -O encrypt /dev/nvme0n1
 # mount /dev/nvme0n1 /mnt/crypto -o inlinecrypt
 # head -c 64 /dev/urandom > /tmp/master_key
 # fscryptctl add_key /mnt/crypto/ < /tmp/master_key
 # mkdir /mnt/crypto/test1
 # fscryptctl set_policy 152c41b2ea39fa3d90ea06448456e7fb /mnt/crypto/test1
   ** “152c41b2ea39fa3d90ea06448456e7fb” is the output of the
      “fscryptctl add_key” command.
 # echo foo > /mnt/crypto/test1/foo

Notes:
 - At plaintext mode only, the user set a master key and the fscrypt
   driver derived from it the DEK and the key identifier.
 - 152c41b2ea39fa3d90ea06448456e7fb is the derived key identifier
 - Only on the first IO, nvme-rdma gets a callback to load the derived DEK. 

There is no special configuration to support crypto at nvme modules.

Thanks

Israel Rukshin (13):
  net/mlx5: Introduce crypto IFC bits and structures
  net/mlx5: Introduce crypto capabilities macro
  RDMA: Split kernel-only create QP flags from uverbs create QP flags
  RDMA/core: Add cryptographic device capabilities
  RDMA/core: Add DEK management API
  RDMA/core: Introduce MR type for crypto operations
  RDMA/core: Add support for creating crypto enabled QPs
  RDMA/mlx5: Add cryptographic device capabilities
  RDMA/mlx5: Add DEK management API
  RDMA/mlx5: Add AES-XTS crypto support
  nvme: Introduce a local variable
  nvme: Add crypto profile at nvme controller
  nvme-rdma: Add inline encryption support

 drivers/infiniband/core/device.c              |   3 +
 drivers/infiniband/core/mr_pool.c             |   2 +
 drivers/infiniband/core/uverbs_std_types_qp.c |  12 +-
 drivers/infiniband/core/verbs.c               |  91 ++++++-
 drivers/infiniband/hw/bnxt_re/ib_verbs.c      |   2 +-
 drivers/infiniband/hw/mlx4/mlx4_ib.h          |   4 +-
 drivers/infiniband/hw/mlx4/qp.c               |   4 +-
 drivers/infiniband/hw/mlx5/Makefile           |   1 +
 drivers/infiniband/hw/mlx5/crypto.c           | 115 +++++++++
 drivers/infiniband/hw/mlx5/crypto.h           |  46 ++++
 drivers/infiniband/hw/mlx5/main.c             |   5 +
 drivers/infiniband/hw/mlx5/mlx5_ib.h          |   5 +-
 drivers/infiniband/hw/mlx5/mr.c               |  33 +++
 drivers/infiniband/hw/mlx5/qp.c               |  15 +-
 drivers/infiniband/hw/mlx5/wr.c               | 164 +++++++++++-
 drivers/infiniband/hw/vmw_pvrdma/pvrdma_qp.c  |   2 +-
 drivers/net/ethernet/mellanox/mlx5/core/fw.c  |   6 +
 .../net/ethernet/mellanox/mlx5/core/main.c    |   1 +
 drivers/nvme/host/core.c                      |  10 +-
 drivers/nvme/host/nvme.h                      |   4 +
 drivers/nvme/host/rdma.c                      | 236 +++++++++++++++++-
 include/linux/mlx5/device.h                   |   4 +
 include/linux/mlx5/mlx5_ifc.h                 |  36 ++-
 include/rdma/crypto.h                         | 118 +++++++++
 include/rdma/ib_verbs.h                       |  46 +++-
 include/trace/events/rdma_core.h              |  33 +++
 26 files changed, 954 insertions(+), 44 deletions(-)
 create mode 100644 drivers/infiniband/hw/mlx5/crypto.c
 create mode 100644 drivers/infiniband/hw/mlx5/crypto.h
 create mode 100644 include/rdma/crypto.h

-- 
2.39.0


From mboxrd@z Thu Jan  1 00:00:00 1970
Return-Path: <netdev-owner@vger.kernel.org>
X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on
	aws-us-west-2-korg-lkml-1.web.codeaurora.org
Received: from vger.kernel.org (vger.kernel.org [23.128.96.18])
	by smtp.lore.kernel.org (Postfix) with ESMTP id 09C51C678D9
	for <netdev@archiver.kernel.org>; Mon, 16 Jan 2023 13:07:03 +0000 (UTC)
Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand
        id S231346AbjAPNHA (ORCPT <rfc822;netdev@archiver.kernel.org>);
        Mon, 16 Jan 2023 08:07:00 -0500
Received: from lindbergh.monkeyblade.net ([23.128.96.19]:55224 "EHLO
        lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org
        with ESMTP id S229749AbjAPNGj (ORCPT
        <rfc822;netdev@vger.kernel.org>); Mon, 16 Jan 2023 08:06:39 -0500
Received: from ams.source.kernel.org (ams.source.kernel.org [IPv6:2604:1380:4601:e00::1])
        by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 76B371710;
        Mon, 16 Jan 2023 05:06:12 -0800 (PST)
Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140])
        (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
        (No client certificate requested)
        by ams.source.kernel.org (Postfix) with ESMTPS id 345E4B80E37;
        Mon, 16 Jan 2023 13:06:11 +0000 (UTC)
Received: by smtp.kernel.org (Postfix) with ESMTPSA id 1CDE9C433EF;
        Mon, 16 Jan 2023 13:06:09 +0000 (UTC)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org;
        s=k20201202; t=1673874369;
        bh=3vqVXUNkAHRx8zvBd2AMUzlpHePNDOP0eQM8jTGqV00=;
        h=From:To:Cc:Subject:Date:In-Reply-To:References:From;
        b=A+Ku8TJauHSmodpnxMWblBL/kJGeyDexnWB+oiYm9dRCuEjlso7zxFTrRg0d6jTQg
         rmfHIM76X4oKr/6gENePgPcHXo05aDmeH1eVo0lGnSCqZ7iqNotTQqd9WjtTSmAWGI
         baXQE1khBkW58+c72IQllvg0MD1qZBh4p/kFjO//+T53eeao6z7Bg/lLG0nYMCbXDh
         EETstXcwgprT46aNeJODiKtP5G2c1HNF/gH+Q9yQXur2G56UPf/CnKa7ef1Bwvq0Xh
         wC9F3wXlGdAAvaLdySmmiFlfOy1fBeELz6QXrn6xhWmMMtcfl0YP5dzFkoPmJYBM5Z
         bMuaTEw7Cm0og==
From:   Leon Romanovsky <leon@kernel.org>
To:     Jason Gunthorpe <jgg@nvidia.com>
Cc:     Israel Rukshin <israelr@nvidia.com>,
        Bryan Tan <bryantan@vmware.com>,
        Christoph Hellwig <hch@lst.de>,
        Eric Dumazet <edumazet@google.com>,
        Jakub Kicinski <kuba@kernel.org>, Jens Axboe <axboe@fb.com>,
        Keith Busch <kbusch@kernel.org>, linux-kernel@vger.kernel.org,
        linux-nvme@lists.infradead.org, linux-rdma@vger.kernel.org,
        linux-trace-kernel@vger.kernel.org,
        Masami Hiramatsu <mhiramat@kernel.org>,
        Max Gurtovoy <mgurtovoy@nvidia.com>, netdev@vger.kernel.org,
        Paolo Abeni <pabeni@redhat.com>,
        Saeed Mahameed <saeedm@nvidia.com>,
        Sagi Grimberg <sagi@grimberg.me>,
        Selvin Xavier <selvin.xavier@broadcom.com>,
        Steven Rostedt <rostedt@goodmis.org>,
        Vishnu Dasa <vdasa@vmware.com>,
        Yishai Hadas <yishaih@nvidia.com>
Subject: [PATCH mlx5-next 01/13] net/mlx5: Introduce crypto IFC bits and structures
Date:   Mon, 16 Jan 2023 15:05:48 +0200
Message-Id: <92da0db17a6106230c9a1938bc43071c119b7e7f.1673873422.git.leon@kernel.org>
X-Mailer: git-send-email 2.39.0
In-Reply-To: <cover.1673873422.git.leon@kernel.org>
References: <cover.1673873422.git.leon@kernel.org>
MIME-Version: 1.0
Content-Transfer-Encoding: 8bit
Precedence: bulk
List-ID: <netdev.vger.kernel.org>
X-Mailing-List: netdev@vger.kernel.org

From: Israel Rukshin <israelr@nvidia.com>

Add crypto related IFC structs, layouts and enumerations.

Signed-off-by: Israel Rukshin <israelr@nvidia.com>
Signed-off-by: Leon Romanovsky <leon@kernel.org>
---
 include/linux/mlx5/mlx5_ifc.h | 36 ++++++++++++++++++++++++++++++++---
 1 file changed, 33 insertions(+), 3 deletions(-)

diff --git a/include/linux/mlx5/mlx5_ifc.h b/include/linux/mlx5/mlx5_ifc.h
index 8bbf15433bb2..170fe1081820 100644
--- a/include/linux/mlx5/mlx5_ifc.h
+++ b/include/linux/mlx5/mlx5_ifc.h
@@ -1331,6 +1331,29 @@ struct mlx5_ifc_macsec_cap_bits {
 	u8    reserved_at_40[0x7c0];
 };
 
+enum {
+	MLX5_CRYPTO_WRAPPED_IMPORT_METHOD_CAP_AES_XTS = 0x4,
+};
+
+struct mlx5_ifc_crypto_cap_bits {
+	u8         wrapped_crypto_operational[0x1];
+	u8         reserved_at_1[0x17];
+	u8         wrapped_import_method[0x8];
+
+	u8         reserved_at_20[0xb];
+	u8         log_max_num_deks[0x5];
+	u8         reserved_at_30[0x3];
+	u8         log_max_num_import_keks[0x5];
+	u8         reserved_at_38[0x3];
+	u8         log_max_num_creds[0x5];
+
+	u8         failed_selftests[0x10];
+	u8         num_nv_import_keks[0x8];
+	u8         num_nv_credentials[0x8];
+
+	u8         reserved_at_60[0x7a0];
+};
+
 enum {
 	MLX5_WQ_TYPE_LINKED_LIST  = 0x0,
 	MLX5_WQ_TYPE_CYCLIC       = 0x1,
@@ -1758,7 +1781,9 @@ struct mlx5_ifc_cmd_hca_cap_bits {
 	u8         reserved_at_3e8[0x2];
 	u8         vhca_state[0x1];
 	u8         log_max_vlan_list[0x5];
-	u8         reserved_at_3f0[0x3];
+	u8         reserved_at_3f0[0x1];
+	u8         aes_xts_single_block_le_tweak[0x1];
+	u8         aes_xts_multi_block_be_tweak[0x1];
 	u8         log_max_current_mc_list[0x5];
 	u8         reserved_at_3f8[0x3];
 	u8         log_max_current_uc_list[0x5];
@@ -1774,7 +1799,8 @@ struct mlx5_ifc_cmd_hca_cap_bits {
 	u8         ats[0x1];
 	u8         reserved_at_462[0x1];
 	u8         log_max_uctx[0x5];
-	u8         reserved_at_468[0x2];
+	u8         aes_xts_multi_block_le_tweak[0x1];
+	u8         crypto[0x1];
 	u8         ipsec_offload[0x1];
 	u8         log_max_umem[0x5];
 	u8         max_num_eqs[0x10];
@@ -3377,6 +3403,7 @@ union mlx5_ifc_hca_cap_union_bits {
 	struct mlx5_ifc_virtio_emulation_cap_bits virtio_emulation_cap;
 	struct mlx5_ifc_shampo_cap_bits shampo_cap;
 	struct mlx5_ifc_macsec_cap_bits macsec_cap;
+	struct mlx5_ifc_crypto_cap_bits crypto_cap;
 	u8         reserved_at_0[0x8000];
 };
 
@@ -3995,7 +4022,9 @@ struct mlx5_ifc_mkc_bits {
 	u8         reserved_at_1d9[0x1];
 	u8         log_page_size[0x5];
 
-	u8         reserved_at_1e0[0x20];
+	u8         reserved_at_1e0[0x3];
+	u8         crypto_en[0x2];
+	u8         reserved_at_1e5[0x1b];
 };
 
 struct mlx5_ifc_pkey_bits {
@@ -11978,6 +12007,7 @@ enum {
 enum {
 	MLX5_GENERAL_OBJECT_TYPE_ENCRYPTION_KEY_TYPE_TLS = 0x1,
 	MLX5_GENERAL_OBJECT_TYPE_ENCRYPTION_KEY_TYPE_IPSEC = 0x2,
+	MLX5_GENERAL_OBJECT_TYPE_ENCRYPTION_KEY_TYPE_AES_XTS = 0x3,
 	MLX5_GENERAL_OBJECT_TYPE_ENCRYPTION_KEY_TYPE_MACSEC = 0x4,
 };
 
-- 
2.39.0


From mboxrd@z Thu Jan  1 00:00:00 1970
Return-Path: <netdev-owner@vger.kernel.org>
X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on
	aws-us-west-2-korg-lkml-1.web.codeaurora.org
Received: from vger.kernel.org (vger.kernel.org [23.128.96.18])
	by smtp.lore.kernel.org (Postfix) with ESMTP id CD8F5C54EBE
	for <netdev@archiver.kernel.org>; Mon, 16 Jan 2023 13:07:03 +0000 (UTC)
Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand
        id S231422AbjAPNHC (ORCPT <rfc822;netdev@archiver.kernel.org>);
        Mon, 16 Jan 2023 08:07:02 -0500
Received: from lindbergh.monkeyblade.net ([23.128.96.19]:56666 "EHLO
        lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org
        with ESMTP id S231290AbjAPNGj (ORCPT
        <rfc822;netdev@vger.kernel.org>); Mon, 16 Jan 2023 08:06:39 -0500
Received: from dfw.source.kernel.org (dfw.source.kernel.org [IPv6:2604:1380:4641:c500::1])
        by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 6D3AA272C;
        Mon, 16 Jan 2023 05:06:15 -0800 (PST)
Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140])
        (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
        (No client certificate requested)
        by dfw.source.kernel.org (Postfix) with ESMTPS id BFC8960F9C;
        Mon, 16 Jan 2023 13:06:14 +0000 (UTC)
Received: by smtp.kernel.org (Postfix) with ESMTPSA id 59EFCC433D2;
        Mon, 16 Jan 2023 13:06:13 +0000 (UTC)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org;
        s=k20201202; t=1673874374;
        bh=WAO+ONgqCraZR3yFcifYnEjOC2lBkvnTXTYmd++BW0s=;
        h=From:To:Cc:Subject:Date:In-Reply-To:References:From;
        b=GM0T4s2IJ0FDTKt1QqH1Ja6+4wlUM9LuKIb7UeMv8Is4hBnKRT4LPtYykvf2yXcHP
         sS4B8DzDvptm/UcueHCYuLj41X/BDBs+xCV3rcY9avIHMClwgyXjeBIu7VNSd52Wbw
         ZaSUr4nBJnGEohv+f1PoH9FjXAME+MWVEHZZsfGSQHAuy+4H9qjbp5TfhpD6XJ475R
         iEja0YyoCJsFuwHcgXOqnNGMfEp7XlrsDay1zJfd4DGNrtRgfA617UrNC4BmhUpVlz
         pGYE5bUmFZt3Oi2qaZPmEpeRpkM0Gc8jL5sgA49rP9N7Vhuugkj2OAaFx0WeQ4Jf0k
         DVEONyK8FEurA==
From:   Leon Romanovsky <leon@kernel.org>
To:     Jason Gunthorpe <jgg@nvidia.com>
Cc:     Israel Rukshin <israelr@nvidia.com>,
        Bryan Tan <bryantan@vmware.com>,
        Christoph Hellwig <hch@lst.de>,
        Eric Dumazet <edumazet@google.com>,
        Jakub Kicinski <kuba@kernel.org>, Jens Axboe <axboe@fb.com>,
        Keith Busch <kbusch@kernel.org>, linux-kernel@vger.kernel.org,
        linux-nvme@lists.infradead.org, linux-rdma@vger.kernel.org,
        linux-trace-kernel@vger.kernel.org,
        Masami Hiramatsu <mhiramat@kernel.org>,
        Max Gurtovoy <mgurtovoy@nvidia.com>, netdev@vger.kernel.org,
        Paolo Abeni <pabeni@redhat.com>,
        Saeed Mahameed <saeedm@nvidia.com>,
        Sagi Grimberg <sagi@grimberg.me>,
        Selvin Xavier <selvin.xavier@broadcom.com>,
        Steven Rostedt <rostedt@goodmis.org>,
        Vishnu Dasa <vdasa@vmware.com>,
        Yishai Hadas <yishaih@nvidia.com>
Subject: [PATCH rdma-next 03/13] RDMA: Split kernel-only create QP flags from uverbs create QP flags
Date:   Mon, 16 Jan 2023 15:05:50 +0200
Message-Id: <6e46859a58645d9f16a63ff76592487aabc9971d.1673873422.git.leon@kernel.org>
X-Mailer: git-send-email 2.39.0
In-Reply-To: <cover.1673873422.git.leon@kernel.org>
References: <cover.1673873422.git.leon@kernel.org>
MIME-Version: 1.0
Content-Transfer-Encoding: 8bit
Precedence: bulk
List-ID: <netdev.vger.kernel.org>
X-Mailing-List: netdev@vger.kernel.org

From: Israel Rukshin <israelr@nvidia.com>

The current code limits the create_flags to being the same bitmap that
will be copied to user space. Extend the create QP flags to 64 bits and
move all the flags that are only used internally in the kernel to the
upper 32 bits. This cleanly split out the uverbs flags from the kernel
flags to avoid confusion in the flags bitmap. Also add some short
comments describing which each of the kernel flags is connected to.

Signed-off-by: Israel Rukshin <israelr@nvidia.com>
Signed-off-by: Leon Romanovsky <leon@kernel.org>
---
 drivers/infiniband/core/uverbs_std_types_qp.c | 12 +++++-----
 drivers/infiniband/hw/bnxt_re/ib_verbs.c      |  2 +-
 drivers/infiniband/hw/mlx4/mlx4_ib.h          |  4 ++--
 drivers/infiniband/hw/mlx4/qp.c               |  4 ++--
 drivers/infiniband/hw/mlx5/mlx5_ib.h          |  2 +-
 drivers/infiniband/hw/mlx5/qp.c               |  9 ++++----
 drivers/infiniband/hw/vmw_pvrdma/pvrdma_qp.c  |  2 +-
 include/rdma/ib_verbs.h                       | 22 ++++++++++++++-----
 8 files changed, 35 insertions(+), 22 deletions(-)

diff --git a/drivers/infiniband/core/uverbs_std_types_qp.c b/drivers/infiniband/core/uverbs_std_types_qp.c
index 7b4773fa4bc0..fe2c7427eac4 100644
--- a/drivers/infiniband/core/uverbs_std_types_qp.c
+++ b/drivers/infiniband/core/uverbs_std_types_qp.c
@@ -97,6 +97,7 @@ static int UVERBS_HANDLER(UVERBS_METHOD_QP_CREATE)(
 	struct ib_uobject *xrcd_uobj = NULL;
 	struct ib_device *device;
 	u64 user_handle;
+	u32 create_flags;
 	int ret;
 
 	ret = uverbs_copy_from_or_zero(&cap, attrs,
@@ -191,7 +192,7 @@ static int UVERBS_HANDLER(UVERBS_METHOD_QP_CREATE)(
 		return -EINVAL;
 	}
 
-	ret = uverbs_get_flags32(&attr.create_flags, attrs,
+	ret = uverbs_get_flags32(&create_flags, attrs,
 			 UVERBS_ATTR_CREATE_QP_FLAGS,
 			 IB_UVERBS_QP_CREATE_BLOCK_MULTICAST_LOOPBACK |
 			 IB_UVERBS_QP_CREATE_SCATTER_FCS |
@@ -201,7 +202,7 @@ static int UVERBS_HANDLER(UVERBS_METHOD_QP_CREATE)(
 	if (ret)
 		return ret;
 
-	ret = check_creation_flags(attr.qp_type, attr.create_flags);
+	ret = check_creation_flags(attr.qp_type, create_flags);
 	if (ret)
 		return ret;
 
@@ -211,7 +212,7 @@ static int UVERBS_HANDLER(UVERBS_METHOD_QP_CREATE)(
 				       UVERBS_ATTR_CREATE_QP_SOURCE_QPN);
 		if (ret)
 			return ret;
-		attr.create_flags |= IB_QP_CREATE_SOURCE_QPN;
+		create_flags |= IB_QP_CREATE_SOURCE_QPN;
 	}
 
 	srq = uverbs_attr_get_obj(attrs,
@@ -234,16 +235,17 @@ static int UVERBS_HANDLER(UVERBS_METHOD_QP_CREATE)(
 	attr.send_cq = send_cq;
 	attr.recv_cq = recv_cq;
 	attr.xrcd = xrcd;
-	if (attr.create_flags & IB_UVERBS_QP_CREATE_SQ_SIG_ALL) {
+	if (create_flags & IB_UVERBS_QP_CREATE_SQ_SIG_ALL) {
 		/* This creation bit is uverbs one, need to mask before
 		 * calling drivers. It was added to prevent an extra user attr
 		 * only for that when using ioctl.
 		 */
-		attr.create_flags &= ~IB_UVERBS_QP_CREATE_SQ_SIG_ALL;
+		create_flags &= ~IB_UVERBS_QP_CREATE_SQ_SIG_ALL;
 		attr.sq_sig_type = IB_SIGNAL_ALL_WR;
 	} else {
 		attr.sq_sig_type = IB_SIGNAL_REQ_WR;
 	}
+	attr.create_flags = create_flags;
 
 	set_caps(&attr, &cap, true);
 	mutex_init(&obj->mcast_lock);
diff --git a/drivers/infiniband/hw/bnxt_re/ib_verbs.c b/drivers/infiniband/hw/bnxt_re/ib_verbs.c
index 989edc789633..1493ee9ed2b8 100644
--- a/drivers/infiniband/hw/bnxt_re/ib_verbs.c
+++ b/drivers/infiniband/hw/bnxt_re/ib_verbs.c
@@ -1273,7 +1273,7 @@ static int bnxt_re_init_qp_attr(struct bnxt_re_qp *qp, struct bnxt_re_pd *pd,
 	qplqp->dpi = &rdev->dpi_privileged; /* Doorbell page */
 	if (init_attr->create_flags) {
 		ibdev_dbg(&rdev->ibdev,
-			  "QP create flags 0x%x not supported",
+			  "QP create flags 0x%llx not supported",
 			  init_attr->create_flags);
 		return -EOPNOTSUPP;
 	}
diff --git a/drivers/infiniband/hw/mlx4/mlx4_ib.h b/drivers/infiniband/hw/mlx4/mlx4_ib.h
index 17fee1e73a45..c553bf0eb257 100644
--- a/drivers/infiniband/hw/mlx4/mlx4_ib.h
+++ b/drivers/infiniband/hw/mlx4/mlx4_ib.h
@@ -184,7 +184,7 @@ enum mlx4_ib_qp_flags {
 	/* Mellanox specific flags start from IB_QP_CREATE_RESERVED_START */
 	MLX4_IB_ROCE_V2_GSI_QP = MLX4_IB_QP_CREATE_ROCE_V2_GSI,
 	MLX4_IB_SRIOV_TUNNEL_QP = 1 << 30,
-	MLX4_IB_SRIOV_SQP = 1 << 31,
+	MLX4_IB_SRIOV_SQP = 1ULL << 31,
 };
 
 struct mlx4_ib_gid_entry {
@@ -342,7 +342,7 @@ struct mlx4_ib_qp {
 	int			buf_size;
 	struct mutex		mutex;
 	u16			xrcdn;
-	u32			flags;
+	u64			flags;
 	u8			port;
 	u8			alt_port;
 	u8			atomic_rd_en;
diff --git a/drivers/infiniband/hw/mlx4/qp.c b/drivers/infiniband/hw/mlx4/qp.c
index 884825b2e5f7..f3ad436bf5c9 100644
--- a/drivers/infiniband/hw/mlx4/qp.c
+++ b/drivers/infiniband/hw/mlx4/qp.c
@@ -287,7 +287,7 @@ static void mlx4_ib_wq_event(struct mlx4_qp *qp, enum mlx4_event type)
 			    type, qp->qpn);
 }
 
-static int send_wqe_overhead(enum mlx4_ib_qp_type type, u32 flags)
+static int send_wqe_overhead(enum mlx4_ib_qp_type type, u64 flags)
 {
 	/*
 	 * UD WQEs must have a datagram segment.
@@ -1514,7 +1514,7 @@ static int _mlx4_ib_create_qp(struct ib_pd *pd, struct mlx4_ib_qp *qp,
 			      struct ib_udata *udata)
 {
 	int err;
-	int sup_u_create_flags = MLX4_IB_QP_BLOCK_MULTICAST_LOOPBACK;
+	u64 sup_u_create_flags = MLX4_IB_QP_BLOCK_MULTICAST_LOOPBACK;
 	u16 xrcdn = 0;
 
 	if (init_attr->rwq_ind_tbl)
diff --git a/drivers/infiniband/hw/mlx5/mlx5_ib.h b/drivers/infiniband/hw/mlx5/mlx5_ib.h
index ddb36c757074..295502692da2 100644
--- a/drivers/infiniband/hw/mlx5/mlx5_ib.h
+++ b/drivers/infiniband/hw/mlx5/mlx5_ib.h
@@ -500,7 +500,7 @@ struct mlx5_ib_qp {
 	 */
 	struct mutex		mutex;
 	/* cached variant of create_flags from struct ib_qp_init_attr */
-	u32			flags;
+	u64			flags;
 	u32			port;
 	u8			state;
 	int			max_inline_data;
diff --git a/drivers/infiniband/hw/mlx5/qp.c b/drivers/infiniband/hw/mlx5/qp.c
index 80504db92ee9..f04adc18e63b 100644
--- a/drivers/infiniband/hw/mlx5/qp.c
+++ b/drivers/infiniband/hw/mlx5/qp.c
@@ -2938,7 +2938,7 @@ static int process_vendor_flags(struct mlx5_ib_dev *dev, struct mlx5_ib_qp *qp,
 	return (flags) ? -EINVAL : 0;
 	}
 
-static void process_create_flag(struct mlx5_ib_dev *dev, int *flags, int flag,
+static void process_create_flag(struct mlx5_ib_dev *dev, u64 *flags, u64 flag,
 				bool cond, struct mlx5_ib_qp *qp)
 {
 	if (!(*flags & flag))
@@ -2958,7 +2958,8 @@ static void process_create_flag(struct mlx5_ib_dev *dev, int *flags, int flag,
 		*flags &= ~MLX5_IB_QP_CREATE_WC_TEST;
 		return;
 	}
-	mlx5_ib_dbg(dev, "Verbs create QP flag 0x%X is not supported\n", flag);
+	mlx5_ib_dbg(dev, "Verbs create QP flag 0x%llX is not supported\n",
+		    flag);
 }
 
 static int process_create_flags(struct mlx5_ib_dev *dev, struct mlx5_ib_qp *qp,
@@ -2966,7 +2967,7 @@ static int process_create_flags(struct mlx5_ib_dev *dev, struct mlx5_ib_qp *qp,
 {
 	enum ib_qp_type qp_type = qp->type;
 	struct mlx5_core_dev *mdev = dev->mdev;
-	int create_flags = attr->create_flags;
+	u64 create_flags = attr->create_flags;
 	bool cond;
 
 	if (qp_type == MLX5_IB_QPT_DCT)
@@ -3024,7 +3025,7 @@ static int process_create_flags(struct mlx5_ib_dev *dev, struct mlx5_ib_qp *qp,
 			    true, qp);
 
 	if (create_flags) {
-		mlx5_ib_dbg(dev, "Create QP has unsupported flags 0x%X\n",
+		mlx5_ib_dbg(dev, "Create QP has unsupported flags 0x%llX\n",
 			    create_flags);
 		return -EOPNOTSUPP;
 	}
diff --git a/drivers/infiniband/hw/vmw_pvrdma/pvrdma_qp.c b/drivers/infiniband/hw/vmw_pvrdma/pvrdma_qp.c
index f83cd4a9d992..0dbfc3c9e274 100644
--- a/drivers/infiniband/hw/vmw_pvrdma/pvrdma_qp.c
+++ b/drivers/infiniband/hw/vmw_pvrdma/pvrdma_qp.c
@@ -206,7 +206,7 @@ int pvrdma_create_qp(struct ib_qp *ibqp, struct ib_qp_init_attr *init_attr,
 
 	if (init_attr->create_flags) {
 		dev_warn(&dev->pdev->dev,
-			 "invalid create queuepair flags %#x\n",
+			 "invalid create queuepair flags 0x%llx\n",
 			 init_attr->create_flags);
 		return -EOPNOTSUPP;
 	}
diff --git a/include/rdma/ib_verbs.h b/include/rdma/ib_verbs.h
index 949cf4ffc536..cc2ddd4e6c12 100644
--- a/include/rdma/ib_verbs.h
+++ b/include/rdma/ib_verbs.h
@@ -1140,16 +1140,15 @@ enum ib_qp_type {
 	IB_QPT_RESERVED10,
 };
 
+/*
+ * bits 0, 5, 6 and 7 may be set by old kernels and should not be used.
+ */
 enum ib_qp_create_flags {
-	IB_QP_CREATE_IPOIB_UD_LSO		= 1 << 0,
 	IB_QP_CREATE_BLOCK_MULTICAST_LOOPBACK	=
 		IB_UVERBS_QP_CREATE_BLOCK_MULTICAST_LOOPBACK,
 	IB_QP_CREATE_CROSS_CHANNEL              = 1 << 2,
 	IB_QP_CREATE_MANAGED_SEND               = 1 << 3,
 	IB_QP_CREATE_MANAGED_RECV               = 1 << 4,
-	IB_QP_CREATE_NETIF_QP			= 1 << 5,
-	IB_QP_CREATE_INTEGRITY_EN		= 1 << 6,
-	IB_QP_CREATE_NETDEV_USE			= 1 << 7,
 	IB_QP_CREATE_SCATTER_FCS		=
 		IB_UVERBS_QP_CREATE_SCATTER_FCS,
 	IB_QP_CREATE_CVLAN_STRIPPING		=
@@ -1159,7 +1158,18 @@ enum ib_qp_create_flags {
 		IB_UVERBS_QP_CREATE_PCI_WRITE_END_PADDING,
 	/* reserve bits 26-31 for low level drivers' internal use */
 	IB_QP_CREATE_RESERVED_START		= 1 << 26,
-	IB_QP_CREATE_RESERVED_END		= 1 << 31,
+	IB_QP_CREATE_RESERVED_END		= 1ULL << 31,
+
+	/* The below flags are used only by the kernel */
+
+	 /* The created QP will be used for IPoIB UD LSO */
+	IB_QP_CREATE_IPOIB_UD_LSO		= 1ULL << 32,
+	/* Create a QP that supports flow-steering */
+	IB_QP_CREATE_NETIF_QP			= 1ULL << 33,
+	/* The created QP can carry out integrity handover operations */
+	IB_QP_CREATE_INTEGRITY_EN		= 1ULL << 34,
+	/* Create an accelerated UD QP */
+	IB_QP_CREATE_NETDEV_USE			= 1ULL << 35,
 };
 
 /*
@@ -1179,7 +1189,7 @@ struct ib_qp_init_attr {
 	struct ib_qp_cap	cap;
 	enum ib_sig_type	sq_sig_type;
 	enum ib_qp_type		qp_type;
-	u32			create_flags;
+	u64			create_flags;
 
 	/*
 	 * Only needed for special QP types, or when using the RW API.
-- 
2.39.0


From mboxrd@z Thu Jan  1 00:00:00 1970
Return-Path: <netdev-owner@vger.kernel.org>
X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on
	aws-us-west-2-korg-lkml-1.web.codeaurora.org
Received: from vger.kernel.org (vger.kernel.org [23.128.96.18])
	by smtp.lore.kernel.org (Postfix) with ESMTP id 1E575C678D7
	for <netdev@archiver.kernel.org>; Mon, 16 Jan 2023 13:07:07 +0000 (UTC)
Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand
        id S231455AbjAPNHD (ORCPT <rfc822;netdev@archiver.kernel.org>);
        Mon, 16 Jan 2023 08:07:03 -0500
Received: from lindbergh.monkeyblade.net ([23.128.96.19]:55186 "EHLO
        lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org
        with ESMTP id S231300AbjAPNGj (ORCPT
        <rfc822;netdev@vger.kernel.org>); Mon, 16 Jan 2023 08:06:39 -0500
Received: from ams.source.kernel.org (ams.source.kernel.org [145.40.68.75])
        by lindbergh.monkeyblade.net (Postfix) with ESMTPS id D700F3A92;
        Mon, 16 Jan 2023 05:06:20 -0800 (PST)
Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140])
        (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
        (No client certificate requested)
        by ams.source.kernel.org (Postfix) with ESMTPS id 8885CB80D31;
        Mon, 16 Jan 2023 13:06:19 +0000 (UTC)
Received: by smtp.kernel.org (Postfix) with ESMTPSA id 7E2E2C433D2;
        Mon, 16 Jan 2023 13:06:17 +0000 (UTC)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org;
        s=k20201202; t=1673874378;
        bh=FGqAyVpLDpa6MeOFTzSFbDozHcqVNx/tnBsc6ttpwoM=;
        h=From:To:Cc:Subject:Date:In-Reply-To:References:From;
        b=in1GLNrzq+wqFOE9AjaQ/K0YCORb2NHrxF8tjgbcNj/kIKb2H6NHGODGVottYTQMO
         hSCapPblNrwwp6gJsN5FxW6/Vks9AoCr/LmQoiE/dkc4qxBb0U8Oj1ndZjy5CjDjiI
         PQmjUt1pRAF+CAlIhFstqLSerYxKXNovbzYn8WNAuAnfSP5IPDEBLB8xArwA+3tCn8
         FW/qet4RhByW7xKJPoGynWXG84hWb2hIT/wBSpa/LG7tzX6k71bUS+lW6pwuNvJB/m
         AyROUjPdB2yGTQpVzslf6xLN3OfmWOXVz4lISPONbX2LmTreM8inz9x683LsyBUvN3
         pVoCrD78rk9hw==
From:   Leon Romanovsky <leon@kernel.org>
To:     Jason Gunthorpe <jgg@nvidia.com>
Cc:     Israel Rukshin <israelr@nvidia.com>,
        Bryan Tan <bryantan@vmware.com>,
        Christoph Hellwig <hch@lst.de>,
        Eric Dumazet <edumazet@google.com>,
        Jakub Kicinski <kuba@kernel.org>, Jens Axboe <axboe@fb.com>,
        Keith Busch <kbusch@kernel.org>, linux-kernel@vger.kernel.org,
        linux-nvme@lists.infradead.org, linux-rdma@vger.kernel.org,
        linux-trace-kernel@vger.kernel.org,
        Masami Hiramatsu <mhiramat@kernel.org>,
        Max Gurtovoy <mgurtovoy@nvidia.com>, netdev@vger.kernel.org,
        Paolo Abeni <pabeni@redhat.com>,
        Saeed Mahameed <saeedm@nvidia.com>,
        Sagi Grimberg <sagi@grimberg.me>,
        Selvin Xavier <selvin.xavier@broadcom.com>,
        Steven Rostedt <rostedt@goodmis.org>,
        Vishnu Dasa <vdasa@vmware.com>,
        Yishai Hadas <yishaih@nvidia.com>
Subject: [PATCH rdma-next 04/13] RDMA/core: Add cryptographic device capabilities
Date:   Mon, 16 Jan 2023 15:05:51 +0200
Message-Id: <4be0048cfe54548acc3730d733009237d8a896f8.1673873422.git.leon@kernel.org>
X-Mailer: git-send-email 2.39.0
In-Reply-To: <cover.1673873422.git.leon@kernel.org>
References: <cover.1673873422.git.leon@kernel.org>
MIME-Version: 1.0
Content-Transfer-Encoding: 8bit
Precedence: bulk
List-ID: <netdev.vger.kernel.org>
X-Mailing-List: netdev@vger.kernel.org

From: Israel Rukshin <israelr@nvidia.com>

Some advanced RDMA devices have HW engines with cryptographic
capabilities. Those devices can encrypt/decrypt data when transmitting
from memory domain to wire domain and when receiving data from wire
domain to memory domain. Expose these capabilities via common RDMA
device attributes. For now, add only AES-XTS cryptographic support.

Signed-off-by: Israel Rukshin <israelr@nvidia.com>
Reviewed-by: Max Gurtovoy <mgurtovoy@nvidia.com>
Signed-off-by: Leon Romanovsky <leon@kernel.org>
---
 include/rdma/crypto.h   | 37 +++++++++++++++++++++++++++++++++++++
 include/rdma/ib_verbs.h |  2 ++
 2 files changed, 39 insertions(+)
 create mode 100644 include/rdma/crypto.h

diff --git a/include/rdma/crypto.h b/include/rdma/crypto.h
new file mode 100644
index 000000000000..4779eacb000e
--- /dev/null
+++ b/include/rdma/crypto.h
@@ -0,0 +1,37 @@
+/* SPDX-License-Identifier: GPL-2.0 OR Linux-OpenIB */
+/*
+ * Copyright (c) 2022, NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+ */
+
+#ifndef _RDMA_CRYPTO_H_
+#define _RDMA_CRYPTO_H_
+
+#include <linux/types.h>
+
+/**
+ * Encryption and decryption operations are done by attaching crypto properties
+ * to a memory region. Once done, every access to the memory via the crypto
+ * memory region will result in inline encryption or decryption of the data
+ * by the RDMA device. The crypto properties contain the Data Encryption Key
+ * (DEK) and the crypto standard that should be used and its attributes.
+ */
+
+/**
+ * Cryptographic engines in clear text mode capabilities.
+ * @IB_CRYPTO_ENGINES_CAP_AES_XTS: Support AES-XTS engine.
+ */
+enum {
+	IB_CRYPTO_ENGINES_CAP_AES_XTS = 1 << 0,
+};
+
+/**
+ * struct ib_crypto_caps - Cryptographic capabilities
+ * @crypto_engines: From enum ib_crypto_engines_cap_bits.
+ * @max_num_deks: Maximum number of Data Encryption Keys.
+ */
+struct ib_crypto_caps {
+	u32 crypto_engines;
+	u32 max_num_deks;
+};
+
+#endif /* _RDMA_CRYPTO_H_ */
diff --git a/include/rdma/ib_verbs.h b/include/rdma/ib_verbs.h
index cc2ddd4e6c12..83be7e49c5f7 100644
--- a/include/rdma/ib_verbs.h
+++ b/include/rdma/ib_verbs.h
@@ -40,6 +40,7 @@
 #include <rdma/rdma_counter.h>
 #include <rdma/restrack.h>
 #include <rdma/signature.h>
+#include <rdma/crypto.h>
 #include <uapi/rdma/rdma_user_ioctl.h>
 #include <uapi/rdma/ib_user_ioctl_verbs.h>
 
@@ -450,6 +451,7 @@ struct ib_device_attr {
 	u64			max_dm_size;
 	/* Max entries for sgl for optimized performance per READ */
 	u32			max_sgl_rd;
+	struct ib_crypto_caps	crypto_caps;
 };
 
 enum ib_mtu {
-- 
2.39.0


From mboxrd@z Thu Jan  1 00:00:00 1970
Return-Path: <netdev-owner@vger.kernel.org>
X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on
	aws-us-west-2-korg-lkml-1.web.codeaurora.org
Received: from vger.kernel.org (vger.kernel.org [23.128.96.18])
	by smtp.lore.kernel.org (Postfix) with ESMTP id 41E55C54EBE
	for <netdev@archiver.kernel.org>; Mon, 16 Jan 2023 13:07:09 +0000 (UTC)
Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand
        id S231497AbjAPNHF (ORCPT <rfc822;netdev@archiver.kernel.org>);
        Mon, 16 Jan 2023 08:07:05 -0500
Received: from lindbergh.monkeyblade.net ([23.128.96.19]:56846 "EHLO
        lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org
        with ESMTP id S231454AbjAPNGk (ORCPT
        <rfc822;netdev@vger.kernel.org>); Mon, 16 Jan 2023 08:06:40 -0500
Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217])
        by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 8F5674690;
        Mon, 16 Jan 2023 05:06:27 -0800 (PST)
Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140])
        (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
        (No client certificate requested)
        by dfw.source.kernel.org (Postfix) with ESMTPS id 0F59B60F8A;
        Mon, 16 Jan 2023 13:06:27 +0000 (UTC)
Received: by smtp.kernel.org (Postfix) with ESMTPSA id AB33EC433D2;
        Mon, 16 Jan 2023 13:06:25 +0000 (UTC)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org;
        s=k20201202; t=1673874386;
        bh=PHPfzcF12XN8XXrVZEoA/iEMhAWm5Si9T+rPS/xKCts=;
        h=From:To:Cc:Subject:Date:In-Reply-To:References:From;
        b=mZYTjnfg8+FBLPGRtA0+2CT/2NZiCZAPxlQotdBoQc+7iqOpGJ4kjallcHjbv/LR9
         0GhJFlb4ZWZMMxjFdWAx1XgKiNuBzq3s47KxCfkZiqMCbRqYpYfyakYufFMJEiSNHH
         MzmiRJfjtEmbBuchyPbd6TMJ4OC56i99jm/Q5h1zXjmoSFQqaGVkpJX5XUQKSHxbsz
         qhmjUCMBS+2IRBX2oNd07bmDxndH/+okE4/k8qxjKbE+dUT1dBdViz0tKlTyrldWB0
         Ug+Mhk9jzZJ8HkjQBVA0DZiOZafAu0dgraTV+13maJcQ0rKwe9SB2KBx62t0RnALe3
         l3OZXwnlWC/gg==
From:   Leon Romanovsky <leon@kernel.org>
To:     Jason Gunthorpe <jgg@nvidia.com>
Cc:     Israel Rukshin <israelr@nvidia.com>,
        Bryan Tan <bryantan@vmware.com>,
        Christoph Hellwig <hch@lst.de>,
        Eric Dumazet <edumazet@google.com>,
        Jakub Kicinski <kuba@kernel.org>, Jens Axboe <axboe@fb.com>,
        Keith Busch <kbusch@kernel.org>, linux-kernel@vger.kernel.org,
        linux-nvme@lists.infradead.org, linux-rdma@vger.kernel.org,
        linux-trace-kernel@vger.kernel.org,
        Masami Hiramatsu <mhiramat@kernel.org>,
        Max Gurtovoy <mgurtovoy@nvidia.com>, netdev@vger.kernel.org,
        Paolo Abeni <pabeni@redhat.com>,
        Saeed Mahameed <saeedm@nvidia.com>,
        Sagi Grimberg <sagi@grimberg.me>,
        Selvin Xavier <selvin.xavier@broadcom.com>,
        Steven Rostedt <rostedt@goodmis.org>,
        Vishnu Dasa <vdasa@vmware.com>,
        Yishai Hadas <yishaih@nvidia.com>
Subject: [PATCH rdma-next 06/13] RDMA/core: Introduce MR type for crypto operations
Date:   Mon, 16 Jan 2023 15:05:53 +0200
Message-Id: <5b8fadc00c0fcc0c0ba3a5dcc9e7b9012c6b5859.1673873422.git.leon@kernel.org>
X-Mailer: git-send-email 2.39.0
In-Reply-To: <cover.1673873422.git.leon@kernel.org>
References: <cover.1673873422.git.leon@kernel.org>
MIME-Version: 1.0
Content-Transfer-Encoding: 8bit
Precedence: bulk
List-ID: <netdev.vger.kernel.org>
X-Mailing-List: netdev@vger.kernel.org

From: Israel Rukshin <israelr@nvidia.com>

Add crypto attributes for MKey. With those attributes, the device
can encrypt/decrypt data when transmitting from memory domain to wire
domain and when receiving data from wire domain to memory domain.

Signed-off-by: Israel Rukshin <israelr@nvidia.com>
Reviewed-by: Max Gurtovoy <mgurtovoy@nvidia.com>
Signed-off-by: Leon Romanovsky <leon@kernel.org>
---
 drivers/infiniband/core/device.c  |  1 +
 drivers/infiniband/core/mr_pool.c |  2 ++
 drivers/infiniband/core/verbs.c   | 56 ++++++++++++++++++++++++++++++-
 include/rdma/crypto.h             | 43 ++++++++++++++++++++++++
 include/rdma/ib_verbs.h           |  7 ++++
 include/trace/events/rdma_core.h  | 33 ++++++++++++++++++
 6 files changed, 141 insertions(+), 1 deletion(-)

diff --git a/drivers/infiniband/core/device.c b/drivers/infiniband/core/device.c
index b2016725c3d8..d9e98fe92b3c 100644
--- a/drivers/infiniband/core/device.c
+++ b/drivers/infiniband/core/device.c
@@ -2599,6 +2599,7 @@ void ib_set_device_ops(struct ib_device *dev, const struct ib_device_ops *ops)
 	SET_DEVICE_OP(dev_ops, alloc_hw_device_stats);
 	SET_DEVICE_OP(dev_ops, alloc_hw_port_stats);
 	SET_DEVICE_OP(dev_ops, alloc_mr);
+	SET_DEVICE_OP(dev_ops, alloc_mr_crypto);
 	SET_DEVICE_OP(dev_ops, alloc_mr_integrity);
 	SET_DEVICE_OP(dev_ops, alloc_mw);
 	SET_DEVICE_OP(dev_ops, alloc_pd);
diff --git a/drivers/infiniband/core/mr_pool.c b/drivers/infiniband/core/mr_pool.c
index c0e2df128b34..d102cb4caefd 100644
--- a/drivers/infiniband/core/mr_pool.c
+++ b/drivers/infiniband/core/mr_pool.c
@@ -44,6 +44,8 @@ int ib_mr_pool_init(struct ib_qp *qp, struct list_head *list, int nr,
 		if (type == IB_MR_TYPE_INTEGRITY)
 			mr = ib_alloc_mr_integrity(qp->pd, max_num_sg,
 						   max_num_meta_sg);
+		else if (type == IB_MR_TYPE_CRYPTO)
+			mr = ib_alloc_mr_crypto(qp->pd, max_num_sg);
 		else
 			mr = ib_alloc_mr(qp->pd, type, max_num_sg);
 		if (IS_ERR(mr)) {
diff --git a/drivers/infiniband/core/verbs.c b/drivers/infiniband/core/verbs.c
index 03633d706106..61473fee4b54 100644
--- a/drivers/infiniband/core/verbs.c
+++ b/drivers/infiniband/core/verbs.c
@@ -2179,6 +2179,7 @@ int ib_dereg_mr_user(struct ib_mr *mr, struct ib_udata *udata)
 	struct ib_pd *pd = mr->pd;
 	struct ib_dm *dm = mr->dm;
 	struct ib_sig_attrs *sig_attrs = mr->sig_attrs;
+	struct ib_crypto_attrs *crypto_attrs = mr->crypto_attrs;
 	int ret;
 
 	trace_mr_dereg(mr);
@@ -2189,6 +2190,7 @@ int ib_dereg_mr_user(struct ib_mr *mr, struct ib_udata *udata)
 		if (dm)
 			atomic_dec(&dm->usecnt);
 		kfree(sig_attrs);
+		kfree(crypto_attrs);
 	}
 
 	return ret;
@@ -2217,7 +2219,8 @@ struct ib_mr *ib_alloc_mr(struct ib_pd *pd, enum ib_mr_type mr_type,
 		goto out;
 	}
 
-	if (mr_type == IB_MR_TYPE_INTEGRITY) {
+	if (mr_type == IB_MR_TYPE_INTEGRITY ||
+	    mr_type == IB_MR_TYPE_CRYPTO) {
 		WARN_ON_ONCE(1);
 		mr = ERR_PTR(-EINVAL);
 		goto out;
@@ -2294,6 +2297,7 @@ struct ib_mr *ib_alloc_mr_integrity(struct ib_pd *pd,
 	mr->uobject = NULL;
 	atomic_inc(&pd->usecnt);
 	mr->need_inval = false;
+	mr->crypto_attrs = NULL;
 	mr->type = IB_MR_TYPE_INTEGRITY;
 	mr->sig_attrs = sig_attrs;
 
@@ -2306,6 +2310,56 @@ struct ib_mr *ib_alloc_mr_integrity(struct ib_pd *pd,
 }
 EXPORT_SYMBOL(ib_alloc_mr_integrity);
 
+/**
+ * ib_alloc_mr_crypto() - Allocates a crypto memory region
+ * @pd:                   protection domain associated with the region
+ * @max_num_sg:           maximum sg entries available for registration
+ *
+ * Notes:
+ * Memory registration page/sg lists must not exceed max_num_sg.
+ *
+ */
+struct ib_mr *ib_alloc_mr_crypto(struct ib_pd *pd, u32 max_num_sg)
+{
+	struct ib_mr *mr;
+	struct ib_crypto_attrs *crypto_attrs;
+
+	if (!pd->device->ops.alloc_mr_crypto) {
+		mr = ERR_PTR(-EOPNOTSUPP);
+		goto out;
+	}
+
+	crypto_attrs = kzalloc(sizeof(*crypto_attrs), GFP_KERNEL);
+	if (!crypto_attrs) {
+		mr = ERR_PTR(-ENOMEM);
+		goto out;
+	}
+
+	mr = pd->device->ops.alloc_mr_crypto(pd, max_num_sg);
+	if (IS_ERR(mr)) {
+		kfree(crypto_attrs);
+		goto out;
+	}
+
+	mr->device = pd->device;
+	mr->pd = pd;
+	mr->dm = NULL;
+	mr->uobject = NULL;
+	atomic_inc(&pd->usecnt);
+	mr->need_inval = false;
+	mr->sig_attrs = NULL;
+	mr->type = IB_MR_TYPE_CRYPTO;
+	mr->crypto_attrs = crypto_attrs;
+
+	rdma_restrack_new(&mr->res, RDMA_RESTRACK_MR);
+	rdma_restrack_parent_name(&mr->res, &pd->res);
+	rdma_restrack_add(&mr->res);
+out:
+	trace_mr_crypto_alloc(pd, max_num_sg, mr);
+	return mr;
+}
+EXPORT_SYMBOL(ib_alloc_mr_crypto);
+
 /**
  * ib_create_dek - Create a DEK (Data Encryption Key) associated with the
  * specific protection domain.
diff --git a/include/rdma/crypto.h b/include/rdma/crypto.h
index cdf287c94737..ba1c6576a8ba 100644
--- a/include/rdma/crypto.h
+++ b/include/rdma/crypto.h
@@ -34,6 +34,49 @@ struct ib_crypto_caps {
 	u32 max_num_deks;
 };
 
+/**
+ * enum ib_crypto_domain - Encryption domain
+ * According to the encryption domain and the data direction, the HW can
+ * conclude if need to encrypt or decrypt the data.
+ * @IB_CRYPTO_ENCRYPTED_WIRE_DOMAIN: encrypted data is in the wire domain.
+ * @IB_CRYPTO_ENCRYPTED_MEM_DOMAIN: encrypted data is in the memory domain.
+ */
+enum ib_crypto_domain {
+	IB_CRYPTO_ENCRYPTED_WIRE_DOMAIN,
+	IB_CRYPTO_ENCRYPTED_MEM_DOMAIN,
+};
+
+/**
+ * enum ib_crypto_standard - Encryption standard
+ * @IB_CRYPTO_AES_XTS: AES-XTS encryption.
+ */
+enum ib_crypto_standard {
+	IB_CRYPTO_AES_XTS,
+};
+
+/* XTS initial tweak size is up to 128 bits, 16 bytes. */
+#define IB_CRYPTO_XTS_TWEAK_MAX_SIZE 16
+
+/**
+ * struct ib_crypto_attrs - Parameters for crypto handover operation
+ * @encrypt_domain: specific encryption domain.
+ * @encrypt_standard: specific encryption standard.
+ * @data_unit_size: data unit size in bytes. It might be e.g., the filesystem
+ *     block size or the disk sector size.
+ * @xts_init_tweak: a value to be used during encryption of each data unit.
+ *     This value is incremented by the device for every data_unit_size in the
+ *     message.
+ * @dek: Data Encryption Key index.
+ */
+struct ib_crypto_attrs {
+	enum ib_crypto_domain encrypt_domain;
+	enum ib_crypto_standard encrypt_standard;
+	int data_unit_size;
+	/* Today we support only AES-XTS */
+	u32 xts_init_tweak[IB_CRYPTO_XTS_TWEAK_MAX_SIZE / sizeof(u32)];
+	u32 dek;
+};
+
 /**
  * enum ib_crypto_key_type - Cryptographic key types
  * @IB_CRYPTO_KEY_TYPE_AES_XTS: Key of type AES-XTS, which can be used when
diff --git a/include/rdma/ib_verbs.h b/include/rdma/ib_verbs.h
index 5fb42d553ca1..7507661c78d0 100644
--- a/include/rdma/ib_verbs.h
+++ b/include/rdma/ib_verbs.h
@@ -876,6 +876,8 @@ __attribute_const__ int ib_rate_to_mbps(enum ib_rate rate);
  *                            without address translations (VA=PA)
  * @IB_MR_TYPE_INTEGRITY:     memory region that is used for
  *                            data integrity operations
+ * @IB_MR_TYPE_CRYPTO:        memory region that is used for cryptographic
+ *                            operations
  */
 enum ib_mr_type {
 	IB_MR_TYPE_MEM_REG,
@@ -884,6 +886,7 @@ enum ib_mr_type {
 	IB_MR_TYPE_USER,
 	IB_MR_TYPE_DMA,
 	IB_MR_TYPE_INTEGRITY,
+	IB_MR_TYPE_CRYPTO,
 };
 
 enum ib_mr_status_check {
@@ -1854,6 +1857,7 @@ struct ib_mr {
 
 	struct ib_dm      *dm;
 	struct ib_sig_attrs *sig_attrs; /* only for IB_MR_TYPE_INTEGRITY MRs */
+	struct ib_crypto_attrs *crypto_attrs; /* only for IB_MR_TYPE_CRYPTO */
 	/*
 	 * Implementation details of the RDMA core, don't use in drivers:
 	 */
@@ -2512,6 +2516,7 @@ struct ib_device_ops {
 	struct ib_mr *(*alloc_mr_integrity)(struct ib_pd *pd,
 					    u32 max_num_data_sg,
 					    u32 max_num_meta_sg);
+	struct ib_mr *(*alloc_mr_crypto)(struct ib_pd *pd, u32 max_num_sg);
 	struct ib_dek *(*create_dek)(struct ib_pd *pd,
 				     struct ib_dek_attr *attr);
 	void (*destroy_dek)(struct ib_dek *dek);
@@ -4295,6 +4300,8 @@ struct ib_mr *ib_alloc_mr_integrity(struct ib_pd *pd,
 				    u32 max_num_data_sg,
 				    u32 max_num_meta_sg);
 
+struct ib_mr *ib_alloc_mr_crypto(struct ib_pd *pd, u32 max_num_sg);
+
 /**
  * ib_update_fast_reg_key - updates the key portion of the fast_reg MR
  *   R_Key and L_Key.
diff --git a/include/trace/events/rdma_core.h b/include/trace/events/rdma_core.h
index 17642aa54437..b6a3d82b89ca 100644
--- a/include/trace/events/rdma_core.h
+++ b/include/trace/events/rdma_core.h
@@ -371,6 +371,39 @@ TRACE_EVENT(mr_integ_alloc,
 		__entry->max_num_meta_sg, __entry->rc)
 );
 
+TRACE_EVENT(mr_crypto_alloc,
+	TP_PROTO(
+		const struct ib_pd *pd,
+		u32 max_num_sg,
+		const struct ib_mr *mr
+	),
+
+	TP_ARGS(pd, max_num_sg, mr),
+
+	TP_STRUCT__entry(
+		__field(u32, pd_id)
+		__field(u32, mr_id)
+		__field(u32, max_num_sg)
+		__field(int, rc)
+	),
+
+	TP_fast_assign(
+		__entry->pd_id = pd->res.id;
+		if (IS_ERR(mr)) {
+			__entry->mr_id = 0;
+			__entry->rc = PTR_ERR(mr);
+		} else {
+			__entry->mr_id = mr->res.id;
+			__entry->rc = 0;
+		}
+		__entry->max_num_sg = max_num_sg;
+	),
+
+	TP_printk("pd.id=%u mr.id=%u max_num_sg=%u rc=%d",
+		__entry->pd_id, __entry->mr_id, __entry->max_num_sg,
+		__entry->rc)
+);
+
 TRACE_EVENT(mr_dereg,
 	TP_PROTO(
 		const struct ib_mr *mr
-- 
2.39.0


From mboxrd@z Thu Jan  1 00:00:00 1970
Return-Path: <netdev-owner@vger.kernel.org>
X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on
	aws-us-west-2-korg-lkml-1.web.codeaurora.org
Received: from vger.kernel.org (vger.kernel.org [23.128.96.18])
	by smtp.lore.kernel.org (Postfix) with ESMTP id 1AB55C46467
	for <netdev@archiver.kernel.org>; Mon, 16 Jan 2023 13:07:11 +0000 (UTC)
Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand
        id S231517AbjAPNHI (ORCPT <rfc822;netdev@archiver.kernel.org>);
        Mon, 16 Jan 2023 08:07:08 -0500
Received: from lindbergh.monkeyblade.net ([23.128.96.19]:56848 "EHLO
        lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org
        with ESMTP id S231478AbjAPNGk (ORCPT
        <rfc822;netdev@vger.kernel.org>); Mon, 16 Jan 2023 08:06:40 -0500
Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217])
        by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 8CEF146AD;
        Mon, 16 Jan 2023 05:06:31 -0800 (PST)
Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140])
        (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
        (No client certificate requested)
        by dfw.source.kernel.org (Postfix) with ESMTPS id 298E360F8C;
        Mon, 16 Jan 2023 13:06:31 +0000 (UTC)
Received: by smtp.kernel.org (Postfix) with ESMTPSA id C5839C433EF;
        Mon, 16 Jan 2023 13:06:29 +0000 (UTC)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org;
        s=k20201202; t=1673874390;
        bh=IB+veTmklBKw7PiVo8v/NfpEJifNCfbu55nd+mTVPUg=;
        h=From:To:Cc:Subject:Date:In-Reply-To:References:From;
        b=Zq1PeBxWasNH9Dc8TC5hkRkJlGnfrUF4cWKlF4/fG9vW4QUdGsdaxsghZtSzGekgL
         Ql33dt1b+c8IiQ26vu0nGPhqhXCrfk9adXxFZXDxdtwDHR4x6iJ3SaSyZYBepeCwBZ
         Kc1h7HaqLwA7JpvpcHUpLHDiYeVw/LVwcm3Qkh0MIpAuak9cURPQnFRsQWsR/l+0Yk
         wUKGvViiPRzzYkrTDFpzf2TnxIG8excmNjlptAdTQM7ps0xm/RB3itb2zPmaoGw6tz
         ZlhHeHiLEOPWtUTi0h9Lawmhs2MdwmLgHHbEwdPoDN5CS39q3fayP03G9NKAQ6Vrgl
         DHhkfMNo2J7fQ==
From:   Leon Romanovsky <leon@kernel.org>
To:     Jason Gunthorpe <jgg@nvidia.com>
Cc:     Israel Rukshin <israelr@nvidia.com>,
        Bryan Tan <bryantan@vmware.com>,
        Christoph Hellwig <hch@lst.de>,
        Eric Dumazet <edumazet@google.com>,
        Jakub Kicinski <kuba@kernel.org>, Jens Axboe <axboe@fb.com>,
        Keith Busch <kbusch@kernel.org>, linux-kernel@vger.kernel.org,
        linux-nvme@lists.infradead.org, linux-rdma@vger.kernel.org,
        linux-trace-kernel@vger.kernel.org,
        Masami Hiramatsu <mhiramat@kernel.org>,
        Max Gurtovoy <mgurtovoy@nvidia.com>, netdev@vger.kernel.org,
        Paolo Abeni <pabeni@redhat.com>,
        Saeed Mahameed <saeedm@nvidia.com>,
        Sagi Grimberg <sagi@grimberg.me>,
        Selvin Xavier <selvin.xavier@broadcom.com>,
        Steven Rostedt <rostedt@goodmis.org>,
        Vishnu Dasa <vdasa@vmware.com>,
        Yishai Hadas <yishaih@nvidia.com>
Subject: [PATCH rdma-next 07/13] RDMA/core: Add support for creating crypto enabled QPs
Date:   Mon, 16 Jan 2023 15:05:54 +0200
Message-Id: <7a772388d517a28052fa5f0b8ea507cb3fe471fe.1673873422.git.leon@kernel.org>
X-Mailer: git-send-email 2.39.0
In-Reply-To: <cover.1673873422.git.leon@kernel.org>
References: <cover.1673873422.git.leon@kernel.org>
MIME-Version: 1.0
Content-Transfer-Encoding: 8bit
Precedence: bulk
List-ID: <netdev.vger.kernel.org>
X-Mailing-List: netdev@vger.kernel.org

From: Israel Rukshin <israelr@nvidia.com>

Add a list of crypto MRs and introduce a crypto WR type to post
on those QPs.

Signed-off-by: Israel Rukshin <israelr@nvidia.com>
Signed-off-by: Leon Romanovsky <leon@kernel.org>
---
 drivers/infiniband/core/verbs.c |  3 +++
 include/rdma/ib_verbs.h         | 12 +++++++++++-
 2 files changed, 14 insertions(+), 1 deletion(-)

diff --git a/drivers/infiniband/core/verbs.c b/drivers/infiniband/core/verbs.c
index 61473fee4b54..01aefff6760e 100644
--- a/drivers/infiniband/core/verbs.c
+++ b/drivers/infiniband/core/verbs.c
@@ -1223,6 +1223,7 @@ static struct ib_qp *create_qp(struct ib_device *dev, struct ib_pd *pd,
 	spin_lock_init(&qp->mr_lock);
 	INIT_LIST_HEAD(&qp->rdma_mrs);
 	INIT_LIST_HEAD(&qp->sig_mrs);
+	INIT_LIST_HEAD(&qp->crypto_mrs);
 
 	qp->send_cq = attr->send_cq;
 	qp->recv_cq = attr->recv_cq;
@@ -1363,6 +1364,8 @@ struct ib_qp *ib_create_qp_kernel(struct ib_pd *pd,
 				 device->attrs.max_sge_rd);
 	if (qp_init_attr->create_flags & IB_QP_CREATE_INTEGRITY_EN)
 		qp->integrity_en = true;
+	if (qp_init_attr->create_flags & IB_QP_CREATE_CRYPTO_EN)
+		qp->crypto_en = true;
 
 	return qp;
 
diff --git a/include/rdma/ib_verbs.h b/include/rdma/ib_verbs.h
index 7507661c78d0..1770cd30c0f0 100644
--- a/include/rdma/ib_verbs.h
+++ b/include/rdma/ib_verbs.h
@@ -1175,6 +1175,8 @@ enum ib_qp_create_flags {
 	IB_QP_CREATE_INTEGRITY_EN		= 1ULL << 34,
 	/* Create an accelerated UD QP */
 	IB_QP_CREATE_NETDEV_USE			= 1ULL << 35,
+	/* The created QP can carry out cryptographic handover operations */
+	IB_QP_CREATE_CRYPTO_EN			= 1ULL << 36,
 };
 
 /*
@@ -1352,6 +1354,12 @@ enum ib_wr_opcode {
 	/* These are kernel only and can not be issued by userspace */
 	IB_WR_REG_MR = 0x20,
 	IB_WR_REG_MR_INTEGRITY,
+	/*
+	 * It is used to assign crypto properties to a MKey. Use the MKey in
+	 * any RDMA transaction (SEND/RECV/READ/WRITE) to encrypt/decrypt data
+	 * on-the-fly.
+	 */
+	IB_WR_REG_MR_CRYPTO,
 
 	/* reserve values for low level drivers' internal use.
 	 * These values will not be used at all in the ib core layer.
@@ -1800,6 +1808,7 @@ struct ib_qp {
 	int			mrs_used;
 	struct list_head	rdma_mrs;
 	struct list_head	sig_mrs;
+	struct list_head	crypto_mrs;
 	struct ib_srq	       *srq;
 	struct ib_xrcd	       *xrcd; /* XRC TGT QPs only */
 	struct list_head	xrcd_list;
@@ -1822,7 +1831,8 @@ struct ib_qp {
 	struct ib_qp_security  *qp_sec;
 	u32			port;
 
-	bool			integrity_en;
+	u8			integrity_en:1;
+	u8			crypto_en:1;
 	/*
 	 * Implementation details of the RDMA core, don't use in drivers:
 	 */
-- 
2.39.0


From mboxrd@z Thu Jan  1 00:00:00 1970
Return-Path: <netdev-owner@vger.kernel.org>
X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on
	aws-us-west-2-korg-lkml-1.web.codeaurora.org
Received: from vger.kernel.org (vger.kernel.org [23.128.96.18])
	by smtp.lore.kernel.org (Postfix) with ESMTP id B2674C63797
	for <netdev@archiver.kernel.org>; Mon, 16 Jan 2023 13:07:13 +0000 (UTC)
Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand
        id S231527AbjAPNHK (ORCPT <rfc822;netdev@archiver.kernel.org>);
        Mon, 16 Jan 2023 08:07:10 -0500
Received: from lindbergh.monkeyblade.net ([23.128.96.19]:55526 "EHLO
        lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org
        with ESMTP id S230213AbjAPNGu (ORCPT
        <rfc822;netdev@vger.kernel.org>); Mon, 16 Jan 2023 08:06:50 -0500
Received: from ams.source.kernel.org (ams.source.kernel.org [IPv6:2604:1380:4601:e00::1])
        by lindbergh.monkeyblade.net (Postfix) with ESMTPS id E3164A5E5;
        Mon, 16 Jan 2023 05:06:49 -0800 (PST)
Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140])
        (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
        (No client certificate requested)
        by ams.source.kernel.org (Postfix) with ESMTPS id 9DA52B80D31;
        Mon, 16 Jan 2023 13:06:48 +0000 (UTC)
Received: by smtp.kernel.org (Postfix) with ESMTPSA id 84A50C433EF;
        Mon, 16 Jan 2023 13:06:46 +0000 (UTC)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org;
        s=k20201202; t=1673874407;
        bh=JlISh7ZpJM1JouHMtly+4IcB63O1MxsyN1Eywkp8Wsg=;
        h=From:To:Cc:Subject:Date:In-Reply-To:References:From;
        b=aEBoKJSxCjguG2DHqyA7O8qM4GE3x282g/b/CwTfZAS4zWD4B6wASQ+jEB5v/9zR/
         ghrEPqNMvev2QZmBhfekV3oRIuWjuTMKqu+Tzfqn8kldJwTA8QYAxUbuyHvhGL80BM
         QdA4d1/nx3Ly5CVTs3zgz2eyIgWBsTZ5eAcizz+q/laOZkFfW1DiI9wgCVeZrkKahJ
         +g3SoDr4gJw/IVa0yBpVcsJEhstqA9mYnolCld9y2SO7DzLDOhhyLv3ueEl/plNU8I
         RLCCMIed/IZ9bzeQdgF5gf/J2ek6Vtx4EP26J7GKfgPLdqlLQWH17T/t1+hT8LCDHf
         vrmptUx/xttaw==
From:   Leon Romanovsky <leon@kernel.org>
To:     Jason Gunthorpe <jgg@nvidia.com>
Cc:     Israel Rukshin <israelr@nvidia.com>,
        Bryan Tan <bryantan@vmware.com>,
        Christoph Hellwig <hch@lst.de>,
        Eric Dumazet <edumazet@google.com>,
        Jakub Kicinski <kuba@kernel.org>, Jens Axboe <axboe@fb.com>,
        Keith Busch <kbusch@kernel.org>, linux-kernel@vger.kernel.org,
        linux-nvme@lists.infradead.org, linux-rdma@vger.kernel.org,
        linux-trace-kernel@vger.kernel.org,
        Masami Hiramatsu <mhiramat@kernel.org>,
        Max Gurtovoy <mgurtovoy@nvidia.com>, netdev@vger.kernel.org,
        Paolo Abeni <pabeni@redhat.com>,
        Saeed Mahameed <saeedm@nvidia.com>,
        Sagi Grimberg <sagi@grimberg.me>,
        Selvin Xavier <selvin.xavier@broadcom.com>,
        Steven Rostedt <rostedt@goodmis.org>,
        Vishnu Dasa <vdasa@vmware.com>,
        Yishai Hadas <yishaih@nvidia.com>
Subject: [PATCH rdma-next 11/13] nvme: Introduce a local variable
Date:   Mon, 16 Jan 2023 15:05:58 +0200
Message-Id: <cf5bc542e014f465f7ae443e52e70def2993aef1.1673873422.git.leon@kernel.org>
X-Mailer: git-send-email 2.39.0
In-Reply-To: <cover.1673873422.git.leon@kernel.org>
References: <cover.1673873422.git.leon@kernel.org>
MIME-Version: 1.0
Content-Transfer-Encoding: 8bit
Precedence: bulk
List-ID: <netdev.vger.kernel.org>
X-Mailing-List: netdev@vger.kernel.org

From: Israel Rukshin <israelr@nvidia.com>

The patch doesn't change any logic.

Signed-off-by: Israel Rukshin <israelr@nvidia.com>
Signed-off-by: Leon Romanovsky <leon@kernel.org>
---
 drivers/nvme/host/core.c | 7 ++++---
 1 file changed, 4 insertions(+), 3 deletions(-)

diff --git a/drivers/nvme/host/core.c b/drivers/nvme/host/core.c
index 7be562a4e1aa..51a9880db6ce 100644
--- a/drivers/nvme/host/core.c
+++ b/drivers/nvme/host/core.c
@@ -1870,6 +1870,7 @@ static void nvme_update_disk_info(struct gendisk *disk,
 	sector_t capacity = nvme_lba_to_sect(ns, le64_to_cpu(id->nsze));
 	unsigned short bs = 1 << ns->lba_shift;
 	u32 atomic_bs, phys_bs, io_opt = 0;
+	struct nvme_ctrl *ctrl = ns->ctrl;
 
 	/*
 	 * The block layer can't support LBA sizes larger than the page size
@@ -1892,7 +1893,7 @@ static void nvme_update_disk_info(struct gendisk *disk,
 		if (id->nsfeat & NVME_NS_FEAT_ATOMICS && id->nawupf)
 			atomic_bs = (1 + le16_to_cpu(id->nawupf)) * bs;
 		else
-			atomic_bs = (1 + ns->ctrl->subsys->awupf) * bs;
+			atomic_bs = (1 + ctrl->subsys->awupf) * bs;
 	}
 
 	if (id->nsfeat & NVME_NS_FEAT_IO_OPT) {
@@ -1922,7 +1923,7 @@ static void nvme_update_disk_info(struct gendisk *disk,
 		if (IS_ENABLED(CONFIG_BLK_DEV_INTEGRITY) &&
 		    (ns->features & NVME_NS_METADATA_SUPPORTED))
 			nvme_init_integrity(disk, ns,
-					    ns->ctrl->max_integrity_segments);
+					    ctrl->max_integrity_segments);
 		else if (!nvme_ns_has_pi(ns))
 			capacity = 0;
 	}
@@ -1931,7 +1932,7 @@ static void nvme_update_disk_info(struct gendisk *disk,
 
 	nvme_config_discard(disk, ns);
 	blk_queue_max_write_zeroes_sectors(disk->queue,
-					   ns->ctrl->max_zeroes_sectors);
+					   ctrl->max_zeroes_sectors);
 }
 
 static bool nvme_ns_is_readonly(struct nvme_ns *ns, struct nvme_ns_info *info)
-- 
2.39.0


From mboxrd@z Thu Jan  1 00:00:00 1970
Return-Path: <netdev-owner@vger.kernel.org>
X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on
	aws-us-west-2-korg-lkml-1.web.codeaurora.org
Received: from vger.kernel.org (vger.kernel.org [23.128.96.18])
	by smtp.lore.kernel.org (Postfix) with ESMTP id C4CCDC678D8
	for <netdev@archiver.kernel.org>; Mon, 16 Jan 2023 13:07:14 +0000 (UTC)
Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand
        id S231290AbjAPNHM (ORCPT <rfc822;netdev@archiver.kernel.org>);
        Mon, 16 Jan 2023 08:07:12 -0500
Received: from lindbergh.monkeyblade.net ([23.128.96.19]:56720 "EHLO
        lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org
        with ESMTP id S231420AbjAPNHB (ORCPT
        <rfc822;netdev@vger.kernel.org>); Mon, 16 Jan 2023 08:07:01 -0500
Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217])
        by lindbergh.monkeyblade.net (Postfix) with ESMTPS id EAA661710;
        Mon, 16 Jan 2023 05:07:00 -0800 (PST)
Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140])
        (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
        (No client certificate requested)
        by dfw.source.kernel.org (Postfix) with ESMTPS id 645F660F9B;
        Mon, 16 Jan 2023 13:07:00 +0000 (UTC)
Received: by smtp.kernel.org (Postfix) with ESMTPSA id DA0DFC433F1;
        Mon, 16 Jan 2023 13:06:58 +0000 (UTC)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org;
        s=k20201202; t=1673874419;
        bh=Z9+KomyeN4vAjModCpaAbKOjkyqP4m8dRccCmgRLn8k=;
        h=From:To:Cc:Subject:Date:In-Reply-To:References:From;
        b=OPeQnGdMKL0qvPIWYWzLqm//vIJ6xVrQ+zSrw9NRNMJnbSzuJa9VEEYupUejVnt06
         HRNhIbjJVVOzzBSXSeoUjnDrYIDLqtF3CjSOmEgHFu37MIVWOwUbq/DFyGfdnJ++Yg
         BvhvBUhhKztQYlOb92vy7VS8Uiq44148kyxwVg4auh30C5QonsBIXdBs6dnWQlVAaH
         p/08idib6ImCyqQwz1xJ8CyMsMgn2pwOnLcSRdNel6waQkTHTGQTYUJsOWFJsf4Ywz
         ++JUX3bjsvRlVxxLqSrPNBs7farZKtwyKf2xjpumDEoLbmQvbxXcM2CM0TkhY3sduc
         8JxubTZg4TsUw==
From:   Leon Romanovsky <leon@kernel.org>
To:     Jason Gunthorpe <jgg@nvidia.com>
Cc:     Israel Rukshin <israelr@nvidia.com>,
        Bryan Tan <bryantan@vmware.com>,
        Christoph Hellwig <hch@lst.de>,
        Eric Dumazet <edumazet@google.com>,
        Jakub Kicinski <kuba@kernel.org>, Jens Axboe <axboe@fb.com>,
        Keith Busch <kbusch@kernel.org>, linux-kernel@vger.kernel.org,
        linux-nvme@lists.infradead.org, linux-rdma@vger.kernel.org,
        linux-trace-kernel@vger.kernel.org,
        Masami Hiramatsu <mhiramat@kernel.org>,
        Max Gurtovoy <mgurtovoy@nvidia.com>, netdev@vger.kernel.org,
        Paolo Abeni <pabeni@redhat.com>,
        Saeed Mahameed <saeedm@nvidia.com>,
        Sagi Grimberg <sagi@grimberg.me>,
        Selvin Xavier <selvin.xavier@broadcom.com>,
        Steven Rostedt <rostedt@goodmis.org>,
        Vishnu Dasa <vdasa@vmware.com>,
        Yishai Hadas <yishaih@nvidia.com>
Subject: [PATCH rdma-next 12/13] nvme: Add crypto profile at nvme controller
Date:   Mon, 16 Jan 2023 15:05:59 +0200
Message-Id: <efc8dc5952baa096a14db1761f84a5ab2e76654a.1673873422.git.leon@kernel.org>
X-Mailer: git-send-email 2.39.0
In-Reply-To: <cover.1673873422.git.leon@kernel.org>
References: <cover.1673873422.git.leon@kernel.org>
MIME-Version: 1.0
Content-Transfer-Encoding: 8bit
Precedence: bulk
List-ID: <netdev.vger.kernel.org>
X-Mailing-List: netdev@vger.kernel.org

From: Israel Rukshin <israelr@nvidia.com>

The crypto profile will be filled by the transport drivers. This
is a preparation patch for adding support of inline encryption
at nvme-rdma driver.

Signed-off-by: Israel Rukshin <israelr@nvidia.com>
Signed-off-by: Leon Romanovsky <leon@kernel.org>
---
 drivers/nvme/host/core.c | 3 +++
 drivers/nvme/host/nvme.h | 4 ++++
 2 files changed, 7 insertions(+)

diff --git a/drivers/nvme/host/core.c b/drivers/nvme/host/core.c
index 51a9880db6ce..f09e4e0216b3 100644
--- a/drivers/nvme/host/core.c
+++ b/drivers/nvme/host/core.c
@@ -1928,6 +1928,9 @@ static void nvme_update_disk_info(struct gendisk *disk,
 			capacity = 0;
 	}
 
+	if (ctrl->crypto_enable)
+		blk_crypto_register(&ctrl->crypto_profile, disk->queue);
+
 	set_capacity_and_notify(disk, capacity);
 
 	nvme_config_discard(disk, ns);
diff --git a/drivers/nvme/host/nvme.h b/drivers/nvme/host/nvme.h
index 424c8a467a0c..591380f53744 100644
--- a/drivers/nvme/host/nvme.h
+++ b/drivers/nvme/host/nvme.h
@@ -16,6 +16,7 @@
 #include <linux/rcupdate.h>
 #include <linux/wait.h>
 #include <linux/t10-pi.h>
+#include <linux/blk-crypto-profile.h>
 
 #include <trace/events/block.h>
 
@@ -374,6 +375,9 @@ struct nvme_ctrl {
 
 	enum nvme_ctrl_type cntrltype;
 	enum nvme_dctype dctype;
+
+	bool crypto_enable;
+	struct blk_crypto_profile crypto_profile;
 };
 
 enum nvme_iopolicy {
-- 
2.39.0


From mboxrd@z Thu Jan  1 00:00:00 1970
Return-Path: <netdev-owner@vger.kernel.org>
X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on
	aws-us-west-2-korg-lkml-1.web.codeaurora.org
Received: from vger.kernel.org (vger.kernel.org [23.128.96.18])
	by smtp.lore.kernel.org (Postfix) with ESMTP id 1F257C678DB
	for <netdev@archiver.kernel.org>; Mon, 16 Jan 2023 13:07:16 +0000 (UTC)
Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand
        id S231556AbjAPNHO (ORCPT <rfc822;netdev@archiver.kernel.org>);
        Mon, 16 Jan 2023 08:07:14 -0500
Received: from lindbergh.monkeyblade.net ([23.128.96.19]:55186 "EHLO
        lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org
        with ESMTP id S231287AbjAPNG6 (ORCPT
        <rfc822;netdev@vger.kernel.org>); Mon, 16 Jan 2023 08:06:58 -0500
Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217])
        by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 920897EE1;
        Mon, 16 Jan 2023 05:06:56 -0800 (PST)
Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140])
        (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
        (No client certificate requested)
        by dfw.source.kernel.org (Postfix) with ESMTPS id 203C460F9B;
        Mon, 16 Jan 2023 13:06:56 +0000 (UTC)
Received: by smtp.kernel.org (Postfix) with ESMTPSA id A8A0EC433F0;
        Mon, 16 Jan 2023 13:06:54 +0000 (UTC)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org;
        s=k20201202; t=1673874415;
        bh=3onQ/rBBcurHQvS6iIFYjN6t3VLGjjleEvgmJokrbgA=;
        h=From:To:Cc:Subject:Date:In-Reply-To:References:From;
        b=NUJkHjzulOFJcMkWyEr4QZGeeBrMIcy+EHSm7ZNEzgiWesNYaqRow8CXR3AlkAQ8i
         dJvjT8FNvlrkqVq9fs5JF1v5O8p2AJXQrGk7x8i+newd1L8Qc7oAM4Y8faxFFuMm3I
         qqHPzLr8ApM9FLcq0iPB+rYjUNNpKdpQ1pstknMWu7AfW0R7Y8HpZXx4Em51+CPR8b
         ZB3Fbt5NLbLtolW/kHQfgw1BIVr7TB3wHpobxPSgBfk55jPas04n8Oz1TpsI+e0FKC
         rLnD8WKyrb3HBRB++nAeBaQIanu1gD4gW6dZCc76bECmRRIMEPgYDKKGHewU0YMchi
         fee5Llt2FnYWA==
From:   Leon Romanovsky <leon@kernel.org>
To:     Jason Gunthorpe <jgg@nvidia.com>
Cc:     Israel Rukshin <israelr@nvidia.com>,
        Bryan Tan <bryantan@vmware.com>,
        Christoph Hellwig <hch@lst.de>,
        Eric Dumazet <edumazet@google.com>,
        Jakub Kicinski <kuba@kernel.org>, Jens Axboe <axboe@fb.com>,
        Keith Busch <kbusch@kernel.org>, linux-kernel@vger.kernel.org,
        linux-nvme@lists.infradead.org, linux-rdma@vger.kernel.org,
        linux-trace-kernel@vger.kernel.org,
        Masami Hiramatsu <mhiramat@kernel.org>,
        Max Gurtovoy <mgurtovoy@nvidia.com>, netdev@vger.kernel.org,
        Paolo Abeni <pabeni@redhat.com>,
        Saeed Mahameed <saeedm@nvidia.com>,
        Sagi Grimberg <sagi@grimberg.me>,
        Selvin Xavier <selvin.xavier@broadcom.com>,
        Steven Rostedt <rostedt@goodmis.org>,
        Vishnu Dasa <vdasa@vmware.com>,
        Yishai Hadas <yishaih@nvidia.com>
Subject: [PATCH rdma-next 13/13] nvme-rdma: Add inline encryption support
Date:   Mon, 16 Jan 2023 15:06:00 +0200
Message-Id: <ab0817fac98d498372fd32e950d1ca6e157b575d.1673873422.git.leon@kernel.org>
X-Mailer: git-send-email 2.39.0
In-Reply-To: <cover.1673873422.git.leon@kernel.org>
References: <cover.1673873422.git.leon@kernel.org>
MIME-Version: 1.0
Content-Transfer-Encoding: 8bit
Precedence: bulk
List-ID: <netdev.vger.kernel.org>
X-Mailing-List: netdev@vger.kernel.org

From: Israel Rukshin <israelr@nvidia.com>

Add support for inline encryption/decryption of the data at a
similar way like integrity is used via a unique Mkey. The feature
is enabled only when CONFIG_BLK_INLINE_ENCRYPTION is configured.
There is no special configuration to enable crypto at nvme modules.

The code was tested with fscrypt and BF-3 HW, which had more than
50% CPU utilization improvement at 64K and bigger I/O sizes when
comparing to the SW only solution at this case.

Signed-off-by: Israel Rukshin <israelr@nvidia.com>
Signed-off-by: Leon Romanovsky <leon@kernel.org>
---
 drivers/nvme/host/rdma.c | 236 ++++++++++++++++++++++++++++++++++++++-
 1 file changed, 232 insertions(+), 4 deletions(-)

diff --git a/drivers/nvme/host/rdma.c b/drivers/nvme/host/rdma.c
index bbad26b82b56..8bea38eb976f 100644
--- a/drivers/nvme/host/rdma.c
+++ b/drivers/nvme/host/rdma.c
@@ -40,6 +40,10 @@
 #define NVME_RDMA_METADATA_SGL_SIZE \
 	(sizeof(struct scatterlist) * NVME_INLINE_METADATA_SG_CNT)
 
+#define NVME_RDMA_NUM_CRYPTO_KEYSLOTS	(32U)
+/* Bitmask for 512B and 4K data unit sizes */
+#define NVME_RDMA_DATA_UNIT_SIZES	(512U | 4096U)
+
 struct nvme_rdma_device {
 	struct ib_device	*dev;
 	struct ib_pd		*pd;
@@ -75,6 +79,7 @@ struct nvme_rdma_request {
 	struct nvme_rdma_sgl	data_sgl;
 	struct nvme_rdma_sgl	*metadata_sgl;
 	bool			use_sig_mr;
+	bool			use_crypto;
 };
 
 enum nvme_rdma_queue_flags {
@@ -97,6 +102,7 @@ struct nvme_rdma_queue {
 	int			cm_error;
 	struct completion	cm_done;
 	bool			pi_support;
+	bool			crypto_support;
 	int			cq_size;
 	struct mutex		queue_lock;
 };
@@ -126,6 +132,8 @@ struct nvme_rdma_ctrl {
 	struct nvme_ctrl	ctrl;
 	bool			use_inline_data;
 	u32			io_queues[HCTX_MAX_TYPES];
+
+	struct ib_dek		*dek[NVME_RDMA_NUM_CRYPTO_KEYSLOTS];
 };
 
 static inline struct nvme_rdma_ctrl *to_rdma_ctrl(struct nvme_ctrl *ctrl)
@@ -275,6 +283,8 @@ static int nvme_rdma_create_qp(struct nvme_rdma_queue *queue, const int factor)
 	init_attr.recv_cq = queue->ib_cq;
 	if (queue->pi_support)
 		init_attr.create_flags |= IB_QP_CREATE_INTEGRITY_EN;
+	if (queue->crypto_support)
+		init_attr.create_flags |= IB_QP_CREATE_CRYPTO_EN;
 	init_attr.qp_context = queue;
 
 	ret = rdma_create_qp(queue->cm_id, dev->pd, &init_attr);
@@ -364,6 +374,77 @@ static int nvme_rdma_dev_get(struct nvme_rdma_device *dev)
 	return kref_get_unless_zero(&dev->ref);
 }
 
+static int nvme_rdma_crypto_keyslot_program(struct blk_crypto_profile *profile,
+					    const struct blk_crypto_key *key,
+					    unsigned int slot)
+{
+	struct nvme_ctrl *nctrl =
+		container_of(profile, struct nvme_ctrl, crypto_profile);
+	struct nvme_rdma_ctrl *ctrl = to_rdma_ctrl(nctrl);
+	struct ib_dek_attr dek_attr = { };
+	int err = 0;
+
+	if (slot > NVME_RDMA_NUM_CRYPTO_KEYSLOTS) {
+		dev_err(nctrl->device, "slot index reached the limit %d/%d",
+			slot, NVME_RDMA_NUM_CRYPTO_KEYSLOTS);
+		return -EOPNOTSUPP;
+	}
+
+	if (WARN_ON_ONCE(key->crypto_cfg.crypto_mode !=
+			 BLK_ENCRYPTION_MODE_AES_256_XTS))
+		return -EOPNOTSUPP;
+
+	if (ctrl->dek[slot]) {
+		dev_dbg(nctrl->device, "slot %d is taken, free DEK ID %d\n",
+			slot, ctrl->dek[slot]->id);
+		ib_destroy_dek(ctrl->dek[slot]);
+	}
+
+	dek_attr.key_blob = key->raw;
+	dek_attr.key_blob_size = key->size;
+	dek_attr.key_type = IB_CRYPTO_KEY_TYPE_AES_XTS;
+	ctrl->dek[slot] = ib_create_dek(ctrl->device->pd, &dek_attr);
+	if (IS_ERR(ctrl->dek[slot])) {
+		err = PTR_ERR(ctrl->dek[slot]);
+		dev_err(nctrl->device,
+			"failed to create DEK at slot %d, err %d\n", slot, err);
+		ctrl->dek[slot] = NULL;
+	} else {
+		dev_dbg(nctrl->device, "DEK ID %d was created at slot %d\n",
+			ctrl->dek[slot]->id, slot);
+	}
+
+	return err;
+}
+
+static int nvme_rdma_crypto_keyslot_evict(struct blk_crypto_profile *profile,
+					  const struct blk_crypto_key *key,
+					  unsigned int slot)
+{
+	struct nvme_ctrl *nctrl =
+		container_of(profile, struct nvme_ctrl, crypto_profile);
+	struct nvme_rdma_ctrl *ctrl = to_rdma_ctrl(nctrl);
+
+	if (slot > NVME_RDMA_NUM_CRYPTO_KEYSLOTS) {
+		dev_err(nctrl->device, "slot index reached the limit %d/%d\n",
+			slot, NVME_RDMA_NUM_CRYPTO_KEYSLOTS);
+		return -EOPNOTSUPP;
+	}
+
+	if (!ctrl->dek[slot]) {
+		dev_err(nctrl->device, "slot %d is empty\n", slot);
+		return -EINVAL;
+	}
+
+	dev_dbg(nctrl->device, "Destroy DEK ID %d slot %d\n",
+		ctrl->dek[slot]->id, slot);
+
+	ib_destroy_dek(ctrl->dek[slot]);
+	ctrl->dek[slot] = NULL;
+
+	return 0;
+}
+
 static struct nvme_rdma_device *
 nvme_rdma_find_get_device(struct rdma_cm_id *cm_id)
 {
@@ -430,6 +511,8 @@ static void nvme_rdma_destroy_queue_ib(struct nvme_rdma_queue *queue)
 	dev = queue->device;
 	ibdev = dev->dev;
 
+	if (queue->crypto_support)
+		ib_mr_pool_destroy(queue->qp, &queue->qp->crypto_mrs);
 	if (queue->pi_support)
 		ib_mr_pool_destroy(queue->qp, &queue->qp->sig_mrs);
 	ib_mr_pool_destroy(queue->qp, &queue->qp->rdma_mrs);
@@ -553,10 +636,24 @@ static int nvme_rdma_create_queue_ib(struct nvme_rdma_queue *queue)
 		}
 	}
 
+	if (queue->crypto_support) {
+		ret = ib_mr_pool_init(queue->qp, &queue->qp->crypto_mrs,
+				      queue->queue_size, IB_MR_TYPE_CRYPTO,
+				      pages_per_mr, 0);
+		if (ret) {
+			dev_err(queue->ctrl->ctrl.device,
+				"failed to initialize crypto MR pool sized %d for QID %d\n",
+				queue->queue_size, nvme_rdma_queue_idx(queue));
+			goto out_destroy_sig_mr_pool;
+		}
+	}
+
 	set_bit(NVME_RDMA_Q_TR_READY, &queue->flags);
 
 	return 0;
 
+out_destroy_sig_mr_pool:
+	ib_mr_pool_destroy(queue->qp, &queue->qp->sig_mrs);
 out_destroy_mr_pool:
 	ib_mr_pool_destroy(queue->qp, &queue->qp->rdma_mrs);
 out_destroy_ring:
@@ -585,6 +682,9 @@ static int nvme_rdma_alloc_queue(struct nvme_rdma_ctrl *ctrl,
 		queue->pi_support = true;
 	else
 		queue->pi_support = false;
+
+	queue->crypto_support = idx && ctrl->ctrl.crypto_enable;
+
 	init_completion(&queue->cm_done);
 
 	if (idx > 0)
@@ -805,15 +905,109 @@ static int nvme_rdma_alloc_tag_set(struct nvme_ctrl *ctrl)
 
 static void nvme_rdma_destroy_admin_queue(struct nvme_rdma_ctrl *ctrl)
 {
+	unsigned int i;
+
 	if (ctrl->async_event_sqe.data) {
 		cancel_work_sync(&ctrl->ctrl.async_event_work);
 		nvme_rdma_free_qe(ctrl->device->dev, &ctrl->async_event_sqe,
 				sizeof(struct nvme_command), DMA_TO_DEVICE);
 		ctrl->async_event_sqe.data = NULL;
 	}
+
+	for (i = 0; i < NVME_RDMA_NUM_CRYPTO_KEYSLOTS; i++) {
+		if (!ctrl->dek[i])
+			continue;
+		ib_destroy_dek(ctrl->dek[i]);
+		ctrl->dek[i] = NULL;
+	}
+
 	nvme_rdma_free_queue(&ctrl->queues[0]);
 }
 
+#ifdef CONFIG_BLK_INLINE_ENCRYPTION
+static const struct blk_crypto_ll_ops nvme_rdma_crypto_ops = {
+	.keyslot_program	= nvme_rdma_crypto_keyslot_program,
+	.keyslot_evict		= nvme_rdma_crypto_keyslot_evict,
+};
+
+static int nvme_rdma_crypto_profile_init(struct nvme_rdma_ctrl *ctrl, bool new)
+{
+	struct blk_crypto_profile *profile = &ctrl->ctrl.crypto_profile;
+	int ret;
+
+	if (!new) {
+		blk_crypto_reprogram_all_keys(profile);
+		return 0;
+	}
+
+	ret = devm_blk_crypto_profile_init(ctrl->ctrl.device, profile,
+					   NVME_RDMA_NUM_CRYPTO_KEYSLOTS);
+	if (ret) {
+		dev_err(ctrl->ctrl.device,
+			"devm_blk_crypto_profile_init failed err %d\n", ret);
+		return ret;
+	}
+
+	profile->ll_ops = nvme_rdma_crypto_ops;
+	profile->dev = ctrl->ctrl.device;
+	profile->max_dun_bytes_supported = IB_CRYPTO_XTS_TWEAK_MAX_SIZE;
+	profile->modes_supported[BLK_ENCRYPTION_MODE_AES_256_XTS] =
+		NVME_RDMA_DATA_UNIT_SIZES;
+
+	return 0;
+}
+
+static void nvme_rdma_set_crypto_attrs(struct ib_crypto_attrs *crypto_attrs,
+		struct nvme_rdma_queue *queue, struct nvme_rdma_request *req)
+{
+	struct request *rq = blk_mq_rq_from_pdu(req);
+	u32 slot_idx = blk_crypto_keyslot_index(rq->crypt_keyslot);
+
+	memset(crypto_attrs, 0, sizeof(*crypto_attrs));
+
+	crypto_attrs->encrypt_domain = IB_CRYPTO_ENCRYPTED_WIRE_DOMAIN;
+	crypto_attrs->encrypt_standard = IB_CRYPTO_AES_XTS;
+	crypto_attrs->data_unit_size =
+		rq->crypt_ctx->bc_key->crypto_cfg.data_unit_size;
+	crypto_attrs->dek = queue->ctrl->dek[slot_idx]->id;
+	memcpy(crypto_attrs->xts_init_tweak, rq->crypt_ctx->bc_dun,
+	       sizeof(crypto_attrs->xts_init_tweak));
+}
+
+static int nvme_rdma_fill_crypto_caps(struct nvme_rdma_ctrl *ctrl, bool new)
+{
+	struct ib_crypto_caps *caps = &ctrl->device->dev->attrs.crypto_caps;
+
+	if (caps->crypto_engines & IB_CRYPTO_ENGINES_CAP_AES_XTS) {
+		ctrl->ctrl.crypto_enable = true;
+		return 0;
+	}
+
+	if (!new && ctrl->ctrl.crypto_enable) {
+		dev_err(ctrl->ctrl.device, "Must support crypto!\n");
+		return -EOPNOTSUPP;
+	}
+	ctrl->ctrl.crypto_enable = false;
+
+	return 0;
+}
+#else
+static int nvme_rdma_crypto_profile_init(struct nvme_rdma_ctrl *ctrl, bool new)
+{
+	return -EOPNOTSUPP;
+}
+
+static void nvme_rdma_set_crypto_attrs(struct ib_crypto_attrs *crypto_attrs,
+		struct nvme_rdma_queue *queue, struct nvme_rdma_request *req)
+{
+}
+
+static int nvme_rdma_fill_crypto_caps(struct nvme_rdma_ctrl *ctrl, bool new)
+{
+	return 0;
+}
+#endif /* CONFIG_BLK_INLINE_ENCRYPTION */
+
 static int nvme_rdma_configure_admin_queue(struct nvme_rdma_ctrl *ctrl,
 		bool new)
 {
@@ -835,6 +1029,10 @@ static int nvme_rdma_configure_admin_queue(struct nvme_rdma_ctrl *ctrl,
 	ctrl->max_fr_pages = nvme_rdma_get_max_fr_pages(ctrl->device->dev,
 							pi_capable);
 
+	error = nvme_rdma_fill_crypto_caps(ctrl, new);
+	if (error)
+		goto out_free_queue;
+
 	/*
 	 * Bind the async event SQE DMA mapping to the admin queue lifetime.
 	 * It's safe, since any chage in the underlying RDMA device will issue
@@ -870,6 +1068,12 @@ static int nvme_rdma_configure_admin_queue(struct nvme_rdma_ctrl *ctrl,
 	else
 		ctrl->ctrl.max_integrity_segments = 0;
 
+	if (ctrl->ctrl.crypto_enable) {
+		error = nvme_rdma_crypto_profile_init(ctrl, new);
+		if (error)
+			goto out_quiesce_queue;
+	}
+
 	nvme_unquiesce_admin_queue(&ctrl->ctrl);
 
 	error = nvme_init_ctrl_finish(&ctrl->ctrl, false);
@@ -1268,6 +1472,8 @@ static void nvme_rdma_unmap_data(struct nvme_rdma_queue *queue,
 
 	if (req->use_sig_mr)
 		pool = &queue->qp->sig_mrs;
+	else if (req->use_crypto)
+		pool = &queue->qp->crypto_mrs;
 
 	if (req->mr) {
 		ib_mr_pool_put(queue->qp, pool, req->mr);
@@ -1331,9 +1537,13 @@ static int nvme_rdma_map_sg_fr(struct nvme_rdma_queue *queue,
 		int count)
 {
 	struct nvme_keyed_sgl_desc *sg = &c->common.dptr.ksgl;
+	struct list_head *pool = &queue->qp->rdma_mrs;
 	int nr;
 
-	req->mr = ib_mr_pool_get(queue->qp, &queue->qp->rdma_mrs);
+	if (req->use_crypto)
+		pool = &queue->qp->crypto_mrs;
+
+	req->mr = ib_mr_pool_get(queue->qp, pool);
 	if (WARN_ON_ONCE(!req->mr))
 		return -EAGAIN;
 
@@ -1344,18 +1554,24 @@ static int nvme_rdma_map_sg_fr(struct nvme_rdma_queue *queue,
 	nr = ib_map_mr_sg(req->mr, req->data_sgl.sg_table.sgl, count, NULL,
 			  SZ_4K);
 	if (unlikely(nr < count)) {
-		ib_mr_pool_put(queue->qp, &queue->qp->rdma_mrs, req->mr);
+		ib_mr_pool_put(queue->qp, pool, req->mr);
 		req->mr = NULL;
 		if (nr < 0)
 			return nr;
 		return -EINVAL;
 	}
 
+	if (req->use_crypto)
+		nvme_rdma_set_crypto_attrs(req->mr->crypto_attrs, queue, req);
+
 	ib_update_fast_reg_key(req->mr, ib_inc_rkey(req->mr->rkey));
 
 	req->reg_cqe.done = nvme_rdma_memreg_done;
 	memset(&req->reg_wr, 0, sizeof(req->reg_wr));
-	req->reg_wr.wr.opcode = IB_WR_REG_MR;
+	if (req->use_crypto)
+		req->reg_wr.wr.opcode = IB_WR_REG_MR_CRYPTO;
+	else
+		req->reg_wr.wr.opcode = IB_WR_REG_MR;
 	req->reg_wr.wr.wr_cqe = &req->reg_cqe;
 	req->reg_wr.wr.num_sge = 0;
 	req->reg_wr.mr = req->mr;
@@ -1571,7 +1787,7 @@ static int nvme_rdma_map_data(struct nvme_rdma_queue *queue,
 		goto out;
 	}
 
-	if (count <= dev->num_inline_segments) {
+	if (count <= dev->num_inline_segments && !req->use_crypto) {
 		if (rq_data_dir(rq) == WRITE && nvme_rdma_queue_idx(queue) &&
 		    queue->ctrl->use_inline_data &&
 		    blk_rq_payload_bytes(rq) <=
@@ -2052,6 +2268,18 @@ static blk_status_t nvme_rdma_queue_rq(struct blk_mq_hw_ctx *hctx,
 	else
 		req->use_sig_mr = false;
 
+#ifdef CONFIG_BLK_INLINE_ENCRYPTION
+	if (rq->crypt_ctx) {
+		if (WARN_ON_ONCE(!queue->crypto_support)) {
+			err = -EIO;
+			goto err;
+		}
+		req->use_crypto = true;
+	} else {
+		req->use_crypto = false;
+	}
+#endif
+
 	err = nvme_rdma_map_data(queue, rq, c);
 	if (unlikely(err < 0)) {
 		dev_err(queue->ctrl->ctrl.device,
-- 
2.39.0


From mboxrd@z Thu Jan  1 00:00:00 1970
Return-Path: <netdev-owner@vger.kernel.org>
X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on
	aws-us-west-2-korg-lkml-1.web.codeaurora.org
Received: from vger.kernel.org (vger.kernel.org [23.128.96.18])
	by smtp.lore.kernel.org (Postfix) with ESMTP id 2019BC677F1
	for <netdev@archiver.kernel.org>; Mon, 16 Jan 2023 13:07:19 +0000 (UTC)
Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand
        id S230238AbjAPNHP (ORCPT <rfc822;netdev@archiver.kernel.org>);
        Mon, 16 Jan 2023 08:07:15 -0500
Received: from lindbergh.monkeyblade.net ([23.128.96.19]:55576 "EHLO
        lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org
        with ESMTP id S229817AbjAPNGx (ORCPT
        <rfc822;netdev@vger.kernel.org>); Mon, 16 Jan 2023 08:06:53 -0500
Received: from dfw.source.kernel.org (dfw.source.kernel.org [IPv6:2604:1380:4641:c500::1])
        by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 5A2803A90;
        Mon, 16 Jan 2023 05:06:52 -0800 (PST)
Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140])
        (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
        (No client certificate requested)
        by dfw.source.kernel.org (Postfix) with ESMTPS id E4F1060F97;
        Mon, 16 Jan 2023 13:06:51 +0000 (UTC)
Received: by smtp.kernel.org (Postfix) with ESMTPSA id 88B85C433D2;
        Mon, 16 Jan 2023 13:06:50 +0000 (UTC)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org;
        s=k20201202; t=1673874411;
        bh=yXankd3ADXRZerTodbpyETr0H4Qj01JdsQgtwpw0Dpo=;
        h=From:To:Cc:Subject:Date:In-Reply-To:References:From;
        b=fgIjibUjt1g+aWE/ux7w8lldiqti9MPTOYrg664GTgeLLIko4iPlQIWTchtK4PPS2
         E/Nj7bdo+U8PXiZB22s69BYxYSo2jkj8gYzgB/2jWsva2ntMUuI2Aq3657DptC5dE5
         4Jr0jMc+OseLWn678RNGP/CbyWwuU5rWZ4YS+wtBZ9xex7dzm7e7WMNr0g149oyV5g
         bV1mGiJJ9b8fsOyoNu+nM/BzhRhn01fxJYL0KHLx7WVH9RdBd24jGTRwxpOzoj8iDi
         A66sE/zByovan3DalETwgdA4lfwZfOcUxm25T0qqmVAvszBo4pnTSvZLqnjof6c1O+
         WQcxe4oVbyRIg==
From:   Leon Romanovsky <leon@kernel.org>
To:     Jason Gunthorpe <jgg@nvidia.com>
Cc:     Israel Rukshin <israelr@nvidia.com>,
        Bryan Tan <bryantan@vmware.com>,
        Christoph Hellwig <hch@lst.de>,
        Eric Dumazet <edumazet@google.com>,
        Jakub Kicinski <kuba@kernel.org>, Jens Axboe <axboe@fb.com>,
        Keith Busch <kbusch@kernel.org>, linux-kernel@vger.kernel.org,
        linux-nvme@lists.infradead.org, linux-rdma@vger.kernel.org,
        linux-trace-kernel@vger.kernel.org,
        Masami Hiramatsu <mhiramat@kernel.org>,
        Max Gurtovoy <mgurtovoy@nvidia.com>, netdev@vger.kernel.org,
        Paolo Abeni <pabeni@redhat.com>,
        Saeed Mahameed <saeedm@nvidia.com>,
        Sagi Grimberg <sagi@grimberg.me>,
        Selvin Xavier <selvin.xavier@broadcom.com>,
        Steven Rostedt <rostedt@goodmis.org>,
        Vishnu Dasa <vdasa@vmware.com>,
        Yishai Hadas <yishaih@nvidia.com>
Subject: [PATCH rdma-next 08/13] RDMA/mlx5: Add cryptographic device capabilities
Date:   Mon, 16 Jan 2023 15:05:55 +0200
Message-Id: <39ba2f3cd1786e47f2541f4a7be59cc5af4b03c7.1673873422.git.leon@kernel.org>
X-Mailer: git-send-email 2.39.0
In-Reply-To: <cover.1673873422.git.leon@kernel.org>
References: <cover.1673873422.git.leon@kernel.org>
MIME-Version: 1.0
Content-Transfer-Encoding: 8bit
Precedence: bulk
List-ID: <netdev.vger.kernel.org>
X-Mailing-List: netdev@vger.kernel.org

From: Israel Rukshin <israelr@nvidia.com>

The capabilities provide information on general cryptographic support,
maximum number of DEKs and status for RDMA devices. Also, they include
the supported cryptographic engines and their import method (wrapped or
plaintext). Wrapped crypto operational flag indicates the import method
mode that can be used. For now, add only AES-XTS cryptographic support.

Signed-off-by: Israel Rukshin <israelr@nvidia.com>
Signed-off-by: Leon Romanovsky <leon@kernel.org>
---
 drivers/infiniband/hw/mlx5/Makefile  |  1 +
 drivers/infiniband/hw/mlx5/crypto.c  | 31 ++++++++++++++++++++++++++++
 drivers/infiniband/hw/mlx5/crypto.h  | 11 ++++++++++
 drivers/infiniband/hw/mlx5/main.c    |  5 +++++
 drivers/infiniband/hw/mlx5/mlx5_ib.h |  2 ++
 5 files changed, 50 insertions(+)
 create mode 100644 drivers/infiniband/hw/mlx5/crypto.c
 create mode 100644 drivers/infiniband/hw/mlx5/crypto.h

diff --git a/drivers/infiniband/hw/mlx5/Makefile b/drivers/infiniband/hw/mlx5/Makefile
index 612ee8190a2d..d6ae1a08b5b2 100644
--- a/drivers/infiniband/hw/mlx5/Makefile
+++ b/drivers/infiniband/hw/mlx5/Makefile
@@ -6,6 +6,7 @@ mlx5_ib-y := ah.o \
 	     cong.o \
 	     counters.o \
 	     cq.o \
+	     crypto.o \
 	     dm.o \
 	     doorbell.o \
 	     gsi.o \
diff --git a/drivers/infiniband/hw/mlx5/crypto.c b/drivers/infiniband/hw/mlx5/crypto.c
new file mode 100644
index 000000000000..6fad9084877e
--- /dev/null
+++ b/drivers/infiniband/hw/mlx5/crypto.c
@@ -0,0 +1,31 @@
+// SPDX-License-Identifier: GPL-2.0 OR Linux-OpenIB
+/* Copyright (c) 2022, NVIDIA CORPORATION & AFFILIATES. */
+
+#include "crypto.h"
+
+void mlx5r_crypto_caps_init(struct mlx5_ib_dev *dev)
+{
+	struct ib_crypto_caps *caps = &dev->crypto_caps;
+	struct mlx5_core_dev *mdev = dev->mdev;
+
+	if (!(MLX5_CAP_GEN_64(dev->mdev, general_obj_types) &
+	      MLX5_HCA_CAP_GENERAL_OBJECT_TYPES_ENCRYPTION_KEY))
+		return;
+
+	if (!MLX5_CAP_GEN(mdev, aes_xts_multi_block_le_tweak) &&
+	    !MLX5_CAP_GEN(mdev, aes_xts_multi_block_be_tweak))
+		return;
+
+	if (MLX5_CAP_CRYPTO(mdev, wrapped_import_method) &
+	    MLX5_CRYPTO_WRAPPED_IMPORT_METHOD_CAP_AES_XTS)
+		return;
+
+	if (MLX5_CAP_CRYPTO(mdev, failed_selftests)) {
+		mlx5_ib_warn(dev, "crypto self-tests failed with error 0x%x\n",
+			     MLX5_CAP_CRYPTO(mdev, failed_selftests));
+		return;
+	}
+
+	caps->crypto_engines |= IB_CRYPTO_ENGINES_CAP_AES_XTS;
+	caps->max_num_deks = 1 << MLX5_CAP_CRYPTO(mdev, log_max_num_deks);
+}
diff --git a/drivers/infiniband/hw/mlx5/crypto.h b/drivers/infiniband/hw/mlx5/crypto.h
new file mode 100644
index 000000000000..8686ac6fb0b0
--- /dev/null
+++ b/drivers/infiniband/hw/mlx5/crypto.h
@@ -0,0 +1,11 @@
+/* SPDX-License-Identifier: GPL-2.0 OR Linux-OpenIB */
+/* Copyright (c) 2022, NVIDIA CORPORATION & AFFILIATES. */
+
+#ifndef _MLX5_IB_CRYPTO_H
+#define _MLX5_IB_CRYPTO_H
+
+#include "mlx5_ib.h"
+
+void mlx5r_crypto_caps_init(struct mlx5_ib_dev *dev);
+
+#endif /* _MLX5_IB_CRYPTO_H */
diff --git a/drivers/infiniband/hw/mlx5/main.c b/drivers/infiniband/hw/mlx5/main.c
index fb0d97bd4074..10f12e9a4dc3 100644
--- a/drivers/infiniband/hw/mlx5/main.c
+++ b/drivers/infiniband/hw/mlx5/main.c
@@ -39,6 +39,7 @@
 #include "srq.h"
 #include "qp.h"
 #include "wr.h"
+#include "crypto.h"
 #include "restrack.h"
 #include "counters.h"
 #include "umr.h"
@@ -989,6 +990,7 @@ static int mlx5_ib_query_device(struct ib_device *ibdev,
 	props->max_ah = INT_MAX;
 	props->hca_core_clock = MLX5_CAP_GEN(mdev, device_frequency_khz);
 	props->timestamp_mask = 0x7FFFFFFFFFFFFFFFULL;
+	props->crypto_caps = dev->crypto_caps;
 
 	if (IS_ENABLED(CONFIG_INFINIBAND_ON_DEMAND_PAGING)) {
 		if (dev->odp_caps.general_caps & IB_ODP_SUPPORT)
@@ -3826,6 +3828,9 @@ static int mlx5_ib_stage_caps_init(struct mlx5_ib_dev *dev)
 	if (MLX5_CAP_GEN(mdev, xrc))
 		ib_set_device_ops(&dev->ib_dev, &mlx5_ib_dev_xrc_ops);
 
+	if (MLX5_CAP_GEN(mdev, crypto))
+		mlx5r_crypto_caps_init(dev);
+
 	if (MLX5_CAP_DEV_MEM(mdev, memic) ||
 	    MLX5_CAP_GEN_64(dev->mdev, general_obj_types) &
 	    MLX5_GENERAL_OBJ_TYPES_CAP_SW_ICM)
diff --git a/drivers/infiniband/hw/mlx5/mlx5_ib.h b/drivers/infiniband/hw/mlx5/mlx5_ib.h
index 295502692da2..8f6850539542 100644
--- a/drivers/infiniband/hw/mlx5/mlx5_ib.h
+++ b/drivers/infiniband/hw/mlx5/mlx5_ib.h
@@ -1100,6 +1100,8 @@ struct mlx5_ib_dev {
 	struct mlx5_ib_delay_drop	delay_drop;
 	const struct mlx5_ib_profile	*profile;
 
+	struct ib_crypto_caps		crypto_caps;
+
 	struct mlx5_ib_lb_state		lb;
 	u8			umr_fence;
 	struct list_head	ib_dev_list;
-- 
2.39.0


From mboxrd@z Thu Jan  1 00:00:00 1970
Return-Path: <netdev-owner@vger.kernel.org>
X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on
	aws-us-west-2-korg-lkml-1.web.codeaurora.org
Received: from vger.kernel.org (vger.kernel.org [23.128.96.18])
	by smtp.lore.kernel.org (Postfix) with ESMTP id A83D4C678D8
	for <netdev@archiver.kernel.org>; Mon, 16 Jan 2023 13:07:20 +0000 (UTC)
Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand
        id S231576AbjAPNHS (ORCPT <rfc822;netdev@archiver.kernel.org>);
        Mon, 16 Jan 2023 08:07:18 -0500
Received: from lindbergh.monkeyblade.net ([23.128.96.19]:55416 "EHLO
        lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org
        with ESMTP id S229700AbjAPNGs (ORCPT
        <rfc822;netdev@vger.kernel.org>); Mon, 16 Jan 2023 08:06:48 -0500
Received: from ams.source.kernel.org (ams.source.kernel.org [IPv6:2604:1380:4601:e00::1])
        by lindbergh.monkeyblade.net (Postfix) with ESMTPS id F1C0A468D;
        Mon, 16 Jan 2023 05:06:45 -0800 (PST)
Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140])
        (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
        (No client certificate requested)
        by ams.source.kernel.org (Postfix) with ESMTPS id 81FD9B80E37;
        Mon, 16 Jan 2023 13:06:44 +0000 (UTC)
Received: by smtp.kernel.org (Postfix) with ESMTPSA id 63900C433EF;
        Mon, 16 Jan 2023 13:06:42 +0000 (UTC)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org;
        s=k20201202; t=1673874403;
        bh=Wfrh+cFzRa6bL8VsIjXTHMZp54Ti5ONFcFNaROn2N0s=;
        h=From:To:Cc:Subject:Date:In-Reply-To:References:From;
        b=BE6tEMsI+O5FK8qa8wAtpmy+ZOkSqtR5gHiDdYaBYb0ant+YSdskbqREQ1w4EPXxG
         EIy7m1WWsgybCSyttOwAHsYEn1pcFeUYPTmV6vKmbWU5XmfbcXlRXohT3y3fx02ZNg
         S8etQ60IlT5YaDoKVuwr0g1cy/N52o0bRNI5ut1FPTydIwGP15BtgV4rK9dsyUlAm9
         qMdZQlN1S2k9WiLELz5sNoHnXjsjxp/OvZurZhneSDNe8SiMUI8r7oGb0AP3S6CTnw
         J8HkCMJ+BJrYJhhhU2os4McmuIj4uoe8HhAUpyQXfD73IaruVVIKzXqybteHruAcEo
         ETpmtsfRT8Gfg==
From:   Leon Romanovsky <leon@kernel.org>
To:     Jason Gunthorpe <jgg@nvidia.com>
Cc:     Israel Rukshin <israelr@nvidia.com>,
        Bryan Tan <bryantan@vmware.com>,
        Christoph Hellwig <hch@lst.de>,
        Eric Dumazet <edumazet@google.com>,
        Jakub Kicinski <kuba@kernel.org>, Jens Axboe <axboe@fb.com>,
        Keith Busch <kbusch@kernel.org>, linux-kernel@vger.kernel.org,
        linux-nvme@lists.infradead.org, linux-rdma@vger.kernel.org,
        linux-trace-kernel@vger.kernel.org,
        Masami Hiramatsu <mhiramat@kernel.org>,
        Max Gurtovoy <mgurtovoy@nvidia.com>, netdev@vger.kernel.org,
        Paolo Abeni <pabeni@redhat.com>,
        Saeed Mahameed <saeedm@nvidia.com>,
        Sagi Grimberg <sagi@grimberg.me>,
        Selvin Xavier <selvin.xavier@broadcom.com>,
        Steven Rostedt <rostedt@goodmis.org>,
        Vishnu Dasa <vdasa@vmware.com>,
        Yishai Hadas <yishaih@nvidia.com>
Subject: [PATCH rdma-next 10/13] RDMA/mlx5: Add AES-XTS crypto support
Date:   Mon, 16 Jan 2023 15:05:57 +0200
Message-Id: <0a7ea7b225b3a62be84de852b1fc126508d07ee6.1673873422.git.leon@kernel.org>
X-Mailer: git-send-email 2.39.0
In-Reply-To: <cover.1673873422.git.leon@kernel.org>
References: <cover.1673873422.git.leon@kernel.org>
MIME-Version: 1.0
Content-Transfer-Encoding: 8bit
Precedence: bulk
List-ID: <netdev.vger.kernel.org>
X-Mailing-List: netdev@vger.kernel.org

From: Israel Rukshin <israelr@nvidia.com>

Add crypto attributes for MKey and QP. With this, the device
can encrypt/decrypt data when transmitting data from memory to
network and when receiving data from network to memory.

Signed-off-by: Israel Rukshin <israelr@nvidia.com>
Signed-off-by: Leon Romanovsky <leon@kernel.org>
---
 drivers/infiniband/hw/mlx5/crypto.c  |   1 +
 drivers/infiniband/hw/mlx5/crypto.h  |  27 +++++
 drivers/infiniband/hw/mlx5/mlx5_ib.h |   1 +
 drivers/infiniband/hw/mlx5/mr.c      |  33 ++++++
 drivers/infiniband/hw/mlx5/qp.c      |   6 +
 drivers/infiniband/hw/mlx5/wr.c      | 164 +++++++++++++++++++++++++--
 6 files changed, 222 insertions(+), 10 deletions(-)

diff --git a/drivers/infiniband/hw/mlx5/crypto.c b/drivers/infiniband/hw/mlx5/crypto.c
index 36e978c0fb85..28bc661bd5bc 100644
--- a/drivers/infiniband/hw/mlx5/crypto.c
+++ b/drivers/infiniband/hw/mlx5/crypto.c
@@ -82,6 +82,7 @@ static void mlx5r_destroy_dek(struct ib_dek *dek)
 static const struct ib_device_ops mlx5r_dev_crypto_ops = {
 	.create_dek = mlx5r_create_dek,
 	.destroy_dek = mlx5r_destroy_dek,
+	.alloc_mr_crypto = mlx5r_alloc_mr_crypto,
 };
 
 void mlx5r_crypto_caps_init(struct mlx5_ib_dev *dev)
diff --git a/drivers/infiniband/hw/mlx5/crypto.h b/drivers/infiniband/hw/mlx5/crypto.h
index b132b780030f..33f7e3b8bcce 100644
--- a/drivers/infiniband/hw/mlx5/crypto.h
+++ b/drivers/infiniband/hw/mlx5/crypto.h
@@ -6,6 +6,33 @@
 
 #include "mlx5_ib.h"
 
+enum {
+	MLX5_CRYPTO_ENCRYPTED_WIRE = 0x0,
+	MLX5_CRYPTO_ENCRYPTED_MEM = 0x1,
+};
+
+enum {
+	MLX5_CRYPTO_AES_XTS = 0x0,
+};
+
+struct mlx5_bsf_crypto {
+	u8 bsf_size_type;
+	u8 encryption_order;
+	u8 rsvd0;
+	u8 encryption_standard;
+	__be32 raw_data_size;
+	u8 block_size_p;
+	u8 rsvd1[7];
+	union {
+		__be32 be_xts_init_tweak[4];
+		__le32 le_xts_init_tweak[4];
+	};
+	__be32 rsvd_dek_pointer;
+	u8 rsvd2[4];
+	u8 keytag[8];
+	u8 rsvd3[16];
+};
+
 /*
  * The standard AES-XTS key blob composed of two keys.
  * AES-128-XTS key blob composed of two 128-bit keys, which is 32 bytes and
diff --git a/drivers/infiniband/hw/mlx5/mlx5_ib.h b/drivers/infiniband/hw/mlx5/mlx5_ib.h
index 8f6850539542..6e5b1a65f91b 100644
--- a/drivers/infiniband/hw/mlx5/mlx5_ib.h
+++ b/drivers/infiniband/hw/mlx5/mlx5_ib.h
@@ -1291,6 +1291,7 @@ struct ib_mr *mlx5_ib_alloc_mr(struct ib_pd *pd, enum ib_mr_type mr_type,
 struct ib_mr *mlx5_ib_alloc_mr_integrity(struct ib_pd *pd,
 					 u32 max_num_sg,
 					 u32 max_num_meta_sg);
+struct ib_mr *mlx5r_alloc_mr_crypto(struct ib_pd *pd, u32 max_num_sg);
 int mlx5_ib_map_mr_sg(struct ib_mr *ibmr, struct scatterlist *sg, int sg_nents,
 		      unsigned int *sg_offset);
 int mlx5_ib_map_mr_sg_pi(struct ib_mr *ibmr, struct scatterlist *data_sg,
diff --git a/drivers/infiniband/hw/mlx5/mr.c b/drivers/infiniband/hw/mlx5/mr.c
index e3b0f41aef0d..8e2f32ae4196 100644
--- a/drivers/infiniband/hw/mlx5/mr.c
+++ b/drivers/infiniband/hw/mlx5/mr.c
@@ -40,6 +40,7 @@
 #include <linux/dma-buf.h>
 #include <linux/dma-resv.h>
 #include <rdma/ib_umem_odp.h>
+#include "crypto.h"
 #include "dm.h"
 #include "mlx5_ib.h"
 #include "umr.h"
@@ -1553,6 +1554,8 @@ mlx5_alloc_priv_descs(struct ib_device *device,
 	int add_size;
 	int ret;
 
+	if (mr->ibmr.type == IB_MR_TYPE_CRYPTO)
+		size += sizeof(struct mlx5_bsf_crypto);
 	add_size = max_t(int, MLX5_UMR_ALIGN - ARCH_KMALLOC_MINALIGN, 0);
 
 	mr->descs_alloc = kzalloc(size + add_size, GFP_KERNEL);
@@ -1582,6 +1585,8 @@ mlx5_free_priv_descs(struct mlx5_ib_mr *mr)
 		int size = mr->max_descs * mr->desc_size;
 		struct mlx5_ib_dev *dev = to_mdev(device);
 
+		if (mr->ibmr.type == IB_MR_TYPE_CRYPTO)
+			size += sizeof(struct mlx5_bsf_crypto);
 		dma_unmap_single(&dev->mdev->pdev->dev, mr->desc_map, size,
 				 DMA_TO_DEVICE);
 		kfree(mr->descs_alloc);
@@ -1862,6 +1867,21 @@ static int mlx5_alloc_integrity_descs(struct ib_pd *pd, struct mlx5_ib_mr *mr,
 	return err;
 }
 
+static int mlx5_alloc_crypto_descs(struct ib_pd *pd, struct mlx5_ib_mr *mr,
+				   int ndescs, u32 *in, int inlen)
+{
+	void *mkc;
+
+	mkc = MLX5_ADDR_OF(create_mkey_in, in, memory_key_mkey_entry);
+	MLX5_SET(mkc, mkc, crypto_en, 1);
+	/* Set bsf descriptors for mkey */
+	MLX5_SET(mkc, mkc, bsf_en, 1);
+	MLX5_SET(mkc, mkc, bsf_octword_size, MLX5_MKEY_BSF_OCTO_SIZE);
+
+	return _mlx5_alloc_mkey_descs(pd, mr, ndescs, sizeof(struct mlx5_klm),
+				      0, MLX5_MKC_ACCESS_MODE_KLMS, in, inlen);
+}
+
 static struct ib_mr *__mlx5_ib_alloc_mr(struct ib_pd *pd,
 					enum ib_mr_type mr_type, u32 max_num_sg,
 					u32 max_num_meta_sg)
@@ -1897,6 +1917,9 @@ static struct ib_mr *__mlx5_ib_alloc_mr(struct ib_pd *pd,
 		err = mlx5_alloc_integrity_descs(pd, mr, max_num_sg,
 						 max_num_meta_sg, in, inlen);
 		break;
+	case IB_MR_TYPE_CRYPTO:
+		err = mlx5_alloc_crypto_descs(pd, mr, ndescs, in, inlen);
+		break;
 	default:
 		mlx5_ib_warn(dev, "Invalid mr type %d\n", mr_type);
 		err = -EINVAL;
@@ -1929,6 +1952,11 @@ struct ib_mr *mlx5_ib_alloc_mr_integrity(struct ib_pd *pd,
 				  max_num_meta_sg);
 }
 
+struct ib_mr *mlx5r_alloc_mr_crypto(struct ib_pd *pd, u32 max_num_sg)
+{
+	return __mlx5_ib_alloc_mr(pd, IB_MR_TYPE_CRYPTO, max_num_sg, 0);
+}
+
 int mlx5_ib_alloc_mw(struct ib_mw *ibmw, struct ib_udata *udata)
 {
 	struct mlx5_ib_dev *dev = to_mdev(ibmw->device);
@@ -2368,6 +2396,11 @@ int mlx5_ib_map_mr_sg(struct ib_mr *ibmr, struct scatterlist *sg, int sg_nents,
 				      mr->desc_size * mr->max_descs,
 				      DMA_TO_DEVICE);
 
+	if (ibmr->type == IB_MR_TYPE_CRYPTO) {
+		/* This is zero-based memory region */
+		ibmr->iova = 0;
+	}
+
 	return n;
 }
 
diff --git a/drivers/infiniband/hw/mlx5/qp.c b/drivers/infiniband/hw/mlx5/qp.c
index f04adc18e63b..8eb777d422e4 100644
--- a/drivers/infiniband/hw/mlx5/qp.c
+++ b/drivers/infiniband/hw/mlx5/qp.c
@@ -40,6 +40,7 @@
 #include "ib_rep.h"
 #include "counters.h"
 #include "cmd.h"
+#include "crypto.h"
 #include "umr.h"
 #include "qp.h"
 #include "wr.h"
@@ -554,6 +555,8 @@ static int calc_send_wqe(struct ib_qp_init_attr *attr)
 	}
 
 	size += attr->cap.max_send_sge * sizeof(struct mlx5_wqe_data_seg);
+	if (attr->create_flags & IB_QP_CREATE_CRYPTO_EN)
+		size += sizeof(struct mlx5_bsf_crypto);
 	if (attr->create_flags & IB_QP_CREATE_INTEGRITY_EN &&
 	    ALIGN(max_t(int, inl_size, size), MLX5_SEND_WQE_BB) < MLX5_SIG_WQE_SIZE)
 		return MLX5_SIG_WQE_SIZE;
@@ -3024,6 +3027,9 @@ static int process_create_flags(struct mlx5_ib_dev *dev, struct mlx5_ib_qp *qp,
 	process_create_flag(dev, &create_flags, MLX5_IB_QP_CREATE_SQPN_QP1,
 			    true, qp);
 
+	process_create_flag(dev, &create_flags, IB_QP_CREATE_CRYPTO_EN,
+			    !!dev->crypto_caps.crypto_engines, qp);
+
 	if (create_flags) {
 		mlx5_ib_dbg(dev, "Create QP has unsupported flags 0x%llX\n",
 			    create_flags);
diff --git a/drivers/infiniband/hw/mlx5/wr.c b/drivers/infiniband/hw/mlx5/wr.c
index df1d1b0a3ef7..e047b8aabceb 100644
--- a/drivers/infiniband/hw/mlx5/wr.c
+++ b/drivers/infiniband/hw/mlx5/wr.c
@@ -6,6 +6,7 @@
 #include <linux/gfp.h>
 #include <linux/mlx5/qp.h>
 #include <linux/mlx5/driver.h>
+#include "crypto.h"
 #include "wr.h"
 #include "umr.h"
 
@@ -21,6 +22,7 @@ static const u32 mlx5_ib_opcode[] = {
 	[IB_WR_SEND_WITH_INV]			= MLX5_OPCODE_SEND_INVAL,
 	[IB_WR_LOCAL_INV]			= MLX5_OPCODE_UMR,
 	[IB_WR_REG_MR]				= MLX5_OPCODE_UMR,
+	[IB_WR_REG_MR_CRYPTO]			= MLX5_OPCODE_UMR,
 	[IB_WR_MASKED_ATOMIC_CMP_AND_SWP]	= MLX5_OPCODE_ATOMIC_MASKED_CS,
 	[IB_WR_MASKED_ATOMIC_FETCH_AND_ADD]	= MLX5_OPCODE_ATOMIC_MASKED_FA,
 	[MLX5_IB_WR_UMR]			= MLX5_OPCODE_UMR,
@@ -115,7 +117,7 @@ static void set_data_ptr_seg(struct mlx5_wqe_data_seg *dseg, struct ib_sge *sg)
 	dseg->addr       = cpu_to_be64(sg->addr);
 }
 
-static __be64 frwr_mkey_mask(bool atomic)
+static __be64 frwr_mkey_mask(bool atomic, bool crypto)
 {
 	u64 result;
 
@@ -134,6 +136,9 @@ static __be64 frwr_mkey_mask(bool atomic)
 	if (atomic)
 		result |= MLX5_MKEY_MASK_A;
 
+	if (crypto)
+		result |= MLX5_MKEY_MASK_BSF_EN;
+
 	return cpu_to_be64(result);
 }
 
@@ -159,7 +164,8 @@ static __be64 sig_mkey_mask(void)
 }
 
 static void set_reg_umr_seg(struct mlx5_wqe_umr_ctrl_seg *umr,
-			    struct mlx5_ib_mr *mr, u8 flags, bool atomic)
+			    struct mlx5_ib_mr *mr, u8 flags, bool atomic,
+			    bool crypto)
 {
 	int size = (mr->mmkey.ndescs + mr->meta_ndescs) * mr->desc_size;
 
@@ -167,7 +173,9 @@ static void set_reg_umr_seg(struct mlx5_wqe_umr_ctrl_seg *umr,
 
 	umr->flags = flags;
 	umr->xlt_octowords = cpu_to_be16(mlx5r_umr_get_xlt_octo(size));
-	umr->mkey_mask = frwr_mkey_mask(atomic);
+	umr->mkey_mask = frwr_mkey_mask(atomic, crypto);
+	if (crypto)
+		umr->bsf_octowords = cpu_to_be16(MLX5_MKEY_BSF_OCTO_SIZE);
 }
 
 static void set_linv_umr_seg(struct mlx5_wqe_umr_ctrl_seg *umr)
@@ -188,7 +196,7 @@ static u8 get_umr_flags(int acc)
 
 static void set_reg_mkey_seg(struct mlx5_mkey_seg *seg,
 			     struct mlx5_ib_mr *mr,
-			     u32 key, int access)
+			     u32 key, int access, bool crypto)
 {
 	int ndescs = ALIGN(mr->mmkey.ndescs + mr->meta_ndescs, 8) >> 1;
 
@@ -203,6 +211,8 @@ static void set_reg_mkey_seg(struct mlx5_mkey_seg *seg,
 	seg->flags = get_umr_flags(access) | mr->access_mode;
 	seg->qpn_mkey7_0 = cpu_to_be32((key & 0xff) | 0xffffff00);
 	seg->flags_pd = cpu_to_be32(MLX5_MKEY_REMOTE_INVAL);
+	if (crypto)
+		seg->flags_pd |= cpu_to_be32(MLX5_MKEY_BSF_EN);
 	seg->start_addr = cpu_to_be64(mr->ibmr.iova);
 	seg->len = cpu_to_be64(mr->ibmr.length);
 	seg->xlt_oct_size = cpu_to_be32(ndescs);
@@ -353,6 +363,66 @@ static void mlx5_fill_inl_bsf(struct ib_sig_domain *domain,
 		cpu_to_be16(domain->sig.dif.apptag_check_mask);
 }
 
+static void mlx5_set_xts_tweak(struct ib_crypto_attrs *crypto_attrs,
+			       struct mlx5_bsf_crypto *bsf, bool is_be)
+{
+	int tweak_array_size = sizeof(bsf->be_xts_init_tweak) / sizeof(u32);
+	int i, j;
+
+	/* The endianness of the initial tweak in the kernel is LE */
+	if (is_be) {
+		for (i = 0; i < tweak_array_size; i++) {
+			j = tweak_array_size - i - 1;
+			bsf->be_xts_init_tweak[i] =
+				cpu_to_be32(crypto_attrs->xts_init_tweak[j]);
+		}
+	} else {
+		for (i = 0; i < tweak_array_size; i++)
+			bsf->le_xts_init_tweak[i] =
+				cpu_to_le32(crypto_attrs->xts_init_tweak[i]);
+	}
+}
+
+static int mlx5_set_bsf_crypto(struct ib_mr *ibmr, struct mlx5_bsf_crypto *bsf)
+{
+	struct mlx5_ib_dev *dev = to_mdev(ibmr->device);
+	struct mlx5_core_dev *mdev = dev->mdev;
+	struct ib_crypto_attrs *crypto_attrs = ibmr->crypto_attrs;
+	u64 data_size = ibmr->length;
+
+	if (crypto_attrs->encrypt_standard != IB_CRYPTO_AES_XTS)
+		return -EINVAL;
+
+	memset(bsf, 0, sizeof(*bsf));
+
+	/* Crypto only */
+	bsf->bsf_size_type = 1 << 7;
+	/* Crypto type */
+	bsf->bsf_size_type |= 1;
+
+	switch (crypto_attrs->encrypt_domain) {
+	case IB_CRYPTO_ENCRYPTED_WIRE_DOMAIN:
+		bsf->encryption_order = MLX5_CRYPTO_ENCRYPTED_WIRE;
+		break;
+	case IB_CRYPTO_ENCRYPTED_MEM_DOMAIN:
+		bsf->encryption_order = MLX5_CRYPTO_ENCRYPTED_MEM;
+		break;
+	default:
+		WARN_ONCE(1, "Bad encryption domain (%d) is given.\n",
+			  crypto_attrs->encrypt_domain);
+		return -EINVAL;
+	}
+
+	bsf->encryption_standard = MLX5_CRYPTO_AES_XTS;
+	bsf->raw_data_size = cpu_to_be32(data_size);
+	bsf->block_size_p = bs_selector(crypto_attrs->data_unit_size);
+	mlx5_set_xts_tweak(crypto_attrs, bsf,
+			   MLX5_CAP_GEN(mdev, aes_xts_multi_block_be_tweak));
+	bsf->rsvd_dek_pointer |= cpu_to_be32(crypto_attrs->dek & 0xffffff);
+
+	return 0;
+}
+
 static int mlx5_set_bsf(struct ib_mr *sig_mr,
 			struct ib_sig_attrs *sig_attrs,
 			struct mlx5_bsf *bsf, u32 data_size)
@@ -632,10 +702,53 @@ static int set_psv_wr(struct ib_sig_domain *domain,
 	return 0;
 }
 
+static int set_crypto_data_segment(struct mlx5_ib_qp *qp, struct mlx5_ib_mr *mr,
+				   void **seg, int *size, void **cur_edge)
+{
+	int mr_list_size = (mr->mmkey.ndescs + mr->meta_ndescs) * mr->desc_size;
+	int tmp_size;
+
+	mlx5r_memcpy_send_wqe(&qp->sq, cur_edge, seg, size, mr->descs,
+			      mr_list_size);
+	tmp_size = *size;
+	*size = ALIGN(*size, MLX5_SEND_WQE_BB >> 4);
+	*seg += (*size - tmp_size) * 16;
+	handle_post_send_edge(&qp->sq, seg, *size, cur_edge);
+
+	if (mlx5_set_bsf_crypto(&mr->ibmr, *seg))
+		return -EINVAL;
+
+	*seg += sizeof(struct mlx5_bsf_crypto);
+	*size += sizeof(struct mlx5_bsf_crypto) / 16;
+	handle_post_send_edge(&qp->sq, seg, *size, cur_edge);
+
+	return 0;
+}
+
+static int set_dma_bsf_crypto(struct mlx5_ib_mr *mr)
+{
+	int mr_list_size = (mr->mmkey.ndescs + mr->meta_ndescs) * mr->desc_size;
+	int aligned_size = ALIGN(mr_list_size, MLX5_SEND_WQE_BB);
+
+	ib_dma_sync_single_for_cpu(mr->ibmr.device, mr->desc_map + aligned_size,
+				   sizeof(struct mlx5_bsf_crypto),
+				   DMA_TO_DEVICE);
+
+	if (mlx5_set_bsf_crypto(&mr->ibmr, mr->descs + aligned_size))
+		return -EINVAL;
+
+	ib_dma_sync_single_for_device(mr->ibmr.device,
+				      mr->desc_map + aligned_size,
+				      sizeof(struct mlx5_bsf_crypto),
+				      DMA_TO_DEVICE);
+
+	return 0;
+}
+
 static int set_reg_wr(struct mlx5_ib_qp *qp,
 		      const struct ib_reg_wr *wr,
 		      void **seg, int *size, void **cur_edge,
-		      bool check_not_free)
+		      bool check_not_free, bool crypto)
 {
 	struct mlx5_ib_mr *mr = to_mmr(wr->mr);
 	struct mlx5_ib_pd *pd = to_mpd(qp->ibqp.pd);
@@ -667,17 +780,21 @@ static int set_reg_wr(struct mlx5_ib_qp *qp,
 	if (umr_inline)
 		flags |= MLX5_UMR_INLINE;
 
-	set_reg_umr_seg(*seg, mr, flags, atomic);
+	set_reg_umr_seg(*seg, mr, flags, atomic, crypto);
 	*seg += sizeof(struct mlx5_wqe_umr_ctrl_seg);
 	*size += sizeof(struct mlx5_wqe_umr_ctrl_seg) / 16;
 	handle_post_send_edge(&qp->sq, seg, *size, cur_edge);
 
-	set_reg_mkey_seg(*seg, mr, wr->key, wr->access);
+	set_reg_mkey_seg(*seg, mr, wr->key, wr->access, crypto);
 	*seg += sizeof(struct mlx5_mkey_seg);
 	*size += sizeof(struct mlx5_mkey_seg) / 16;
 	handle_post_send_edge(&qp->sq, seg, *size, cur_edge);
 
 	if (umr_inline) {
+		if (crypto)
+			return set_crypto_data_segment(qp, mr, seg, size,
+						       cur_edge);
+
 		mlx5r_memcpy_send_wqe(&qp->sq, cur_edge, seg, size, mr->descs,
 				      mr_list_size);
 		*size = ALIGN(*size, MLX5_SEND_WQE_BB >> 4);
@@ -685,6 +802,9 @@ static int set_reg_wr(struct mlx5_ib_qp *qp,
 		set_reg_data_seg(*seg, mr, pd);
 		*seg += sizeof(struct mlx5_wqe_data_seg);
 		*size += (sizeof(struct mlx5_wqe_data_seg) / 16);
+
+		if (crypto)
+			return set_dma_bsf_crypto(mr);
 	}
 	return 0;
 }
@@ -806,7 +926,7 @@ static int handle_reg_mr(struct mlx5_ib_qp *qp, const struct ib_send_wr *wr,
 {
 	qp->sq.wr_data[idx] = IB_WR_REG_MR;
 	(*ctrl)->imm = cpu_to_be32(reg_wr(wr)->key);
-	return set_reg_wr(qp, reg_wr(wr), seg, size, cur_edge, true);
+	return set_reg_wr(qp, reg_wr(wr), seg, size, cur_edge, true, false);
 }
 
 static int handle_psv(struct mlx5_ib_dev *dev, struct mlx5_ib_qp *qp,
@@ -870,7 +990,8 @@ static int handle_reg_mr_integrity(struct mlx5_ib_dev *dev,
 
 		(*ctrl)->imm = cpu_to_be32(reg_pi_wr.key);
 		/* UMR for data + prot registration */
-		err = set_reg_wr(qp, &reg_pi_wr, seg, size, cur_edge, false);
+		err = set_reg_wr(qp, &reg_pi_wr, seg, size, cur_edge, false,
+				 false);
 		if (unlikely(err))
 			goto out;
 
@@ -928,6 +1049,20 @@ static int handle_reg_mr_integrity(struct mlx5_ib_dev *dev,
 	return err;
 }
 
+static int handle_reg_mr_crypto(struct mlx5_ib_qp *qp,
+				const struct ib_send_wr *wr,
+				struct mlx5_wqe_ctrl_seg **ctrl, void **seg,
+				int *size, void **cur_edge, unsigned int idx)
+{
+	qp->sq.wr_data[idx] = IB_WR_REG_MR_CRYPTO;
+	(*ctrl)->imm = cpu_to_be32(reg_wr(wr)->key);
+
+	if (unlikely(!qp->ibqp.crypto_en))
+		return -EINVAL;
+
+	return set_reg_wr(qp, reg_wr(wr), seg, size, cur_edge, true, true);
+}
+
 static int handle_qpt_rc(struct mlx5_ib_dev *dev, struct mlx5_ib_qp *qp,
 			 const struct ib_send_wr *wr,
 			 struct mlx5_wqe_ctrl_seg **ctrl, void **seg, int *size,
@@ -971,6 +1106,14 @@ static int handle_qpt_rc(struct mlx5_ib_dev *dev, struct mlx5_ib_qp *qp,
 		*num_sge = 0;
 		break;
 
+	case IB_WR_REG_MR_CRYPTO:
+		err = handle_reg_mr_crypto(qp, wr, ctrl, seg, size, cur_edge,
+					   *idx);
+		if (unlikely(err))
+			goto out;
+		*num_sge = 0;
+		break;
+
 	default:
 		break;
 	}
@@ -1105,7 +1248,8 @@ int mlx5_ib_post_send(struct ib_qp *ibqp, const struct ib_send_wr *wr,
 		}
 
 		if (wr->opcode == IB_WR_REG_MR ||
-		    wr->opcode == IB_WR_REG_MR_INTEGRITY) {
+		    wr->opcode == IB_WR_REG_MR_INTEGRITY ||
+		    wr->opcode == IB_WR_REG_MR_CRYPTO) {
 			fence = dev->umr_fence;
 			next_fence = MLX5_FENCE_MODE_INITIATOR_SMALL;
 		} else  {
-- 
2.39.0


From mboxrd@z Thu Jan  1 00:00:00 1970
Return-Path: <netdev-owner@vger.kernel.org>
X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on
	aws-us-west-2-korg-lkml-1.web.codeaurora.org
Received: from vger.kernel.org (vger.kernel.org [23.128.96.18])
	by smtp.lore.kernel.org (Postfix) with ESMTP id D788BC46467
	for <netdev@archiver.kernel.org>; Mon, 16 Jan 2023 13:07:21 +0000 (UTC)
Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand
        id S231588AbjAPNHU (ORCPT <rfc822;netdev@archiver.kernel.org>);
        Mon, 16 Jan 2023 08:07:20 -0500
Received: from lindbergh.monkeyblade.net ([23.128.96.19]:55260 "EHLO
        lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org
        with ESMTP id S231393AbjAPNGn (ORCPT
        <rfc822;netdev@vger.kernel.org>); Mon, 16 Jan 2023 08:06:43 -0500
Received: from dfw.source.kernel.org (dfw.source.kernel.org [IPv6:2604:1380:4641:c500::1])
        by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 372237AA8;
        Mon, 16 Jan 2023 05:06:40 -0800 (PST)
Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140])
        (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
        (No client certificate requested)
        by dfw.source.kernel.org (Postfix) with ESMTPS id C9FCD60F97;
        Mon, 16 Jan 2023 13:06:39 +0000 (UTC)
Received: by smtp.kernel.org (Postfix) with ESMTPSA id 73C5AC433D2;
        Mon, 16 Jan 2023 13:06:38 +0000 (UTC)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org;
        s=k20201202; t=1673874399;
        bh=e1ONfQusN53JOuI2VALrT8a03P5hTd6F5/2Z57t4AgQ=;
        h=From:To:Cc:Subject:Date:In-Reply-To:References:From;
        b=NjeEAf6K1ZTvA5sAmlWSOfHcOZLUrT+sap+XphlaKKggOYN+QPNxrmcUQFaf8Dl37
         jll6CyVdwpaeyFDzoSXUsGerkF+82sSLOpgHsFSW2WtWgtwJMLcHaAiA6RTD5GBCbx
         86cQry3okdTrEYP8hjBbdqF6lPFBVPXghRJi7RFvZ6PTTQYXvTBN4IltOejDi8eJk6
         Nta/DpHh7p8t9iM8YgyfWjXykc+Ukxya3aBzhuKzE4xgJOrt6VAaL5D3fHIxi4KxrX
         MAeRmaIk3WdH9vIkaRjmhLj0Q91G5hk3gQwxrf7Tj4CBTH0laj42RSCxQqTJ/xNxCR
         UDxsDd3CQeeFQ==
From:   Leon Romanovsky <leon@kernel.org>
To:     Jason Gunthorpe <jgg@nvidia.com>
Cc:     Israel Rukshin <israelr@nvidia.com>,
        Bryan Tan <bryantan@vmware.com>,
        Christoph Hellwig <hch@lst.de>,
        Eric Dumazet <edumazet@google.com>,
        Jakub Kicinski <kuba@kernel.org>, Jens Axboe <axboe@fb.com>,
        Keith Busch <kbusch@kernel.org>, linux-kernel@vger.kernel.org,
        linux-nvme@lists.infradead.org, linux-rdma@vger.kernel.org,
        linux-trace-kernel@vger.kernel.org,
        Masami Hiramatsu <mhiramat@kernel.org>,
        Max Gurtovoy <mgurtovoy@nvidia.com>, netdev@vger.kernel.org,
        Paolo Abeni <pabeni@redhat.com>,
        Saeed Mahameed <saeedm@nvidia.com>,
        Sagi Grimberg <sagi@grimberg.me>,
        Selvin Xavier <selvin.xavier@broadcom.com>,
        Steven Rostedt <rostedt@goodmis.org>,
        Vishnu Dasa <vdasa@vmware.com>,
        Yishai Hadas <yishaih@nvidia.com>
Subject: [PATCH rdma-next 09/13] RDMA/mlx5: Add DEK management API
Date:   Mon, 16 Jan 2023 15:05:56 +0200
Message-Id: <447a02ca42116a422d5727e725bf90551dd0c8ba.1673873422.git.leon@kernel.org>
X-Mailer: git-send-email 2.39.0
In-Reply-To: <cover.1673873422.git.leon@kernel.org>
References: <cover.1673873422.git.leon@kernel.org>
MIME-Version: 1.0
Content-Transfer-Encoding: 8bit
Precedence: bulk
List-ID: <netdev.vger.kernel.org>
X-Mailing-List: netdev@vger.kernel.org

From: Israel Rukshin <israelr@nvidia.com>

Add an API to manage Data Encryption Keys (DEKs). The API allows
creating and destroying a DEK. DEKs allow encryption and decryption
of transmitted data and are used in MKeys for crypto operations.

Signed-off-by: Israel Rukshin <israelr@nvidia.com>
Signed-off-by: Leon Romanovsky <leon@kernel.org>
---
 drivers/infiniband/hw/mlx5/crypto.c | 83 +++++++++++++++++++++++++++++
 drivers/infiniband/hw/mlx5/crypto.h |  8 +++
 2 files changed, 91 insertions(+)

diff --git a/drivers/infiniband/hw/mlx5/crypto.c b/drivers/infiniband/hw/mlx5/crypto.c
index 6fad9084877e..36e978c0fb85 100644
--- a/drivers/infiniband/hw/mlx5/crypto.c
+++ b/drivers/infiniband/hw/mlx5/crypto.c
@@ -3,6 +3,87 @@
 
 #include "crypto.h"
 
+static struct ib_dek *mlx5r_create_dek(struct ib_pd *pd,
+				       struct ib_dek_attr *attr)
+{
+	u32 in[MLX5_ST_SZ_DW(create_encryption_key_in)] = {};
+	u32 out[MLX5_ST_SZ_DW(general_obj_out_cmd_hdr)];
+	struct mlx5_ib_dev *dev = to_mdev(pd->device);
+	u32 key_blob_size = attr->key_blob_size;
+	void *ptr, *key_addr;
+	struct ib_dek *dek;
+	u8 key_size;
+	int err;
+
+	if (attr->key_type != IB_CRYPTO_KEY_TYPE_AES_XTS)
+		return ERR_PTR(-EOPNOTSUPP);
+
+	switch (key_blob_size) {
+	case MLX5_IB_CRYPTO_AES_128_XTS_KEY_SIZE:
+		key_size = MLX5_GENERAL_OBJECT_TYPE_ENCRYPTION_KEY_KEY_SIZE_128;
+		break;
+	case MLX5_IB_CRYPTO_AES_256_XTS_KEY_SIZE:
+		key_size = MLX5_GENERAL_OBJECT_TYPE_ENCRYPTION_KEY_KEY_SIZE_256;
+		break;
+	default:
+		return ERR_PTR(-EOPNOTSUPP);
+	}
+
+	dek = kzalloc(sizeof(*dek), GFP_KERNEL);
+	if (!dek)
+		return ERR_PTR(-ENOMEM);
+
+	ptr = MLX5_ADDR_OF(create_encryption_key_in, in,
+			   general_obj_in_cmd_hdr);
+	MLX5_SET(general_obj_in_cmd_hdr, ptr, opcode,
+		 MLX5_CMD_OP_CREATE_GENERAL_OBJECT);
+	MLX5_SET(general_obj_in_cmd_hdr, ptr, obj_type,
+		 MLX5_GENERAL_OBJECT_TYPES_ENCRYPTION_KEY);
+	ptr = MLX5_ADDR_OF(create_encryption_key_in, in, encryption_key_object);
+	MLX5_SET(encryption_key_obj, ptr, key_size, key_size);
+	MLX5_SET(encryption_key_obj, ptr, key_type,
+		 MLX5_GENERAL_OBJECT_TYPE_ENCRYPTION_KEY_TYPE_AES_XTS);
+	MLX5_SET(encryption_key_obj, ptr, pd, to_mpd(pd)->pdn);
+	key_addr = MLX5_ADDR_OF(encryption_key_obj, ptr, key);
+	memcpy(key_addr, attr->key_blob, key_blob_size);
+
+	err = mlx5_cmd_exec(dev->mdev, in, sizeof(in), out, sizeof(out));
+	/* avoid leaking key on the stack */
+	memzero_explicit(in, sizeof(in));
+	if (err)
+		goto err_free;
+
+	dek->id = MLX5_GET(general_obj_out_cmd_hdr, out, obj_id);
+	dek->pd = pd;
+
+	return dek;
+
+err_free:
+	kfree(dek);
+	return ERR_PTR(err);
+}
+
+static void mlx5r_destroy_dek(struct ib_dek *dek)
+{
+	struct mlx5_ib_dev *dev = to_mdev(dek->pd->device);
+	u32 in[MLX5_ST_SZ_DW(general_obj_in_cmd_hdr)] = {};
+	u32 out[MLX5_ST_SZ_DW(general_obj_out_cmd_hdr)];
+
+	MLX5_SET(general_obj_in_cmd_hdr, in, opcode,
+		 MLX5_CMD_OP_DESTROY_GENERAL_OBJECT);
+	MLX5_SET(general_obj_in_cmd_hdr, in, obj_type,
+		 MLX5_GENERAL_OBJECT_TYPES_ENCRYPTION_KEY);
+	MLX5_SET(general_obj_in_cmd_hdr, in, obj_id, dek->id);
+
+	mlx5_cmd_exec(dev->mdev, in, sizeof(in), out, sizeof(out));
+	kfree(dek);
+}
+
+static const struct ib_device_ops mlx5r_dev_crypto_ops = {
+	.create_dek = mlx5r_create_dek,
+	.destroy_dek = mlx5r_destroy_dek,
+};
+
 void mlx5r_crypto_caps_init(struct mlx5_ib_dev *dev)
 {
 	struct ib_crypto_caps *caps = &dev->crypto_caps;
@@ -28,4 +109,6 @@ void mlx5r_crypto_caps_init(struct mlx5_ib_dev *dev)
 
 	caps->crypto_engines |= IB_CRYPTO_ENGINES_CAP_AES_XTS;
 	caps->max_num_deks = 1 << MLX5_CAP_CRYPTO(mdev, log_max_num_deks);
+
+	ib_set_device_ops(&dev->ib_dev, &mlx5r_dev_crypto_ops);
 }
diff --git a/drivers/infiniband/hw/mlx5/crypto.h b/drivers/infiniband/hw/mlx5/crypto.h
index 8686ac6fb0b0..b132b780030f 100644
--- a/drivers/infiniband/hw/mlx5/crypto.h
+++ b/drivers/infiniband/hw/mlx5/crypto.h
@@ -6,6 +6,14 @@
 
 #include "mlx5_ib.h"
 
+/*
+ * The standard AES-XTS key blob composed of two keys.
+ * AES-128-XTS key blob composed of two 128-bit keys, which is 32 bytes and
+ * AES-256-XTS key blob composed of two 256-bit keys, which is 64 bytes.
+ */
+#define MLX5_IB_CRYPTO_AES_128_XTS_KEY_SIZE	32
+#define MLX5_IB_CRYPTO_AES_256_XTS_KEY_SIZE	64
+
 void mlx5r_crypto_caps_init(struct mlx5_ib_dev *dev);
 
 #endif /* _MLX5_IB_CRYPTO_H */
-- 
2.39.0


From mboxrd@z Thu Jan  1 00:00:00 1970
Return-Path: <netdev-owner@vger.kernel.org>
X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on
	aws-us-west-2-korg-lkml-1.web.codeaurora.org
Received: from vger.kernel.org (vger.kernel.org [23.128.96.18])
	by smtp.lore.kernel.org (Postfix) with ESMTP id 6B6D2C63797
	for <netdev@archiver.kernel.org>; Mon, 16 Jan 2023 13:07:23 +0000 (UTC)
Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand
        id S231598AbjAPNHV (ORCPT <rfc822;netdev@archiver.kernel.org>);
        Mon, 16 Jan 2023 08:07:21 -0500
Received: from lindbergh.monkeyblade.net ([23.128.96.19]:56840 "EHLO
        lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org
        with ESMTP id S231312AbjAPNGj (ORCPT
        <rfc822;netdev@vger.kernel.org>); Mon, 16 Jan 2023 08:06:39 -0500
Received: from ams.source.kernel.org (ams.source.kernel.org [145.40.68.75])
        by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 05E0FAD;
        Mon, 16 Jan 2023 05:06:25 -0800 (PST)
Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140])
        (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
        (No client certificate requested)
        by ams.source.kernel.org (Postfix) with ESMTPS id A4272B80D20;
        Mon, 16 Jan 2023 13:06:23 +0000 (UTC)
Received: by smtp.kernel.org (Postfix) with ESMTPSA id 91A16C433D2;
        Mon, 16 Jan 2023 13:06:21 +0000 (UTC)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org;
        s=k20201202; t=1673874382;
        bh=vAkRimTAOToGcO2FbKESWn7KypzSRbBcz5r9JH5CQXA=;
        h=From:To:Cc:Subject:Date:In-Reply-To:References:From;
        b=ADlWSkwHACNAO2DLud+o3x07DseJaWthyhMStPks9JKa3AMvFD7TCTSQ8TZM0fQ3R
         TXWvlPdUnDpnYl24ITlNcRdLUTeTLOuuVf9xcSA9ISNpqlNKsN/d5rGkznbmjvxyE9
         qZiBv/tQlQfwjsPz960Yrq13RsUph72exN6wRTnzpQyRbuVxqrZA+czP5JviwZapIH
         8BWLxK82/4kZPDyD8SyFfgxbcdcubFqUEE97PWhIiPLmUqRmsm2JDukeuGGnDcrUma
         aPQAEvN4YMtPALZapqy/W/8UppovupAUSNeAIgEwmJ4niNNuj8tJ9dq2bl4kwefxCs
         3BkpzLZuxGDVw==
From:   Leon Romanovsky <leon@kernel.org>
To:     Jason Gunthorpe <jgg@nvidia.com>
Cc:     Israel Rukshin <israelr@nvidia.com>,
        Bryan Tan <bryantan@vmware.com>,
        Christoph Hellwig <hch@lst.de>,
        Eric Dumazet <edumazet@google.com>,
        Jakub Kicinski <kuba@kernel.org>, Jens Axboe <axboe@fb.com>,
        Keith Busch <kbusch@kernel.org>, linux-kernel@vger.kernel.org,
        linux-nvme@lists.infradead.org, linux-rdma@vger.kernel.org,
        linux-trace-kernel@vger.kernel.org,
        Masami Hiramatsu <mhiramat@kernel.org>,
        Max Gurtovoy <mgurtovoy@nvidia.com>, netdev@vger.kernel.org,
        Paolo Abeni <pabeni@redhat.com>,
        Saeed Mahameed <saeedm@nvidia.com>,
        Sagi Grimberg <sagi@grimberg.me>,
        Selvin Xavier <selvin.xavier@broadcom.com>,
        Steven Rostedt <rostedt@goodmis.org>,
        Vishnu Dasa <vdasa@vmware.com>,
        Yishai Hadas <yishaih@nvidia.com>
Subject: [PATCH mlx5-next 02/13] net/mlx5: Introduce crypto capabilities macro
Date:   Mon, 16 Jan 2023 15:05:49 +0200
Message-Id: <561d70fff0ab0ebc5594e371d9faec2a2c934972.1673873422.git.leon@kernel.org>
X-Mailer: git-send-email 2.39.0
In-Reply-To: <cover.1673873422.git.leon@kernel.org>
References: <cover.1673873422.git.leon@kernel.org>
MIME-Version: 1.0
Content-Transfer-Encoding: 8bit
Precedence: bulk
List-ID: <netdev.vger.kernel.org>
X-Mailing-List: netdev@vger.kernel.org

From: Israel Rukshin <israelr@nvidia.com>

Add MLX5_CAP_CRYPTO() macro to the list of capabilities.

Signed-off-by: Israel Rukshin <israelr@nvidia.com>
Reviewed-by: Max Gurtovoy <mgurtovoy@nvidia.com>
Signed-off-by: Leon Romanovsky <leon@kernel.org>
---
 drivers/net/ethernet/mellanox/mlx5/core/fw.c   | 6 ++++++
 drivers/net/ethernet/mellanox/mlx5/core/main.c | 1 +
 include/linux/mlx5/device.h                    | 4 ++++
 3 files changed, 11 insertions(+)

diff --git a/drivers/net/ethernet/mellanox/mlx5/core/fw.c b/drivers/net/ethernet/mellanox/mlx5/core/fw.c
index f34e758a2f1f..4603f7ffd8d6 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/fw.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/fw.c
@@ -147,6 +147,12 @@ int mlx5_query_hca_caps(struct mlx5_core_dev *dev)
 	if (err)
 		return err;
 
+	if (MLX5_CAP_GEN(dev, crypto)) {
+		err = mlx5_core_get_caps(dev, MLX5_CAP_CRYPTO);
+		if (err)
+			return err;
+	}
+
 	if (MLX5_CAP_GEN(dev, port_selection_cap)) {
 		err = mlx5_core_get_caps(dev, MLX5_CAP_PORT_SELECTION);
 		if (err)
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/main.c b/drivers/net/ethernet/mellanox/mlx5/core/main.c
index df134f6d32dc..81348a009666 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/main.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/main.c
@@ -1555,6 +1555,7 @@ static const int types[] = {
 	MLX5_CAP_DEV_SHAMPO,
 	MLX5_CAP_MACSEC,
 	MLX5_CAP_ADV_VIRTUALIZATION,
+	MLX5_CAP_CRYPTO,
 };
 
 static void mlx5_hca_caps_free(struct mlx5_core_dev *dev)
diff --git a/include/linux/mlx5/device.h b/include/linux/mlx5/device.h
index 29d4b201c7b2..fd095f0ed3ec 100644
--- a/include/linux/mlx5/device.h
+++ b/include/linux/mlx5/device.h
@@ -1204,6 +1204,7 @@ enum mlx5_cap_type {
 	MLX5_CAP_VDPA_EMULATION = 0x13,
 	MLX5_CAP_DEV_EVENT = 0x14,
 	MLX5_CAP_IPSEC,
+	MLX5_CAP_CRYPTO = 0x1a,
 	MLX5_CAP_DEV_SHAMPO = 0x1d,
 	MLX5_CAP_MACSEC = 0x1f,
 	MLX5_CAP_GENERAL_2 = 0x20,
@@ -1466,6 +1467,9 @@ enum mlx5_qcam_feature_groups {
 #define MLX5_CAP_MACSEC(mdev, cap)\
 	MLX5_GET(macsec_cap, (mdev)->caps.hca[MLX5_CAP_MACSEC]->cur, cap)
 
+#define MLX5_CAP_CRYPTO(mdev, cap)\
+	MLX5_GET(crypto_cap, (mdev)->caps.hca[MLX5_CAP_CRYPTO]->cur, cap)
+
 enum {
 	MLX5_CMD_STAT_OK			= 0x0,
 	MLX5_CMD_STAT_INT_ERR			= 0x1,
-- 
2.39.0


From mboxrd@z Thu Jan  1 00:00:00 1970
Return-Path: <netdev-owner@vger.kernel.org>
X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on
	aws-us-west-2-korg-lkml-1.web.codeaurora.org
Received: from vger.kernel.org (vger.kernel.org [23.128.96.18])
	by smtp.lore.kernel.org (Postfix) with ESMTP id 4D6A5C678DC
	for <netdev@archiver.kernel.org>; Mon, 16 Jan 2023 13:07:24 +0000 (UTC)
Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand
        id S231612AbjAPNHW (ORCPT <rfc822;netdev@archiver.kernel.org>);
        Mon, 16 Jan 2023 08:07:22 -0500
Received: from lindbergh.monkeyblade.net ([23.128.96.19]:56720 "EHLO
        lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org
        with ESMTP id S229810AbjAPNGm (ORCPT
        <rfc822;netdev@vger.kernel.org>); Mon, 16 Jan 2023 08:06:42 -0500
Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217])
        by lindbergh.monkeyblade.net (Postfix) with ESMTPS id EDB2B10AB7;
        Mon, 16 Jan 2023 05:06:35 -0800 (PST)
Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140])
        (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
        (No client certificate requested)
        by dfw.source.kernel.org (Postfix) with ESMTPS id 78B6360F97;
        Mon, 16 Jan 2023 13:06:35 +0000 (UTC)
Received: by smtp.kernel.org (Postfix) with ESMTPSA id 17886C433D2;
        Mon, 16 Jan 2023 13:06:34 +0000 (UTC)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org;
        s=k20201202; t=1673874394;
        bh=AUs3KMfwul5/qoENqpBBsgdpaDQZs+4sjPEYd3EWOgw=;
        h=From:To:Cc:Subject:Date:In-Reply-To:References:From;
        b=SZ2qMwj8IZWq9M1yXgCtDldTsT2NjF+yGxfuxQc6UhuH6wTSCTXdN8hzxp4iR5+kx
         TAs8fTSTQeD+D8cq/Ge6NnfU1ObfO0vh8sL+fE1NrUgSUPSB/hIfwg8T8DR6cXgAN8
         HaZm5yn7T4ObQhbtMtAuzAPR5t3jE2eMTo9BsF3IDPB/6FuplJNv+huJKbmeS7gjdU
         SeeCkTOFp03faiW1jae/4mHefqoY/AR6QUy9NgxR+Co+CNVB5Y+zKNylpEHNx23pCI
         Nrk4md70FmjZ+U4kzzG/nNSdyRbf6vgaoj+wE+KZoYnYo0MvKynZ8kvqgNXJvkpqGQ
         tJOQAlFnKwrmQ==
From:   Leon Romanovsky <leon@kernel.org>
To:     Jason Gunthorpe <jgg@nvidia.com>
Cc:     Israel Rukshin <israelr@nvidia.com>,
        Bryan Tan <bryantan@vmware.com>,
        Christoph Hellwig <hch@lst.de>,
        Eric Dumazet <edumazet@google.com>,
        Jakub Kicinski <kuba@kernel.org>, Jens Axboe <axboe@fb.com>,
        Keith Busch <kbusch@kernel.org>, linux-kernel@vger.kernel.org,
        linux-nvme@lists.infradead.org, linux-rdma@vger.kernel.org,
        linux-trace-kernel@vger.kernel.org,
        Masami Hiramatsu <mhiramat@kernel.org>,
        Max Gurtovoy <mgurtovoy@nvidia.com>, netdev@vger.kernel.org,
        Paolo Abeni <pabeni@redhat.com>,
        Saeed Mahameed <saeedm@nvidia.com>,
        Sagi Grimberg <sagi@grimberg.me>,
        Selvin Xavier <selvin.xavier@broadcom.com>,
        Steven Rostedt <rostedt@goodmis.org>,
        Vishnu Dasa <vdasa@vmware.com>,
        Yishai Hadas <yishaih@nvidia.com>
Subject: [PATCH rdma-next 05/13] RDMA/core: Add DEK management API
Date:   Mon, 16 Jan 2023 15:05:52 +0200
Message-Id: <58e678103d910efbe3481d698169af9dadf70d4b.1673873422.git.leon@kernel.org>
X-Mailer: git-send-email 2.39.0
In-Reply-To: <cover.1673873422.git.leon@kernel.org>
References: <cover.1673873422.git.leon@kernel.org>
MIME-Version: 1.0
Content-Transfer-Encoding: 8bit
Precedence: bulk
List-ID: <netdev.vger.kernel.org>
X-Mailing-List: netdev@vger.kernel.org

From: Israel Rukshin <israelr@nvidia.com>

Add an API to manage Data Encryption Keys (DEKs). The API allows
creating and destroying a DEK. DEKs allow encryption and decryption
of transmitted data and are used in MKeys for crypto operations.
A crypto setter for the MKey configuration API will be added in the
following commit.

Signed-off-by: Israel Rukshin <israelr@nvidia.com>
Reviewed-by: Max Gurtovoy <mgurtovoy@nvidia.com>
Signed-off-by: Leon Romanovsky <leon@kernel.org>
---
 drivers/infiniband/core/device.c |  2 ++
 drivers/infiniband/core/verbs.c  | 32 +++++++++++++++++++++++++++
 include/rdma/crypto.h            | 38 ++++++++++++++++++++++++++++++++
 include/rdma/ib_verbs.h          |  3 +++
 4 files changed, 75 insertions(+)

diff --git a/drivers/infiniband/core/device.c b/drivers/infiniband/core/device.c
index a666847bd714..b2016725c3d8 100644
--- a/drivers/infiniband/core/device.c
+++ b/drivers/infiniband/core/device.c
@@ -2615,6 +2615,7 @@ void ib_set_device_ops(struct ib_device *dev, const struct ib_device_ops *ops)
 	SET_DEVICE_OP(dev_ops, create_ah);
 	SET_DEVICE_OP(dev_ops, create_counters);
 	SET_DEVICE_OP(dev_ops, create_cq);
+	SET_DEVICE_OP(dev_ops, create_dek);
 	SET_DEVICE_OP(dev_ops, create_flow);
 	SET_DEVICE_OP(dev_ops, create_qp);
 	SET_DEVICE_OP(dev_ops, create_rwq_ind_table);
@@ -2632,6 +2633,7 @@ void ib_set_device_ops(struct ib_device *dev, const struct ib_device_ops *ops)
 	SET_DEVICE_OP(dev_ops, destroy_ah);
 	SET_DEVICE_OP(dev_ops, destroy_counters);
 	SET_DEVICE_OP(dev_ops, destroy_cq);
+	SET_DEVICE_OP(dev_ops, destroy_dek);
 	SET_DEVICE_OP(dev_ops, destroy_flow);
 	SET_DEVICE_OP(dev_ops, destroy_flow_action);
 	SET_DEVICE_OP(dev_ops, destroy_qp);
diff --git a/drivers/infiniband/core/verbs.c b/drivers/infiniband/core/verbs.c
index 26b021f43ba4..03633d706106 100644
--- a/drivers/infiniband/core/verbs.c
+++ b/drivers/infiniband/core/verbs.c
@@ -2306,6 +2306,38 @@ struct ib_mr *ib_alloc_mr_integrity(struct ib_pd *pd,
 }
 EXPORT_SYMBOL(ib_alloc_mr_integrity);
 
+/**
+ * ib_create_dek - Create a DEK (Data Encryption Key) associated with the
+ * specific protection domain.
+ * @pd: The protection domain associated with the DEK.
+ * @attr: The attributes of the DEK.
+ *
+ * Return: Allocated DEK in case of success; IS_ERR() is true in case of an
+ * error, PTR_ERR() returns the error code.
+ */
+struct ib_dek *ib_create_dek(struct ib_pd *pd, struct ib_dek_attr *attr)
+{
+	struct ib_device *device = pd->device;
+
+	if (!device->ops.create_dek || !device->ops.destroy_dek)
+		return ERR_PTR(-EOPNOTSUPP);
+
+	return device->ops.create_dek(pd, attr);
+}
+EXPORT_SYMBOL(ib_create_dek);
+
+/**
+ * ib_destroy_dek - Destroys the specified DEK.
+ * @dek: The DEK to destroy.
+ */
+void ib_destroy_dek(struct ib_dek *dek)
+{
+	struct ib_device *device = dek->pd->device;
+
+	device->ops.destroy_dek(dek);
+}
+EXPORT_SYMBOL(ib_destroy_dek);
+
 /* Multicast groups */
 
 static bool is_valid_mcast_lid(struct ib_qp *qp, u16 lid)
diff --git a/include/rdma/crypto.h b/include/rdma/crypto.h
index 4779eacb000e..cdf287c94737 100644
--- a/include/rdma/crypto.h
+++ b/include/rdma/crypto.h
@@ -34,4 +34,42 @@ struct ib_crypto_caps {
 	u32 max_num_deks;
 };
 
+/**
+ * enum ib_crypto_key_type - Cryptographic key types
+ * @IB_CRYPTO_KEY_TYPE_AES_XTS: Key of type AES-XTS, which can be used when
+ * IB_CRYPTO_AES_XTS is supported.
+ */
+enum ib_crypto_key_type {
+	IB_CRYPTO_KEY_TYPE_AES_XTS,
+};
+
+/**
+ * struct ib_dek_attr - Parameters for DEK (Data Encryption Key)
+ * @key_blob: the key blob that will be used for encryption and decryption of
+ *     transmitted data. Actual size and layout of this field depends on the
+ *     provided key_type and key_blob_size.
+ *     The layout of AES_XTS key is: key1_128b + key2_128b or key1_256b +
+ *     key2_256b.
+ * @key_blob_size: size of the key blob in bytes.
+ * @key_type: specific cryptographic key type.
+ */
+struct ib_dek_attr {
+	const void *key_blob;
+	u32 key_blob_size;
+	enum ib_crypto_key_type key_type;
+};
+
+/**
+ * struct ib_dek - Data Encryption Key
+ * @pd: The protection domain associated with the DEK.
+ * @id: DEK identifier.
+ */
+struct ib_dek {
+	struct ib_pd *pd;
+	u32 id;
+};
+
+struct ib_dek *ib_create_dek(struct ib_pd *pd, struct ib_dek_attr *attr);
+void ib_destroy_dek(struct ib_dek *dek);
+
 #endif /* _RDMA_CRYPTO_H_ */
diff --git a/include/rdma/ib_verbs.h b/include/rdma/ib_verbs.h
index 83be7e49c5f7..5fb42d553ca1 100644
--- a/include/rdma/ib_verbs.h
+++ b/include/rdma/ib_verbs.h
@@ -2512,6 +2512,9 @@ struct ib_device_ops {
 	struct ib_mr *(*alloc_mr_integrity)(struct ib_pd *pd,
 					    u32 max_num_data_sg,
 					    u32 max_num_meta_sg);
+	struct ib_dek *(*create_dek)(struct ib_pd *pd,
+				     struct ib_dek_attr *attr);
+	void (*destroy_dek)(struct ib_dek *dek);
 	int (*advise_mr)(struct ib_pd *pd,
 			 enum ib_uverbs_advise_mr_advice advice, u32 flags,
 			 struct ib_sge *sg_list, u32 num_sge,
-- 
2.39.0


From mboxrd@z Thu Jan  1 00:00:00 1970
Return-Path: <netdev-owner@vger.kernel.org>
X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on
	aws-us-west-2-korg-lkml-1.web.codeaurora.org
Received: from vger.kernel.org (vger.kernel.org [23.128.96.18])
	by smtp.lore.kernel.org (Postfix) with ESMTP id 13BD2C46467
	for <netdev@archiver.kernel.org>; Mon, 16 Jan 2023 17:59:06 +0000 (UTC)
Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand
        id S230056AbjAPR7C (ORCPT <rfc822;netdev@archiver.kernel.org>);
        Mon, 16 Jan 2023 12:59:02 -0500
Received: from lindbergh.monkeyblade.net ([23.128.96.19]:44884 "EHLO
        lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org
        with ESMTP id S233461AbjAPR6V (ORCPT
        <rfc822;netdev@vger.kernel.org>); Mon, 16 Jan 2023 12:58:21 -0500
Received: from NAM04-MW2-obe.outbound.protection.outlook.com (mail-mw2nam04on2065.outbound.protection.outlook.com [40.107.101.65])
        by lindbergh.monkeyblade.net (Postfix) with ESMTPS id EA172305CB;
        Mon, 16 Jan 2023 09:39:41 -0800 (PST)
ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none;
 b=VM4FJYNDA+/qBwofI3xjWsiq5wyBDDD+TBCWf+yccNF/ix7bwrqq6Uzex4dIQWkjv8Lf7QQ2KVZlcvDobCsT9n+szYt2tCW2SoLVx1+0r5x1kjwFhRztjXtaHDu4pOlg12g9ZH2BHpzKwimVz22DKFcqRb1tk1awbsUeOV+6kNLPjkgnMhpbO58YN2K5FLcAmxg2UivpD6038mFgsgNz/uLl9WU03ZAeouJ/ImxWbdR4svvPqZZn+muk9OJZZ3bWbkQQ/4Osi2Euhi7YZeK04txzz3zqJpB7e3rhcY5Z4olxXxpOcaQcmr3t/Ptnhvvl8BATWYg76auMumtFrCbgPA==
ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com;
 s=arcselector9901;
 h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1;
 bh=cj7he6aEHAdV7kojaxUUo6huinQ8kqwxP++8RACu1BY=;
 b=lxsBYa8LTiG+Q5B6tQziR90zV5e+BMxOXx8XYWmnAP8FA/MopeFWMXWda1hcW3zzppEUc+i3lbuRss48BSlRpS1XBkdLlCgOjaslMmvKZkJxt+iXpH8RZXTsIzW0aa9dT+dk73+ZNAStdoWd1QJCxOkKhaylBpPP+6UmasaboQvXh7J3vLt0WxgxlEKE41S7uC9vov3L+cD0upRxYkmr+pdbot5oqrZ/27POTucgHNLhdI2Kh+UtOUs1Q7djGUH+fdFxNvib7XYUswDVDxp1hyOGKI2gnb6KUuH6OQLmj0ezZItZ/xvuxXcynvIMpVCTFl8o/N3Fc45NuLLs0xCt+g==
ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass
 smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com;
 dkim=pass header.d=nvidia.com; arc=none
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com;
 s=selector2;
 h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck;
 bh=cj7he6aEHAdV7kojaxUUo6huinQ8kqwxP++8RACu1BY=;
 b=uQExVIfnnlp3oCDmFRY66UEWe6MytmeG+C1g1JD+Pf6s4T3IH+Y8+FLujEmbkElB2c9eQDpyXLtM0UIMdkvUCcOvt2wCwZL1r4vdTVWVGaE9M52uRCK/W3xItHm+xue+fN8dJkMacJtEeWvmRS1uPvVP6/zS/ny9egzThJZ1u2j9CLTEmRS0ciPn1MTeVieTicrEuNcxPhef8rfWKf6TLTRm8cCOUokbOKg1KOkenCuejZx8Sph9iEP3f1I0wYII6lM4ege2DAyCYEvUvfwi3QNxQX323TS+O5wIrEHTUKUpz6w1jo21usPKHZs+4I1qjIfY7fHnt6RIOv9s0BHi/g==
Authentication-Results: dkim=none (message not signed)
 header.d=none;dmarc=none action=none header.from=nvidia.com;
Received: from LV2PR12MB5869.namprd12.prod.outlook.com (2603:10b6:408:176::16)
 by PH7PR12MB6564.namprd12.prod.outlook.com (2603:10b6:510:210::22) with
 Microsoft SMTP Server (version=TLS1_2,
 cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5986.18; Mon, 16 Jan
 2023 17:39:40 +0000
Received: from LV2PR12MB5869.namprd12.prod.outlook.com
 ([fe80::f8b0:df13:5f8d:12a]) by LV2PR12MB5869.namprd12.prod.outlook.com
 ([fe80::f8b0:df13:5f8d:12a%9]) with mapi id 15.20.6002.013; Mon, 16 Jan 2023
 17:39:40 +0000
Date:   Mon, 16 Jan 2023 13:39:38 -0400
From:   Jason Gunthorpe <jgg@nvidia.com>
To:     Leon Romanovsky <leon@kernel.org>
Cc:     Israel Rukshin <israelr@nvidia.com>,
        Bryan Tan <bryantan@vmware.com>,
        Christoph Hellwig <hch@lst.de>,
        Eric Dumazet <edumazet@google.com>,
        Jakub Kicinski <kuba@kernel.org>, Jens Axboe <axboe@fb.com>,
        Keith Busch <kbusch@kernel.org>, linux-kernel@vger.kernel.org,
        linux-nvme@lists.infradead.org, linux-rdma@vger.kernel.org,
        linux-trace-kernel@vger.kernel.org,
        Masami Hiramatsu <mhiramat@kernel.org>,
        Max Gurtovoy <mgurtovoy@nvidia.com>, netdev@vger.kernel.org,
        Paolo Abeni <pabeni@redhat.com>,
        Saeed Mahameed <saeedm@nvidia.com>,
        Sagi Grimberg <sagi@grimberg.me>,
        Selvin Xavier <selvin.xavier@broadcom.com>,
        Steven Rostedt <rostedt@goodmis.org>,
        Vishnu Dasa <vdasa@vmware.com>,
        Yishai Hadas <yishaih@nvidia.com>
Subject: Re: [PATCH rdma-next 03/13] RDMA: Split kernel-only create QP flags
 from uverbs create QP flags
Message-ID: <Y8WL2gQqqeZdMvr6@nvidia.com>
References: <cover.1673873422.git.leon@kernel.org>
 <6e46859a58645d9f16a63ff76592487aabc9971d.1673873422.git.leon@kernel.org>
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <6e46859a58645d9f16a63ff76592487aabc9971d.1673873422.git.leon@kernel.org>
X-ClientProxiedBy: BL0PR01CA0026.prod.exchangelabs.com (2603:10b6:208:71::39)
 To LV2PR12MB5869.namprd12.prod.outlook.com (2603:10b6:408:176::16)
MIME-Version: 1.0
X-MS-PublicTrafficType: Email
X-MS-TrafficTypeDiagnostic: LV2PR12MB5869:EE_|PH7PR12MB6564:EE_
X-MS-Office365-Filtering-Correlation-Id: 9709d8e8-c433-4848-edcf-08daf7e8a4d8
X-MS-Exchange-SenderADCheck: 1
X-MS-Exchange-AntiSpam-Relay: 0
X-Microsoft-Antispam: BCL:0;
X-Microsoft-Antispam-Message-Info:  
X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:LV2PR12MB5869.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230022)(4636009)(39860400002)(136003)(376002)(366004)(346002)(396003)(451199015)(6916009)(54906003)(41300700001)(2906002)(38100700002)(316002)(6486002)(107886003)(6506007)(478600001)(83380400001)(66476007)(4326008)(8676002)(66556008)(186003)(26005)(86362001)(6512007)(2616005)(5660300002)(36756003)(7416002)(66946007)(8936002);DIR:OUT;SFP:1101;
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?2jhN+mgCWVdPrrtULIfZoKmWJd04av+CfUuzhvhIDTyxHDy6tXNv2VyFgAYo?=
 =?us-ascii?Q?6RGri+Z181oeoFM3pZybeX9qOnhv2pyBGypHEBsP1C4rG5xO9nKB/s+2grrx?=
 =?us-ascii?Q?1vj/WQtJefMNLTFsniOig1yF8XB8dioNVr0ntwy700wslMArXfsRowNvn9BI?=
 =?us-ascii?Q?YojU4BkcYjE7QujcdG0yLIVNJEollOG1Mhhkyosw2/qWxFpmkBUWDiRweM8h?=
 =?us-ascii?Q?PmghzLU/B6+JJ+4Lgofozx1WY/h/aLcOu96yicZQyV6u+yiQpOWN7uh65THV?=
 =?us-ascii?Q?dlB3CGOLy6B1esSDaFY0ThbtnUvAS02lS9+Lv9UzOCJ9KXxuEU04S499oB9q?=
 =?us-ascii?Q?pXhhsYP/nZ3OBi1hil0kuLY8FUBdKr0tV0k5hZIE8+3sfkPtF3Kir6ImyMIG?=
 =?us-ascii?Q?OFnCBSbnAxinVmf+jzmKc6ejQMgs/2oJ6YcZ7JFsQdq+IJAHgzYz4uaPqYfZ?=
 =?us-ascii?Q?fFqrk/HcfPE9I+yKfb+m2+ZvglyKrtTOG4gzRCmj7DTaDHka7xxo/OjEasii?=
 =?us-ascii?Q?bULCaV7e2wxZinS50BPLnXclOldbUAaykhn0abH+l3V1nffSsoxLYOVoVoGq?=
 =?us-ascii?Q?X3A9p16KR2ASb+oSr1JR4Z/cMJ2CfCzfLII/TJimEcK4FabU+5WQ8VcwvEDE?=
 =?us-ascii?Q?EquRzzKWyNSrZ8R8ELP51Sqec/qTdkdwFlv9Fpf19BNIIVFJ4bxRFhrkPpT7?=
 =?us-ascii?Q?wVP5gcx09kfQjw5o8HUXZo5s/zX9eVMLoHFqpBiuuQRytPwtGJqE/rJjaea+?=
 =?us-ascii?Q?tBVYnlyKUdnuf+85/UhDgHmLb46Q761KUrpSQKhnrLzypikVipL9dt1ajzS5?=
 =?us-ascii?Q?P+J1F0u1wJeuXPFPpnub7EN6GaCTjsJiOrHOL+fiXgB+I6DWQeIbxzJlUHxQ?=
 =?us-ascii?Q?0s+et1Go1J92LwkG9h4xdC8+GMmJhxGXpNGByRvY9OU2k2nHDAJwqDXqwpMS?=
 =?us-ascii?Q?zQdnLKcH7w4523wu4nFyFCJLeJWmESRnH8l5ou/aDBVcuSEt+lyXyuEzM6+d?=
 =?us-ascii?Q?iuZIUcAfZic8yxqcxtNRSUTD7NG9URQyCa6sPtSS7/ApKnozHRR27dAjmT8d?=
 =?us-ascii?Q?+fLXROHEtGBRP8Ufg12ZNn364tMFd2d8hK/fALjdLOmXB/c3j7QBywCNBwfn?=
 =?us-ascii?Q?JcKnfN5TUIQkK2GhHUraUX9dOk06DEkzS8JNx/y8hSRMpyjYICCbzZvUMbye?=
 =?us-ascii?Q?0kmraPecHxHiDv0NUPOsIprxxe+h8LJ9BLkr0FsZsK5NyC7kkJyyGOH6Qtbr?=
 =?us-ascii?Q?TTRMPXAEq05aIYnONqeRDS/lwpDbOq2sXftwBzHEHXsW8xwX/mAEUKpn67FK?=
 =?us-ascii?Q?ZSIQ9JrPNFbHNOAGiUI7m46cbXXFuj1bIl3MW42JbuYvX7LPJHuPDSS91zyv?=
 =?us-ascii?Q?QPjwwZLqzKDc9x798ArjzEwrOYHTTD73J01/FyCWl7n4c/es2nIRSq+Qe7nF?=
 =?us-ascii?Q?mZrBALgqrknyNPuFAV7E31WXScqttJetD+poWDplG487kj3FV562J/QkiXAn?=
 =?us-ascii?Q?cwlrl5NiAwknmr82lAlyQwSttqtT1Ff0gQ/Eu6jJ3ZYIHQB5z6qu616JI3JS?=
 =?us-ascii?Q?5rZ9VoaXaJrU1Lmhif6JdRfyGtaOGZj2n10HmcjT?=
X-OriginatorOrg: Nvidia.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 9709d8e8-c433-4848-edcf-08daf7e8a4d8
X-MS-Exchange-CrossTenant-AuthSource: LV2PR12MB5869.namprd12.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 16 Jan 2023 17:39:39.8999
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: qZ9fbuGKhpd9XSKckBlzMqDFIbGo7YfCP1H0f5o6hPVuaC0duDmXP4bFKtgFhZOT
X-MS-Exchange-Transport-CrossTenantHeadersStamped: PH7PR12MB6564
Precedence: bulk
List-ID: <netdev.vger.kernel.org>
X-Mailing-List: netdev@vger.kernel.org

On Mon, Jan 16, 2023 at 03:05:50PM +0200, Leon Romanovsky wrote:

> diff --git a/drivers/infiniband/hw/mlx4/mlx4_ib.h b/drivers/infiniband/hw/mlx4/mlx4_ib.h
> index 17fee1e73a45..c553bf0eb257 100644
> --- a/drivers/infiniband/hw/mlx4/mlx4_ib.h
> +++ b/drivers/infiniband/hw/mlx4/mlx4_ib.h
> @@ -184,7 +184,7 @@ enum mlx4_ib_qp_flags {
>  	/* Mellanox specific flags start from IB_QP_CREATE_RESERVED_START */
>  	MLX4_IB_ROCE_V2_GSI_QP = MLX4_IB_QP_CREATE_ROCE_V2_GSI,
>  	MLX4_IB_SRIOV_TUNNEL_QP = 1 << 30,
> -	MLX4_IB_SRIOV_SQP = 1 << 31,
> +	MLX4_IB_SRIOV_SQP = 1ULL << 31,
>  };

These should be moved to a uapi if we are saying they are userspace
available

But I'm not sure they are?


> diff --git a/include/rdma/ib_verbs.h b/include/rdma/ib_verbs.h
> index 949cf4ffc536..cc2ddd4e6c12 100644
> --- a/include/rdma/ib_verbs.h
> +++ b/include/rdma/ib_verbs.h
> @@ -1140,16 +1140,15 @@ enum ib_qp_type {
>  	IB_QPT_RESERVED10,
>  };
>  
> +/*
> + * bits 0, 5, 6 and 7 may be set by old kernels and should not be used.
> + */

This is backwards "bits 0 5 6 7 were understood by older kernels and
should not be used"

>  enum ib_qp_create_flags {
> -	IB_QP_CREATE_IPOIB_UD_LSO		= 1 << 0,
>  	IB_QP_CREATE_BLOCK_MULTICAST_LOOPBACK	=
>  		IB_UVERBS_QP_CREATE_BLOCK_MULTICAST_LOOPBACK,
>  	IB_QP_CREATE_CROSS_CHANNEL              = 1 << 2,
>  	IB_QP_CREATE_MANAGED_SEND               = 1 << 3,
>  	IB_QP_CREATE_MANAGED_RECV               = 1 << 4,
> -	IB_QP_CREATE_NETIF_QP			= 1 << 5,
> -	IB_QP_CREATE_INTEGRITY_EN		= 1 << 6,
> -	IB_QP_CREATE_NETDEV_USE			= 1 << 7,
>  	IB_QP_CREATE_SCATTER_FCS		=
>  		IB_UVERBS_QP_CREATE_SCATTER_FCS,
>  	IB_QP_CREATE_CVLAN_STRIPPING		=
> @@ -1159,7 +1158,18 @@ enum ib_qp_create_flags {
>  		IB_UVERBS_QP_CREATE_PCI_WRITE_END_PADDING,
>  	/* reserve bits 26-31 for low level drivers' internal use */
>  	IB_QP_CREATE_RESERVED_START		= 1 << 26,
> -	IB_QP_CREATE_RESERVED_END		= 1 << 31,
> +	IB_QP_CREATE_RESERVED_END		= 1ULL << 31,

And these should be shifted to the uapi header

Jason

From mboxrd@z Thu Jan  1 00:00:00 1970
Return-Path: <netdev-owner@vger.kernel.org>
X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on
	aws-us-west-2-korg-lkml-1.web.codeaurora.org
Received: from vger.kernel.org (vger.kernel.org [23.128.96.18])
	by smtp.lore.kernel.org (Postfix) with ESMTP id CE35EC67871
	for <netdev@archiver.kernel.org>; Tue, 17 Jan 2023 00:31:33 +0000 (UTC)
Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand
        id S234168AbjAQAba (ORCPT <rfc822;netdev@archiver.kernel.org>);
        Mon, 16 Jan 2023 19:31:30 -0500
Received: from lindbergh.monkeyblade.net ([23.128.96.19]:58138 "EHLO
        lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org
        with ESMTP id S233746AbjAQAb2 (ORCPT
        <rfc822;netdev@vger.kernel.org>); Mon, 16 Jan 2023 19:31:28 -0500
Received: from NAM10-BN7-obe.outbound.protection.outlook.com (mail-bn7nam10on2082.outbound.protection.outlook.com [40.107.92.82])
        by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 2AA7C2886E;
        Mon, 16 Jan 2023 16:31:26 -0800 (PST)
ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none;
 b=FPRT4RxL/yKJ7lBittGlx9YTxldADAEWq204A0friLn8TSj4i3n6E/5u1kLV/zBO0EoJVKxAqyYfz7ZLcETvF2EorNhH4xXUotpFQ0YsmD1krdcI84l6Hh4ndYTv1xa7akDjZmou0DzekKjticoAv1sK20MGGBk2URU7hFL39Ug+QO78Dz8DfCKARIIFuPPfvQCjdYYvrRc0KzVH6Mj22eJnMZ3BUsOgzzudxiaL50kHmE19paTenGvBIooQni4vLhqDZyDBRNGAugRchxPAcqsmGPQlNHJauCY9geiyT5QL0JGkp4KLqYaghbCGFVRqHBXxPFqmoI6MqEHLmoIWFA==
ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com;
 s=arcselector9901;
 h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1;
 bh=D13cpUPDWVPFB1CJshF2IBtCKb3RO7UY1tX2ORD5O4Y=;
 b=oTXFR5YZ1qEW0bZVC2LoT/CE3cF1yfNzRn//oxq65i4R+I4AkNojbawtFGdwLZ5ufc3dpDIO4SUNh+EZ48MVNkyZzmpaT5fi3IAS75WrGMCZutpWS7hvCGF3ORu0g76smHK1kz6GvM6KOKG6J5fE+97QbvJpLIFgazMbTV+LIELjMljw27Z6KVYFd57oa6L5IZyuSomPXDsheMboCvOj8Yfyr/q7Z3i8OkP8Yy/vklQN4iGrWPdYuN/Jclf+GNveLlM7FWj/dx5NK12taeHdGUbEK7VJ4RRtR88Bu1F+/LozQFUrYsTzHTvaqTe72Fe0jLKds1kyp8ZKjTOYU2kGMA==
ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass
 smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com;
 dkim=pass header.d=nvidia.com; arc=none
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com;
 s=selector2;
 h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck;
 bh=D13cpUPDWVPFB1CJshF2IBtCKb3RO7UY1tX2ORD5O4Y=;
 b=qS9u9gQLSzXBAPO+8AT5o6Z8oxmvzYNQ4bIyMAbPKJTIQSysrnsSzf5gavmcTZA4HRDMhvDOP0Q7e/FueHNs/BcuSrm6u48tIe3VJP333eMbzJFkSHAdjoKlzbXrkuPdWo0VS99U8BhoqeMMPz2SSxl3OX9Fvr0PlzueZ3+e4RfgqQKew3RYQYksI9wMLxrIeTdqzf4NhHdMkRWToeNE37+XV5G+sjrwURYhJ1gu6v02zsJFuRzRKACec8YFec8aJ+a5R0UsPsM7fz7qbNmbIexK05Wje/HtchY8iAwpjC345zZhRD5jFu8v9Xg6kTKKRdF9Uk9Bl7QComU1n2t1nQ==
Received: from MW2PR12MB4667.namprd12.prod.outlook.com (2603:10b6:302:12::28)
 by DM4PR12MB5390.namprd12.prod.outlook.com (2603:10b6:5:389::15) with
 Microsoft SMTP Server (version=TLS1_2,
 cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6002.13; Tue, 17 Jan
 2023 00:31:23 +0000
Received: from MW2PR12MB4667.namprd12.prod.outlook.com
 ([fe80::b84c:b6e:dc0:8d7]) by MW2PR12MB4667.namprd12.prod.outlook.com
 ([fe80::b84c:b6e:dc0:8d7%2]) with mapi id 15.20.5986.022; Tue, 17 Jan 2023
 00:31:23 +0000
From:   Chaitanya Kulkarni <chaitanyak@nvidia.com>
To:     Israel Rukshin <israelr@nvidia.com>,
        Jason Gunthorpe <jgg@nvidia.com>
CC:     Leon Romanovsky <leon@kernel.org>, Bryan Tan <bryantan@vmware.com>,
        Christoph Hellwig <hch@lst.de>,
        Eric Dumazet <edumazet@google.com>,
        Jakub Kicinski <kuba@kernel.org>, Jens Axboe <axboe@fb.com>,
        Keith Busch <kbusch@kernel.org>,
        "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
        "linux-nvme@lists.infradead.org" <linux-nvme@lists.infradead.org>,
        "linux-rdma@vger.kernel.org" <linux-rdma@vger.kernel.org>,
        "linux-trace-kernel@vger.kernel.org" 
        <linux-trace-kernel@vger.kernel.org>,
        Masami Hiramatsu <mhiramat@kernel.org>,
        Max Gurtovoy <mgurtovoy@nvidia.com>,
        "netdev@vger.kernel.org" <netdev@vger.kernel.org>,
        Paolo Abeni <pabeni@redhat.com>,
        Saeed Mahameed <saeedm@nvidia.com>,
        Sagi Grimberg <sagi@grimberg.me>,
        Selvin Xavier <selvin.xavier@broadcom.com>,
        Steven Rostedt <rostedt@goodmis.org>,
        Vishnu Dasa <vdasa@vmware.com>,
        Yishai Hadas <yishaih@nvidia.com>
Subject: Re: [PATCH rdma-next 12/13] nvme: Add crypto profile at nvme
 controller
Thread-Topic: [PATCH rdma-next 12/13] nvme: Add crypto profile at nvme
 controller
Thread-Index: AQHZKaytERU2LmOXwU2MWs0vBNPRVK6hwrIA
Date:   Tue, 17 Jan 2023 00:31:22 +0000
Message-ID: <52fef455-fd7a-225f-c03f-b214907ce03b@nvidia.com>
References: <cover.1673873422.git.leon@kernel.org>
 <efc8dc5952baa096a14db1761f84a5ab2e76654a.1673873422.git.leon@kernel.org>
In-Reply-To: <efc8dc5952baa096a14db1761f84a5ab2e76654a.1673873422.git.leon@kernel.org>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: 
user-agent: Mozilla/5.0 (X11; Linux x86_64; rv:91.0) Gecko/20100101
 Thunderbird/91.9.0
authentication-results: dkim=none (message not signed)
 header.d=none;dmarc=none action=none header.from=nvidia.com;
x-ms-publictraffictype: Email
x-ms-traffictypediagnostic: MW2PR12MB4667:EE_|DM4PR12MB5390:EE_
x-ms-office365-filtering-correlation-id: d5b45c2b-1db8-41ed-f9fa-08daf8222931
x-ms-exchange-senderadcheck: 1
x-ms-exchange-antispam-relay: 0
x-microsoft-antispam: BCL:0;
x-microsoft-antispam-message-info:  
x-forefront-antispam-report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:MW2PR12MB4667.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230022)(4636009)(396003)(136003)(346002)(39860400002)(366004)(376002)(451199015)(7416002)(36756003)(2906002)(107886003)(6506007)(76116006)(122000001)(5660300002)(8676002)(31686004)(4326008)(8936002)(91956017)(64756008)(83380400001)(66446008)(6636002)(38100700002)(6512007)(478600001)(41300700001)(38070700005)(6486002)(186003)(66476007)(66946007)(31696002)(66556008)(71200400001)(54906003)(2616005)(86362001)(316002)(110136005)(45980500001)(43740500002);DIR:OUT;SFP:1101;
x-ms-exchange-antispam-messagedata-chunkcount: 1
x-ms-exchange-antispam-messagedata-0: =?utf-8?B?MmErdUlwYmFvL05ndndzcmQwWDNOL0htdGpIa0oxVmc2L240NVRJUVg2ZGpq?=
 =?utf-8?B?Q0U5dVZSTkpMV2dVWEUvcUZwQ0ExbVBHMTFoZzM0bjVIdHIxUEpDTzBBSnpC?=
 =?utf-8?B?d0NXNVJMNVRrRTdKVG1HeWhWUjdyS1pQWWtOT05HaXpob0ZEcjZrUlhzZFhu?=
 =?utf-8?B?OXB1c2VobUtuM21uUVc5M1RtYmgzNnRvcThPbzFrN1ZlZlFWc24rclllZHl4?=
 =?utf-8?B?U2l4MkxCQ0w0cll4eEF2WVVsZ3oyaEhSdXllNG5PNG5XWkhMdXROWmxkSkxR?=
 =?utf-8?B?UEJZTGZ6Nk9TdmdtWU9NQmN5WXNZNGR3eWhhQ1c0cWpWVHBhMThrRjduazZk?=
 =?utf-8?B?SE5UOEx6eFhZdDFZZWZ1alBsWHBoZkxwNkN2OHRjU0tuazkyRlo3cU1BS3Fs?=
 =?utf-8?B?SjBUcDRJVGhCNHRkc0twZjFkM0J6RjNLa2lXTldJUHQ4bXlDT09NMzZNdnky?=
 =?utf-8?B?Sjk5M0hhaStLUC9MQWtwNkVINzE5bExZbk43dFBiVVFWbkRYaURzWkY5NGJa?=
 =?utf-8?B?VS92bEZkejhEK3BVbEpuUEdPUEx5RSsrUUh2TG5OWE9UK3VvdFJyVW1rMUZF?=
 =?utf-8?B?STJWSjVqYzJmY3o4ZkVBSVY1NDQwcWdzM250R3ExS2Y3Z3hNd2FpZlozSkVk?=
 =?utf-8?B?cUEwOHozbVBWNzgvdENheE5LaDFBbmhycEJpd2xoNkNHZ21aTG5mUVJvRFlt?=
 =?utf-8?B?dVpLZGpRZWlpbHE0YW9obDBrVTNLdVRvdy9UWG5HUGdNYi9mNDl0V2U1NDRs?=
 =?utf-8?B?SlJVYmFKakh3VnZNYTZwK0hIdTlqaG9RajVUVVNWaWMvMFcyWkgxUmVVbkls?=
 =?utf-8?B?RjZUTXZzM0pQOSt0T3prMnV0WjdXRjVpT2VMNkVlTE1vU2VQb0E3TUJqZ1V5?=
 =?utf-8?B?RU5BUDlFNjhTWTdTWXZ6ajVFWmF2K3d2ZEFDeXlSMUxvRExER1A4c3c4WTNQ?=
 =?utf-8?B?cGRUWnkyNDZPVjZXWGMwTzZsSGlrTHMwVTBqbnEzODZVRGtqdlovK2VpRjVU?=
 =?utf-8?B?WTJUb2FybTQrRGpDUmo1UW9vOGlIV1dEQXdsOFFxakZRcEJPcjVIQkQ3alNF?=
 =?utf-8?B?NnBqcXdGRC9aaWs2MFpva1BrczhjdXRsNzhTME90TWt2MVFKeHR1blQ3bXQy?=
 =?utf-8?B?b1luWlVIRkVaaTVXaG1zZHdmd2ZqQVdkVUp5RHN2NE5vLzU5d1pMbjAxZ0JJ?=
 =?utf-8?B?QXFESVhSQ0oyVy9lT1RsMmNlem5PMTRTZWxsRno2bzNrUmw3RnhzK002Tjh0?=
 =?utf-8?B?bDdCQ0RXOFVCREYyeUxXblpCUjltVUJ6SUVRUmVVUGx4QWJBNktOaUdGdjBP?=
 =?utf-8?B?ck1WMWc2L216UG1pR0tjdXd6aXRzVTJLSk93L1RBVTNLckhnRVlySk1pYitC?=
 =?utf-8?B?YkRNSkV1L0UrR0RQbkNScytGdk1DZE5uMk0vek1lc1FidVJ3b1JkRXlPemxv?=
 =?utf-8?B?VG1lZjlXaytIYmU1ajNjOVVyRUpLODZDZDhkcWNDSE1ScGd1UEUyQjcxWkRQ?=
 =?utf-8?B?NERCZk0yYnRFb2pyQkhla2lkcjczV28xbXZwMlNoT2t1NUJGS0crOUtHdkhC?=
 =?utf-8?B?cHcwNm5EbWdDeE1yaVhtVFlPQ0YzUVloTkoxdElDNHJHNjVMS2FtVWtLdzFo?=
 =?utf-8?B?MGs4UkNGNlhCcjl5L1E0WDU1Y2xESmRzV05wUDV2SW5EYy80MU0wVDJUeXF5?=
 =?utf-8?B?N2l0NVY1czJPZVFsSU5HTlNuODBvSU9qUEttV0V5Vk00RE41SUN1bERYbEJS?=
 =?utf-8?B?a3FXamlhd0VpeUkzTlJkU1dualE0d3Q1cUhMbjk2TGo1WlhSb1pCdjVvcWcy?=
 =?utf-8?B?YVhPRmxnZ1dnc1BCYmw1aVRlbGdSVzgwS3BQRWh1elNNQzlOSmdZSUZ5a3Vm?=
 =?utf-8?B?VG5rSUpMN29Ga3JWM011ZlU4SE4vK205bFliaHhqTkhKYUFud2dZZEllQXFs?=
 =?utf-8?B?NHdjZWdLdHdzVEtPTkgrQXY4L20ycG1NS0pYaXZ5Tjg2UG1wNm0vTE0xb042?=
 =?utf-8?B?aUlNMTZOVTVSSFdFcUlDbTJvZHdDbEVpSTJaak43MWhwWnpmRmRIMm9BUXNW?=
 =?utf-8?B?Zk8rTWxwOEdvWDdiemZwWng2Q2M1clVuUnp4MnE5cFhxSjB0SXo5bmdUNTFt?=
 =?utf-8?B?dFpkWVBqVCtmaFJpcWhjSHhUSE9pV3Y0dFdLc0d1MTVpdUxqckdFY2dQTG94?=
 =?utf-8?B?S1E9PQ==?=
Content-Type: text/plain; charset="utf-8"
Content-ID: <5E61E4732E77AF498790074B6E30FDC3@namprd12.prod.outlook.com>
Content-Transfer-Encoding: base64
MIME-Version: 1.0
X-OriginatorOrg: Nvidia.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-AuthSource: MW2PR12MB4667.namprd12.prod.outlook.com
X-MS-Exchange-CrossTenant-Network-Message-Id: d5b45c2b-1db8-41ed-f9fa-08daf8222931
X-MS-Exchange-CrossTenant-originalarrivaltime: 17 Jan 2023 00:31:22.9751
 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: 43083d15-7273-40c1-b7db-39efd9ccc17a
X-MS-Exchange-CrossTenant-mailboxtype: HOSTED
X-MS-Exchange-CrossTenant-userprincipalname: 5XU3CHaLYAJykBMeYBbzvIKZTQRgPNaZYz4x1HwCQZjfABZUpjlMG6Vdn9O7We2cRCm21gu+ywhdts2nKNvzSg==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM4PR12MB5390
Precedence: bulk
List-ID: <netdev.vger.kernel.org>
X-Mailing-List: netdev@vger.kernel.org

PiBkaWZmIC0tZ2l0IGEvZHJpdmVycy9udm1lL2hvc3QvY29yZS5jIGIvZHJpdmVycy9udm1lL2hv
c3QvY29yZS5jDQo+IGluZGV4IDUxYTk4ODBkYjZjZS4uZjA5ZTRlMDIxNmIzIDEwMDY0NA0KPiAt
LS0gYS9kcml2ZXJzL252bWUvaG9zdC9jb3JlLmMNCj4gKysrIGIvZHJpdmVycy9udm1lL2hvc3Qv
Y29yZS5jDQo+IEBAIC0xOTI4LDYgKzE5MjgsOSBAQCBzdGF0aWMgdm9pZCBudm1lX3VwZGF0ZV9k
aXNrX2luZm8oc3RydWN0IGdlbmRpc2sgKmRpc2ssDQo+ICAgCQkJY2FwYWNpdHkgPSAwOw0KPiAg
IAl9DQo+ICAgDQo+ICsJaWYgKGN0cmwtPmNyeXB0b19lbmFibGUpDQo+ICsJCWJsa19jcnlwdG9f
cmVnaXN0ZXIoJmN0cmwtPmNyeXB0b19wcm9maWxlLCBkaXNrLT5xdWV1ZSk7DQo+ICsNCj4gICAJ
c2V0X2NhcGFjaXR5X2FuZF9ub3RpZnkoZGlzaywgY2FwYWNpdHkpOw0KPiAgIA0KPiAgIAludm1l
X2NvbmZpZ19kaXNjYXJkKGRpc2ssIG5zKTsNCj4gZGlmZiAtLWdpdCBhL2RyaXZlcnMvbnZtZS9o
b3N0L252bWUuaCBiL2RyaXZlcnMvbnZtZS9ob3N0L252bWUuaA0KPiBpbmRleCA0MjRjOGE0Njdh
MGMuLjU5MTM4MGY1Mzc0NCAxMDA2NDQNCj4gLS0tIGEvZHJpdmVycy9udm1lL2hvc3QvbnZtZS5o
DQo+ICsrKyBiL2RyaXZlcnMvbnZtZS9ob3N0L252bWUuaA0KPiBAQCAtMTYsNiArMTYsNyBAQA0K
PiAgICNpbmNsdWRlIDxsaW51eC9yY3VwZGF0ZS5oPg0KPiAgICNpbmNsdWRlIDxsaW51eC93YWl0
Lmg+DQo+ICAgI2luY2x1ZGUgPGxpbnV4L3QxMC1waS5oPg0KPiArI2luY2x1ZGUgPGxpbnV4L2Js
ay1jcnlwdG8tcHJvZmlsZS5oPg0KPiAgIA0KPiAgICNpbmNsdWRlIDx0cmFjZS9ldmVudHMvYmxv
Y2suaD4NCj4gICANCj4gQEAgLTM3NCw2ICszNzUsOSBAQCBzdHJ1Y3QgbnZtZV9jdHJsIHsNCj4g
ICANCj4gICAJZW51bSBudm1lX2N0cmxfdHlwZSBjbnRybHR5cGU7DQo+ICAgCWVudW0gbnZtZV9k
Y3R5cGUgZGN0eXBlOw0KPiArDQo+ICsJYm9vbCBjcnlwdG9fZW5hYmxlOw0KDQp3aHkgbm90IGRl
Y2FscmUgY3J5cHRvX3Byb2ZpbGUgYSBwb2ludGVyLCBhbGxvY2F0ZSB0aGF0IGF0IGluaXQNCmNv
bnRyb2xsZXIgYW5kIE5VTEwgY2hlY2sgYWdhaW5zdCB0aGF0IHBvaW50ZXIgaW5zdGVhZCBvZiB1
c2luZw0KYW4gZXh0cmEgdmFyaWFibGUgY3J5cHRvX2VuYWJsZSA/DQoNCmUuZy4gOi0NCg0KCWlm
IChjdHJsLT5jcnlwdG9fcHJvZmlsZSkNCgkJYmxrX2NyeXB0b19yZWdpc3RlcihjdHJsLT5jcnlw
dG9fcHJvZmlsZSwgZGlzay0+cXVldWUpOw0KDQo+ICsJc3RydWN0IGJsa19jcnlwdG9fcHJvZmls
ZSBjcnlwdG9fcHJvZmlsZTsNCg0KeW91IGFyZSBpbmNyZWFzaW5nIHRoZSBzaXplIG9mIHN0cnVj
dCBudm1lX2N0cmwgdW5jb25kaXRpb25hbGx5LA0Kd2h5IG5vdCBndWFyZCBhYm92ZSB3aXRoIENP
TkZJR19CTEtfSU5MSU5FX0VOQ1JZUFRJT04gPw0KDQotY2sNCg0K

From mboxrd@z Thu Jan  1 00:00:00 1970
Return-Path: <netdev-owner@vger.kernel.org>
X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on
	aws-us-west-2-korg-lkml-1.web.codeaurora.org
Received: from vger.kernel.org (vger.kernel.org [23.128.96.18])
	by smtp.lore.kernel.org (Postfix) with ESMTP id 9018DC67871
	for <netdev@archiver.kernel.org>; Tue, 17 Jan 2023 00:33:09 +0000 (UTC)
Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand
        id S234911AbjAQAdI (ORCPT <rfc822;netdev@archiver.kernel.org>);
        Mon, 16 Jan 2023 19:33:08 -0500
Received: from lindbergh.monkeyblade.net ([23.128.96.19]:58502 "EHLO
        lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org
        with ESMTP id S234697AbjAQAdE (ORCPT
        <rfc822;netdev@vger.kernel.org>); Mon, 16 Jan 2023 19:33:04 -0500
Received: from NAM10-BN7-obe.outbound.protection.outlook.com (mail-bn7nam10on2072.outbound.protection.outlook.com [40.107.92.72])
        by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 516A82A145;
        Mon, 16 Jan 2023 16:33:01 -0800 (PST)
ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none;
 b=RXPWWaOgJclINj+HCL3IscILcbh5sdIFhHD43Q98j7r+XDfXQu9wXcaSWrif0lby4J9J2d63ECo7AeYPEsHONIeTpwJaCYuGP3h8ub+cuYE2yzLYCCy8/P+UxDiGS/G0n13vogimbhYIjRAcIpANgwBBzhH/FneYWiAgchInuGj5Rg+zRE3/OMT6KzuJWMtDyxdhDduTFl7jUcjyUvOP/hUr2zJoABJT75K50VGN1zdte15izxPaiGvImP6v3scfNPhDOdJJa/7l9WRUiIJ9wwZwEynmSDJmGz3S/WQMjTVVv72uP6RTwzWMUK0UGkdCJTPZq/aXxwQB2FvIWl92Mg==
ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com;
 s=arcselector9901;
 h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1;
 bh=LeO6BRRgbB97O2Q2C3oBmrQdi8ULJsoXXm1jTu5FUwI=;
 b=ZLwLj+xu7E+jqia0ZHQSVFQnthiU7GS5umUej2fD9Gei8spVEq42ENiQoSvqWX5R/K845Mek/LA1c3VF0FXGJpOn/baSrLidIRlNiwcwIRntLF7o53w7tuxyV8ZjFQkzJIpQyOFe4Ti31HiA0voswY4VdtyK7cIQhae8/jP/Vx5Bc8Wf1HL/pYqj3AnzwqPil1yaFARk8Q9rzJ39WHlULM+p0OGJ5fm+a6oQHFJvvo37nPkt6X0fj1LyjPIUGdWhL7A+dJwcPssCY6q2f1/VWSt8Yl0g9no7byeQnDqnwL2zqWiQ5ZKIj8vDEB/vsC0bv7DzDqySw5WNCtbuh5OVKA==
ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass
 smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com;
 dkim=pass header.d=nvidia.com; arc=none
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com;
 s=selector2;
 h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck;
 bh=LeO6BRRgbB97O2Q2C3oBmrQdi8ULJsoXXm1jTu5FUwI=;
 b=SbkK0lxh/rTN2sBi3brGfXNug2PnnkrfixeDA9h0a1NpfFxTYYGl1VYWuuqsYHgHYg5VCwsxxlStZj9Hp7TKw2f+SroHjr+9cQDY6J4QLzvVl0xhqM2NkZLcflAxDCEWVHwdDeuApRHV8DxDGiRYMOeqBWkHsp5xd4TtU4nfqjq/6TyyUSEZW8qNz6fseVgeFxhywg7UimO2SsS64MwmTiwSfBdZEbplEZY61EtmzcsPbLMmDufRPlTBK5sWt7dT27e5quajJ/1QyjcSzCZ02gc6KNTusLM9t3Xc3iLxJ2LubziRkKV91WgD41IjjOjSRkXpg4n9zW/iEqAde4KmpA==
Received: from MW2PR12MB4667.namprd12.prod.outlook.com (2603:10b6:302:12::28)
 by DM4PR12MB5390.namprd12.prod.outlook.com (2603:10b6:5:389::15) with
 Microsoft SMTP Server (version=TLS1_2,
 cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6002.13; Tue, 17 Jan
 2023 00:32:59 +0000
Received: from MW2PR12MB4667.namprd12.prod.outlook.com
 ([fe80::b84c:b6e:dc0:8d7]) by MW2PR12MB4667.namprd12.prod.outlook.com
 ([fe80::b84c:b6e:dc0:8d7%2]) with mapi id 15.20.5986.022; Tue, 17 Jan 2023
 00:32:59 +0000
From:   Chaitanya Kulkarni <chaitanyak@nvidia.com>
To:     Leon Romanovsky <leon@kernel.org>, Jason Gunthorpe <jgg@nvidia.com>
CC:     Israel Rukshin <israelr@nvidia.com>,
        Bryan Tan <bryantan@vmware.com>,
        Christoph Hellwig <hch@lst.de>,
        Eric Dumazet <edumazet@google.com>,
        Jakub Kicinski <kuba@kernel.org>, Jens Axboe <axboe@fb.com>,
        Keith Busch <kbusch@kernel.org>,
        "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
        "linux-nvme@lists.infradead.org" <linux-nvme@lists.infradead.org>,
        "linux-rdma@vger.kernel.org" <linux-rdma@vger.kernel.org>,
        "linux-trace-kernel@vger.kernel.org" 
        <linux-trace-kernel@vger.kernel.org>,
        Masami Hiramatsu <mhiramat@kernel.org>,
        Max Gurtovoy <mgurtovoy@nvidia.com>,
        "netdev@vger.kernel.org" <netdev@vger.kernel.org>,
        Paolo Abeni <pabeni@redhat.com>,
        Saeed Mahameed <saeedm@nvidia.com>,
        Sagi Grimberg <sagi@grimberg.me>,
        Selvin Xavier <selvin.xavier@broadcom.com>,
        Steven Rostedt <rostedt@goodmis.org>,
        Vishnu Dasa <vdasa@vmware.com>,
        Yishai Hadas <yishaih@nvidia.com>
Subject: Re: [PATCH rdma-next 11/13] nvme: Introduce a local variable
Thread-Topic: [PATCH rdma-next 11/13] nvme: Introduce a local variable
Thread-Index: AQHZKaxHVD5zkyAsbECelDFns/Z2ka6hwyeA
Date:   Tue, 17 Jan 2023 00:32:59 +0000
Message-ID: <c095c9a9-0e77-c8ab-d34e-f4ab42b11938@nvidia.com>
References: <cover.1673873422.git.leon@kernel.org>
 <cf5bc542e014f465f7ae443e52e70def2993aef1.1673873422.git.leon@kernel.org>
In-Reply-To: <cf5bc542e014f465f7ae443e52e70def2993aef1.1673873422.git.leon@kernel.org>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: 
user-agent: Mozilla/5.0 (X11; Linux x86_64; rv:91.0) Gecko/20100101
 Thunderbird/91.9.0
authentication-results: dkim=none (message not signed)
 header.d=none;dmarc=none action=none header.from=nvidia.com;
x-ms-publictraffictype: Email
x-ms-traffictypediagnostic: MW2PR12MB4667:EE_|DM4PR12MB5390:EE_
x-ms-office365-filtering-correlation-id: 9edea2fe-6b61-475f-84c6-08daf82262c6
x-ms-exchange-senderadcheck: 1
x-ms-exchange-antispam-relay: 0
x-microsoft-antispam: BCL:0;
x-microsoft-antispam-message-info:  
x-forefront-antispam-report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:MW2PR12MB4667.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230022)(4636009)(396003)(136003)(346002)(39860400002)(366004)(376002)(451199015)(7416002)(36756003)(2906002)(107886003)(6506007)(53546011)(76116006)(122000001)(5660300002)(8676002)(31686004)(4744005)(4326008)(8936002)(91956017)(64756008)(83380400001)(66446008)(6636002)(38100700002)(6512007)(478600001)(41300700001)(38070700005)(6486002)(186003)(66476007)(66946007)(31696002)(66556008)(71200400001)(54906003)(2616005)(86362001)(316002)(110136005)(45980500001)(43740500002);DIR:OUT;SFP:1101;
x-ms-exchange-antispam-messagedata-chunkcount: 1
x-ms-exchange-antispam-messagedata-0: =?utf-8?B?UU5Ecm4zbCtkSG4xYm9qRDdyZ3NCbnZ1ajc0OGZ1ZFBIUVJvd2M0cGg5KzhG?=
 =?utf-8?B?K25mWkhNcFpJSzhzQVl2U3pORktIMllheDhqZHFCdGVUdHUvejdTdFpBS1R2?=
 =?utf-8?B?TkMydVVwdWFaK0s2bXowY2VyQW1Ma3g1OTI2eEFRMHZLb3gzRklLZjYrdXpI?=
 =?utf-8?B?VlMrTFJYMC9CaHE0T01kclpBRSsrWGdGZ0VqY0k3UHJSQzZ0a1lkQkV6WHl1?=
 =?utf-8?B?c3BTSzFUTFhtcjdiV25iNjY1am9YaUtlOVpqNnhWUTJhOUxNNDkzWXBOcG85?=
 =?utf-8?B?RXhFNFhtNmR4RlFwcW1VbU55K3p0TEF5OFoxNy9aYXI4OUtoOUIxeDQrVFR2?=
 =?utf-8?B?UHY5ckVvRUFGUnNacGVodW0yZzdwMUR3eHpxaGNzVk9mN1NiakRSdkhZOFNW?=
 =?utf-8?B?eEhWMjlub3dyQ21WQ3VRWW5CZ3k5TDk1c3Fod3A2bVVjQ0U2T2txcnZpMnk2?=
 =?utf-8?B?Z3FPWTBhQVh0MUxhbmVOYXZEdGRVbHVWRjQvdUw5MjdPczRHVkdZZUh5QitB?=
 =?utf-8?B?OXYrQUo2aXdNVmJsblowVmY4TnZkZG0wenAvMFpYN1RsQ00wT0dYemltb1hr?=
 =?utf-8?B?ZDQyVUtOMnhUOEhqSkNDeFZEYzMwSE1kS3hZcmdpa1dtM2JqT3hCWFkyTVpP?=
 =?utf-8?B?K3F3aXJKZW0yc25oWGtkWWl3b3czZmtEcTd0b1ZYcFBCV0kxRDIvY09DZm1J?=
 =?utf-8?B?VGhybGxSd25oR2dSeEFJN3VOS0ZQZ1FGbFRMS3RJVmVWNnhnTzk5SmNtOEhG?=
 =?utf-8?B?OUZIR2VkdEFyYk1PQk9NdUJGMEZsUWVJWVBGeXhORzkwVXkrYXl5RlhiWnFO?=
 =?utf-8?B?b3prR0d6alFGTnJnY2JMRy9ReGRZSG0rVmVOR1ZsbTdKRjFzQ2pFaUVVSHlQ?=
 =?utf-8?B?UGRpdW9GU3o5bHRWVTJybzRHZ2hxaGx0aW05SnVXUDhQUGdvWHo0Y0xscUlo?=
 =?utf-8?B?cEtJM2Y4aytjNDU5cEh4YkI2RmUvVW8reXN6K1RQMDBxQ00wNnZISi9jY1Vm?=
 =?utf-8?B?MXk1d1BqdXNtQ05BdGtqb3VjRFYrRHRlZVQ4QXJJWFdJQk5UcTVrMTFzZTJR?=
 =?utf-8?B?S3FwQUo3TitYRFp3RlFCbWRLTHVxcjM3alhva25wVXliQ2xFYnl0RlpZU1U5?=
 =?utf-8?B?L2RyVTdrd3F0V0x5Rzh6UTlqN0RpcHZpdkVpaFNra3FFZTZSaGVvN2dTUkVT?=
 =?utf-8?B?U3VpaXZjQlh2S21sd1o3ZXVJd0lzdmZJenR3M202aTlRa1NpZlpEVW1Md2ds?=
 =?utf-8?B?Yk1hbGkvVFd4bk1EMmd1Um84WkNrMVpWckpPbjhpNUdLcjVWOEtkK0lTU0VR?=
 =?utf-8?B?d0hOSlNFUHdhRDN4dHppdWs0MjFrZ2pEV1hyb1ljenpUZlp0VjRvV3M4dDN0?=
 =?utf-8?B?am1ON0ZyUE5KSWRRelZyc0VEb3hVL2t6QjVMOEdiTFJRcU84eXJXNnFka1hI?=
 =?utf-8?B?MngzOGJRK0pxZXBYWktXZ0x0a1h2WUlwb29SK2k3YzROeDIwNEdyUGI0MHBm?=
 =?utf-8?B?UXA4VTV1dEIrUlRVZjVyVlJyQ0RrQWp5Z1Q1TWRvbTU5blNEcnNHYVZmd3Yr?=
 =?utf-8?B?SkZWMjlSa3FSVGVrZmtUVVcyWVRPTHNyR0RYTS95N3J0WEhiRmVJWkJHS1RO?=
 =?utf-8?B?bG5YQXFkYmdBQ2l6WmxHdnFkbFRJS2RuSG1WL2loR2pCNU9MVWVoQThpczhk?=
 =?utf-8?B?VVU0OUlOMUFPckxNQXBCNWpsSUJ6MkM1R3o0QVhGaTBsYlZydGxjKzAzZnMy?=
 =?utf-8?B?YVVvMmx1QlV6eUthTWgrblBpNzlPKzI4UTBxdURyYmV6L201V3pMUUFEK2Qz?=
 =?utf-8?B?bHFNMGJGYytpSjRXd3hlUFREOWtmYXJZeHd0U1N6alVJV1hFR2wxbFhBS0Nx?=
 =?utf-8?B?YjlXTnE4Z0tvZHdDbWVwbHVkZDkyclc4SkhTT01MMG5yRU5BTjhQbWd5dTlZ?=
 =?utf-8?B?anpxQWJHeVF6Y1RSaDZLb1hmLzVjOXNTWkNxYTBUM3YxTFBlRHVTUkt1Q0Jh?=
 =?utf-8?B?MHZhcW9HR3VTZ1VuQ1dlbVRZalkyKzZUR1ZQaG9IRThVUHgySnlZdWVXWXpF?=
 =?utf-8?B?SEFWVHRsL3U2eDNSYllwY2UzcXVSTzg0TS96ejVNKzZzblR4VHF3aFFLU2Z4?=
 =?utf-8?B?dDN3MDZsRWJFU3VGV3Fta25oNi9MRmRrdkQ1YVdMaXprMWc4UlR5THNyRkJw?=
 =?utf-8?B?MFE9PQ==?=
Content-Type: text/plain; charset="utf-8"
Content-ID: <835BEEC0E72A5943AE3BFF7EE05AED0B@namprd12.prod.outlook.com>
Content-Transfer-Encoding: base64
MIME-Version: 1.0
X-OriginatorOrg: Nvidia.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-AuthSource: MW2PR12MB4667.namprd12.prod.outlook.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 9edea2fe-6b61-475f-84c6-08daf82262c6
X-MS-Exchange-CrossTenant-originalarrivaltime: 17 Jan 2023 00:32:59.5787
 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: 43083d15-7273-40c1-b7db-39efd9ccc17a
X-MS-Exchange-CrossTenant-mailboxtype: HOSTED
X-MS-Exchange-CrossTenant-userprincipalname: q7JRV1qOyyndDLMlraLaxsqhYfBL9zDsMHoX5Fn2jFz/gwIK1Y/8AUk2zzejCLjgVl5PhlgFriSzf8XREv5oDA==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM4PR12MB5390
Precedence: bulk
List-ID: <netdev.vger.kernel.org>
X-Mailing-List: netdev@vger.kernel.org

T24gMS8xNi8yMyAwNTowNSwgTGVvbiBSb21hbm92c2t5IHdyb3RlOg0KPiBGcm9tOiBJc3JhZWwg
UnVrc2hpbiA8aXNyYWVsckBudmlkaWEuY29tPg0KPiANCj4gVGhlIHBhdGNoIGRvZXNuJ3QgY2hh
bmdlIGFueSBsb2dpYy4NCj4gDQo+IFNpZ25lZC1vZmYtYnk6IElzcmFlbCBSdWtzaGluIDxpc3Jh
ZWxyQG52aWRpYS5jb20+DQo+IFNpZ25lZC1vZmYtYnk6IExlb24gUm9tYW5vdnNreSA8bGVvbkBr
ZXJuZWwub3JnPg0KPiAtLS0NCj4gICBkcml2ZXJzL252bWUvaG9zdC9jb3JlLmMgfCA3ICsrKyst
LS0NCj4gICAxIGZpbGUgY2hhbmdlZCwgNCBpbnNlcnRpb25zKCspLCAzIGRlbGV0aW9ucygtKQ0K
PiANCj4gZGlmZiAtLWdpdCBhL2RyaXZlcnMvbnZtZS9ob3N0L2NvcmUuYyBiL2RyaXZlcnMvbnZt
ZS9ob3N0L2NvcmUuYw0KPiBpbmRleCA3YmU1NjJhNGUxYWEuLjUxYTk4ODBkYjZjZSAxMDA2NDQN
Cj4gLS0tIGEvZHJpdmVycy9udm1lL2hvc3QvY29yZS5jDQo+ICsrKyBiL2RyaXZlcnMvbnZtZS9o
b3N0L2NvcmUuYw0KPiBAQCAtMTg3MCw2ICsxODcwLDcgQEAgc3RhdGljIHZvaWQgbnZtZV91cGRh
dGVfZGlza19pbmZvKHN0cnVjdCBnZW5kaXNrICpkaXNrLA0KPiAgIAlzZWN0b3JfdCBjYXBhY2l0
eSA9IG52bWVfbGJhX3RvX3NlY3QobnMsIGxlNjRfdG9fY3B1KGlkLT5uc3plKSk7DQo+ICAgCXVu
c2lnbmVkIHNob3J0IGJzID0gMSA8PCBucy0+bGJhX3NoaWZ0Ow0KPiAgIAl1MzIgYXRvbWljX2Jz
LCBwaHlzX2JzLCBpb19vcHQgPSAwOw0KPiArCXN0cnVjdCBudm1lX2N0cmwgKmN0cmwgPSBucy0+
Y3RybDsNCj4gICANCg0KSSBkb24ndCB0aGluayB0aGlzIHBhdGNoIGlzIG5lZWRlZCwgZXhpc3Rp
bmcgY29kZSBpcyBtb3JlIHJlYWRhYmxlIGFuZA0KZ2l2ZXMgbXVjaCBjbGFyaXR5IHRoYXQgd2Ug
YXJlIGFjY2Vzc2luZyBjdHJsIGZyb20gbmFtZXNwYWNlIHNpbmNlIHdlDQphcmUgaW4gbnZtZV91
cGRhdGVfZGlza19pbmZvKCkgd2hpY2ggaXMgbmFtZXNwYWNlIGJhc2VkLg0KDQotY2sNCg0K

From mboxrd@z Thu Jan  1 00:00:00 1970
Return-Path: <netdev-owner@vger.kernel.org>
X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on
	aws-us-west-2-korg-lkml-1.web.codeaurora.org
Received: from vger.kernel.org (vger.kernel.org [23.128.96.18])
	by smtp.lore.kernel.org (Postfix) with ESMTP id C6D61C678D4
	for <netdev@archiver.kernel.org>; Tue, 17 Jan 2023 11:59:47 +0000 (UTC)
Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand
        id S236118AbjAQL7p (ORCPT <rfc822;netdev@archiver.kernel.org>);
        Tue, 17 Jan 2023 06:59:45 -0500
Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39006 "EHLO
        lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org
        with ESMTP id S236908AbjAQL7g (ORCPT
        <rfc822;netdev@vger.kernel.org>); Tue, 17 Jan 2023 06:59:36 -0500
Received: from dfw.source.kernel.org (dfw.source.kernel.org [IPv6:2604:1380:4641:c500::1])
        by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 281F636FDD;
        Tue, 17 Jan 2023 03:59:09 -0800 (PST)
Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140])
        (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
        (No client certificate requested)
        by dfw.source.kernel.org (Postfix) with ESMTPS id 8BEF4612CE;
        Tue, 17 Jan 2023 11:59:09 +0000 (UTC)
Received: by smtp.kernel.org (Postfix) with ESMTPSA id 851B8C433EF;
        Tue, 17 Jan 2023 11:59:08 +0000 (UTC)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org;
        s=k20201202; t=1673956749;
        bh=C4DWw1lnsfHjJvgG7EN7/MBLwzOtKiOqkpCYuIalGOc=;
        h=Date:From:To:Cc:Subject:References:In-Reply-To:From;
        b=CyEJUdpmNRp0vCg9aUxD6eG2NOon+Jg4hJ1Lxm49Daa37qaahNpu1+wK/c6EB1LZq
         r7a6oulLUJ3AhNnHB4x1yQ6L0miJkjQ8W6kW7j2jE2H+gjJcugdZu8QcnzE/HbZ4v1
         HACJ5TUyBlN/IAZ0qHng1uVw9vSqWCZxAZXcbgkhwAwBrQncAzymslz5BL8QvKbcVv
         L+6DODA8TSGQnnK4oV9fioc7Ib8dAt6CfJYz5a1p7sT25RphdnZiB/jIT9wJIK8QMd
         moqGrsy8AWobu8dPl20BmSVBSU0UicFXjLLq+ItmOdmoK05h4GUGy1RQTVkSvHltlg
         s18hH9FMn0v8g==
Date:   Tue, 17 Jan 2023 13:59:05 +0200
From:   Leon Romanovsky <leon@kernel.org>
To:     Chaitanya Kulkarni <chaitanyak@nvidia.com>
Cc:     Israel Rukshin <israelr@nvidia.com>,
        Jason Gunthorpe <jgg@nvidia.com>,
        Bryan Tan <bryantan@vmware.com>,
        Christoph Hellwig <hch@lst.de>,
        Eric Dumazet <edumazet@google.com>,
        Jakub Kicinski <kuba@kernel.org>, Jens Axboe <axboe@fb.com>,
        Keith Busch <kbusch@kernel.org>,
        "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
        "linux-nvme@lists.infradead.org" <linux-nvme@lists.infradead.org>,
        "linux-rdma@vger.kernel.org" <linux-rdma@vger.kernel.org>,
        "linux-trace-kernel@vger.kernel.org" 
        <linux-trace-kernel@vger.kernel.org>,
        Masami Hiramatsu <mhiramat@kernel.org>,
        Max Gurtovoy <mgurtovoy@nvidia.com>,
        "netdev@vger.kernel.org" <netdev@vger.kernel.org>,
        Paolo Abeni <pabeni@redhat.com>,
        Saeed Mahameed <saeedm@nvidia.com>,
        Sagi Grimberg <sagi@grimberg.me>,
        Selvin Xavier <selvin.xavier@broadcom.com>,
        Steven Rostedt <rostedt@goodmis.org>,
        Vishnu Dasa <vdasa@vmware.com>,
        Yishai Hadas <yishaih@nvidia.com>
Subject: Re: [PATCH rdma-next 12/13] nvme: Add crypto profile at nvme
 controller
Message-ID: <Y8aNiUWu9AufXkou@unreal>
References: <cover.1673873422.git.leon@kernel.org>
 <efc8dc5952baa096a14db1761f84a5ab2e76654a.1673873422.git.leon@kernel.org>
 <52fef455-fd7a-225f-c03f-b214907ce03b@nvidia.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <52fef455-fd7a-225f-c03f-b214907ce03b@nvidia.com>
Precedence: bulk
List-ID: <netdev.vger.kernel.org>
X-Mailing-List: netdev@vger.kernel.org

On Tue, Jan 17, 2023 at 12:31:22AM +0000, Chaitanya Kulkarni wrote:
> > diff --git a/drivers/nvme/host/core.c b/drivers/nvme/host/core.c
> > index 51a9880db6ce..f09e4e0216b3 100644
> > --- a/drivers/nvme/host/core.c
> > +++ b/drivers/nvme/host/core.c
> > @@ -1928,6 +1928,9 @@ static void nvme_update_disk_info(struct gendisk *disk,
> >   			capacity = 0;
> >   	}
> >   
> > +	if (ctrl->crypto_enable)
> > +		blk_crypto_register(&ctrl->crypto_profile, disk->queue);
> > +
> >   	set_capacity_and_notify(disk, capacity);
> >   
> >   	nvme_config_discard(disk, ns);
> > diff --git a/drivers/nvme/host/nvme.h b/drivers/nvme/host/nvme.h
> > index 424c8a467a0c..591380f53744 100644
> > --- a/drivers/nvme/host/nvme.h
> > +++ b/drivers/nvme/host/nvme.h
> > @@ -16,6 +16,7 @@
> >   #include <linux/rcupdate.h>
> >   #include <linux/wait.h>
> >   #include <linux/t10-pi.h>
> > +#include <linux/blk-crypto-profile.h>
> >   
> >   #include <trace/events/block.h>
> >   
> > @@ -374,6 +375,9 @@ struct nvme_ctrl {
> >   
> >   	enum nvme_ctrl_type cntrltype;
> >   	enum nvme_dctype dctype;
> > +
> > +	bool crypto_enable;
> 
> why not decalre crypto_profile a pointer, allocate that at init
> controller and NULL check against that pointer instead of using
> an extra variable crypto_enable ?

The embedded struct allows us to rely on container_of() macro.

   377 static int nvme_rdma_crypto_keyslot_program(struct blk_crypto_profile *profile,
   378                                             const struct blk_crypto_key *key,
   379                                             unsigned int slot)
   380 {
   381         struct nvme_ctrl *nctrl =
   382                 container_of(profile, struct nvme_ctrl, crypto_profile);

> 
> e.g. :-
> 
> 	if (ctrl->crypto_profile)
> 		blk_crypto_register(ctrl->crypto_profile, disk->queue);
> 
> > +	struct blk_crypto_profile crypto_profile;
> 
> you are increasing the size of struct nvme_ctrl unconditionally,
> why not guard above with CONFIG_BLK_INLINE_ENCRYPTION ?

We can do it.

Thanks

> 
> -ck
> 

From mboxrd@z Thu Jan  1 00:00:00 1970
Return-Path: <netdev-owner@vger.kernel.org>
X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on
	aws-us-west-2-korg-lkml-1.web.codeaurora.org
Received: from vger.kernel.org (vger.kernel.org [23.128.96.18])
	by smtp.lore.kernel.org (Postfix) with ESMTP id 2978AC678D4
	for <netdev@archiver.kernel.org>; Tue, 17 Jan 2023 12:00:19 +0000 (UTC)
Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand
        id S236605AbjAQMAQ (ORCPT <rfc822;netdev@archiver.kernel.org>);
        Tue, 17 Jan 2023 07:00:16 -0500
Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39022 "EHLO
        lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org
        with ESMTP id S236593AbjAQMAB (ORCPT
        <rfc822;netdev@vger.kernel.org>); Tue, 17 Jan 2023 07:00:01 -0500
Received: from ams.source.kernel.org (ams.source.kernel.org [IPv6:2604:1380:4601:e00::1])
        by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 9C4EA36FD9;
        Tue, 17 Jan 2023 03:59:57 -0800 (PST)
Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140])
        (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
        (No client certificate requested)
        by ams.source.kernel.org (Postfix) with ESMTPS id 16A12B81187;
        Tue, 17 Jan 2023 11:59:56 +0000 (UTC)
Received: by smtp.kernel.org (Postfix) with ESMTPSA id 5B004C433D2;
        Tue, 17 Jan 2023 11:59:54 +0000 (UTC)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org;
        s=k20201202; t=1673956794;
        bh=aTQYYKmLmNf07nxBHwjaA+StqUovg0SS7JYEnCH+EZU=;
        h=Date:From:To:Cc:Subject:References:In-Reply-To:From;
        b=JcktzRZbthNjyz+5qDuk41oZMxJYCCPDkgEXy01vFYTwlnZjxjziBy/fjRBFrmcSa
         E+PMfnpmsxQOvk4Nnoif+2kebtJFzsk6MLmCF0naHyrUxRwLUY0egDjGMueOewYX9U
         jY95uj2yF2tCHsBQgxOnBrKJInMJKFTeLLsroP9Qz+Zmb5S5ohasJ2sIlHVuXyrdKU
         +JQlAVyRuTN0m1l13Oc+TNnPPuP7Sr59v4b9nO4VxcqClgsxIrXN9pyrNM8z/w85BT
         oBD2AKo9teesJp1MkdO9eUCNiMo/mj0umhmkVIhKlpMUQYQSsiQ5U2bWLoyBYl3VOM
         zsK3K8o3KkQoA==
Date:   Tue, 17 Jan 2023 13:59:51 +0200
From:   Leon Romanovsky <leon@kernel.org>
To:     Chaitanya Kulkarni <chaitanyak@nvidia.com>
Cc:     Jason Gunthorpe <jgg@nvidia.com>,
        Israel Rukshin <israelr@nvidia.com>,
        Bryan Tan <bryantan@vmware.com>,
        Christoph Hellwig <hch@lst.de>,
        Eric Dumazet <edumazet@google.com>,
        Jakub Kicinski <kuba@kernel.org>, Jens Axboe <axboe@fb.com>,
        Keith Busch <kbusch@kernel.org>,
        "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
        "linux-nvme@lists.infradead.org" <linux-nvme@lists.infradead.org>,
        "linux-rdma@vger.kernel.org" <linux-rdma@vger.kernel.org>,
        "linux-trace-kernel@vger.kernel.org" 
        <linux-trace-kernel@vger.kernel.org>,
        Masami Hiramatsu <mhiramat@kernel.org>,
        Max Gurtovoy <mgurtovoy@nvidia.com>,
        "netdev@vger.kernel.org" <netdev@vger.kernel.org>,
        Paolo Abeni <pabeni@redhat.com>,
        Saeed Mahameed <saeedm@nvidia.com>,
        Sagi Grimberg <sagi@grimberg.me>,
        Selvin Xavier <selvin.xavier@broadcom.com>,
        Steven Rostedt <rostedt@goodmis.org>,
        Vishnu Dasa <vdasa@vmware.com>,
        Yishai Hadas <yishaih@nvidia.com>
Subject: Re: [PATCH rdma-next 11/13] nvme: Introduce a local variable
Message-ID: <Y8aNt9YB9irSVXNo@unreal>
References: <cover.1673873422.git.leon@kernel.org>
 <cf5bc542e014f465f7ae443e52e70def2993aef1.1673873422.git.leon@kernel.org>
 <c095c9a9-0e77-c8ab-d34e-f4ab42b11938@nvidia.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <c095c9a9-0e77-c8ab-d34e-f4ab42b11938@nvidia.com>
Precedence: bulk
List-ID: <netdev.vger.kernel.org>
X-Mailing-List: netdev@vger.kernel.org

On Tue, Jan 17, 2023 at 12:32:59AM +0000, Chaitanya Kulkarni wrote:
> On 1/16/23 05:05, Leon Romanovsky wrote:
> > From: Israel Rukshin <israelr@nvidia.com>
> > 
> > The patch doesn't change any logic.
> > 
> > Signed-off-by: Israel Rukshin <israelr@nvidia.com>
> > Signed-off-by: Leon Romanovsky <leon@kernel.org>
> > ---
> >   drivers/nvme/host/core.c | 7 ++++---
> >   1 file changed, 4 insertions(+), 3 deletions(-)
> > 
> > diff --git a/drivers/nvme/host/core.c b/drivers/nvme/host/core.c
> > index 7be562a4e1aa..51a9880db6ce 100644
> > --- a/drivers/nvme/host/core.c
> > +++ b/drivers/nvme/host/core.c
> > @@ -1870,6 +1870,7 @@ static void nvme_update_disk_info(struct gendisk *disk,
> >   	sector_t capacity = nvme_lba_to_sect(ns, le64_to_cpu(id->nsze));
> >   	unsigned short bs = 1 << ns->lba_shift;
> >   	u32 atomic_bs, phys_bs, io_opt = 0;
> > +	struct nvme_ctrl *ctrl = ns->ctrl;
> >   
> 
> I don't think this patch is needed, existing code is more readable and
> gives much clarity that we are accessing ctrl from namespace since we
> are in nvme_update_disk_info() which is namespace based.

I don't have strong opinion here.

Thanks

> 
> -ck
> 

From mboxrd@z Thu Jan  1 00:00:00 1970
Return-Path: <netdev-owner@vger.kernel.org>
X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on
	aws-us-west-2-korg-lkml-1.web.codeaurora.org
Received: from vger.kernel.org (vger.kernel.org [23.128.96.18])
	by smtp.lore.kernel.org (Postfix) with ESMTP id 245FBC3DA78
	for <netdev@archiver.kernel.org>; Tue, 17 Jan 2023 12:04:05 +0000 (UTC)
Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand
        id S236618AbjAQMEC (ORCPT <rfc822;netdev@archiver.kernel.org>);
        Tue, 17 Jan 2023 07:04:02 -0500
Received: from lindbergh.monkeyblade.net ([23.128.96.19]:41350 "EHLO
        lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org
        with ESMTP id S236976AbjAQMD0 (ORCPT
        <rfc822;netdev@vger.kernel.org>); Tue, 17 Jan 2023 07:03:26 -0500
Received: from sin.source.kernel.org (sin.source.kernel.org [145.40.73.55])
        by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 6F66936FCD;
        Tue, 17 Jan 2023 04:03:16 -0800 (PST)
Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140])
        (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
        (No client certificate requested)
        by sin.source.kernel.org (Postfix) with ESMTPS id B59E3CE1688;
        Tue, 17 Jan 2023 12:03:14 +0000 (UTC)
Received: by smtp.kernel.org (Postfix) with ESMTPSA id 7E677C433D2;
        Tue, 17 Jan 2023 12:03:12 +0000 (UTC)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org;
        s=k20201202; t=1673956993;
        bh=93tzp+RSOFf3+k4VJuc6vDivUrGEcUxrSIufHOvJ2kk=;
        h=Date:From:To:Cc:Subject:References:In-Reply-To:From;
        b=of5w9Xji5VNt/i6LxdxPZcYFz/Uw2hYN4b1Dd1rF2g8GNW0eiNQDJ+Y8y1aEEeE/a
         NYxCsYslSsLRlIH33cntH+tV20HVHqrJQC/0Q1VBorj/mqOH7yHvAPVNLBD4TnP8bX
         kSdSUi++Jh+XfjjAEcozg0vsEg+6lVk8BcOg+BxODigDLV6JzXKuIWJpSrNS1ZvgsF
         0kfgA+p4Oo3b3+EprhLqWQjlKdQwvR4bGaaB6nGu9tfws4+mG4rXxdh75g16JMVJtM
         FEt2NMAXwJ6dD8alBSLP5SfCNjN1jeHTEg7dJD/Bz3ND9ClbQIwMvdb/UolK6yOVRb
         LFVB1lvT2qqdw==
Date:   Tue, 17 Jan 2023 14:03:07 +0200
From:   Leon Romanovsky <leon@kernel.org>
To:     Jason Gunthorpe <jgg@nvidia.com>
Cc:     Israel Rukshin <israelr@nvidia.com>,
        Bryan Tan <bryantan@vmware.com>,
        Christoph Hellwig <hch@lst.de>,
        Eric Dumazet <edumazet@google.com>,
        Jakub Kicinski <kuba@kernel.org>, Jens Axboe <axboe@fb.com>,
        Keith Busch <kbusch@kernel.org>, linux-kernel@vger.kernel.org,
        linux-nvme@lists.infradead.org, linux-rdma@vger.kernel.org,
        linux-trace-kernel@vger.kernel.org,
        Masami Hiramatsu <mhiramat@kernel.org>,
        Max Gurtovoy <mgurtovoy@nvidia.com>, netdev@vger.kernel.org,
        Paolo Abeni <pabeni@redhat.com>,
        Saeed Mahameed <saeedm@nvidia.com>,
        Sagi Grimberg <sagi@grimberg.me>,
        Selvin Xavier <selvin.xavier@broadcom.com>,
        Steven Rostedt <rostedt@goodmis.org>,
        Vishnu Dasa <vdasa@vmware.com>,
        Yishai Hadas <yishaih@nvidia.com>
Subject: Re: [PATCH rdma-next 03/13] RDMA: Split kernel-only create QP flags
 from uverbs create QP flags
Message-ID: <Y8aOe68Q49lvsjv8@unreal>
References: <cover.1673873422.git.leon@kernel.org>
 <6e46859a58645d9f16a63ff76592487aabc9971d.1673873422.git.leon@kernel.org>
 <Y8WL2gQqqeZdMvr6@nvidia.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <Y8WL2gQqqeZdMvr6@nvidia.com>
Precedence: bulk
List-ID: <netdev.vger.kernel.org>
X-Mailing-List: netdev@vger.kernel.org

On Mon, Jan 16, 2023 at 01:39:38PM -0400, Jason Gunthorpe wrote:
> On Mon, Jan 16, 2023 at 03:05:50PM +0200, Leon Romanovsky wrote:
> 
> > diff --git a/drivers/infiniband/hw/mlx4/mlx4_ib.h b/drivers/infiniband/hw/mlx4/mlx4_ib.h
> > index 17fee1e73a45..c553bf0eb257 100644
> > --- a/drivers/infiniband/hw/mlx4/mlx4_ib.h
> > +++ b/drivers/infiniband/hw/mlx4/mlx4_ib.h
> > @@ -184,7 +184,7 @@ enum mlx4_ib_qp_flags {
> >  	/* Mellanox specific flags start from IB_QP_CREATE_RESERVED_START */
> >  	MLX4_IB_ROCE_V2_GSI_QP = MLX4_IB_QP_CREATE_ROCE_V2_GSI,
> >  	MLX4_IB_SRIOV_TUNNEL_QP = 1 << 30,
> > -	MLX4_IB_SRIOV_SQP = 1 << 31,
> > +	MLX4_IB_SRIOV_SQP = 1ULL << 31,
> >  };
> 
> These should be moved to a uapi if we are saying they are userspace
> available
> 
> But I'm not sure they are?

I don't think so.

> 
> 
> > diff --git a/include/rdma/ib_verbs.h b/include/rdma/ib_verbs.h
> > index 949cf4ffc536..cc2ddd4e6c12 100644
> > --- a/include/rdma/ib_verbs.h
> > +++ b/include/rdma/ib_verbs.h
> > @@ -1140,16 +1140,15 @@ enum ib_qp_type {
> >  	IB_QPT_RESERVED10,
> >  };
> >  
> > +/*
> > + * bits 0, 5, 6 and 7 may be set by old kernels and should not be used.
> > + */
> 
> This is backwards "bits 0 5 6 7 were understood by older kernels and
> should not be used"

will change

> 
> >  enum ib_qp_create_flags {
> > -	IB_QP_CREATE_IPOIB_UD_LSO		= 1 << 0,
> >  	IB_QP_CREATE_BLOCK_MULTICAST_LOOPBACK	=
> >  		IB_UVERBS_QP_CREATE_BLOCK_MULTICAST_LOOPBACK,
> >  	IB_QP_CREATE_CROSS_CHANNEL              = 1 << 2,
> >  	IB_QP_CREATE_MANAGED_SEND               = 1 << 3,
> >  	IB_QP_CREATE_MANAGED_RECV               = 1 << 4,
> > -	IB_QP_CREATE_NETIF_QP			= 1 << 5,
> > -	IB_QP_CREATE_INTEGRITY_EN		= 1 << 6,
> > -	IB_QP_CREATE_NETDEV_USE			= 1 << 7,
> >  	IB_QP_CREATE_SCATTER_FCS		=
> >  		IB_UVERBS_QP_CREATE_SCATTER_FCS,
> >  	IB_QP_CREATE_CVLAN_STRIPPING		=
> > @@ -1159,7 +1158,18 @@ enum ib_qp_create_flags {
> >  		IB_UVERBS_QP_CREATE_PCI_WRITE_END_PADDING,
> >  	/* reserve bits 26-31 for low level drivers' internal use */
> >  	IB_QP_CREATE_RESERVED_START		= 1 << 26,
> > -	IB_QP_CREATE_RESERVED_END		= 1 << 31,
> > +	IB_QP_CREATE_RESERVED_END		= 1ULL << 31,
> 
> And these should be shifted to the uapi header

No problem.

> 
> Jason

From mboxrd@z Thu Jan  1 00:00:00 1970
Return-Path: <netdev-owner@vger.kernel.org>
X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on
	aws-us-west-2-korg-lkml-1.web.codeaurora.org
Received: from vger.kernel.org (vger.kernel.org [23.128.96.18])
	by smtp.lore.kernel.org (Postfix) with ESMTP id 95C23C3DA78
	for <netdev@archiver.kernel.org>; Tue, 17 Jan 2023 13:50:08 +0000 (UTC)
Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand
        id S230080AbjAQNuG (ORCPT <rfc822;netdev@archiver.kernel.org>);
        Tue, 17 Jan 2023 08:50:06 -0500
Received: from lindbergh.monkeyblade.net ([23.128.96.19]:46728 "EHLO
        lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org
        with ESMTP id S230484AbjAQNtt (ORCPT
        <rfc822;netdev@vger.kernel.org>); Tue, 17 Jan 2023 08:49:49 -0500
Received: from NAM10-BN7-obe.outbound.protection.outlook.com (mail-bn7nam10on2071.outbound.protection.outlook.com [40.107.92.71])
        by lindbergh.monkeyblade.net (Postfix) with ESMTPS id C8C5B3BDAD;
        Tue, 17 Jan 2023 05:49:33 -0800 (PST)
ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none;
 b=a8fJB4CBARFXOSWe1+koMrWCvnVz8ZM6lxdtAq8BlxnOsOtE8zIvvMUXRJWfquwNnyvG4zqTU/9D91/kvBrZr1oXp6RJ9V22bExgWD4cQmfG78wjBNV3wVI3bySMXwQuLKUtdXuyEmn9YoKzj0Dd7Hb9UBCAG4oaDspOsbeolHRvaFfefJ2n3GL0mKM2qdfRqOx39aIN0hly5Ch/wHS6OeXZyH0Ayu+SY3rmpIytkTBMGyEkYruQCzdqDwqDC9GEB4hMIFS1SMI6ewpGVynBao3rd7ahp6B6sOMl5SDqmteusAyfPH4Pylj1sD00joTV1RGfCw83R3oU1tJEpAb4tg==
ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com;
 s=arcselector9901;
 h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1;
 bh=qQmchS+i21MjtfvKSkvJ9eVBqvTCctjr7pttEfd51GI=;
 b=dSrUKNvskYfyH/XebEC9x+3auE24vzUh0HGlYbhGhfZ6t7750EbTiVFjknbiaFgyl3AOqH0oYiSswjTczURiOGjl7sJu5deOMsUsw0uaR050qX7WMTZRsM2VL07IeXlpdGHdy52NSr6s8WjYdhVlGmn0QwVRdIMz51UyUIDAUAqsoRlPNZiUAidXY4R/0Q4CmeVtImrWUD8jnM9Tb2m6t6pIEYCFIJEBAJxJ7Fhh2/2RdpSpF2Dt3VxKeGBHsfB3RmB2xe3GKnJycarMTwJXmmBWJuOBJ7RoFOngKOC+Q9MERvAT+19c8qXA/9a6ShxluypTJa4hS8OQtjYczTgKDQ==
ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass
 smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com;
 dkim=pass header.d=nvidia.com; arc=none
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com;
 s=selector2;
 h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck;
 bh=qQmchS+i21MjtfvKSkvJ9eVBqvTCctjr7pttEfd51GI=;
 b=PNEqn8o2Nwxgno7ZaIIdb9x3iyCtgKOM7HGTsJ+pcsRE0Lml5I6lW3Jqp5avtNTe/s6fx+zCvyTyGnE5eEFHQlnaFyyIbUoD499QU3z+HLpnF/KEYSp5NAY6DRdwbPFds6l4NpQc4f20yYXDRB1gYJFRIl3vHInCRegZ/rFIJS89GW3AdS9mkn4AlIvAaptYXhuNyZxjoyyBPmEWf33G0Pp4KnkgAMpADM44m0HtsRnIbxpVmlVMkOWQbrkTt/o32efr7LpJkjBo6HTjG6qrdkPgHA+DXuFBjInRAGc+wDCKkcAG1QbaE51H/G9fdoZtMXEKbfq5a6nUMp+ABpJOPA==
Authentication-Results: dkim=none (message not signed)
 header.d=none;dmarc=none action=none header.from=nvidia.com;
Received: from LV2PR12MB5869.namprd12.prod.outlook.com (2603:10b6:408:176::16)
 by BY5PR12MB4034.namprd12.prod.outlook.com (2603:10b6:a03:205::9) with
 Microsoft SMTP Server (version=TLS1_2,
 cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5986.23; Tue, 17 Jan
 2023 13:49:30 +0000
Received: from LV2PR12MB5869.namprd12.prod.outlook.com
 ([fe80::f8b0:df13:5f8d:12a]) by LV2PR12MB5869.namprd12.prod.outlook.com
 ([fe80::f8b0:df13:5f8d:12a%9]) with mapi id 15.20.6002.013; Tue, 17 Jan 2023
 13:49:30 +0000
Date:   Tue, 17 Jan 2023 09:49:28 -0400
From:   Jason Gunthorpe <jgg@nvidia.com>
To:     Leon Romanovsky <leon@kernel.org>
Cc:     Israel Rukshin <israelr@nvidia.com>,
        Bryan Tan <bryantan@vmware.com>,
        Christoph Hellwig <hch@lst.de>,
        Eric Dumazet <edumazet@google.com>,
        Jakub Kicinski <kuba@kernel.org>, Jens Axboe <axboe@fb.com>,
        Keith Busch <kbusch@kernel.org>, linux-kernel@vger.kernel.org,
        linux-nvme@lists.infradead.org, linux-rdma@vger.kernel.org,
        linux-trace-kernel@vger.kernel.org,
        Masami Hiramatsu <mhiramat@kernel.org>,
        Max Gurtovoy <mgurtovoy@nvidia.com>, netdev@vger.kernel.org,
        Paolo Abeni <pabeni@redhat.com>,
        Saeed Mahameed <saeedm@nvidia.com>,
        Sagi Grimberg <sagi@grimberg.me>,
        Selvin Xavier <selvin.xavier@broadcom.com>,
        Steven Rostedt <rostedt@goodmis.org>,
        Vishnu Dasa <vdasa@vmware.com>,
        Yishai Hadas <yishaih@nvidia.com>
Subject: Re: [PATCH rdma-next 03/13] RDMA: Split kernel-only create QP flags
 from uverbs create QP flags
Message-ID: <Y8anaBBZDOGF471q@nvidia.com>
References: <cover.1673873422.git.leon@kernel.org>
 <6e46859a58645d9f16a63ff76592487aabc9971d.1673873422.git.leon@kernel.org>
 <Y8WL2gQqqeZdMvr6@nvidia.com>
 <Y8aOe68Q49lvsjv8@unreal>
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <Y8aOe68Q49lvsjv8@unreal>
X-ClientProxiedBy: MN2PR17CA0007.namprd17.prod.outlook.com
 (2603:10b6:208:15e::20) To LV2PR12MB5869.namprd12.prod.outlook.com
 (2603:10b6:408:176::16)
MIME-Version: 1.0
X-MS-PublicTrafficType: Email
X-MS-TrafficTypeDiagnostic: LV2PR12MB5869:EE_|BY5PR12MB4034:EE_
X-MS-Office365-Filtering-Correlation-Id: e0f35378-1365-43a5-b404-08daf891a7de
X-MS-Exchange-SenderADCheck: 1
X-MS-Exchange-AntiSpam-Relay: 0
X-Microsoft-Antispam: BCL:0;
X-Microsoft-Antispam-Message-Info:  
X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:LV2PR12MB5869.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230022)(4636009)(376002)(346002)(136003)(366004)(39860400002)(396003)(451199015)(38100700002)(86362001)(7416002)(8936002)(4744005)(4326008)(5660300002)(2906002)(8676002)(6916009)(66946007)(66476007)(66556008)(41300700001)(2616005)(186003)(6506007)(107886003)(26005)(6512007)(316002)(54906003)(478600001)(6486002)(36756003);DIR:OUT;SFP:1101;
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?dUK4Bu/PY52F9WB5T0mxDbblSL3zcucKqQD/o1XR4hdvBFkTdi5vhTBObq/Z?=
 =?us-ascii?Q?Lx2oP0fwNvOr+55yyOz0oirJbhS/ehF1cDUDKuduEzmCfUw4gT2dKoQ5EqwU?=
 =?us-ascii?Q?vQPfw9oW/Ii5QDlRdIFqigV+vDMpN4YN+F3GydOhyquU4tCKaouynBQ2b+o4?=
 =?us-ascii?Q?VPXDrPECzA/Mo1Ucwx5/jU7uyGqyN+CeTiP4Hjv3l4k6RwvaZZ2W+rGaSmnr?=
 =?us-ascii?Q?KHuLs6lhbElTRSdGTPp+xwqym6ntZHzwInqC37f4nYlNhg6rj3ZOtNlMm1Da?=
 =?us-ascii?Q?t52MoXjJIUQkqgdtC+IKTHeDlN5GntfK7hogN36PEpqS8T3Lm5N9KfBUmvCR?=
 =?us-ascii?Q?mj98TRr1EKKKocm879momuQETr4TQ5DwqMueetrC2/QFX8owTZFi5znnKS1N?=
 =?us-ascii?Q?s0xxJNWbKpjZdaiOLF5lQvL48ITb0bNemRO7BMMLeR8WtK4LHGwiSM2IbjVh?=
 =?us-ascii?Q?h3ww+h5C+zGC7X+Eg6Jg/6aI4rwIuVSJ7ZSdl8ry975MfhsDRA4noAiD/dsA?=
 =?us-ascii?Q?9+puLwHmChCp61zaBsjbH7lVqwTIQTeQ6/MTl72OH5K0+WIa/XHXKDtXGf6Z?=
 =?us-ascii?Q?9AolGusYNRlyGNR2rAXXPEaCOwcBpG/CVqtGlCfh3l4dhrFkOHhDoOy+hrWU?=
 =?us-ascii?Q?ZhxZtt5a4BW+fIlIt8eYQ7sg58krb3oQBgiI+NKDIG262H81l/wSLjTepKen?=
 =?us-ascii?Q?VFZCQgbdKi8laJnUsvKc38axaRIOUJf7er58UKhZxEMGHhaPVwWRQc3FYZij?=
 =?us-ascii?Q?qM1km6ppZ+EqlrPsy3HtyPVdIUKU4zkQfqxCnb+zOwdgKmheknBu2LwBnmqo?=
 =?us-ascii?Q?20Zeux89PKfPwOs1FuoyJvQdSkvJAenOIYHufws9Hpjy0iLv3y54o52f2su1?=
 =?us-ascii?Q?5BTjhvJ5MIeD1iDjbPJSTCFuS7X0awDWFRu0mawLCVggnWiGDkfVOxstsJvj?=
 =?us-ascii?Q?j858eyaq7tRo/kqjG9jC4dchUccqrNlATCvY5n6jjXo+fn7yzlQZZ3+rIPYX?=
 =?us-ascii?Q?iAhzg/gtN6/kWxwEBqdNkOuromirLNWWjEwPjhykvsR5g0a8/iV6llXI8K1t?=
 =?us-ascii?Q?K9ma4/eXItZ+6TyNgGNQCa3NAV7B9syrY8OxvA4URTBHpCJzjz8VK1uPlvbs?=
 =?us-ascii?Q?cReulMLuhteSrGowpOpaflR2F1vkJUWuHYwt3wQ2Vuf1T9UaspfEiFJ5BN7/?=
 =?us-ascii?Q?ZJafdQnNQgAFRUDHwRhaQLXVkLH8Dh/c3synp/o2e1GsFu6YaVNgspUH6uy5?=
 =?us-ascii?Q?F9dsUmDh1L9z2CeCFkwjlpSBNEZNS9RsCO9SUa83AvLSkhFb+rHz/YAOfRuN?=
 =?us-ascii?Q?1zgNwDlXYyPOYcZMtlbwGM+uKddWbsHtee4a5gV35m6N/Z9wPHx9HAMaVihs?=
 =?us-ascii?Q?PjM+Wsfk849jozGSjJhUpHle+lrGrsCSmzXNTgAI0m5oYOQqZbJocthNycvX?=
 =?us-ascii?Q?8Vn9vWSIw4UvazO7qd/gME6kUFXUcAVgRqm+sbmelU30DeBWsTezCM4rimyi?=
 =?us-ascii?Q?xjL61E8yjc9NUhNDoRibbFgaW+maHI2wn0NvRjk/HQEKteIh0KtsQEg6pz1p?=
 =?us-ascii?Q?Jrj+7Kg3/87i7I2sTe1X5SKeohm+uthjCdC46or0?=
X-OriginatorOrg: Nvidia.com
X-MS-Exchange-CrossTenant-Network-Message-Id: e0f35378-1365-43a5-b404-08daf891a7de
X-MS-Exchange-CrossTenant-AuthSource: LV2PR12MB5869.namprd12.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 17 Jan 2023 13:49:29.8713
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: UtQWck1Yc0WcJxT9OqncArAw4c4CHWTlqacY5zMEiuDV1TBknzwPffGbjPZjGUZi
X-MS-Exchange-Transport-CrossTenantHeadersStamped: BY5PR12MB4034
Precedence: bulk
List-ID: <netdev.vger.kernel.org>
X-Mailing-List: netdev@vger.kernel.org

On Tue, Jan 17, 2023 at 02:03:07PM +0200, Leon Romanovsky wrote:
> On Mon, Jan 16, 2023 at 01:39:38PM -0400, Jason Gunthorpe wrote:
> > On Mon, Jan 16, 2023 at 03:05:50PM +0200, Leon Romanovsky wrote:
> > 
> > > diff --git a/drivers/infiniband/hw/mlx4/mlx4_ib.h b/drivers/infiniband/hw/mlx4/mlx4_ib.h
> > > index 17fee1e73a45..c553bf0eb257 100644
> > > --- a/drivers/infiniband/hw/mlx4/mlx4_ib.h
> > > +++ b/drivers/infiniband/hw/mlx4/mlx4_ib.h
> > > @@ -184,7 +184,7 @@ enum mlx4_ib_qp_flags {
> > >  	/* Mellanox specific flags start from IB_QP_CREATE_RESERVED_START */
> > >  	MLX4_IB_ROCE_V2_GSI_QP = MLX4_IB_QP_CREATE_ROCE_V2_GSI,
> > >  	MLX4_IB_SRIOV_TUNNEL_QP = 1 << 30,
> > > -	MLX4_IB_SRIOV_SQP = 1 << 31,
> > > +	MLX4_IB_SRIOV_SQP = 1ULL << 31,
> > >  };
> > 
> > These should be moved to a uapi if we are saying they are userspace
> > available
> > 
> > But I'm not sure they are?
> 
> I don't think so.

Then they should be > 32 bits right?

Jason

From mboxrd@z Thu Jan  1 00:00:00 1970
Return-Path: <netdev-owner@vger.kernel.org>
X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on
	aws-us-west-2-korg-lkml-1.web.codeaurora.org
Received: from vger.kernel.org (vger.kernel.org [23.128.96.18])
	by smtp.lore.kernel.org (Postfix) with ESMTP id 8DA70C63797
	for <netdev@archiver.kernel.org>; Tue, 17 Jan 2023 14:14:50 +0000 (UTC)
Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand
        id S231878AbjAQOOr (ORCPT <rfc822;netdev@archiver.kernel.org>);
        Tue, 17 Jan 2023 09:14:47 -0500
Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39350 "EHLO
        lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org
        with ESMTP id S232034AbjAQOOe (ORCPT
        <rfc822;netdev@vger.kernel.org>); Tue, 17 Jan 2023 09:14:34 -0500
Received: from sin.source.kernel.org (sin.source.kernel.org [IPv6:2604:1380:40e1:4800::1])
        by lindbergh.monkeyblade.net (Postfix) with ESMTPS id ABFA13CE0D;
        Tue, 17 Jan 2023 06:14:29 -0800 (PST)
Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140])
        (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
        (No client certificate requested)
        by sin.source.kernel.org (Postfix) with ESMTPS id 01F3FCE1393;
        Tue, 17 Jan 2023 14:14:28 +0000 (UTC)
Received: by smtp.kernel.org (Postfix) with ESMTPSA id D2B4BC433D2;
        Tue, 17 Jan 2023 14:14:25 +0000 (UTC)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org;
        s=k20201202; t=1673964866;
        bh=oO6yTThzMkNKYiBtaWdm5Lzb8UxNpLOg1Jp0NdlEpAY=;
        h=Date:From:To:Cc:Subject:References:In-Reply-To:From;
        b=H7za2Vaq0l6xcNEV0SarMt/DnMBLCzeBWWHimgfSvI2ImwcbagaDOKiGoYjIiVBn2
         Z5/XYNfZU0JPPb5lQk9Fyqx3urt7Z2rVUrJIrDl3SwLQ675XBE66cmf/6EaOs2Bfwy
         YDS/ocT+Z2a0yNJKgZyHAD1NwgcBptZIwjtCrGqY0QlxkBtjqN+NVcWibOoDKYrV21
         HO3Fy2Oxv8suZVnjpwcLAzWp+8wz57WX6dw1iyjFG+85UVZpuAi+TwcjOte50kjEDw
         I1qmrajJrx7UtP5YnXQFOm2l/gT5fh0RLuTYHRhbhKshd1vTmU43lOWdH+f/4/6aHs
         g78xEoge90KtQ==
Date:   Tue, 17 Jan 2023 16:14:22 +0200
From:   Leon Romanovsky <leon@kernel.org>
To:     Jason Gunthorpe <jgg@nvidia.com>
Cc:     Israel Rukshin <israelr@nvidia.com>,
        Bryan Tan <bryantan@vmware.com>,
        Christoph Hellwig <hch@lst.de>,
        Eric Dumazet <edumazet@google.com>,
        Jakub Kicinski <kuba@kernel.org>, Jens Axboe <axboe@fb.com>,
        Keith Busch <kbusch@kernel.org>, linux-kernel@vger.kernel.org,
        linux-nvme@lists.infradead.org, linux-rdma@vger.kernel.org,
        linux-trace-kernel@vger.kernel.org,
        Masami Hiramatsu <mhiramat@kernel.org>,
        Max Gurtovoy <mgurtovoy@nvidia.com>, netdev@vger.kernel.org,
        Paolo Abeni <pabeni@redhat.com>,
        Saeed Mahameed <saeedm@nvidia.com>,
        Sagi Grimberg <sagi@grimberg.me>,
        Selvin Xavier <selvin.xavier@broadcom.com>,
        Steven Rostedt <rostedt@goodmis.org>,
        Vishnu Dasa <vdasa@vmware.com>,
        Yishai Hadas <yishaih@nvidia.com>
Subject: Re: [PATCH rdma-next 03/13] RDMA: Split kernel-only create QP flags
 from uverbs create QP flags
Message-ID: <Y8atPjQ1x75tBdib@unreal>
References: <cover.1673873422.git.leon@kernel.org>
 <6e46859a58645d9f16a63ff76592487aabc9971d.1673873422.git.leon@kernel.org>
 <Y8WL2gQqqeZdMvr6@nvidia.com>
 <Y8aOe68Q49lvsjv8@unreal>
 <Y8anaBBZDOGF471q@nvidia.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <Y8anaBBZDOGF471q@nvidia.com>
Precedence: bulk
List-ID: <netdev.vger.kernel.org>
X-Mailing-List: netdev@vger.kernel.org

On Tue, Jan 17, 2023 at 09:49:28AM -0400, Jason Gunthorpe wrote:
> On Tue, Jan 17, 2023 at 02:03:07PM +0200, Leon Romanovsky wrote:
> > On Mon, Jan 16, 2023 at 01:39:38PM -0400, Jason Gunthorpe wrote:
> > > On Mon, Jan 16, 2023 at 03:05:50PM +0200, Leon Romanovsky wrote:
> > > 
> > > > diff --git a/drivers/infiniband/hw/mlx4/mlx4_ib.h b/drivers/infiniband/hw/mlx4/mlx4_ib.h
> > > > index 17fee1e73a45..c553bf0eb257 100644
> > > > --- a/drivers/infiniband/hw/mlx4/mlx4_ib.h
> > > > +++ b/drivers/infiniband/hw/mlx4/mlx4_ib.h
> > > > @@ -184,7 +184,7 @@ enum mlx4_ib_qp_flags {
> > > >  	/* Mellanox specific flags start from IB_QP_CREATE_RESERVED_START */
> > > >  	MLX4_IB_ROCE_V2_GSI_QP = MLX4_IB_QP_CREATE_ROCE_V2_GSI,
> > > >  	MLX4_IB_SRIOV_TUNNEL_QP = 1 << 30,
> > > > -	MLX4_IB_SRIOV_SQP = 1 << 31,
> > > > +	MLX4_IB_SRIOV_SQP = 1ULL << 31,
> > > >  };
> > > 
> > > These should be moved to a uapi if we are saying they are userspace
> > > available
> > > 
> > > But I'm not sure they are?
> > 
> > I don't think so.
> 
> Then they should be > 32 bits right?

Right now, they are in reserved range:
        /* reserve bits 26-31 for low level drivers' internal use */
        IB_QP_CREATE_RESERVED_START             = 1 << 26,
        IB_QP_CREATE_RESERVED_END               = 1ULL << 31,

If we move them to kernel part, we will need to define reserved range
there too. So we just "burn" extra bits just for mlx4, also I don't see
any reason to promote mlx4 bits to be general ones.

Thanks

> 
> Jason

From mboxrd@z Thu Jan  1 00:00:00 1970
Return-Path: <netdev-owner@vger.kernel.org>
X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on
	aws-us-west-2-korg-lkml-1.web.codeaurora.org
Received: from vger.kernel.org (vger.kernel.org [23.128.96.18])
	by smtp.lore.kernel.org (Postfix) with ESMTP id 53FC2C3DA78
	for <netdev@archiver.kernel.org>; Tue, 17 Jan 2023 14:23:05 +0000 (UTC)
Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand
        id S232172AbjAQOXB (ORCPT <rfc822;netdev@archiver.kernel.org>);
        Tue, 17 Jan 2023 09:23:01 -0500
Received: from lindbergh.monkeyblade.net ([23.128.96.19]:44314 "EHLO
        lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org
        with ESMTP id S231749AbjAQOWb (ORCPT
        <rfc822;netdev@vger.kernel.org>); Tue, 17 Jan 2023 09:22:31 -0500
Received: from NAM10-MW2-obe.outbound.protection.outlook.com (mail-mw2nam10on2081.outbound.protection.outlook.com [40.107.94.81])
        by lindbergh.monkeyblade.net (Postfix) with ESMTPS id D57573FF03;
        Tue, 17 Jan 2023 06:21:21 -0800 (PST)
ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none;
 b=ePirdxuVc9ah3EJ+d04a+va8MLpMoJNmco4fluTz4VF/MslH6dmD4kzSlIjFxT8QCJ49VH1pmpWWUyJDvbZfkRAwfUOtjE5/T3ls/gbFQRzuGS9aVYyDDnkeL0qtn+grhMgDVxNGb88n+JgWU1VQ4Jr8D+TXuTd36wnujGV5ZPViiiUbvaofivl/YQewjekvmP2dLXbGi+WBhLeKCqa/4DNtICyQZRQLnO4QylkXZs3Sg/yid6iX7iQUtal5bxiiyOzyp5wDmlWMaK2BPQIuMzfeUFDdfL7KPYAwWneE3YxQy5Dx54YHLaR1InjIs6+dXrr/9eb5ECKfS79jHqlq2g==
ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com;
 s=arcselector9901;
 h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1;
 bh=GzcxGyxBcI6XwuuXf+1G5uFCGWKr0Dmzf11HvWa/5/w=;
 b=KiEQonOtG5N3su9driETicCTPt/3zn5uyymIqtSTrTM5AUqljUOV/arGVFgHFwdlFXBjgqTQVsMlszVd9GHbscRsDB9jWPHFWCLGJ0PLwfSJrJbDBRpczPZhD0TOu4wjJ43XR3QxlxfkDUBdA5vZcyybApz3a8GLUPIk4NQsBvwymnBHQ+h8JO4m1TdtucQCGcAkK4E87on1AjJax59IYYTJH6GZp6Dy0Ia7KZsQMg/WHcLJNlhPsHVQfR7s8ahabqceqByI0o0GTSYbZcx+T9IWf5uQgjp639XqvQfjTkikSRmfOkE47AsI8GEvekoksDvrQuGE1UkiA9Hiaz235w==
ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass
 smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com;
 dkim=pass header.d=nvidia.com; arc=none
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com;
 s=selector2;
 h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck;
 bh=GzcxGyxBcI6XwuuXf+1G5uFCGWKr0Dmzf11HvWa/5/w=;
 b=Ot/jAa486fwo2cswCAnTYcegqnn3BlsBvdm7INiyez/RvCT7U2Akyvfq06F/BbaKSmzQjtKT0S9n4JKh95mvT5rpppwlYPoE0JZiJ+4ogX8V56KATOpMQzxUnieje/8q8NiGYwmqWzZ4NJ4vBLJkrPv2irziqQFG/JWBrUbrkU74vHusQv73tX5GFUi27v5HJX0kChgzZY+1S1jTBfa9bbWENY3RHZe/Oq5eCjLWR/8jPuwJSHiZfWtiA5VE3DuLR+QWCbPRDOMxmmjkV7MJP9oQP+IcvLxWeVxdUGkywcLigXABPwEABo5LItdS2xzRqz23CSG+msCMf0YMycJasw==
Authentication-Results: dkim=none (message not signed)
 header.d=none;dmarc=none action=none header.from=nvidia.com;
Received: from LV2PR12MB5869.namprd12.prod.outlook.com (2603:10b6:408:176::16)
 by BL1PR12MB5174.namprd12.prod.outlook.com (2603:10b6:208:31c::19) with
 Microsoft SMTP Server (version=TLS1_2,
 cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5986.23; Tue, 17 Jan
 2023 14:21:19 +0000
Received: from LV2PR12MB5869.namprd12.prod.outlook.com
 ([fe80::f8b0:df13:5f8d:12a]) by LV2PR12MB5869.namprd12.prod.outlook.com
 ([fe80::f8b0:df13:5f8d:12a%9]) with mapi id 15.20.6002.013; Tue, 17 Jan 2023
 14:21:19 +0000
Date:   Tue, 17 Jan 2023 10:21:18 -0400
From:   Jason Gunthorpe <jgg@nvidia.com>
To:     Leon Romanovsky <leon@kernel.org>
Cc:     Israel Rukshin <israelr@nvidia.com>,
        Bryan Tan <bryantan@vmware.com>,
        Christoph Hellwig <hch@lst.de>,
        Eric Dumazet <edumazet@google.com>,
        Jakub Kicinski <kuba@kernel.org>, Jens Axboe <axboe@fb.com>,
        Keith Busch <kbusch@kernel.org>, linux-kernel@vger.kernel.org,
        linux-nvme@lists.infradead.org, linux-rdma@vger.kernel.org,
        linux-trace-kernel@vger.kernel.org,
        Masami Hiramatsu <mhiramat@kernel.org>,
        Max Gurtovoy <mgurtovoy@nvidia.com>, netdev@vger.kernel.org,
        Paolo Abeni <pabeni@redhat.com>,
        Saeed Mahameed <saeedm@nvidia.com>,
        Sagi Grimberg <sagi@grimberg.me>,
        Selvin Xavier <selvin.xavier@broadcom.com>,
        Steven Rostedt <rostedt@goodmis.org>,
        Vishnu Dasa <vdasa@vmware.com>,
        Yishai Hadas <yishaih@nvidia.com>
Subject: Re: [PATCH rdma-next 03/13] RDMA: Split kernel-only create QP flags
 from uverbs create QP flags
Message-ID: <Y8au3ni8NVBPI5hu@nvidia.com>
References: <cover.1673873422.git.leon@kernel.org>
 <6e46859a58645d9f16a63ff76592487aabc9971d.1673873422.git.leon@kernel.org>
 <Y8WL2gQqqeZdMvr6@nvidia.com>
 <Y8aOe68Q49lvsjv8@unreal>
 <Y8anaBBZDOGF471q@nvidia.com>
 <Y8atPjQ1x75tBdib@unreal>
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <Y8atPjQ1x75tBdib@unreal>
X-ClientProxiedBy: BLAPR05CA0024.namprd05.prod.outlook.com
 (2603:10b6:208:36e::20) To LV2PR12MB5869.namprd12.prod.outlook.com
 (2603:10b6:408:176::16)
MIME-Version: 1.0
X-MS-PublicTrafficType: Email
X-MS-TrafficTypeDiagnostic: LV2PR12MB5869:EE_|BL1PR12MB5174:EE_
X-MS-Office365-Filtering-Correlation-Id: e46fd2e1-021f-499b-dc62-08daf8961a32
X-MS-Exchange-SenderADCheck: 1
X-MS-Exchange-AntiSpam-Relay: 0
X-Microsoft-Antispam: BCL:0;
X-Microsoft-Antispam-Message-Info:  
X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:LV2PR12MB5869.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230022)(4636009)(39860400002)(376002)(136003)(346002)(366004)(396003)(451199015)(2616005)(66476007)(41300700001)(186003)(66946007)(6916009)(8676002)(6512007)(66556008)(4326008)(26005)(36756003)(86362001)(5660300002)(8936002)(107886003)(54906003)(478600001)(316002)(6506007)(7416002)(2906002)(38100700002)(6486002);DIR:OUT;SFP:1101;
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?ktFCzKxzvxP2CrSGrmcl+tknN27b4DjZwVFRqzZJZH1/PyjfqGzqc17wSipd?=
 =?us-ascii?Q?rbQSxZmbrYeL480aKT+ffHuKSB3z0h2QbZhysFS1xh3Ieh5SUlKTKkLdbAlb?=
 =?us-ascii?Q?K3YdCWOBCoEcuXRLL9KLGb2SxH37WraZKobygxzqGML4uN2FFkPu4PolyMp9?=
 =?us-ascii?Q?KR/GwV6fxk6qwT/eTC7+qq4DWoHmxwVf5tyftgkz7OqfFD2uZIWnLmRyO3HH?=
 =?us-ascii?Q?Hpr2M3QyJESk/342RODMs89YVcOhelQN8YjQYQuCQsNX/lEf5SXOyQgrVqd1?=
 =?us-ascii?Q?IlHR4bSrJUooPzGx4Ybjd0fpPZW6jhFU+FJBW/NHleRq4l6tXzscT9Nd5ygk?=
 =?us-ascii?Q?HnwmRPpZbT3KLpP8EqleOIMLU4JGnCFaDY7HHjmQQTNxX/h1IyjZkiA9wntV?=
 =?us-ascii?Q?ExSISQ5QocSq4Jj0s4t/ClTJkycsEkXDa/i0GBvQkLtxjKs+aF9eYuIcXipw?=
 =?us-ascii?Q?HX45Hxa8w06O4ix1zaG7bedptTjhrRo74b1bT6PMtOaTJwcyJie8ovrVXUKu?=
 =?us-ascii?Q?gXTzZAx29pCmvlHUVQkg33h/m+LRcvgy3wSk2maaJy/bITXMlOExZXgGNOPI?=
 =?us-ascii?Q?g4IGAC6dp5hr9VILUpMudGuUvDEOp5e5pZSvUjZBgR0w03p6C9rFau5IyECM?=
 =?us-ascii?Q?eKQw9UcdaJwGKhi4HL++PNUeehJ4Cy3dPALm4Lea6FX9NY3XDrkZ6Rf8188C?=
 =?us-ascii?Q?TcpJSUToARDU5faLP9si0W3O1cia1bhaDLAjpaXcp6xk/qNJveAjTZ7avM5K?=
 =?us-ascii?Q?7ZpcB5c3npmAcs/ABlnlANzal3IS/Cqb5eJmnnd1mQSmMIg+SxgrePWstFxA?=
 =?us-ascii?Q?sSROGvOW9Hn6niIRotY0P8EsM2VSka9JISJfvMl/QbvYXq2XwIbZeZWOydGJ?=
 =?us-ascii?Q?1xmiVWcU0emfueupi2QNqe/ZRqB/g1omtmkIE3OdQu0hy+rEZCQIpcK1qX2N?=
 =?us-ascii?Q?FMBJNZ0d1xNEbg0WZ4ANF40mi2PZhvpgfXFq4WfnEg9AyV6aYSiGMKhJS6ZE?=
 =?us-ascii?Q?sjM3tZWrYwZF6ZzdeFKmw6qsBMNOJsMaH/4ThwZIWh/Oln+3NwOC0pm53Png?=
 =?us-ascii?Q?DAuy9x4VwWyRKhwv3ohXstx2iiR//WSmv5+B5m9mtoSBCn8yzu0UAFBQJs+e?=
 =?us-ascii?Q?VN0dmn5OUY69xlTte9V9ohGPkB77XwQ+dvH7ah+s8YZKrBJq5NLDzgmgTwqe?=
 =?us-ascii?Q?6gqX/AYIqXvI5Pbjx586/MG+Iv9XJktVPdbZ14zcWiX05J41y5K9aOCwhOMK?=
 =?us-ascii?Q?vP40N9NAPfFSiYcMCo2e36mKQLNyWXWLTC9reR0zWmJs8LaJehE8yhIJ9MBU?=
 =?us-ascii?Q?Dh+x8RLM6WV2CM8RTc6fPTePRDKbYvqhZSP3WrvZuoOCjsWKdXt6Y7tnpYDW?=
 =?us-ascii?Q?KXexJBlJtDz24Nji2USX2kbPKJ3njxzjAkpUFDQ6ZtgA5neETiyWwd4+FRXx?=
 =?us-ascii?Q?ef9CDDEX47PKUqoExl88XzCSTyNiUe20tvoxB5JVXO9Rq0XTf0soF78oA91l?=
 =?us-ascii?Q?w+HZV0AyL37y5NOrCA8wmb1kwDMAc9lR84Zt2HwRbSfpi0pBiHuvgW8PytmO?=
 =?us-ascii?Q?VZVT37FhPIMG1XtYSAYXDrMi+LFWJZ+da66fVZI5?=
X-OriginatorOrg: Nvidia.com
X-MS-Exchange-CrossTenant-Network-Message-Id: e46fd2e1-021f-499b-dc62-08daf8961a32
X-MS-Exchange-CrossTenant-AuthSource: LV2PR12MB5869.namprd12.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 17 Jan 2023 14:21:19.6557
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: vh9Woi2sy1rcOcg2Pi5ukvYefztPS5pzWl05OedFEYdMflPfo0KtnrEkaxzcwba2
X-MS-Exchange-Transport-CrossTenantHeadersStamped: BL1PR12MB5174
Precedence: bulk
List-ID: <netdev.vger.kernel.org>
X-Mailing-List: netdev@vger.kernel.org

On Tue, Jan 17, 2023 at 04:14:22PM +0200, Leon Romanovsky wrote:
> On Tue, Jan 17, 2023 at 09:49:28AM -0400, Jason Gunthorpe wrote:
> > On Tue, Jan 17, 2023 at 02:03:07PM +0200, Leon Romanovsky wrote:
> > > On Mon, Jan 16, 2023 at 01:39:38PM -0400, Jason Gunthorpe wrote:
> > > > On Mon, Jan 16, 2023 at 03:05:50PM +0200, Leon Romanovsky wrote:
> > > > 
> > > > > diff --git a/drivers/infiniband/hw/mlx4/mlx4_ib.h b/drivers/infiniband/hw/mlx4/mlx4_ib.h
> > > > > index 17fee1e73a45..c553bf0eb257 100644
> > > > > --- a/drivers/infiniband/hw/mlx4/mlx4_ib.h
> > > > > +++ b/drivers/infiniband/hw/mlx4/mlx4_ib.h
> > > > > @@ -184,7 +184,7 @@ enum mlx4_ib_qp_flags {
> > > > >  	/* Mellanox specific flags start from IB_QP_CREATE_RESERVED_START */
> > > > >  	MLX4_IB_ROCE_V2_GSI_QP = MLX4_IB_QP_CREATE_ROCE_V2_GSI,
> > > > >  	MLX4_IB_SRIOV_TUNNEL_QP = 1 << 30,
> > > > > -	MLX4_IB_SRIOV_SQP = 1 << 31,
> > > > > +	MLX4_IB_SRIOV_SQP = 1ULL << 31,
> > > > >  };
> > > > 
> > > > These should be moved to a uapi if we are saying they are userspace
> > > > available
> > > > 
> > > > But I'm not sure they are?
> > > 
> > > I don't think so.
> > 
> > Then they should be > 32 bits right?
> 
> Right now, they are in reserved range:
>         /* reserve bits 26-31 for low level drivers' internal use */
>         IB_QP_CREATE_RESERVED_START             = 1 << 26,
>         IB_QP_CREATE_RESERVED_END               = 1ULL << 31,
> 
> If we move them to kernel part, we will need to define reserved range
> there too. So we just "burn" extra bits just for mlx4, also I don't see
> any reason to promote mlx4 bits to be general ones.

Is the reserved range kernel only? It would be nice to clarify that
detail

If yes we should move it so that userspace cannot set it. Do we have a
bug here already?

Jason

From mboxrd@z Thu Jan  1 00:00:00 1970
Return-Path: <netdev-owner@vger.kernel.org>
X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on
	aws-us-west-2-korg-lkml-1.web.codeaurora.org
Received: from vger.kernel.org (vger.kernel.org [23.128.96.18])
	by smtp.lore.kernel.org (Postfix) with ESMTP id 58B46C677F1
	for <netdev@archiver.kernel.org>; Tue, 17 Jan 2023 14:35:06 +0000 (UTC)
Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand
        id S232813AbjAQOfE (ORCPT <rfc822;netdev@archiver.kernel.org>);
        Tue, 17 Jan 2023 09:35:04 -0500
Received: from lindbergh.monkeyblade.net ([23.128.96.19]:54896 "EHLO
        lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org
        with ESMTP id S232766AbjAQOeb (ORCPT
        <rfc822;netdev@vger.kernel.org>); Tue, 17 Jan 2023 09:34:31 -0500
Received: from dfw.source.kernel.org (dfw.source.kernel.org [IPv6:2604:1380:4641:c500::1])
        by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 22F84A25D;
        Tue, 17 Jan 2023 06:34:22 -0800 (PST)
Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140])
        (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
        (No client certificate requested)
        by dfw.source.kernel.org (Postfix) with ESMTPS id B723D6147F;
        Tue, 17 Jan 2023 14:34:21 +0000 (UTC)
Received: by smtp.kernel.org (Postfix) with ESMTPSA id 86555C433EF;
        Tue, 17 Jan 2023 14:34:20 +0000 (UTC)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org;
        s=k20201202; t=1673966061;
        bh=Fz/P1XZDGupDtqAJSEnp0rEfUZAKbWxcCPbBc58E86o=;
        h=Date:From:To:Cc:Subject:References:In-Reply-To:From;
        b=Zd/8A4ZSHTYkosiDKj1TO0FVHt7VXMMtyD28nA+OIKNWV0uToO4VZUsV04f+85zsK
         NPAqR1E4a1Q0CqivGIOi81y5puPNQKSjGzt83iuk9+arxT2fS1FYGXhGrqyY0pos+Z
         OcLCxPp7ys9ubCVCxeMxzYmYNy77ccvTqq24OSgfB3I1ANleDm9EcAC5SJFVPc92BV
         82ZrOmcDrcd12x/w57ND2bTpnoX2P/SLRY7hVBHhu05Shda5VwcyAjC1oJ0Zq2EiV+
         jmFS5CVk+gGXWQSHF1NbuqizTD6yrXxEWa4GU13nqxj7Wk2j8qhBhnUxjl4nGJ/QXE
         mk7N0mBa6Ynxg==
Date:   Tue, 17 Jan 2023 16:34:17 +0200
From:   Leon Romanovsky <leon@kernel.org>
To:     Jason Gunthorpe <jgg@nvidia.com>
Cc:     Israel Rukshin <israelr@nvidia.com>,
        Bryan Tan <bryantan@vmware.com>,
        Christoph Hellwig <hch@lst.de>,
        Eric Dumazet <edumazet@google.com>,
        Jakub Kicinski <kuba@kernel.org>, Jens Axboe <axboe@fb.com>,
        Keith Busch <kbusch@kernel.org>, linux-kernel@vger.kernel.org,
        linux-nvme@lists.infradead.org, linux-rdma@vger.kernel.org,
        linux-trace-kernel@vger.kernel.org,
        Masami Hiramatsu <mhiramat@kernel.org>,
        Max Gurtovoy <mgurtovoy@nvidia.com>, netdev@vger.kernel.org,
        Paolo Abeni <pabeni@redhat.com>,
        Saeed Mahameed <saeedm@nvidia.com>,
        Sagi Grimberg <sagi@grimberg.me>,
        Selvin Xavier <selvin.xavier@broadcom.com>,
        Steven Rostedt <rostedt@goodmis.org>,
        Vishnu Dasa <vdasa@vmware.com>,
        Yishai Hadas <yishaih@nvidia.com>
Subject: Re: [PATCH rdma-next 03/13] RDMA: Split kernel-only create QP flags
 from uverbs create QP flags
Message-ID: <Y8ax6ZD6hhyD41pA@unreal>
References: <cover.1673873422.git.leon@kernel.org>
 <6e46859a58645d9f16a63ff76592487aabc9971d.1673873422.git.leon@kernel.org>
 <Y8WL2gQqqeZdMvr6@nvidia.com>
 <Y8aOe68Q49lvsjv8@unreal>
 <Y8anaBBZDOGF471q@nvidia.com>
 <Y8atPjQ1x75tBdib@unreal>
 <Y8au3ni8NVBPI5hu@nvidia.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <Y8au3ni8NVBPI5hu@nvidia.com>
Precedence: bulk
List-ID: <netdev.vger.kernel.org>
X-Mailing-List: netdev@vger.kernel.org

On Tue, Jan 17, 2023 at 10:21:18AM -0400, Jason Gunthorpe wrote:
> On Tue, Jan 17, 2023 at 04:14:22PM +0200, Leon Romanovsky wrote:
> > On Tue, Jan 17, 2023 at 09:49:28AM -0400, Jason Gunthorpe wrote:
> > > On Tue, Jan 17, 2023 at 02:03:07PM +0200, Leon Romanovsky wrote:
> > > > On Mon, Jan 16, 2023 at 01:39:38PM -0400, Jason Gunthorpe wrote:
> > > > > On Mon, Jan 16, 2023 at 03:05:50PM +0200, Leon Romanovsky wrote:
> > > > > 
> > > > > > diff --git a/drivers/infiniband/hw/mlx4/mlx4_ib.h b/drivers/infiniband/hw/mlx4/mlx4_ib.h
> > > > > > index 17fee1e73a45..c553bf0eb257 100644
> > > > > > --- a/drivers/infiniband/hw/mlx4/mlx4_ib.h
> > > > > > +++ b/drivers/infiniband/hw/mlx4/mlx4_ib.h
> > > > > > @@ -184,7 +184,7 @@ enum mlx4_ib_qp_flags {
> > > > > >  	/* Mellanox specific flags start from IB_QP_CREATE_RESERVED_START */
> > > > > >  	MLX4_IB_ROCE_V2_GSI_QP = MLX4_IB_QP_CREATE_ROCE_V2_GSI,
> > > > > >  	MLX4_IB_SRIOV_TUNNEL_QP = 1 << 30,
> > > > > > -	MLX4_IB_SRIOV_SQP = 1 << 31,
> > > > > > +	MLX4_IB_SRIOV_SQP = 1ULL << 31,
> > > > > >  };
> > > > > 
> > > > > These should be moved to a uapi if we are saying they are userspace
> > > > > available
> > > > > 
> > > > > But I'm not sure they are?
> > > > 
> > > > I don't think so.
> > > 
> > > Then they should be > 32 bits right?
> > 
> > Right now, they are in reserved range:
> >         /* reserve bits 26-31 for low level drivers' internal use */
> >         IB_QP_CREATE_RESERVED_START             = 1 << 26,
> >         IB_QP_CREATE_RESERVED_END               = 1ULL << 31,
> > 
> > If we move them to kernel part, we will need to define reserved range
> > there too. So we just "burn" extra bits just for mlx4, also I don't see
> > any reason to promote mlx4 bits to be general ones.
> 
> Is the reserved range kernel only? It would be nice to clarify that
> detail
> 
> If yes we should move it so that userspace cannot set it. Do we have a
> bug here already?

No, we always checked that users can't provide these bits and fail create QP.

It means that we can safely move that range too.

Thanks

> 
> Jason

From mboxrd@z Thu Jan  1 00:00:00 1970
Return-Path: <netdev-owner@vger.kernel.org>
X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on
	aws-us-west-2-korg-lkml-1.web.codeaurora.org
Received: from vger.kernel.org (vger.kernel.org [23.128.96.18])
	by smtp.lore.kernel.org (Postfix) with ESMTP id 0D77DC3DA78
	for <netdev@archiver.kernel.org>; Tue, 17 Jan 2023 21:22:21 +0000 (UTC)
Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand
        id S229541AbjAQVWH (ORCPT <rfc822;netdev@archiver.kernel.org>);
        Tue, 17 Jan 2023 16:22:07 -0500
Received: from lindbergh.monkeyblade.net ([23.128.96.19]:43758 "EHLO
        lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org
        with ESMTP id S229532AbjAQVRu (ORCPT
        <rfc822;netdev@vger.kernel.org>); Tue, 17 Jan 2023 16:17:50 -0500
Received: from ams.source.kernel.org (ams.source.kernel.org [IPv6:2604:1380:4601:e00::1])
        by lindbergh.monkeyblade.net (Postfix) with ESMTPS id EF3E94994C;
        Tue, 17 Jan 2023 11:40:06 -0800 (PST)
Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140])
        (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
        (No client certificate requested)
        by ams.source.kernel.org (Postfix) with ESMTPS id AA50DB81331;
        Tue, 17 Jan 2023 19:40:05 +0000 (UTC)
Received: by smtp.kernel.org (Postfix) with ESMTPSA id 0E837C433D2;
        Tue, 17 Jan 2023 19:40:01 +0000 (UTC)
Date:   Tue, 17 Jan 2023 14:40:00 -0500
From:   Steven Rostedt <rostedt@goodmis.org>
To:     Leon Romanovsky <leon@kernel.org>
Cc:     Jason Gunthorpe <jgg@nvidia.com>,
        Israel Rukshin <israelr@nvidia.com>,
        Bryan Tan <bryantan@vmware.com>,
        Christoph Hellwig <hch@lst.de>,
        Eric Dumazet <edumazet@google.com>,
        Jakub Kicinski <kuba@kernel.org>, Jens Axboe <axboe@fb.com>,
        Keith Busch <kbusch@kernel.org>, linux-kernel@vger.kernel.org,
        linux-nvme@lists.infradead.org, linux-rdma@vger.kernel.org,
        linux-trace-kernel@vger.kernel.org,
        Masami Hiramatsu <mhiramat@kernel.org>,
        Max Gurtovoy <mgurtovoy@nvidia.com>, netdev@vger.kernel.org,
        Paolo Abeni <pabeni@redhat.com>,
        Saeed Mahameed <saeedm@nvidia.com>,
        Sagi Grimberg <sagi@grimberg.me>,
        Selvin Xavier <selvin.xavier@broadcom.com>,
        Vishnu Dasa <vdasa@vmware.com>,
        Yishai Hadas <yishaih@nvidia.com>
Subject: Re: [PATCH rdma-next 06/13] RDMA/core: Introduce MR type for crypto
 operations
Message-ID: <20230117144000.6efc56b2@gandalf.local.home>
In-Reply-To: <5b8fadc00c0fcc0c0ba3a5dcc9e7b9012c6b5859.1673873422.git.leon@kernel.org>
References: <cover.1673873422.git.leon@kernel.org>
        <5b8fadc00c0fcc0c0ba3a5dcc9e7b9012c6b5859.1673873422.git.leon@kernel.org>
X-Mailer: Claws Mail 3.17.8 (GTK+ 2.24.33; x86_64-pc-linux-gnu)
MIME-Version: 1.0
Content-Type: text/plain; charset=US-ASCII
Content-Transfer-Encoding: 7bit
Precedence: bulk
List-ID: <netdev.vger.kernel.org>
X-Mailing-List: netdev@vger.kernel.org

On Mon, 16 Jan 2023 15:05:53 +0200
Leon Romanovsky <leon@kernel.org> wrote:

> index 17642aa54437..b6a3d82b89ca 100644
> --- a/include/trace/events/rdma_core.h
> +++ b/include/trace/events/rdma_core.h
> @@ -371,6 +371,39 @@ TRACE_EVENT(mr_integ_alloc,
>  		__entry->max_num_meta_sg, __entry->rc)
>  );
>  
> +TRACE_EVENT(mr_crypto_alloc,
> +	TP_PROTO(
> +		const struct ib_pd *pd,
> +		u32 max_num_sg,
> +		const struct ib_mr *mr
> +	),
> +
> +	TP_ARGS(pd, max_num_sg, mr),
> +
> +	TP_STRUCT__entry(
> +		__field(u32, pd_id)
> +		__field(u32, mr_id)
> +		__field(u32, max_num_sg)
> +		__field(int, rc)
> +	),
> +
> +	TP_fast_assign(
> +		__entry->pd_id = pd->res.id;
> +		if (IS_ERR(mr)) {
> +			__entry->mr_id = 0;
> +			__entry->rc = PTR_ERR(mr);
> +		} else {
> +			__entry->mr_id = mr->res.id;
> +			__entry->rc = 0;
> +		}
> +		__entry->max_num_sg = max_num_sg;
> +	),
> +
> +	TP_printk("pd.id=%u mr.id=%u max_num_sg=%u rc=%d",
> +		__entry->pd_id, __entry->mr_id, __entry->max_num_sg,
> +		__entry->rc)
> +);
> +

Reviewed-by: Steven Rostedt (Google) <rostedt@goodmis.org>

-- Steve

>  TRACE_EVENT(mr_dereg,
>  	TP_PROTO(
>  		const struct ib_mr *mr


From mboxrd@z Thu Jan  1 00:00:00 1970
Return-Path: <netdev-owner@vger.kernel.org>
X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on
	aws-us-west-2-korg-lkml-1.web.codeaurora.org
Received: from vger.kernel.org (vger.kernel.org [23.128.96.18])
	by smtp.lore.kernel.org (Postfix) with ESMTP id 3434AC38159
	for <netdev@archiver.kernel.org>; Wed, 18 Jan 2023 07:32:15 +0000 (UTC)
Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand
        id S229537AbjARHcE (ORCPT <rfc822;netdev@archiver.kernel.org>);
        Wed, 18 Jan 2023 02:32:04 -0500
Received: from lindbergh.monkeyblade.net ([23.128.96.19]:36740 "EHLO
        lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org
        with ESMTP id S229837AbjARHa5 (ORCPT
        <rfc822;netdev@vger.kernel.org>); Wed, 18 Jan 2023 02:30:57 -0500
Received: from dfw.source.kernel.org (dfw.source.kernel.org [IPv6:2604:1380:4641:c500::1])
        by lindbergh.monkeyblade.net (Postfix) with ESMTPS id EC8725CFFC;
        Tue, 17 Jan 2023 22:47:57 -0800 (PST)
Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140])
        (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
        (No client certificate requested)
        by dfw.source.kernel.org (Postfix) with ESMTPS id 4353B616D0;
        Wed, 18 Jan 2023 06:47:47 +0000 (UTC)
Received: by smtp.kernel.org (Postfix) with ESMTPSA id E3876C433D2;
        Wed, 18 Jan 2023 06:47:45 +0000 (UTC)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org;
        s=k20201202; t=1674024466;
        bh=+e8Tu1yzNoIuZRv3cfV5EujT7Y0Ki8eK75d8fyjGjYU=;
        h=Date:From:To:Cc:Subject:References:In-Reply-To:From;
        b=up1oXAKUJlDbL/vu1dR4uq8tknJjJA23u+OsMvEiOFOxbAVe+XYnvnejYPlUGbgDo
         lyEe1ps9KO6JDBm4ID1AsIgfS2qHN/ICu64ZJQ5+oY3hEmnohorCfK4BCb4qFCFrjX
         FjJwR91gNSiiwXNqo5HSvTS4Q3ZP6pVTv8xTy7oBt4rZdxU4nR1dgnvnJQ8V50TRQH
         m3ajM+v+eAcHBgs/CxO7qx6Xs+6d8Be90kZ809xg6cvUXLVKCrBoFUx/Kh+dQ1bxCn
         UPMPI6OL3LbjH/9q8wLMCa9nwbOXoqN35WN+81n52gI/+QQCatBVN06vsdwRFLsEj1
         2zEWpihkxf+Nw==
Date:   Tue, 17 Jan 2023 22:47:44 -0800
From:   Eric Biggers <ebiggers@kernel.org>
To:     Leon Romanovsky <leon@kernel.org>
Cc:     Jason Gunthorpe <jgg@nvidia.com>, Bryan Tan <bryantan@vmware.com>,
        Christoph Hellwig <hch@lst.de>,
        Eric Dumazet <edumazet@google.com>,
        Israel Rukshin <israelr@nvidia.com>,
        Jakub Kicinski <kuba@kernel.org>, Jens Axboe <axboe@fb.com>,
        Keith Busch <kbusch@kernel.org>, linux-kernel@vger.kernel.org,
        linux-nvme@lists.infradead.org, linux-rdma@vger.kernel.org,
        linux-trace-kernel@vger.kernel.org,
        Masami Hiramatsu <mhiramat@kernel.org>,
        Max Gurtovoy <mgurtovoy@nvidia.com>, netdev@vger.kernel.org,
        Paolo Abeni <pabeni@redhat.com>,
        Saeed Mahameed <saeedm@nvidia.com>,
        Sagi Grimberg <sagi@grimberg.me>,
        Selvin Xavier <selvin.xavier@broadcom.com>,
        Steven Rostedt <rostedt@goodmis.org>,
        Vishnu Dasa <vdasa@vmware.com>,
        Yishai Hadas <yishaih@nvidia.com>
Subject: Re: [PATCH rdma-next 00/13] Add RDMA inline crypto support
Message-ID: <Y8eWEPZahIFAfnoI@sol.localdomain>
References: <cover.1673873422.git.leon@kernel.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Disposition: inline
Content-Transfer-Encoding: 8bit
In-Reply-To: <cover.1673873422.git.leon@kernel.org>
Precedence: bulk
List-ID: <netdev.vger.kernel.org>
X-Mailing-List: netdev@vger.kernel.org

Hi Leon,

On Mon, Jan 16, 2023 at 03:05:47PM +0200, Leon Romanovsky wrote:
> >From Israel,
> 
> The purpose of this patchset is to add support for inline
> encryption/decryption of the data at storage protocols like nvmf over
> RDMA (at a similar way like integrity is used via unique mkey).
> 
> This patchset adds support for plaintext keys. The patches were tested
> on BF-3 HW with fscrypt tool to test this feature, which showed reduce
> in CPU utilization when comparing at 64k or more IO size. The CPU utilization
> was improved by more than 50% comparing to the SW only solution at this case.
> 
> How to configure fscrypt to enable plaintext keys:
>  # mkfs.ext4 -O encrypt /dev/nvme0n1
>  # mount /dev/nvme0n1 /mnt/crypto -o inlinecrypt
>  # head -c 64 /dev/urandom > /tmp/master_key
>  # fscryptctl add_key /mnt/crypto/ < /tmp/master_key
>  # mkdir /mnt/crypto/test1
>  # fscryptctl set_policy 152c41b2ea39fa3d90ea06448456e7fb /mnt/crypto/test1
>    ** “152c41b2ea39fa3d90ea06448456e7fb” is the output of the
>       “fscryptctl add_key” command.
>  # echo foo > /mnt/crypto/test1/foo
> 
> Notes:
>  - At plaintext mode only, the user set a master key and the fscrypt
>    driver derived from it the DEK and the key identifier.
>  - 152c41b2ea39fa3d90ea06448456e7fb is the derived key identifier
>  - Only on the first IO, nvme-rdma gets a callback to load the derived DEK. 
> 
> There is no special configuration to support crypto at nvme modules.
> 
> Thanks

Very interesting work!  Can you Cc me on future versions?

I'm glad to see that this hardware allows all 16 IV bytes to be specified.

Does it also handle programming and evicting keys efficiently?

Also, just checking: have you tested that the ciphertext that this inline
encryption hardware produces is correct?  That's always super important to test.
There are xfstests that test for it, e.g. generic/582.  Another way to test it
is to just manually test whether encrypted files that were created when the
filesystem was mounted with '-o inlinecrypt' show the same contents when the
filesystem is *not* mounted with '-o inlinecrypt' (or vice versa).

- Eric

From mboxrd@z Thu Jan  1 00:00:00 1970
Return-Path: <netdev-owner@vger.kernel.org>
X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on
	aws-us-west-2-korg-lkml-1.web.codeaurora.org
Received: from vger.kernel.org (vger.kernel.org [23.128.96.18])
	by smtp.lore.kernel.org (Postfix) with ESMTP id 68176C38159
	for <netdev@archiver.kernel.org>; Wed, 18 Jan 2023 07:48:29 +0000 (UTC)
Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand
        id S229774AbjARHsY (ORCPT <rfc822;netdev@archiver.kernel.org>);
        Wed, 18 Jan 2023 02:48:24 -0500
Received: from lindbergh.monkeyblade.net ([23.128.96.19]:42448 "EHLO
        lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org
        with ESMTP id S229787AbjARHqL (ORCPT
        <rfc822;netdev@vger.kernel.org>); Wed, 18 Jan 2023 02:46:11 -0500
Received: from NAM12-MW2-obe.outbound.protection.outlook.com (mail-mw2nam12on2061.outbound.protection.outlook.com [40.107.244.61])
        by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 438514390E;
        Tue, 17 Jan 2023 23:14:32 -0800 (PST)
ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none;
 b=ViqklOAwjcErYIoA83c+io9xoYGT7QjEU0GMAgTWXTuk8a1OXdK5txrjAg+rB0mSISXh/FkLtUCpjK3zRdM+++iCdl7SDLG9JC7m1R7dN8hlgD8MX6NJJarJqPQFrOowU5fFY2wzZn0D8LHcN4R9L8NrXbRNqiO4zDWBUkBRJx9sEqHOnlGyk3OCQTyZLSRsON4O114T4kC8Ibmbu2deGVa2xNwDI6hX0cn1WuRBqHV3C5RhYsOBFxi+NlmIOIlVr6aZ6AIZFIg3CTJIdrNoZ7ASaX6DUHG2x9x2CEHnwVRgiNYOYHoCMwrItz+tGoC7+TjdiNZI6+NOpwjbmF14LQ==
ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com;
 s=arcselector9901;
 h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1;
 bh=tb0YZJoRjRZ4chXv/UIkZFOdm7DfOGAJ+iEbkOrjTUg=;
 b=QHoPqUTwxQ0rhdQorX6WGKaK60OXvvU2YPglxI2aG1YGfDhBTHyYAcykk6ZQmAryfALKzyDuKwcD0rNhuKpolejQfkRgul+EXoBLpaGqm3XFQcUckOrtsFfWIGbak78jzBMaloFtaBRFKRcMbvZ6FtFMMotxR7EKEAylp6Qwhv+GK1/egeyGLqOJaHOq85DoxgOWT8P3t/ZUM0YeUZRJc8VWeckw5iFWOlWZFgVaeyJRZVEnVgr1VID4zCoh0QWpOMj5KFrovYpRDHzNvnViqu49wYz954YsReDmCVOFdzBG528UjZwkZ9h85NK5Hg1/qWXkCHsRFKpBJSwTIys+qQ==
ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass
 smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com;
 dkim=pass header.d=nvidia.com; arc=none
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com;
 s=selector2;
 h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck;
 bh=tb0YZJoRjRZ4chXv/UIkZFOdm7DfOGAJ+iEbkOrjTUg=;
 b=GlrUQQOufrg8xOBSPneJ81eXo5p/PZojIsJbZNHEyTV6ftwqmuYd3Sc6waa1D8EnZWFkHeoBq7H0cIF4oPSkLlROJMcHRu1dBDXHUjnrGC6wbZr63H6OLUUREGxRN8irzvDZjmxXW6sAXbt6FB+Ar0t0G70+raUpjTovfOiTLm2Tz6dV+kwzJDBzl/+8wQCXrFbT1MpP2nIXudJ0d2i7L8eQQO/A6LeMdtk7R9I9Kp6wOGOCR7UtiN7aTnKOidBGfT/33lF97rqCmCacbdBAfku/bEM5wxvuMvNZfBjKkJnSMWSFRDJcJUrssf7xx4JItAL99xW91FcDE/BiQPWYxg==
Received: from MW2PR12MB4667.namprd12.prod.outlook.com (2603:10b6:302:12::28)
 by PH7PR12MB6467.namprd12.prod.outlook.com (2603:10b6:510:1f5::14) with
 Microsoft SMTP Server (version=TLS1_2,
 cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5986.23; Wed, 18 Jan
 2023 07:14:30 +0000
Received: from MW2PR12MB4667.namprd12.prod.outlook.com
 ([fe80::b84c:b6e:dc0:8d7]) by MW2PR12MB4667.namprd12.prod.outlook.com
 ([fe80::b84c:b6e:dc0:8d7%2]) with mapi id 15.20.6002.024; Wed, 18 Jan 2023
 07:14:30 +0000
From:   Chaitanya Kulkarni <chaitanyak@nvidia.com>
To:     Eric Biggers <ebiggers@kernel.org>
CC:     Jason Gunthorpe <jgg@nvidia.com>, Bryan Tan <bryantan@vmware.com>,
        Christoph Hellwig <hch@lst.de>,
        Eric Dumazet <edumazet@google.com>,
        Israel Rukshin <israelr@nvidia.com>,
        Jakub Kicinski <kuba@kernel.org>, Jens Axboe <axboe@fb.com>,
        Keith Busch <kbusch@kernel.org>,
        "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
        "linux-nvme@lists.infradead.org" <linux-nvme@lists.infradead.org>,
        "linux-rdma@vger.kernel.org" <linux-rdma@vger.kernel.org>,
        "linux-trace-kernel@vger.kernel.org" 
        <linux-trace-kernel@vger.kernel.org>,
        Masami Hiramatsu <mhiramat@kernel.org>,
        Max Gurtovoy <mgurtovoy@nvidia.com>,
        "netdev@vger.kernel.org" <netdev@vger.kernel.org>,
        Paolo Abeni <pabeni@redhat.com>,
        Saeed Mahameed <saeedm@nvidia.com>,
        Sagi Grimberg <sagi@grimberg.me>,
        Selvin Xavier <selvin.xavier@broadcom.com>,
        Steven Rostedt <rostedt@goodmis.org>,
        Vishnu Dasa <vdasa@vmware.com>,
        Yishai Hadas <yishaih@nvidia.com>,
        Leon Romanovsky <leon@kernel.org>
Subject: Re: [PATCH rdma-next 00/13] Add RDMA inline crypto support
Thread-Topic: [PATCH rdma-next 00/13] Add RDMA inline crypto support
Thread-Index: AQHZKauXj4Rw0vQLQUCXQK7qRIhrWq6jvjEAgAAHeoA=
Date:   Wed, 18 Jan 2023 07:14:30 +0000
Message-ID: <95692a47-09e7-0055-2006-46d085b2eadb@nvidia.com>
References: <cover.1673873422.git.leon@kernel.org>
 <Y8eWEPZahIFAfnoI@sol.localdomain>
In-Reply-To: <Y8eWEPZahIFAfnoI@sol.localdomain>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: 
user-agent: Mozilla/5.0 (X11; Linux x86_64; rv:91.0) Gecko/20100101
 Thunderbird/91.9.0
authentication-results: dkim=none (message not signed)
 header.d=none;dmarc=none action=none header.from=nvidia.com;
x-ms-publictraffictype: Email
x-ms-traffictypediagnostic: MW2PR12MB4667:EE_|PH7PR12MB6467:EE_
x-ms-office365-filtering-correlation-id: 0d797432-849e-4401-95f2-08daf923a443
x-ms-exchange-senderadcheck: 1
x-ms-exchange-antispam-relay: 0
x-microsoft-antispam: BCL:0;
x-microsoft-antispam-message-info:  
x-forefront-antispam-report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:MW2PR12MB4667.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230022)(4636009)(39860400002)(366004)(376002)(136003)(346002)(396003)(451199015)(31686004)(36756003)(2906002)(38070700005)(86362001)(6916009)(66946007)(8676002)(66556008)(76116006)(31696002)(66446008)(4326008)(64756008)(66476007)(8936002)(5660300002)(7416002)(83380400001)(38100700002)(122000001)(316002)(71200400001)(478600001)(54906003)(41300700001)(91956017)(2616005)(6512007)(6506007)(186003)(6486002)(43740500002)(45980500001);DIR:OUT;SFP:1101;
x-ms-exchange-antispam-messagedata-chunkcount: 1
x-ms-exchange-antispam-messagedata-0: =?utf-8?B?eU42eUZxNWt0bjVSVGNCNXdlRXlQM2l0eStmb21IMlEwSFRueTA2THRxMnFr?=
 =?utf-8?B?cndnQXRsZ08wTDB0bndWMU9NT2FCSlBVdXRaOU1mQVJ4TE1rNHRqMm9SL29E?=
 =?utf-8?B?VHc1eEJoMnEzYzhObGlkTkVKZlZwNXBZdWNzbytvQmd5Z2lhUEVyNGMvNjRl?=
 =?utf-8?B?VDR4NkFWVlNMMzh1bk5TNFJiUVNJVDMyblVPckc2bFZ6U1crdEQ5cS9uZ3g4?=
 =?utf-8?B?M3dLOTBsZkswWUhxUFlBdm16K1FjdlFDSDlLTUFCU0FrZm9ZSG40UVlhdENo?=
 =?utf-8?B?Z0NRQlZ3d0pTTk5MTWFkWHo4UW05WkQ1L3hoREk4ckZkdXZXR2xpdkZUVzlQ?=
 =?utf-8?B?ZDR4UzVVcndNa0RtRi9wNjdleHJqUWtiNU5aYmtxa3JxS29sTllvTldrbTFj?=
 =?utf-8?B?OHJCYzFEYkZwWnZiVi9SaWFQSEY4TEZoZzVxTWcyNUNrYjNOZEZrdlliY25D?=
 =?utf-8?B?L1NuNGxkbFo0TElNQmRSay9KZkwyT211TWhFRFBCcjBHNzYreGNSamlQdFEw?=
 =?utf-8?B?VGVaeGFEMFhuTGlPS3M5QVRMSHBsTnBrZTI3T2pDSHBMcTBVdDVNZXVHK3Yv?=
 =?utf-8?B?aW9QUkdxbVNoOGwrNDdNMFVSRWg1eWp6YUpIbU90OEZ3UnVxWmhhdHZGdUxR?=
 =?utf-8?B?b1dhSmVUNC9IL0tpMG1DMXpVR2pjMFl0YU9rMVJJV24wR0cvUnc3U240MWdK?=
 =?utf-8?B?YWN2QVVFakxQT0J3MG1UR2VzRmVmUUNiQ2VTdGlqeTFTQys2cDZmUVJRUUhR?=
 =?utf-8?B?UzFQQlEzaEZMSytQYzdCUGNFZHJUUFBYQld5SEdGY0ljbGhIcGVVdkk1QjJS?=
 =?utf-8?B?clU5ZzVPV0tCcis3aDNNdnNwZHBtMjFsRGFnalBaNWRsVVJ1WUp6cHRmRy9R?=
 =?utf-8?B?VCtxRFVJN2dyeVRRdkJxMVdSU2NjY2kvOW8wRTdtY05VM3NMNHRJQjFqcm9V?=
 =?utf-8?B?eGt0QmNVOUh2aW15ZVd1Wi9oZExOUFYvUWNvbHh3dkRFVzQ5RXUwbklrN0Jk?=
 =?utf-8?B?RDJHWTlCVDY3UElMemtvWjZWY0ZIcm1lK1g4aVpPVXZhcGErc3AzS0FjK2hQ?=
 =?utf-8?B?d3NjSTA3VDY2Smp6QjR4WHdaNlhCRzdGejhqY1NTeXNMdkRhMjVLM25JN3lm?=
 =?utf-8?B?RnVtbVMvSHhRdVBPTTcveFloTmtaVEtYUlB3bWpxSXBZTXp4WmpSSmx1RWZD?=
 =?utf-8?B?Sy90V1hySjdFSmVabi9oNlJlTGpJMFg2SElhbDJOWW4weEJPam1MRFBlZTY5?=
 =?utf-8?B?N0UxV2xPTytQdElLeGJqUjdUOVRHYzk2TXFTdk53M1JXWVBhKzYyb2RoRmE3?=
 =?utf-8?B?UVZ2R0lJaTRtSFcvUnloVmRnK0JmTUZhLyt2WmlzY2Z6d3MyWmdZSGRaN0pp?=
 =?utf-8?B?N3dLdmszUnFmZW05cjdWZ0EwSVhIZnFKbXB3a3N6UTRzUmd6NnhUWlBFY1JB?=
 =?utf-8?B?OHQ1TWRpd2lLdlhFaGRoN3E4NWd6YllJYjNseHJCZGVjOGZzR3pwMUordkpM?=
 =?utf-8?B?ZFdVUVVUTnlyS3dzbFp0WGhCSjYrUGM2U2Z2ZEVkRy94M2xpZTlDZ0JaQVB2?=
 =?utf-8?B?UTNhYlRmVnc3cnMzOWJhWGo2bU5vWWlrWTQyOHZNOS9FeXBpYTRGNkFWMjdD?=
 =?utf-8?B?cUV4YmhLdGYza2Npbjl3cTNBb0JtZEcrdjdZSys2OVllckh0YlVnNkpCNjVu?=
 =?utf-8?B?Z3hiaXpDYS9mUFBLelp1Zmd4bk9aV2FTRk9MbjhlNWx1NlJhcjBrZElWdjk2?=
 =?utf-8?B?NkVwL3NyU0lyd3ZOcUJxQUE2WGdtY2h6L2d4SXdLYzNtWFZKaVNVak1jTXNL?=
 =?utf-8?B?V0krUUd3R1lsUDlSOGFLN3lJUTM2YVZ1aVlkWTF1ajZLblZodXdhWGVsNXl4?=
 =?utf-8?B?TjRqU0k2US9QbnUzU2Y2NnhsbVhraWxyVEVmc1VhSkFnYjY2eFh6RGZ0Vzdk?=
 =?utf-8?B?b2JjMlJ4RE1sak1EQzAxS2FvcnJIK1lhVVBhZ3NadGxCNjhuZTlZRXFRNGlU?=
 =?utf-8?B?MXhBbm1qR3BweXI2UThpeXpVdjNJdXBJWHlsbDFySFFhZVRjaThkU21BelBO?=
 =?utf-8?B?cFZIcC9VTnd6bk54Um1xUG8zZy9WcTVKNVg5WXBjalVpRlFMdVV4NG5lbFJz?=
 =?utf-8?B?L09TbUpvVGZsSmRIV0JXUFl6ZzNxdzh1Z2pmdkFlbGdIOC8wK05tams2OE9s?=
 =?utf-8?B?WHc9PQ==?=
Content-Type: text/plain; charset="utf-8"
Content-ID: <6046736CA13C0545BDD453ED353654D0@namprd12.prod.outlook.com>
Content-Transfer-Encoding: base64
MIME-Version: 1.0
X-OriginatorOrg: Nvidia.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-AuthSource: MW2PR12MB4667.namprd12.prod.outlook.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 0d797432-849e-4401-95f2-08daf923a443
X-MS-Exchange-CrossTenant-originalarrivaltime: 18 Jan 2023 07:14:30.1098
 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: 43083d15-7273-40c1-b7db-39efd9ccc17a
X-MS-Exchange-CrossTenant-mailboxtype: HOSTED
X-MS-Exchange-CrossTenant-userprincipalname: U2rZaaM1zhdDN5PpLHtyuYogpO8GYD73qobTW/s8Fmym5BWWoDqTcSNZfYJk3XBU5tg5l/zEh+tIJ4zAtYsQVw==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: PH7PR12MB6467
Precedence: bulk
List-ID: <netdev.vger.kernel.org>
X-Mailing-List: netdev@vger.kernel.org

RXJpYywNCg0KPj4gTm90ZXM6DQo+PiAgIC0gQXQgcGxhaW50ZXh0IG1vZGUgb25seSwgdGhlIHVz
ZXIgc2V0IGEgbWFzdGVyIGtleSBhbmQgdGhlIGZzY3J5cHQNCj4+ICAgICBkcml2ZXIgZGVyaXZl
ZCBmcm9tIGl0IHRoZSBERUsgYW5kIHRoZSBrZXkgaWRlbnRpZmllci4NCj4+ICAgLSAxNTJjNDFi
MmVhMzlmYTNkOTBlYTA2NDQ4NDU2ZTdmYiBpcyB0aGUgZGVyaXZlZCBrZXkgaWRlbnRpZmllcg0K
Pj4gICAtIE9ubHkgb24gdGhlIGZpcnN0IElPLCBudm1lLXJkbWEgZ2V0cyBhIGNhbGxiYWNrIHRv
IGxvYWQgdGhlIGRlcml2ZWQgREVLLg0KPj4NCj4+IFRoZXJlIGlzIG5vIHNwZWNpYWwgY29uZmln
dXJhdGlvbiB0byBzdXBwb3J0IGNyeXB0byBhdCBudm1lIG1vZHVsZXMuDQo+Pg0KPj4gVGhhbmtz
DQo+IA0KPiBWZXJ5IGludGVyZXN0aW5nIHdvcmshICBDYW4geW91IENjIG1lIG9uIGZ1dHVyZSB2
ZXJzaW9ucz8NCj4gDQo+IEknbSBnbGFkIHRvIHNlZSB0aGF0IHRoaXMgaGFyZHdhcmUgYWxsb3dz
IGFsbCAxNiBJViBieXRlcyB0byBiZSBzcGVjaWZpZWQuDQo+IA0KPiBEb2VzIGl0IGFsc28gaGFu
ZGxlIHByb2dyYW1taW5nIGFuZCBldmljdGluZyBrZXlzIGVmZmljaWVudGx5Pw0KPiANCj4gQWxz
bywganVzdCBjaGVja2luZzogaGF2ZSB5b3UgdGVzdGVkIHRoYXQgdGhlIGNpcGhlcnRleHQgdGhh
dCB0aGlzIGlubGluZQ0KPiBlbmNyeXB0aW9uIGhhcmR3YXJlIHByb2R1Y2VzIGlzIGNvcnJlY3Q/
ICBUaGF0J3MgYWx3YXlzIHN1cGVyIGltcG9ydGFudCB0byB0ZXN0Lg0KPiBUaGVyZSBhcmUgeGZz
dGVzdHMgdGhhdCB0ZXN0IGZvciBpdCwgZS5nLiBnZW5lcmljLzU4Mi4gIEFub3RoZXIgd2F5IHRv
IHRlc3QgaXQNCj4gaXMgdG8ganVzdCBtYW51YWxseSB0ZXN0IHdoZXRoZXIgZW5jcnlwdGVkIGZp
bGVzIHRoYXQgd2VyZSBjcmVhdGVkIHdoZW4gdGhlDQo+IGZpbGVzeXN0ZW0gd2FzIG1vdW50ZWQg
d2l0aCAnLW8gaW5saW5lY3J5cHQnIHNob3cgdGhlIHNhbWUgY29udGVudHMgd2hlbiB0aGUNCj4g
ZmlsZXN5c3RlbSBpcyAqbm90KiBtb3VudGVkIHdpdGggJy1vIGlubGluZWNyeXB0JyAob3Igdmlj
ZSB2ZXJzYSkuDQo+IA0KPiAtIEVyaWMNCj4gDQoNCkknbSB3b25kZXJpbmcgd2hpY2ggYXJlIHRo
ZSB4ZnN0ZXN0cyB0aGF0IG5lZWRzIHRvIHJ1biBpbiBvcmRlcg0KdG8gZXN0YWJsaXNoIHRoZSBj
b3JyZWN0bmVzcy9zdGFiaWxpdHkgYXBhcnQgZnJvbSBnZW5lcmljLzU4Mg0KdGhpcyB3b3JrID8N
Cg0KLWNrDQoNCg==

From mboxrd@z Thu Jan  1 00:00:00 1970
Return-Path: <netdev-owner@vger.kernel.org>
X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on
	aws-us-west-2-korg-lkml-1.web.codeaurora.org
Received: from vger.kernel.org (vger.kernel.org [23.128.96.18])
	by smtp.lore.kernel.org (Postfix) with ESMTP id D2539C54EBE
	for <netdev@archiver.kernel.org>; Wed, 18 Jan 2023 07:49:34 +0000 (UTC)
Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand
        id S229561AbjARHtc (ORCPT <rfc822;netdev@archiver.kernel.org>);
        Wed, 18 Jan 2023 02:49:32 -0500
Received: from lindbergh.monkeyblade.net ([23.128.96.19]:44666 "EHLO
        lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org
        with ESMTP id S229512AbjARHsI (ORCPT
        <rfc822;netdev@vger.kernel.org>); Wed, 18 Jan 2023 02:48:08 -0500
Received: from ams.source.kernel.org (ams.source.kernel.org [IPv6:2604:1380:4601:e00::1])
        by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 3D82F45F46;
        Tue, 17 Jan 2023 23:17:19 -0800 (PST)
Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140])
        (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
        (No client certificate requested)
        by ams.source.kernel.org (Postfix) with ESMTPS id EDA42B81B3E;
        Wed, 18 Jan 2023 07:17:17 +0000 (UTC)
Received: by smtp.kernel.org (Postfix) with ESMTPSA id EBB21C433EF;
        Wed, 18 Jan 2023 07:17:15 +0000 (UTC)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org;
        s=k20201202; t=1674026236;
        bh=TPFbez4kRFQWTS4pIsn2TRE2Yeh3bO7Q6hZYSQ1W6pw=;
        h=Date:From:To:Cc:Subject:References:In-Reply-To:From;
        b=gBE1GvKyWgekNTYsa39JH90glVPf/zresTYnQ0oYxuizvsX6fM5mfbTi+0R6Ij98l
         YyaDFKnsM6lTyQmL6e2hpKrsC5rvStWq4oOvBnVVT0o/wKzFhIVItTIkr2TYxKT23l
         7rqy5WkpcyR27VdIF1yTxHowykDskkvhjoUUG53ZfOQR2moXA8Jxhb1pFtb7BfhQWW
         P0tpyi+0OtQuTEKt00PruI9rK9RKu8zppqgEd/TVL/h3TTTQsh2MB7hpZmsviYbo8P
         vLSJorCADin7THDqDQl/r4t7iaa9cPH75VNP1IT+BlbB2MeyX2Jv7uPewvM9XoZFkR
         ObUnV48z0C9Bg==
Date:   Tue, 17 Jan 2023 23:17:14 -0800
From:   Eric Biggers <ebiggers@kernel.org>
To:     Chaitanya Kulkarni <chaitanyak@nvidia.com>
Cc:     Jason Gunthorpe <jgg@nvidia.com>, Bryan Tan <bryantan@vmware.com>,
        Christoph Hellwig <hch@lst.de>,
        Eric Dumazet <edumazet@google.com>,
        Israel Rukshin <israelr@nvidia.com>,
        Jakub Kicinski <kuba@kernel.org>, Jens Axboe <axboe@fb.com>,
        Keith Busch <kbusch@kernel.org>,
        "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
        "linux-nvme@lists.infradead.org" <linux-nvme@lists.infradead.org>,
        "linux-rdma@vger.kernel.org" <linux-rdma@vger.kernel.org>,
        "linux-trace-kernel@vger.kernel.org" 
        <linux-trace-kernel@vger.kernel.org>,
        Masami Hiramatsu <mhiramat@kernel.org>,
        Max Gurtovoy <mgurtovoy@nvidia.com>,
        "netdev@vger.kernel.org" <netdev@vger.kernel.org>,
        Paolo Abeni <pabeni@redhat.com>,
        Saeed Mahameed <saeedm@nvidia.com>,
        Sagi Grimberg <sagi@grimberg.me>,
        Selvin Xavier <selvin.xavier@broadcom.com>,
        Steven Rostedt <rostedt@goodmis.org>,
        Vishnu Dasa <vdasa@vmware.com>,
        Yishai Hadas <yishaih@nvidia.com>,
        Leon Romanovsky <leon@kernel.org>
Subject: Re: [PATCH rdma-next 00/13] Add RDMA inline crypto support
Message-ID: <Y8ec+vUqitvvLKHL@sol.localdomain>
References: <cover.1673873422.git.leon@kernel.org>
 <Y8eWEPZahIFAfnoI@sol.localdomain>
 <95692a47-09e7-0055-2006-46d085b2eadb@nvidia.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <95692a47-09e7-0055-2006-46d085b2eadb@nvidia.com>
Precedence: bulk
List-ID: <netdev.vger.kernel.org>
X-Mailing-List: netdev@vger.kernel.org

On Wed, Jan 18, 2023 at 07:14:30AM +0000, Chaitanya Kulkarni wrote:
> Eric,
> 
> >> Notes:
> >>   - At plaintext mode only, the user set a master key and the fscrypt
> >>     driver derived from it the DEK and the key identifier.
> >>   - 152c41b2ea39fa3d90ea06448456e7fb is the derived key identifier
> >>   - Only on the first IO, nvme-rdma gets a callback to load the derived DEK.
> >>
> >> There is no special configuration to support crypto at nvme modules.
> >>
> >> Thanks
> > 
> > Very interesting work!  Can you Cc me on future versions?
> > 
> > I'm glad to see that this hardware allows all 16 IV bytes to be specified.
> > 
> > Does it also handle programming and evicting keys efficiently?
> > 
> > Also, just checking: have you tested that the ciphertext that this inline
> > encryption hardware produces is correct?  That's always super important to test.
> > There are xfstests that test for it, e.g. generic/582.  Another way to test it
> > is to just manually test whether encrypted files that were created when the
> > filesystem was mounted with '-o inlinecrypt' show the same contents when the
> > filesystem is *not* mounted with '-o inlinecrypt' (or vice versa).
> > 
> > - Eric
> > 
> 
> I'm wondering which are the xfstests that needs to run in order
> to establish the correctness/stability apart from generic/582
> this work ?
> 

See https://docs.kernel.org/filesystems/fscrypt.html#tests.

- Eric

From mboxrd@z Thu Jan  1 00:00:00 1970
Return-Path: <netdev-owner@vger.kernel.org>
X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on
	aws-us-west-2-korg-lkml-1.web.codeaurora.org
Received: from vger.kernel.org (vger.kernel.org [23.128.96.18])
	by smtp.lore.kernel.org (Postfix) with ESMTP id 11AF0C38159
	for <netdev@archiver.kernel.org>; Wed, 18 Jan 2023 08:08:54 +0000 (UTC)
Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand
        id S229717AbjARIIp (ORCPT <rfc822;netdev@archiver.kernel.org>);
        Wed, 18 Jan 2023 03:08:45 -0500
Received: from lindbergh.monkeyblade.net ([23.128.96.19]:60122 "EHLO
        lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org
        with ESMTP id S230264AbjARICX (ORCPT
        <rfc822;netdev@vger.kernel.org>); Wed, 18 Jan 2023 03:02:23 -0500
Received: from verein.lst.de (verein.lst.de [213.95.11.211])
        by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 9B0855DC1D;
        Tue, 17 Jan 2023 23:36:59 -0800 (PST)
Received: by verein.lst.de (Postfix, from userid 2407)
        id CE78267373; Wed, 18 Jan 2023 08:36:54 +0100 (CET)
Date:   Wed, 18 Jan 2023 08:36:54 +0100
From:   Christoph Hellwig <hch@lst.de>
To:     Leon Romanovsky <leon@kernel.org>
Cc:     Jason Gunthorpe <jgg@nvidia.com>, Bryan Tan <bryantan@vmware.com>,
        Christoph Hellwig <hch@lst.de>,
        Eric Dumazet <edumazet@google.com>,
        Israel Rukshin <israelr@nvidia.com>,
        Jakub Kicinski <kuba@kernel.org>, Jens Axboe <axboe@fb.com>,
        Keith Busch <kbusch@kernel.org>, linux-kernel@vger.kernel.org,
        linux-nvme@lists.infradead.org, linux-rdma@vger.kernel.org,
        linux-trace-kernel@vger.kernel.org,
        Masami Hiramatsu <mhiramat@kernel.org>,
        Max Gurtovoy <mgurtovoy@nvidia.com>, netdev@vger.kernel.org,
        Paolo Abeni <pabeni@redhat.com>,
        Saeed Mahameed <saeedm@nvidia.com>,
        Sagi Grimberg <sagi@grimberg.me>,
        Selvin Xavier <selvin.xavier@broadcom.com>,
        Steven Rostedt <rostedt@goodmis.org>,
        Vishnu Dasa <vdasa@vmware.com>,
        Yishai Hadas <yishaih@nvidia.com>
Subject: Re: [PATCH rdma-next 00/13] Add RDMA inline crypto support
Message-ID: <20230118073654.GC27048@lst.de>
References: <cover.1673873422.git.leon@kernel.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <cover.1673873422.git.leon@kernel.org>
User-Agent: Mutt/1.5.17 (2007-11-01)
Precedence: bulk
List-ID: <netdev.vger.kernel.org>
X-Mailing-List: netdev@vger.kernel.org

On Mon, Jan 16, 2023 at 03:05:47PM +0200, Leon Romanovsky wrote:
> >From Israel,
> 
> The purpose of this patchset is to add support for inline
> encryption/decryption of the data at storage protocols like nvmf over
> RDMA (at a similar way like integrity is used via unique mkey).
> 
> This patchset adds support for plaintext keys. The patches were tested
> on BF-3 HW with fscrypt tool to test this feature, which showed reduce
> in CPU utilization when comparing at 64k or more IO size. The CPU utilization
> was improved by more than 50% comparing to the SW only solution at this case.

One thing that needs to be solved before we can look into this is the
interaction with protection information (or integrity data in Linux
terms).  Currently inline encryption and protection information are
mutally incompatible.

From mboxrd@z Thu Jan  1 00:00:00 1970
Return-Path: <netdev-owner@vger.kernel.org>
X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on
	aws-us-west-2-korg-lkml-1.web.codeaurora.org
Received: from vger.kernel.org (vger.kernel.org [23.128.96.18])
	by smtp.lore.kernel.org (Postfix) with ESMTP id 34250C004D4
	for <netdev@archiver.kernel.org>; Wed, 18 Jan 2023 09:05:44 +0000 (UTC)
Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand
        id S229788AbjARJFb (ORCPT <rfc822;netdev@archiver.kernel.org>);
        Wed, 18 Jan 2023 04:05:31 -0500
Received: from lindbergh.monkeyblade.net ([23.128.96.19]:33776 "EHLO
        lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org
        with ESMTP id S230051AbjARJES (ORCPT
        <rfc822;netdev@vger.kernel.org>); Wed, 18 Jan 2023 04:04:18 -0500
Received: from ams.source.kernel.org (ams.source.kernel.org [145.40.68.75])
        by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 6569C303F8;
        Wed, 18 Jan 2023 00:22:38 -0800 (PST)
Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140])
        (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
        (No client certificate requested)
        by ams.source.kernel.org (Postfix) with ESMTPS id 26EC4B81BA2;
        Wed, 18 Jan 2023 08:22:37 +0000 (UTC)
Received: by smtp.kernel.org (Postfix) with ESMTPSA id 0C176C433EF;
        Wed, 18 Jan 2023 08:22:35 +0000 (UTC)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org;
        s=k20201202; t=1674030155;
        bh=J3P2jLNBlTxVtW6CzAaV5JkcOWy8oIduGW1AIFf0FvM=;
        h=Date:From:To:Cc:Subject:References:In-Reply-To:From;
        b=RrmsUcrdRqqVIlnk7HhrT7HkNkthDZLdAmJAbvi5YDXIY/MF7FZv7NLBXeF4D5emv
         5L2jF3QW8mYDpL17de3mblOvmVPuLZWT1KBQnnv6nLLYB7DdBp4hkmmA3n7yCBr8Et
         2abB0rQoHKGYzfmSFbLBBVVVZ4W68YsQ5XUcPY1n9Zv26lgCb7KN3n3SsJB6cI9q8v
         TXOSpovnW+MSNrF/HvLcd7jYv4u/vgbv9yuRPaOm7JQ8kQt5JVGfg464elsybVphlM
         GJILETQ0yFevJ5Bb7NB0QBJ5g6k0Qy0BpmbwvOwlvxvlybJIQdIILu94VA0iNGYJLN
         zZfXJgAM2Cg6Q==
Date:   Wed, 18 Jan 2023 10:22:31 +0200
From:   Leon Romanovsky <leon@kernel.org>
To:     Eric Biggers <ebiggers@kernel.org>
Cc:     Jason Gunthorpe <jgg@nvidia.com>, Bryan Tan <bryantan@vmware.com>,
        Christoph Hellwig <hch@lst.de>,
        Eric Dumazet <edumazet@google.com>,
        Israel Rukshin <israelr@nvidia.com>,
        Jakub Kicinski <kuba@kernel.org>, Jens Axboe <axboe@fb.com>,
        Keith Busch <kbusch@kernel.org>, linux-kernel@vger.kernel.org,
        linux-nvme@lists.infradead.org, linux-rdma@vger.kernel.org,
        linux-trace-kernel@vger.kernel.org,
        Masami Hiramatsu <mhiramat@kernel.org>,
        Max Gurtovoy <mgurtovoy@nvidia.com>, netdev@vger.kernel.org,
        Paolo Abeni <pabeni@redhat.com>,
        Saeed Mahameed <saeedm@nvidia.com>,
        Sagi Grimberg <sagi@grimberg.me>,
        Selvin Xavier <selvin.xavier@broadcom.com>,
        Steven Rostedt <rostedt@goodmis.org>,
        Vishnu Dasa <vdasa@vmware.com>,
        Yishai Hadas <yishaih@nvidia.com>
Subject: Re: [PATCH rdma-next 00/13] Add RDMA inline crypto support
Message-ID: <Y8esR55x5ozo1tV6@unreal>
References: <cover.1673873422.git.leon@kernel.org>
 <Y8eWEPZahIFAfnoI@sol.localdomain>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Disposition: inline
Content-Transfer-Encoding: 8bit
In-Reply-To: <Y8eWEPZahIFAfnoI@sol.localdomain>
Precedence: bulk
List-ID: <netdev.vger.kernel.org>
X-Mailing-List: netdev@vger.kernel.org

On Tue, Jan 17, 2023 at 10:47:44PM -0800, Eric Biggers wrote:
> Hi Leon,
> 
> On Mon, Jan 16, 2023 at 03:05:47PM +0200, Leon Romanovsky wrote:
> > >From Israel,
> > 
> > The purpose of this patchset is to add support for inline
> > encryption/decryption of the data at storage protocols like nvmf over
> > RDMA (at a similar way like integrity is used via unique mkey).
> > 
> > This patchset adds support for plaintext keys. The patches were tested
> > on BF-3 HW with fscrypt tool to test this feature, which showed reduce
> > in CPU utilization when comparing at 64k or more IO size. The CPU utilization
> > was improved by more than 50% comparing to the SW only solution at this case.
> > 
> > How to configure fscrypt to enable plaintext keys:
> >  # mkfs.ext4 -O encrypt /dev/nvme0n1
> >  # mount /dev/nvme0n1 /mnt/crypto -o inlinecrypt
> >  # head -c 64 /dev/urandom > /tmp/master_key
> >  # fscryptctl add_key /mnt/crypto/ < /tmp/master_key
> >  # mkdir /mnt/crypto/test1
> >  # fscryptctl set_policy 152c41b2ea39fa3d90ea06448456e7fb /mnt/crypto/test1
> >    ** “152c41b2ea39fa3d90ea06448456e7fb” is the output of the
> >       “fscryptctl add_key” command.
> >  # echo foo > /mnt/crypto/test1/foo
> > 
> > Notes:
> >  - At plaintext mode only, the user set a master key and the fscrypt
> >    driver derived from it the DEK and the key identifier.
> >  - 152c41b2ea39fa3d90ea06448456e7fb is the derived key identifier
> >  - Only on the first IO, nvme-rdma gets a callback to load the derived DEK. 
> > 
> > There is no special configuration to support crypto at nvme modules.
> > 
> > Thanks
> 
> Very interesting work!  Can you Cc me on future versions?

Sure

> 
> I'm glad to see that this hardware allows all 16 IV bytes to be specified.
> 
> Does it also handle programming and evicting keys efficiently?

"efficiently" is a very subjective term. We are using FW command
interface to program keys and this interface can do hundreds/thousands
commands per-second.

Thanks

From mboxrd@z Thu Jan  1 00:00:00 1970
Return-Path: <netdev-owner@vger.kernel.org>
X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on
	aws-us-west-2-korg-lkml-1.web.codeaurora.org
Received: from vger.kernel.org (vger.kernel.org [23.128.96.18])
	by smtp.lore.kernel.org (Postfix) with ESMTP id C82D7C004D4
	for <netdev@archiver.kernel.org>; Wed, 18 Jan 2023 09:43:44 +0000 (UTC)
Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand
        id S230216AbjARJni (ORCPT <rfc822;netdev@archiver.kernel.org>);
        Wed, 18 Jan 2023 04:43:38 -0500
Received: from lindbergh.monkeyblade.net ([23.128.96.19]:32810 "EHLO
        lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org
        with ESMTP id S230364AbjARJlM (ORCPT
        <rfc822;netdev@vger.kernel.org>); Wed, 18 Jan 2023 04:41:12 -0500
Received: from NAM02-DM3-obe.outbound.protection.outlook.com (mail-dm3nam02on2057.outbound.protection.outlook.com [40.107.95.57])
        by lindbergh.monkeyblade.net (Postfix) with ESMTPS id A6FBB656E1;
        Wed, 18 Jan 2023 00:58:15 -0800 (PST)
ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none;
 b=jRc7yRlkyUUB+1d7HTLTPuKwmL+Esd+rhM+DM5qltUrIwUyH8r4bHYYOptoAwcpffV0D8Ccdml3ww8WEgK7bk1e3Cl/PwxUfAW82WqPqgag1ByGg7v3iCwMOzfFXh2Xk1cYj0uM3o7xYNNWrfEdRJKko4s+2Cq55NvG7aNAZ87G2s/x35M9dlWED3zGxfrQhgpMAdyV/v52LoTaz1lgJ0CH1FZYoXvVxRpaIR4F99+/BYgCcSrp5ienPurtAnebu1xgFGYVb84WGkBnJPoRxEavmwmtmB3bfETJe3WCpMgzhG6JQ7mJGJfv2e5qMorPH1lIVbsDrl1gg8Py6cDln5A==
ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com;
 s=arcselector9901;
 h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1;
 bh=F2c32Ygh25XfwuNohv3aPEHfgzsmiE+Z/us/yUOHMxk=;
 b=h0Els8seMxDIIKXYqPi0jGk+q7Vp5zIzJxMmHjDOCb6FobrYeQ4O7yDTd5KwmC+CmW1DRY0Jptgwy7I40WSKIqnFAwZWxOj4MWfa4V3g5q8eZAQlWHGm5sC47c5dcgkinvR9W9h6eloX7wifDwVef3Q/RWzPu8dfLLvG5zqXxK/CAxFTjbhP3hC794KbxzaxNdTLeFblkhfwO7ITVwCG6iy1Y5UO0ilJGkwj+fAF+/n0xicG3N2KqeQ76ZljUBSiUlkYNSW8rr1xvrvPtcwqWLutuH4D7Z/rP8whqm/6kEBOgo+xs8gvmUGqzgf778SyhfQnPJoMV8tshycup1XjWg==
ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass
 smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com;
 dkim=pass header.d=nvidia.com; arc=none
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com;
 s=selector2;
 h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck;
 bh=F2c32Ygh25XfwuNohv3aPEHfgzsmiE+Z/us/yUOHMxk=;
 b=ZKE/KcigGt4hG5SdZeHwTiqK60bkQ8U2FmcmgIDuVRAW57NmDEBTSUpRoWXIL1iURA79//2PfOSBlv6Yh7Tk8VbPdUGQup2RyXw9JEVBqXj6WQE56sUa1IIWKNrrO83SbzXBNg4QQhp7/app1ZIlHsQ1AlKoh4OFRXSY2me4YaIPrg5VETc6UoIctuqR6gAXaYjUFpxaf/fSk2pq65ILr8mHzM4zsJ7eGk/D5jdd7jTDv4FoPPQrjQ5LRm3LT1fy4+BmObnTYCwqj83zvbU+teE1RE/8DpLa6C9IEbngnxMzuvOlhx593HwD3UlJx9slINayQ4CYnyIQ7d8SypMLVg==
Authentication-Results: dkim=none (message not signed)
 header.d=none;dmarc=none action=none header.from=nvidia.com;
Received: from MW3PR12MB4409.namprd12.prod.outlook.com (2603:10b6:303:2d::23)
 by MW4PR12MB6874.namprd12.prod.outlook.com (2603:10b6:303:20b::13) with
 Microsoft SMTP Server (version=TLS1_2,
 cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5986.18; Wed, 18 Jan
 2023 08:58:13 +0000
Received: from MW3PR12MB4409.namprd12.prod.outlook.com
 ([fe80::f803:f951:a68f:663a]) by MW3PR12MB4409.namprd12.prod.outlook.com
 ([fe80::f803:f951:a68f:663a%5]) with mapi id 15.20.5986.023; Wed, 18 Jan 2023
 08:58:13 +0000
Message-ID: <f6e661b2-f2f8-1747-2180-b7e6aa0b4bcd@nvidia.com>
Date:   Wed, 18 Jan 2023 10:58:02 +0200
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:102.0) Gecko/20100101
 Thunderbird/102.6.1
Subject: Re: [PATCH rdma-next 00/13] Add RDMA inline crypto support
Content-Language: en-US
To:     Eric Biggers <ebiggers@kernel.org>,
        Leon Romanovsky <leon@kernel.org>
Cc:     Jason Gunthorpe <jgg@nvidia.com>, Bryan Tan <bryantan@vmware.com>,
        Christoph Hellwig <hch@lst.de>,
        Eric Dumazet <edumazet@google.com>,
        Jakub Kicinski <kuba@kernel.org>, Jens Axboe <axboe@fb.com>,
        Keith Busch <kbusch@kernel.org>, linux-kernel@vger.kernel.org,
        linux-nvme@lists.infradead.org, linux-rdma@vger.kernel.org,
        linux-trace-kernel@vger.kernel.org,
        Masami Hiramatsu <mhiramat@kernel.org>,
        Max Gurtovoy <mgurtovoy@nvidia.com>, netdev@vger.kernel.org,
        Paolo Abeni <pabeni@redhat.com>,
        Saeed Mahameed <saeedm@nvidia.com>,
        Sagi Grimberg <sagi@grimberg.me>,
        Selvin Xavier <selvin.xavier@broadcom.com>,
        Steven Rostedt <rostedt@goodmis.org>,
        Vishnu Dasa <vdasa@vmware.com>,
        Yishai Hadas <yishaih@nvidia.com>
References: <cover.1673873422.git.leon@kernel.org>
 <Y8eWEPZahIFAfnoI@sol.localdomain>
From:   Israel Rukshin <israelr@nvidia.com>
In-Reply-To: <Y8eWEPZahIFAfnoI@sol.localdomain>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 8bit
X-ClientProxiedBy: FR0P281CA0120.DEUP281.PROD.OUTLOOK.COM
 (2603:10a6:d10:a8::10) To MW3PR12MB4409.namprd12.prod.outlook.com
 (2603:10b6:303:2d::23)
MIME-Version: 1.0
X-MS-PublicTrafficType: Email
X-MS-TrafficTypeDiagnostic: MW3PR12MB4409:EE_|MW4PR12MB6874:EE_
X-MS-Office365-Filtering-Correlation-Id: 0e25eb8d-4f9b-435f-0dde-08daf9322188
X-MS-Exchange-SenderADCheck: 1
X-MS-Exchange-AntiSpam-Relay: 0
X-Microsoft-Antispam: BCL:0;
X-Microsoft-Antispam-Message-Info:  
X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:MW3PR12MB4409.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230022)(4636009)(136003)(346002)(39860400002)(396003)(366004)(376002)(451199015)(66476007)(38100700002)(86362001)(8936002)(8676002)(2906002)(4326008)(7416002)(31696002)(5660300002)(107886003)(66556008)(478600001)(66946007)(41300700001)(6506007)(6512007)(26005)(186003)(2616005)(53546011)(110136005)(54906003)(6666004)(6486002)(316002)(31686004)(36756003)(45980500001)(43740500002);DIR:OUT;SFP:1101;
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?UHhSZkQzdlp4SFJ2YzhudTVYeVJRTlhZZW5weDExWXFmMUM3UDlVVFBGUEow?=
 =?utf-8?B?TDNNK0Q4MWRuc3ZYNFNKZTEySjc4a2J1dlVsWnJYZEh6SVAzWi9jeHQ0SDYy?=
 =?utf-8?B?QnlQTThvdUZIVU9yblNJclpFZnJmVlpRUEt1NTZGV29CYys0NVV4UmZqTlEz?=
 =?utf-8?B?Tk5Fbi9KelZlZU5WNVRZNElmVzNESGRuVkY3Z3VOaWlHWkd2SmordFR4UU13?=
 =?utf-8?B?dFA2b3FwNUdXWGNMVVVuVzV3blFjQWE2QmZqeS9aUVJ3ckl3L29ubjdKaGJF?=
 =?utf-8?B?cDlnbmVWdnZ5eGlvemNNOWppUFhWUW1STVp6RnlDUkF4NHFuV1dNT3gvazl5?=
 =?utf-8?B?SUpiWHBCK0QvdUFuNXdjV1BjWEdJZWZvM1Zvb2NWY3RCMkU4aUNUd2Z6MEdO?=
 =?utf-8?B?dEJVa28vZXIzQVAvMlRHOXZ3TEkwcXVXNFdHYmhJdGpQcjdKTTBCWUhFbVRj?=
 =?utf-8?B?dno0U0RrM3MvUytFYTljQ0J5YU5MT2kwQ3FXbi9lZUNlQjFSNEwwaTA4QWRX?=
 =?utf-8?B?QTZtbE8wYTlQTEJhM09DZE5QQ0pHck9PcVI1QlQzbEpFODhxb1ArMndMMEQr?=
 =?utf-8?B?czB0d3VYZVI4MWxmQ082eTdDY3kzNkgzeU94WEdieXVSRnh4VUtZODdvTXcx?=
 =?utf-8?B?RjUrSStPSmRtaDJVTW9TbXh4R2RyUWliTDkzTUw0eGdianlJdWFOQlJWY2Mz?=
 =?utf-8?B?OGgzQk44enpCK05FazZMdSt2dGtqd2ZGMWZWWmxMNUg4c29FbktUUlpLOVZS?=
 =?utf-8?B?QkZRMGVpdUZ1cGJpeTVmOTVjNXgvOW95OGNUZ1JZTUxIWnJLbEZhNThoNXlO?=
 =?utf-8?B?WGNNWUdjUWRoTWFLd08rWG1ySG8vWUlpSGRXcDgwZC91enRhZXAwaFVPbkwr?=
 =?utf-8?B?cmdTU1FuZC9RMnpMeU1ma2YyK29xNTNCZWs5cXoyeStpL3VVaFZDbjF5clFQ?=
 =?utf-8?B?WGRMYTdHc2F3L3ZvVmVieTJOVFQ2ZktpZTNVUjNkT1NUWEthQmVkbTRoNElj?=
 =?utf-8?B?aHpHYlRCVHNoa2dVUFcvZXpoS2pZZXpvVXl3RW42YUNZcGNaNHpoNzd1SEJk?=
 =?utf-8?B?SExWRUZPMWJxUERNR3huREFqeHN4Z2oyVWNIUzZzeE02a1RjNFVVL3F4dURC?=
 =?utf-8?B?ZWd6OUJvdUpPNjNqOWZ0R1hrYk5ESm8vTHpORlZkOW1IS0lidGgrZlM1UVJX?=
 =?utf-8?B?azJzTDZNZGdjMzEwbEFiS1dVNmhBUXd3T1phM0NOWVFQdlkrUm14S1FtcmN3?=
 =?utf-8?B?VVAwT1plVHdWK3owYlZXTUVQeFh6RHlabUxpTjFpMko2M0pCdlJKa290YWw1?=
 =?utf-8?B?NFZoaDJIL0pFSDc2Y0ZxYnNYdnkvWjJ3N0VMaVNEUTJHWWFmbTFKSXJsdjlm?=
 =?utf-8?B?YWZCdE11ajhTd1RNRWtLSkkwSXlWZ3V3N1diNWRyNXVrWXhpZVNWT1JldThO?=
 =?utf-8?B?SjRPVTBWZ2lOUzErZFYzdWFSc0V4QzZvdCtLazVDblFORkZFQWo2RHBrUk9t?=
 =?utf-8?B?dlFKT1pibHU0b3VnOVVpNUlxQUpGWnJPMjQ4SzgySjV3QUNsQnlOMVVoWnpJ?=
 =?utf-8?B?Q0g1M3lZdURHVk1kLy9FalBtMG5ZS0JpSGhRQnNIREZPeXpubDdpdFh1aEVB?=
 =?utf-8?B?dDAwUnJtZ0cxSUpESk41OWxreldzRFZPWG1lY1FJQVV5QXhqYnR2VEdvNlJh?=
 =?utf-8?B?RkdaeU03RXRrUmdzWWd4RUwzMWJaNDhaSm5YblFRcUtDRjBBcVNQZU1YQTJJ?=
 =?utf-8?B?Wm9TQm5aelNzcGEvdDRiRlFVZHBUUi9lRkkxVmVRbHJFWmdxc05ONEdqTVBF?=
 =?utf-8?B?WDRDdmdHNEJIeWFMK3BkbGJJMnlQeUJ5ZXdqcjk4eHNpSWNRbFNyZDNOSTlQ?=
 =?utf-8?B?U3YzeHdEVHFSYVBubEpERUkyRmQyL0lEbE84QytqcUVtZW9HVkJGR1BRMU9w?=
 =?utf-8?B?SlcyVjZCdUpSbW9NdjlDSmJBc3pJL2ZWdGxhMVFvbURBSk5sUUJlRzZIOUla?=
 =?utf-8?B?bHF1amw3VzhVdm9pUEVCTUYyZ0VDMXdJdU5BaWlnSXRmRWFsb1dHdWY2UjdW?=
 =?utf-8?B?bFkwaGpFKzNHeTd0Rm5LdjVUc3E4U3RjbHFhZ0pmSHBRdWovQUlrSW83NTBD?=
 =?utf-8?Q?BLyX7UkB4XAGKSByryw3h0rki?=
X-OriginatorOrg: Nvidia.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 0e25eb8d-4f9b-435f-0dde-08daf9322188
X-MS-Exchange-CrossTenant-AuthSource: MW3PR12MB4409.namprd12.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 18 Jan 2023 08:58:13.6028
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: EMYwc1XkaAiHPcEZfSI4pEpYkL8EE/fB+DybznFbZ6R/5kSO0yzrKzqkFGGk0RG0I8+oAayALNm/jrzBJ3WC2w==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: MW4PR12MB6874
Precedence: bulk
List-ID: <netdev.vger.kernel.org>
X-Mailing-List: netdev@vger.kernel.org

Hi Eric,

On 1/18/2023 8:47 AM, Eric Biggers wrote:
> Hi Leon,
>
> On Mon, Jan 16, 2023 at 03:05:47PM +0200, Leon Romanovsky wrote:
>> >From Israel,
>>
>> The purpose of this patchset is to add support for inline
>> encryption/decryption of the data at storage protocols like nvmf over
>> RDMA (at a similar way like integrity is used via unique mkey).
>>
>> This patchset adds support for plaintext keys. The patches were tested
>> on BF-3 HW with fscrypt tool to test this feature, which showed reduce
>> in CPU utilization when comparing at 64k or more IO size. The CPU utilization
>> was improved by more than 50% comparing to the SW only solution at this case.
>>
>> How to configure fscrypt to enable plaintext keys:
>>   # mkfs.ext4 -O encrypt /dev/nvme0n1
>>   # mount /dev/nvme0n1 /mnt/crypto -o inlinecrypt
>>   # head -c 64 /dev/urandom > /tmp/master_key
>>   # fscryptctl add_key /mnt/crypto/ < /tmp/master_key
>>   # mkdir /mnt/crypto/test1
>>   # fscryptctl set_policy 152c41b2ea39fa3d90ea06448456e7fb /mnt/crypto/test1
>>     ** “152c41b2ea39fa3d90ea06448456e7fb” is the output of the
>>        “fscryptctl add_key” command.
>>   # echo foo > /mnt/crypto/test1/foo
>>
>> Notes:
>>   - At plaintext mode only, the user set a master key and the fscrypt
>>     driver derived from it the DEK and the key identifier.
>>   - 152c41b2ea39fa3d90ea06448456e7fb is the derived key identifier
>>   - Only on the first IO, nvme-rdma gets a callback to load the derived DEK.
>>
>> There is no special configuration to support crypto at nvme modules.
>>
>> Thanks
> Very interesting work!  Can you Cc me on future versions?
>
> I'm glad to see that this hardware allows all 16 IV bytes to be specified.
>
> Does it also handle programming and evicting keys efficiently?
>
> Also, just checking: have you tested that the ciphertext that this inline
> encryption hardware produces is correct?  That's always super important to test.
> There are xfstests that test for it, e.g. generic/582.  Another way to test it
> is to just manually test whether encrypted files that were created when the
> filesystem was mounted with '-o inlinecrypt' show the same contents when the
> filesystem is *not* mounted with '-o inlinecrypt' (or vice versa).
sure, I ran the manual test of comparing the encrypted files content 
with and without the '-o inlinecrypt' at the mount command.
>
> - Eric
  - Israel

From mboxrd@z Thu Jan  1 00:00:00 1970
Return-Path: <netdev-owner@vger.kernel.org>
X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on
	aws-us-west-2-korg-lkml-1.web.codeaurora.org
Received: from vger.kernel.org (vger.kernel.org [23.128.96.18])
	by smtp.lore.kernel.org (Postfix) with ESMTP id D3A7CC38147
	for <netdev@archiver.kernel.org>; Wed, 18 Jan 2023 14:32:47 +0000 (UTC)
Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand
        id S231167AbjAROcp (ORCPT <rfc822;netdev@archiver.kernel.org>);
        Wed, 18 Jan 2023 09:32:45 -0500
Received: from lindbergh.monkeyblade.net ([23.128.96.19]:33264 "EHLO
        lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org
        with ESMTP id S230405AbjAROc3 (ORCPT
        <rfc822;netdev@vger.kernel.org>); Wed, 18 Jan 2023 09:32:29 -0500
Received: from NAM02-DM3-obe.outbound.protection.outlook.com (mail-dm3nam02on2056.outbound.protection.outlook.com [40.107.95.56])
        by lindbergh.monkeyblade.net (Postfix) with ESMTPS id D3FC63C29D;
        Wed, 18 Jan 2023 06:20:39 -0800 (PST)
ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none;
 b=cGMCFC6QF50qnq+l8jTr8PsTpM1qsM/7x9aS1E7FJ4E52Ep72h7MpKQCECfaNDFPkEtk95CO7EWDxxza4+Em0nYQplgD6B8CmltTfyU8LQNgde4+xb6ojoetE5dxLONViMHymQKNxl+Ef+bE0FOXHos5CJRbU4ktP+r6nkAFS4qmU/gFnOH0PKW7lJ6DMDUgq8HE9CRCHuv+1NmZiCEhbItf9tp5tMaTZDwUdW6feeTI9+kNKtq8B9CZ/coDKXlIbQNOki0HLoX2bAWBx8cauArwP7ZBL85MA/sLw9yW1z5xbNqsQNga7yplthp9U54qUSqTL7g0Mme6M2NPoakE3g==
ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com;
 s=arcselector9901;
 h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1;
 bh=2DImGXTFklUh486i6aV6dtFC1amhb4zg5fM5iwiDMA0=;
 b=aljgRJ4qCx3Yr/rxVmpBwpufqBrd49ijweeNoupyJuSwrRBVXGZsVwB9H1yo3DaGP7+/Qqk8VyyaJCAKbbo4skG3ESRm/hiyKZyGMiWqfa6LHCMhalaYF2ur2i4OjvR988EJkWBtmZh0TRL6RXJyDsT0sF87TT8cveGXZYkH55PhvryZNlpjbJ5k3cidihnRIsnn8ANi7qMdsY3Swsj2y8YASrH9PrWu2EWO8ER1S9tE8w7Rws9E6tbwQET56/YTQGsNCjjJ4u2s9kyEaYEdlHXsCFie0AydDtOcT2Zf8EYk1HWXStD0FJsvkEdUF8bFOrRKfL29p4UnBjoo9XI26g==
ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass
 smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com;
 dkim=pass header.d=nvidia.com; arc=none
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com;
 s=selector2;
 h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck;
 bh=2DImGXTFklUh486i6aV6dtFC1amhb4zg5fM5iwiDMA0=;
 b=QjPhqJGjH4ji/TTGI2OwrJw7oY+jhPDJg6MnuMy2JP5opqintck47jTHnFw0otnwTMpC1+3pCa+YauRPqf9m68i/q69iL8qyjY3+9u6zkOaq5tMz3I4JdRdxO6k96MvvdGQjbb6ao1l0LE0htNMtvu4t1GtkwOo3j17zjEyrorNTA4f3nra+KzTEh/RnoeLeRmaeR9qFrCKHRrF3KlCKNIXSU4rP/Br8LMyX56ckHyliCEWLCCAXluSilPYvGrU3HFEng9/vIco3ZDtI4GYggVhANVzhzUreN9YhL231eLw9rHUYaBZvnOSUQHZ92gk8Ac/sgKEdvzaCUYULF/FCdA==
Authentication-Results: dkim=none (message not signed)
 header.d=none;dmarc=none action=none header.from=nvidia.com;
Received: from DM4PR12MB5040.namprd12.prod.outlook.com (2603:10b6:5:38b::19)
 by LV2PR12MB5846.namprd12.prod.outlook.com (2603:10b6:408:175::6) with
 Microsoft SMTP Server (version=TLS1_2,
 cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5986.23; Wed, 18 Jan
 2023 14:20:38 +0000
Received: from DM4PR12MB5040.namprd12.prod.outlook.com
 ([fe80::9444:21e0:2a65:da95]) by DM4PR12MB5040.namprd12.prod.outlook.com
 ([fe80::9444:21e0:2a65:da95%6]) with mapi id 15.20.6002.024; Wed, 18 Jan 2023
 14:20:37 +0000
Message-ID: <ac48766a-b861-4fc0-3171-7b23f175c672@nvidia.com>
Date:   Wed, 18 Jan 2023 16:20:27 +0200
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:102.0) Gecko/20100101
 Thunderbird/102.6.1
Subject: Re: [PATCH rdma-next 00/13] Add RDMA inline crypto support
Content-Language: en-US
To:     Christoph Hellwig <hch@lst.de>, Leon Romanovsky <leon@kernel.org>
Cc:     Jason Gunthorpe <jgg@nvidia.com>, Bryan Tan <bryantan@vmware.com>,
        Eric Dumazet <edumazet@google.com>,
        Israel Rukshin <israelr@nvidia.com>,
        Jakub Kicinski <kuba@kernel.org>, Jens Axboe <axboe@fb.com>,
        Keith Busch <kbusch@kernel.org>, linux-kernel@vger.kernel.org,
        linux-nvme@lists.infradead.org, linux-rdma@vger.kernel.org,
        linux-trace-kernel@vger.kernel.org,
        Masami Hiramatsu <mhiramat@kernel.org>, netdev@vger.kernel.org,
        Paolo Abeni <pabeni@redhat.com>,
        Saeed Mahameed <saeedm@nvidia.com>,
        Sagi Grimberg <sagi@grimberg.me>,
        Selvin Xavier <selvin.xavier@broadcom.com>,
        Steven Rostedt <rostedt@goodmis.org>,
        Vishnu Dasa <vdasa@vmware.com>,
        Yishai Hadas <yishaih@nvidia.com>
References: <cover.1673873422.git.leon@kernel.org>
 <20230118073654.GC27048@lst.de>
From:   Max Gurtovoy <mgurtovoy@nvidia.com>
In-Reply-To: <20230118073654.GC27048@lst.de>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit
X-ClientProxiedBy: LO4P123CA0194.GBRP123.PROD.OUTLOOK.COM
 (2603:10a6:600:1a4::19) To DM4PR12MB5040.namprd12.prod.outlook.com
 (2603:10b6:5:38b::19)
MIME-Version: 1.0
X-MS-PublicTrafficType: Email
X-MS-TrafficTypeDiagnostic: DM4PR12MB5040:EE_|LV2PR12MB5846:EE_
X-MS-Office365-Filtering-Correlation-Id: 0465538a-5bc5-4ef5-5468-08daf95f2b56
X-MS-Exchange-SenderADCheck: 1
X-MS-Exchange-AntiSpam-Relay: 0
X-Microsoft-Antispam: BCL:0;
X-Microsoft-Antispam-Message-Info:  
X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:DM4PR12MB5040.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230022)(4636009)(366004)(39860400002)(346002)(396003)(136003)(376002)(451199015)(83380400001)(38100700002)(5660300002)(2906002)(86362001)(4326008)(66476007)(66946007)(8676002)(66556008)(7416002)(8936002)(31696002)(2616005)(478600001)(26005)(53546011)(186003)(6506007)(6512007)(107886003)(316002)(54906003)(6486002)(110136005)(6666004)(41300700001)(31686004)(66899015)(36756003)(45980500001)(43740500002);DIR:OUT;SFP:1101;
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?eFR5bFVSNUxvbHBMSU1ac3IrNVYwQmNDUTZjOUFVNE82bUsyYVJiNWZUVHF5?=
 =?utf-8?B?OEpkNnRYUmNtRitjWlBianlrU2kyTnFMNlNaR0VhZkhZY2NFRndHUzZTYjFh?=
 =?utf-8?B?NDFOVTdiWmZDVUhUc1BYS2hHQjlOWllSS09IQWxOVlpMUG54VlNZYVh6VXY3?=
 =?utf-8?B?UFNqR1JpeWVzRDRST29ENlR4blR0WmZvMTNzY1h5ajRpTmlkNk0xMTFFOUNQ?=
 =?utf-8?B?MjByRDFTZDZmRzVPanltaVlLSUtIN0Z4T3B2dnJXbFkzUzR0Nm9lNmFQcTZU?=
 =?utf-8?B?OFZWcjJ1OU9QWVRGRE81UEd4NVRnQWZpRTduQ0F6b3A1VWliMmNXYmx0S04v?=
 =?utf-8?B?SkZTOWVsK0VlaWRHWlcrdWRCdHF4eG9zNGpNVEZvbXFFWllOL3hIL09SU1Ft?=
 =?utf-8?B?N2VNTmVpbTM3cnBRM2s5bHhQb1pHRDVZaTdwSEpUT092aHlGODZFMXRHRWVZ?=
 =?utf-8?B?RlpzMTZheFN4UWhwTHJpcHgxdTNxK3ZZWVpuTkZodkgzZk5vZm9Rbmg1aEVE?=
 =?utf-8?B?N3Z4VXFPak9pZytudElMRmhMaTVTbzlvRmlFckFrL2RZRkhmdGpiUmpJdFhu?=
 =?utf-8?B?dnBCVmtQQlo1YkNsdFI3M3gzM3NBamEvQkNuU0J4Zk5iMEdJbHQxMjU5SVAr?=
 =?utf-8?B?dUZOaFZjMkRCdE8rOHkyR0ZDcmN1bWk0Y1Y4NU9DcklHTjdsT2lubVlocXFZ?=
 =?utf-8?B?dUU1MDk0N3lZYzZtMVI4cmgyYWpsdmRkektvZE5JK3orSXBraE9GeXg0bGhE?=
 =?utf-8?B?TVgrc0dXNWc2UlhhRVBOcjNFQnFqbUlTR2Z4WHJFMzVqWHRXb0ZIUWVVTU9B?=
 =?utf-8?B?NkplSkxKRGJSNmxuRisvRU5UMlg5VlNybXVpU3VuZnQrY3BYamMxWExoRXdC?=
 =?utf-8?B?OS9uWlQ3dDB2MVhWSElWdVNpSEZSU1VrV3NxNEJUUHBGU3dMYTl3dmlNNFNF?=
 =?utf-8?B?K0ROZEZRUmF2aWh3ckNOU2loUDhOSUpsL0R3UzNTRVBza2RYaFJyMDkxVHpK?=
 =?utf-8?B?Zm0yRXU0R2ozR2RlOXpMWlBpaWVrZTRSMG9KZzU1Tjd3bnBTOWhVb2YwOCtY?=
 =?utf-8?B?M0hyT29PR2hQN1FmT3NWTGQyYkVLMWptT0xuZ0N0eHpoQzNhdDVBYnlIVERC?=
 =?utf-8?B?blVOOVN3RDBpVGdyZy9kTytnQVJEd2p4ZmVPT1RCdTI4eGZWU2xJOW1QdlZl?=
 =?utf-8?B?YkRwcWdHdG1BeW8wN3FKend6WTg3MlF2Mys3VnJXOXFnYWVNdHpmSFl6Rndp?=
 =?utf-8?B?ZDlLZ1Y2WUU3YXRqS0hBZXpRelQzRDhwQmIvZlhWVVRVdkxCcTg2dUV5SXVu?=
 =?utf-8?B?cnVPMjd3TnluNUJWdGVUSUNkSmFXbEk4R2k4c1RrdFNlZnlONzVZN084VkpC?=
 =?utf-8?B?RHYwUHVZVlZURnB3VXJXSUtiMjBXbXRYVW15RURXc0ZUdWxZWHR3MlVSNThE?=
 =?utf-8?B?dUFTUDJpT3ZVZWJoYzgxUVFpV1Mvcmlqd204MU9zUnkyQmpBanBzQzlsSUZz?=
 =?utf-8?B?ZjB4U0V3VmsxVUJvWHlUZ2JDcHI5UUM0ZnRtbUNCdTF3U2tCaUNNejBja21M?=
 =?utf-8?B?VGpoWU02TW9BeHhrMnJreHpkeEJFRWFYNjNoVnBmT3lRZ3NOSHBzSklxZEdx?=
 =?utf-8?B?V3pQaHBlbFZlRjIzTWlNdkV1amZZckZJS2RCRS9yNmVycWZ0S1A3cWlqbTlw?=
 =?utf-8?B?L0E3d2pqdEVXblYvME55bmFWSmdSSTJFb3llbUNLbEcveDYrd3Z3aTlsM2lI?=
 =?utf-8?B?K2FpanlRL2Q1RTAxUzIxMWdUb25JaWVRNzV4cTVHQitJMDlIb24xYXE4R0gz?=
 =?utf-8?B?a0JPb3NjeTltSDB4b0ZDaWNxckNCbzJ3SUUwSWduMXJMM0ZQYzhrcmRQRWxi?=
 =?utf-8?B?cTJJSzN0RzBVaEoweHp0WlU0NFlxV3RhdzVNditYSit5QlhlZHlTUkFhbTgr?=
 =?utf-8?B?Wmk2T3VUUy9UM3lmejAyZVhEL21ydk1Kb1pZV0FpbzBSMWJpZVNQUzdaOC9i?=
 =?utf-8?B?dVJoR0F3VzNBOEVSODFvbE10MHpycTl4SXBXOGpsWEtlSEwwTzZReTJiMHFP?=
 =?utf-8?B?ZUNGKys5aklaR2NNTHJBanJXN1dUcmI5b1NNZ0VQais5U3pFbTdRcE5nRDIr?=
 =?utf-8?B?V1E2VkxGdzVRdVB6WTZjVi9LZ3pQSUZaZWpiMy9VK2FaVnZqUWpZMkE5R3VL?=
 =?utf-8?B?eUE9PQ==?=
X-OriginatorOrg: Nvidia.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 0465538a-5bc5-4ef5-5468-08daf95f2b56
X-MS-Exchange-CrossTenant-AuthSource: DM4PR12MB5040.namprd12.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 18 Jan 2023 14:20:37.4519
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: mON4edscv0JRPJmX8BQSu/hP3sHmPbUQLbMY5tQb1I64bA0DTlRFy0lZehOWKQNHgyRCfDEtxm4ftCn+ofsoaw==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: LV2PR12MB5846
Precedence: bulk
List-ID: <netdev.vger.kernel.org>
X-Mailing-List: netdev@vger.kernel.org


On 18/01/2023 9:36, Christoph Hellwig wrote:
> On Mon, Jan 16, 2023 at 03:05:47PM +0200, Leon Romanovsky wrote:
>> >From Israel,
>>
>> The purpose of this patchset is to add support for inline
>> encryption/decryption of the data at storage protocols like nvmf over
>> RDMA (at a similar way like integrity is used via unique mkey).
>>
>> This patchset adds support for plaintext keys. The patches were tested
>> on BF-3 HW with fscrypt tool to test this feature, which showed reduce
>> in CPU utilization when comparing at 64k or more IO size. The CPU utilization
>> was improved by more than 50% comparing to the SW only solution at this case.
> One thing that needs to be solved before we can look into this is the
> interaction with protection information (or integrity data in Linux
> terms).  Currently inline encryption and protection information are
> mutally incompatible.

Well, for sure this should be solved. Not sure that it should be done 
before this series.

This patch set doesn't break/change existing behavior of PI and 
Encryption offloads so the above can be done incrementally.
It's already big enough series and 2 subsystems are involved.

Today, if one would like to run both features using a capable device 
then the PI will be offloaded (no SW fallback) and the Encryption will 
be using SW fallback.
There should be a serious instrumentation in the block layer to make 
both operations offloaded.
We'll start looking into it. Any suggestions and designs are welcomed.

We also prepared patches to extend the blk lavel encryption (in addition 
to fscrypt exist today).


From mboxrd@z Thu Jan  1 00:00:00 1970
Return-Path: <netdev-owner@vger.kernel.org>
X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on
	aws-us-west-2-korg-lkml-1.web.codeaurora.org
Received: from vger.kernel.org (vger.kernel.org [23.128.96.18])
	by smtp.lore.kernel.org (Postfix) with ESMTP id 84258C05027
	for <netdev@archiver.kernel.org>; Mon, 23 Jan 2023 11:27:18 +0000 (UTC)
Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand
        id S231838AbjAWL1R (ORCPT <rfc822;netdev@archiver.kernel.org>);
        Mon, 23 Jan 2023 06:27:17 -0500
Received: from lindbergh.monkeyblade.net ([23.128.96.19]:60254 "EHLO
        lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org
        with ESMTP id S229557AbjAWL1Q (ORCPT
        <rfc822;netdev@vger.kernel.org>); Mon, 23 Jan 2023 06:27:16 -0500
Received: from mail-wm1-f43.google.com (mail-wm1-f43.google.com [209.85.128.43])
        by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 04D60233FD;
        Mon, 23 Jan 2023 03:27:15 -0800 (PST)
Received: by mail-wm1-f43.google.com with SMTP id d4-20020a05600c3ac400b003db1de2aef0so8294471wms.2;
        Mon, 23 Jan 2023 03:27:14 -0800 (PST)
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=1e100.net; s=20210112;
        h=content-transfer-encoding:in-reply-to:from:references:cc:to
         :content-language:subject:user-agent:mime-version:date:message-id
         :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to;
        bh=4yTdcDBJTrs5Cb/iupgl8JwyhPAoj/hrBf//3NRcvVg=;
        b=lVec4DTVF5z/qPr2zRKruSAD81T/QgRBukIE1vI/EBKwcPV79CP1utJcaQs6Qdvnjr
         otAzdUu1bFw4vtKJqwqkOUYvZOkhJ+bfbrx6jPNM/5exoNVV1BflpImODESZVXCScr3c
         tVxtYHPU6M5NwfgICzzLFoI6E1OfEuGSA27uto9jsrzfaKEpfgk7i1GVNt0IkrIYHzLH
         9r+HKdCxfWfLDlsPYtQy9SOHiDjheEHhDc3NimxI/v3JgsJCMvHejHSLJRJQo4UOLabu
         YgVqYnRwHE/4frs2/kmI4a91wpGpcPQFQp44+b5lOH2JWv/X62iXLl37dseIYXFqdsIN
         z8zA==
X-Gm-Message-State: AFqh2koAdau4LYRGE2PQ6OSQldiT7n2D/0Pbh5RCcF3jPc1OfC0sfirZ
        TdmOlDdDC4pD1IyByYZ+W98=
X-Google-Smtp-Source: AMrXdXvdjw20Lv7I4+W+Hg4dvw9yXtwgL3LEooExW5vSfd6qFA+Kr64gkQWRX15yiaon5VnaaEWpJA==
X-Received: by 2002:a05:600c:3551:b0:3d2:813:138a with SMTP id i17-20020a05600c355100b003d20813138amr31584379wmq.35.1674473233474;
        Mon, 23 Jan 2023 03:27:13 -0800 (PST)
Received: from [192.168.64.80] (bzq-219-42-90.isdn.bezeqint.net. [62.219.42.90])
        by smtp.gmail.com with ESMTPSA id q7-20020a05600c46c700b003c6bbe910fdsm13400157wmo.9.2023.01.23.03.27.10
        (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128);
        Mon, 23 Jan 2023 03:27:12 -0800 (PST)
Message-ID: <6f9da88c-f01e-156b-eb19-0b275c46c6b5@grimberg.me>
Date:   Mon, 23 Jan 2023 13:27:10 +0200
MIME-Version: 1.0
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101
 Thunderbird/102.4.2
Subject: Re: [PATCH rdma-next 00/13] Add RDMA inline crypto support
Content-Language: en-US
To:     Leon Romanovsky <leon@kernel.org>, Jason Gunthorpe <jgg@nvidia.com>
Cc:     Bryan Tan <bryantan@vmware.com>, Christoph Hellwig <hch@lst.de>,
        Eric Dumazet <edumazet@google.com>,
        Israel Rukshin <israelr@nvidia.com>,
        Jakub Kicinski <kuba@kernel.org>, Jens Axboe <axboe@fb.com>,
        Keith Busch <kbusch@kernel.org>, linux-kernel@vger.kernel.org,
        linux-nvme@lists.infradead.org, linux-rdma@vger.kernel.org,
        linux-trace-kernel@vger.kernel.org,
        Masami Hiramatsu <mhiramat@kernel.org>,
        Max Gurtovoy <mgurtovoy@nvidia.com>, netdev@vger.kernel.org,
        Paolo Abeni <pabeni@redhat.com>,
        Saeed Mahameed <saeedm@nvidia.com>,
        Selvin Xavier <selvin.xavier@broadcom.com>,
        Steven Rostedt <rostedt@goodmis.org>,
        Vishnu Dasa <vdasa@vmware.com>,
        Yishai Hadas <yishaih@nvidia.com>
References: <cover.1673873422.git.leon@kernel.org>
From:   Sagi Grimberg <sagi@grimberg.me>
In-Reply-To: <cover.1673873422.git.leon@kernel.org>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 8bit
Precedence: bulk
List-ID: <netdev.vger.kernel.org>
X-Mailing-List: netdev@vger.kernel.org


>  From Israel,
> 
> The purpose of this patchset is to add support for inline
> encryption/decryption of the data at storage protocols like nvmf over
> RDMA (at a similar way like integrity is used via unique mkey).
> 
> This patchset adds support for plaintext keys. The patches were tested
> on BF-3 HW with fscrypt tool to test this feature, which showed reduce
> in CPU utilization when comparing at 64k or more IO size. The CPU utilization
> was improved by more than 50% comparing to the SW only solution at this case.
> 
> How to configure fscrypt to enable plaintext keys:
>   # mkfs.ext4 -O encrypt /dev/nvme0n1
>   # mount /dev/nvme0n1 /mnt/crypto -o inlinecrypt
>   # head -c 64 /dev/urandom > /tmp/master_key
>   # fscryptctl add_key /mnt/crypto/ < /tmp/master_key
>   # mkdir /mnt/crypto/test1
>   # fscryptctl set_policy 152c41b2ea39fa3d90ea06448456e7fb /mnt/crypto/test1
>     ** “152c41b2ea39fa3d90ea06448456e7fb” is the output of the
>        “fscryptctl add_key” command.
>   # echo foo > /mnt/crypto/test1/foo
> 
> Notes:
>   - At plaintext mode only, the user set a master key and the fscrypt
>     driver derived from it the DEK and the key identifier.
>   - 152c41b2ea39fa3d90ea06448456e7fb is the derived key identifier
>   - Only on the first IO, nvme-rdma gets a callback to load the derived DEK.
> 
> There is no special configuration to support crypto at nvme modules.

Hey, this looks sane to me in a very first glance.

Few high level questions:
- what happens with multipathing? when if not all devices are
capable. SW fallback?
- Does the crypt stuff stay intact when bio is requeued?

I'm assuming you tested this with multipathing? This is not very
useful if it is incompatible with it.

From mboxrd@z Thu Jan  1 00:00:00 1970
Return-Path: <netdev-owner@vger.kernel.org>
X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on
	aws-us-west-2-korg-lkml-1.web.codeaurora.org
Received: from vger.kernel.org (vger.kernel.org [23.128.96.18])
	by smtp.lore.kernel.org (Postfix) with ESMTP id 96B34C38142
	for <netdev@archiver.kernel.org>; Mon, 23 Jan 2023 12:57:58 +0000 (UTC)
Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand
        id S231983AbjAWM55 (ORCPT <rfc822;netdev@archiver.kernel.org>);
        Mon, 23 Jan 2023 07:57:57 -0500
Received: from lindbergh.monkeyblade.net ([23.128.96.19]:60868 "EHLO
        lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org
        with ESMTP id S231917AbjAWM54 (ORCPT
        <rfc822;netdev@vger.kernel.org>); Mon, 23 Jan 2023 07:57:56 -0500
Received: from NAM10-MW2-obe.outbound.protection.outlook.com (mail-mw2nam10on2041.outbound.protection.outlook.com [40.107.94.41])
        by lindbergh.monkeyblade.net (Postfix) with ESMTPS id D87526E91;
        Mon, 23 Jan 2023 04:57:31 -0800 (PST)
ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none;
 b=LZ7hmf9aIEDjBTsURem33U/6cCSPc+mwQM7zdXzsry1zl8yHbMHZQVmBqnk/D4LL2rP6yJY+4LyagMn9HrCkzDShh4VdlK9ikMxRjZbUrMKel1ngvqhAk+q3pSbhLFs7STb1+Ka2n8gnyVDlaJxS4QZLtcSeKm4qCTGdtKc4cyp8LBR5hbdZ2sjo/KbOq4u41U7Wn92F8L4rBSRq+Ks3+ejKUQOqTjYW5TBhfiPy9TWENpMaOl6bWi0ZlUJn6R30j1I25jBBMKAX3TmcpO2BClcA+AmEjBKKHghedIonixi7z+IxoQ/wdaXRisJSqOSveJoCmJCq+ReSFVD2u6jYIA==
ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com;
 s=arcselector9901;
 h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1;
 bh=8oH81ja54IaX+Zu/3u41kSiSAeCse4yAf5fOkQMZdbE=;
 b=f8VajZ5j4pfbNdWX4lQLq/91nBK5HAYnh3vumnMS/AusPZGd/ehNY8pCQg39cxKDkWouLSZ6yjaG5nHR7GweVo7he9vPSIzcs7fpPlk+Q4ifBhXnKwiYYbFExR2mk+u1eEP1hwHPgRiG1SFNyclhq5WwL7P84WpbPSgQJfykvSbXwvNnF28Wua3upaG4jx3ecZUIIZ3nhTVvYr0tR01EG/efkmiEoNGnwztYwLDYQZV6rX/rf54jfl3S+OZzOQCMCOdxQbECFN1m753mlNbw0zi1d2Dc8y5cmQCMmLWc9nKCkReNWTpY2HtyoRnnBOSySPTrQVysrOSGeQOgD05NWA==
ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass
 smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com;
 dkim=pass header.d=nvidia.com; arc=none
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com;
 s=selector2;
 h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck;
 bh=8oH81ja54IaX+Zu/3u41kSiSAeCse4yAf5fOkQMZdbE=;
 b=rV/9xDED0pqJqp/kEh+TzpTyGmJgfU16Ck0LzVqdRjMvm8WbtZaevGCx3hHILeTWlD4qYmhV1wLXI1x60xK3+zu7RTGq3QRNd6Zl7GL0A7a26cBni0PIdHwlEs4s8IQEsf5wAhDCxSddLMnfkJbo4EOqtM1BlIV1IYdVgq0kk3rrVifjfodbV4XaAZjFwmVjOa3KV80uce7SZfd9XEWIz7rh0Gm6nISRoPVKs7OE+XtOlamzzBNgyuhQhB/p4BWPYRbWw9yX5Z6ZzmnSXLDo3HlzeLjfgwEEPraUA20PAt9O86eLTebGxIuJfcBinoVho1gMhuj6qng7DJ8aKbTNBg==
Authentication-Results: dkim=none (message not signed)
 header.d=none;dmarc=none action=none header.from=nvidia.com;
Received: from MW3PR12MB4409.namprd12.prod.outlook.com (2603:10b6:303:2d::23)
 by SJ0PR12MB5438.namprd12.prod.outlook.com (2603:10b6:a03:3ba::23) with
 Microsoft SMTP Server (version=TLS1_2,
 cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6002.33; Mon, 23 Jan
 2023 12:57:30 +0000
Received: from MW3PR12MB4409.namprd12.prod.outlook.com
 ([fe80::f803:f951:a68f:663a]) by MW3PR12MB4409.namprd12.prod.outlook.com
 ([fe80::f803:f951:a68f:663a%6]) with mapi id 15.20.6002.033; Mon, 23 Jan 2023
 12:57:30 +0000
Message-ID: <e37fe712-4e0e-229a-a07e-52a0d486819c@nvidia.com>
Date:   Mon, 23 Jan 2023 14:57:18 +0200
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:102.0) Gecko/20100101
 Thunderbird/102.6.1
Subject: Re: [PATCH rdma-next 00/13] Add RDMA inline crypto support
To:     Sagi Grimberg <sagi@grimberg.me>,
        Leon Romanovsky <leon@kernel.org>,
        Jason Gunthorpe <jgg@nvidia.com>
Cc:     Bryan Tan <bryantan@vmware.com>, Christoph Hellwig <hch@lst.de>,
        Eric Dumazet <edumazet@google.com>,
        Jakub Kicinski <kuba@kernel.org>, Jens Axboe <axboe@fb.com>,
        Keith Busch <kbusch@kernel.org>, linux-kernel@vger.kernel.org,
        linux-nvme@lists.infradead.org, linux-rdma@vger.kernel.org,
        linux-trace-kernel@vger.kernel.org,
        Masami Hiramatsu <mhiramat@kernel.org>,
        Max Gurtovoy <mgurtovoy@nvidia.com>, netdev@vger.kernel.org,
        Paolo Abeni <pabeni@redhat.com>,
        Saeed Mahameed <saeedm@nvidia.com>,
        Selvin Xavier <selvin.xavier@broadcom.com>,
        Steven Rostedt <rostedt@goodmis.org>,
        Vishnu Dasa <vdasa@vmware.com>,
        Yishai Hadas <yishaih@nvidia.com>
References: <cover.1673873422.git.leon@kernel.org>
 <6f9da88c-f01e-156b-eb19-0b275c46c6b5@grimberg.me>
Content-Language: en-US
From:   Israel Rukshin <israelr@nvidia.com>
In-Reply-To: <6f9da88c-f01e-156b-eb19-0b275c46c6b5@grimberg.me>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 8bit
X-ClientProxiedBy: LO4P265CA0194.GBRP265.PROD.OUTLOOK.COM
 (2603:10a6:600:318::9) To MW3PR12MB4409.namprd12.prod.outlook.com
 (2603:10b6:303:2d::23)
MIME-Version: 1.0
X-MS-PublicTrafficType: Email
X-MS-TrafficTypeDiagnostic: MW3PR12MB4409:EE_|SJ0PR12MB5438:EE_
X-MS-Office365-Filtering-Correlation-Id: b6380cc0-602c-4a8b-1dbe-08dafd41629e
X-MS-Exchange-SenderADCheck: 1
X-MS-Exchange-AntiSpam-Relay: 0
X-Microsoft-Antispam: BCL:0;
X-Microsoft-Antispam-Message-Info:  
X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:MW3PR12MB4409.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230022)(4636009)(366004)(39860400002)(396003)(346002)(136003)(376002)(451199015)(31686004)(36756003)(86362001)(66556008)(66476007)(8676002)(2906002)(5660300002)(66946007)(4326008)(7416002)(31696002)(8936002)(38100700002)(6666004)(478600001)(107886003)(316002)(54906003)(110136005)(6636002)(6486002)(41300700001)(53546011)(26005)(6512007)(83380400001)(186003)(2616005)(6506007)(43740500002)(45980500001);DIR:OUT;SFP:1101;
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?R0VzSjlXN0VsbUtwclF0c1AyZGJnQTJrUFlGN1FoNGZzMGd5SlBFZUpMQTFk?=
 =?utf-8?B?OURvZ1pjTTh0VGJtNnR2bEIrUHlkNGppQjd2VXVTOEMxa0pWRDh5Tm1xNXBa?=
 =?utf-8?B?LzdiaFBScU1ycCtqcW5XZk5TdVZkT2wxT2M0U1cwN0hvQllySC9jbVFVK3JV?=
 =?utf-8?B?T1l0d1RlU0duMWV2ajFjZVBRSzFuRUxtK3p4bExaWFAwOG9DbEpwZllTY3Za?=
 =?utf-8?B?cWxNSDc5THBNdXR4b25Db05KeHB4b0RyWGFXd1JEdzFOVXVUS3pVU01KS0FV?=
 =?utf-8?B?eWQvcVlidEV0Z2JDc1pxWUZxdGxtd052TlBybm5qK08zZHJmYW8weFl2aGxN?=
 =?utf-8?B?aG92aVdnRDJudk9rK0s0RkdBUTQzaVZsZVMyRXlXbXlsODFWdHp0K3lhQjdq?=
 =?utf-8?B?SFUzN1d5dlRYN2NxUStrQ2VZQUFjaXkyaHpER0pGbkc2YmJhODluYVk2d1Vh?=
 =?utf-8?B?bnA2NHN3dlRFYXBWbWhHODVOZExHdHFwRFhCVjdnMDJWMVNTaFo1Ky9ha2k1?=
 =?utf-8?B?QlZCVHI0SXdIQ2t5UWpCUFgrbGVCL0IzR3JUSk9Kd1BzQURReG0rK3Buam9o?=
 =?utf-8?B?RzBMUk96bXJrNWJaVFRLeVFXNm1UQllMeWRjQzhlR1ZIazdDYlFpZU1TTHl6?=
 =?utf-8?B?Lys4cDRFKzRSRFJxR3JuRXhOU1VmRVRRMW1QcUdDME5lYnNJRkRKQUQ3RFRO?=
 =?utf-8?B?TERBcnZwZGR6OTdIb2hZRHVHdDgvY2swMjV0U3htc3BDdHhvUUdhUkh4WXlJ?=
 =?utf-8?B?bXNTUmVxVnErQ1V6emlBa0ZLZzNPYWUyR0toa0lhRThXNEdMaGJmSXlQU0Nr?=
 =?utf-8?B?dExWTVFqVFBCcThoV1d2bVhlWnpEQkRNeW02YTZxa0F1RVBNMlYzd2pQWEZB?=
 =?utf-8?B?eWwwMVpCZFJ0V1NwRWpkOEdORG5tZTRKanVZb2NUM0wrMFpuNmpRUXRFQmI5?=
 =?utf-8?B?YnIwamtIQlVuMHFia1BkdzliZmVjRVNEMkxBZ0tIOHNZZVdSVUk3NzhFdStE?=
 =?utf-8?B?VDFkdUtXYmJSQXRCem9sUWpRSTFCWlVVajdQVEVrZmtYMnhMSTdHZ2s4dHlM?=
 =?utf-8?B?M1I0RU45RmEvZTFnT05naUFBd1E0dTVLNmVoaTJNMFo1cEJNQkxXMmlUNTJz?=
 =?utf-8?B?RjFYQ2w1TUE4VWVFd0M2RnBXYzhYbzZJK3RVaStUQktYSERjdmxZVlA0V2Qr?=
 =?utf-8?B?QUNrNXozVnNjYTlMU3k2NFNaNGMxYm5vOXd2dTBoK01LU0J3QWpoMFVGYWtu?=
 =?utf-8?B?cmJSdmd5YkxVRmVCV2diSk56Q3cxakxnVUdkY01jSmFwL1NNQlptUEQ5dVpU?=
 =?utf-8?B?MWxHbCtVTGZCekZCaEo1TGhVYUpSWGtrTHBQdjNCVmVWWXVKKzlXNGg5eC8z?=
 =?utf-8?B?TkxCdUNzcklaUjlVRW9TVVNXdVp4VCtrWXl4N29vWWx4SjlSNEYxdGhrcXJh?=
 =?utf-8?B?TXVKNFNjb0RIdUpBbXZ6TWd4QXpPZk1zWHRKSVZ5eTMrUTJTczRMQkp3L3Nz?=
 =?utf-8?B?WjBjdUx1d2lGUHFOWnErQ3B6VWFSS20yVjhjRGJLQUhDRXprRnkraS9SY21T?=
 =?utf-8?B?bGhIeng3K0lPbTZXQXRjdDNRSTkvVWFhbzdtWHc5dTJaWDA4cStlOTZHMUdS?=
 =?utf-8?B?VG16NzRxblg0ejYxY25nQzQxZm9HSnlSZzlLbkUzTUU3cW5FYlNtRzhrL3ZV?=
 =?utf-8?B?RGhPclk4MUdGZUZ5c21qaDRvTHJkdnYzc3pHSzlia1hFTlFSTFJ6bDJ0U1pX?=
 =?utf-8?B?dVpEY25zR1V6aTVRcjJmQ0g5dDZCbUt1c3pkK2tCN0htR1ZsdGlhTkNuNlR0?=
 =?utf-8?B?OU9wUzNFZFlCYVNoTHNRYnlqOVBkcS9JTVk1NGF4TW5wU0ZVMEdXbTcwKy9z?=
 =?utf-8?B?eWp5WkhRaWtqQ255RisrcU9RazJjUHU1ZFBWZy9QYmNyaUpXY2luc0VLL3hz?=
 =?utf-8?B?NFA1TmlFeWR2ZU9XNklrZVBENUExdnNuS3dkMVdvdnhhcHZnQSs3UmI1aEps?=
 =?utf-8?B?QXlJblNLbjNVZTI4ZWUxRmhYOVpLMlpZNVVHQk44RHFueE93TkxMK2FTWEZv?=
 =?utf-8?B?dVlQb3llOU9PYlZqVWR4Tld4UTNSTWpTT1VzMFhZUHlyTTNoV2x0RHRCalAr?=
 =?utf-8?Q?p1ZrOtVKx6m2O0KpptCXRmnDu?=
X-OriginatorOrg: Nvidia.com
X-MS-Exchange-CrossTenant-Network-Message-Id: b6380cc0-602c-4a8b-1dbe-08dafd41629e
X-MS-Exchange-CrossTenant-AuthSource: MW3PR12MB4409.namprd12.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 23 Jan 2023 12:57:30.0108
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: gXI5E2EUx/sl4Q7tXzvGLBcxsyOJZpPcPYYi5ldrRXHlKWn8a0YKfselsVnifixPznHCQiVANqPBrmGcQFZpvg==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: SJ0PR12MB5438
Precedence: bulk
List-ID: <netdev.vger.kernel.org>
X-Mailing-List: netdev@vger.kernel.org

Hi Sagi,

On 1/23/2023 1:27 PM, Sagi Grimberg wrote:
>
>>  From Israel,
>>
>> The purpose of this patchset is to add support for inline
>> encryption/decryption of the data at storage protocols like nvmf over
>> RDMA (at a similar way like integrity is used via unique mkey).
>>
>> This patchset adds support for plaintext keys. The patches were tested
>> on BF-3 HW with fscrypt tool to test this feature, which showed reduce
>> in CPU utilization when comparing at 64k or more IO size. The CPU 
>> utilization
>> was improved by more than 50% comparing to the SW only solution at 
>> this case.
>>
>> How to configure fscrypt to enable plaintext keys:
>>   # mkfs.ext4 -O encrypt /dev/nvme0n1
>>   # mount /dev/nvme0n1 /mnt/crypto -o inlinecrypt
>>   # head -c 64 /dev/urandom > /tmp/master_key
>>   # fscryptctl add_key /mnt/crypto/ < /tmp/master_key
>>   # mkdir /mnt/crypto/test1
>>   # fscryptctl set_policy 152c41b2ea39fa3d90ea06448456e7fb 
>> /mnt/crypto/test1
>>     ** “152c41b2ea39fa3d90ea06448456e7fb” is the output of the
>>        “fscryptctl add_key” command.
>>   # echo foo > /mnt/crypto/test1/foo
>>
>> Notes:
>>   - At plaintext mode only, the user set a master key and the fscrypt
>>     driver derived from it the DEK and the key identifier.
>>   - 152c41b2ea39fa3d90ea06448456e7fb is the derived key identifier
>>   - Only on the first IO, nvme-rdma gets a callback to load the 
>> derived DEK.
>>
>> There is no special configuration to support crypto at nvme modules.
>
> Hey, this looks sane to me in a very first glance.
>
> Few high level questions:
> - what happens with multipathing? when if not all devices are
> capable. SW fallback?
SW fallback happens every time the device doesn't support the specific 
crypto request (which include data-unit-size, mode and dun_bytes).
So with multipathing, one path uses the HW crypto offload and the other 
one uses the SW fallback.
> - Does the crypt stuff stay intact when bio is requeued?
Yes, the crypto ctx is copied when cloning the bio.
>
> I'm assuming you tested this with multipathing? This is not very
> useful if it is incompatible with it.
Yes, sure.
You can see the call to  blk_crypto_reprogram_all_keys(), which is 
called when the controller was reconnected after port toggling.

- Israel



From mboxrd@z Thu Jan  1 00:00:00 1970
Return-Path: <netdev-owner@vger.kernel.org>
X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on
	aws-us-west-2-korg-lkml-1.web.codeaurora.org
Received: from vger.kernel.org (vger.kernel.org [23.128.96.18])
	by smtp.lore.kernel.org (Postfix) with ESMTP id 1E9C5C61DA4
	for <netdev@archiver.kernel.org>; Mon, 30 Jan 2023 12:35:17 +0000 (UTC)
Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand
        id S236886AbjA3MfP (ORCPT <rfc822;netdev@archiver.kernel.org>);
        Mon, 30 Jan 2023 07:35:15 -0500
Received: from lindbergh.monkeyblade.net ([23.128.96.19]:33826 "EHLO
        lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org
        with ESMTP id S236337AbjA3MfO (ORCPT
        <rfc822;netdev@vger.kernel.org>); Mon, 30 Jan 2023 07:35:14 -0500
Received: from verein.lst.de (verein.lst.de [213.95.11.211])
        by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 78EE14211;
        Mon, 30 Jan 2023 04:35:10 -0800 (PST)
Received: by verein.lst.de (Postfix, from userid 2407)
        id 1ED3168D09; Mon, 30 Jan 2023 13:35:06 +0100 (CET)
Date:   Mon, 30 Jan 2023 13:35:05 +0100
From:   Christoph Hellwig <hch@lst.de>
To:     Max Gurtovoy <mgurtovoy@nvidia.com>
Cc:     Christoph Hellwig <hch@lst.de>, Leon Romanovsky <leon@kernel.org>,
        Jason Gunthorpe <jgg@nvidia.com>,
        Bryan Tan <bryantan@vmware.com>,
        Eric Dumazet <edumazet@google.com>,
        Israel Rukshin <israelr@nvidia.com>,
        Jakub Kicinski <kuba@kernel.org>, Jens Axboe <axboe@fb.com>,
        Keith Busch <kbusch@kernel.org>, linux-kernel@vger.kernel.org,
        linux-nvme@lists.infradead.org, linux-rdma@vger.kernel.org,
        linux-trace-kernel@vger.kernel.org,
        Masami Hiramatsu <mhiramat@kernel.org>, netdev@vger.kernel.org,
        Paolo Abeni <pabeni@redhat.com>,
        Saeed Mahameed <saeedm@nvidia.com>,
        Sagi Grimberg <sagi@grimberg.me>,
        Selvin Xavier <selvin.xavier@broadcom.com>,
        Steven Rostedt <rostedt@goodmis.org>,
        Vishnu Dasa <vdasa@vmware.com>,
        Yishai Hadas <yishaih@nvidia.com>
Subject: Re: [PATCH rdma-next 00/13] Add RDMA inline crypto support
Message-ID: <20230130123505.GA19948@lst.de>
References: <cover.1673873422.git.leon@kernel.org> <20230118073654.GC27048@lst.de> <ac48766a-b861-4fc0-3171-7b23f175c672@nvidia.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <ac48766a-b861-4fc0-3171-7b23f175c672@nvidia.com>
User-Agent: Mutt/1.5.17 (2007-11-01)
Precedence: bulk
List-ID: <netdev.vger.kernel.org>
X-Mailing-List: netdev@vger.kernel.org

On Wed, Jan 18, 2023 at 04:20:27PM +0200, Max Gurtovoy wrote:
> Today, if one would like to run both features using a capable device then 
> the PI will be offloaded (no SW fallback) and the Encryption will be using 
> SW fallback.

It's not just running both - it's offering both as currently registering
these fatures is mutally incompatible.

From mboxrd@z Thu Jan  1 00:00:00 1970
Return-Path: <netdev-owner@vger.kernel.org>
X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on
	aws-us-west-2-korg-lkml-1.web.codeaurora.org
Received: from vger.kernel.org (vger.kernel.org [23.128.96.18])
	by smtp.lore.kernel.org (Postfix) with ESMTP id 3E714C61DA4
	for <netdev@archiver.kernel.org>; Mon, 30 Jan 2023 12:36:19 +0000 (UTC)
Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand
        id S236904AbjA3MgR (ORCPT <rfc822;netdev@archiver.kernel.org>);
        Mon, 30 Jan 2023 07:36:17 -0500
Received: from lindbergh.monkeyblade.net ([23.128.96.19]:35230 "EHLO
        lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org
        with ESMTP id S235273AbjA3MgQ (ORCPT
        <rfc822;netdev@vger.kernel.org>); Mon, 30 Jan 2023 07:36:16 -0500
Received: from verein.lst.de (verein.lst.de [213.95.11.211])
        by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 1C636135;
        Mon, 30 Jan 2023 04:36:14 -0800 (PST)
Received: by verein.lst.de (Postfix, from userid 2407)
        id 0544868D0A; Mon, 30 Jan 2023 13:36:11 +0100 (CET)
Date:   Mon, 30 Jan 2023 13:36:10 +0100
From:   Christoph Hellwig <hch@lst.de>
To:     Israel Rukshin <israelr@nvidia.com>
Cc:     Sagi Grimberg <sagi@grimberg.me>,
        Leon Romanovsky <leon@kernel.org>,
        Jason Gunthorpe <jgg@nvidia.com>,
        Bryan Tan <bryantan@vmware.com>,
        Christoph Hellwig <hch@lst.de>,
        Eric Dumazet <edumazet@google.com>,
        Jakub Kicinski <kuba@kernel.org>, Jens Axboe <axboe@fb.com>,
        Keith Busch <kbusch@kernel.org>, linux-kernel@vger.kernel.org,
        linux-nvme@lists.infradead.org, linux-rdma@vger.kernel.org,
        linux-trace-kernel@vger.kernel.org,
        Masami Hiramatsu <mhiramat@kernel.org>,
        Max Gurtovoy <mgurtovoy@nvidia.com>, netdev@vger.kernel.org,
        Paolo Abeni <pabeni@redhat.com>,
        Saeed Mahameed <saeedm@nvidia.com>,
        Selvin Xavier <selvin.xavier@broadcom.com>,
        Steven Rostedt <rostedt@goodmis.org>,
        Vishnu Dasa <vdasa@vmware.com>,
        Yishai Hadas <yishaih@nvidia.com>
Subject: Re: [PATCH rdma-next 00/13] Add RDMA inline crypto support
Message-ID: <20230130123610.GB19948@lst.de>
References: <cover.1673873422.git.leon@kernel.org> <6f9da88c-f01e-156b-eb19-0b275c46c6b5@grimberg.me> <e37fe712-4e0e-229a-a07e-52a0d486819c@nvidia.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <e37fe712-4e0e-229a-a07e-52a0d486819c@nvidia.com>
User-Agent: Mutt/1.5.17 (2007-11-01)
Precedence: bulk
List-ID: <netdev.vger.kernel.org>
X-Mailing-List: netdev@vger.kernel.org

On Mon, Jan 23, 2023 at 02:57:18PM +0200, Israel Rukshin wrote:
>> - what happens with multipathing? when if not all devices are
>> capable. SW fallback?
> SW fallback happens every time the device doesn't support the specific 
> crypto request (which include data-unit-size, mode and dun_bytes).
> So with multipathing, one path uses the HW crypto offload and the other one 
> uses the SW fallback.

That's a big no-go.  The blk-crypto-fallback code is just a toy example
and not actually safe to use in prodution.  Most importantly it just
kmallocs a bio clone and pages for it without any mempool that guarantees
forward progress.

From mboxrd@z Thu Jan  1 00:00:00 1970
Return-Path: <netdev-owner@vger.kernel.org>
X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on
	aws-us-west-2-korg-lkml-1.web.codeaurora.org
Received: from vger.kernel.org (vger.kernel.org [23.128.96.18])
	by smtp.lore.kernel.org (Postfix) with ESMTP id 31B9FC61DA4
	for <netdev@archiver.kernel.org>; Mon, 30 Jan 2023 14:35:07 +0000 (UTC)
Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand
        id S237179AbjA3OfF (ORCPT <rfc822;netdev@archiver.kernel.org>);
        Mon, 30 Jan 2023 09:35:05 -0500
Received: from lindbergh.monkeyblade.net ([23.128.96.19]:54058 "EHLO
        lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org
        with ESMTP id S237883AbjA3Oev (ORCPT
        <rfc822;netdev@vger.kernel.org>); Mon, 30 Jan 2023 09:34:51 -0500
Received: from NAM10-DM6-obe.outbound.protection.outlook.com (mail-dm6nam10on2073.outbound.protection.outlook.com [40.107.93.73])
        by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 2D63913F;
        Mon, 30 Jan 2023 06:34:12 -0800 (PST)
ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none;
 b=djEFT30QkoJBYpA/gZAl6HAJZbyNdqzmnCH3ofGUGt7ioleaeOh5phjZ28BFRsqFDe3szh9/DKu2BPXBSiR5O7W7sLlaAcOqGRQKW2IScRwQMId8Kx3btD/6lo9q12MVLNAg5QmA98PZPCRW0Ha3sKdnyu9kzvJgtzGzjF0zyp0TZwKJCOHOZFUi1m/NbxnlunMqg0C6LIAxZnJIZC68NryQ2zmHpDoG1mZbR1JZEb58sD+5cbNmwD0IGksIkvTTVVI84M9jOuRB4jB9DTo4/+W/RXs8kZDOazyqaqK8VNi5WfT1uT8Besla0Y1zSdd/DU19Bev3EJ3+CS41zuC0Gw==
ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com;
 s=arcselector9901;
 h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1;
 bh=38noOuXBRVv1yORAes3OgK5ciZ1M5aWFJaSDbjCqkBo=;
 b=lX47hkSQvZbdqR+zwJXufh4NhyKIkCLgPFD4G5AQmgKnWM1/q9jvW13d81RXjW7YkXbYMpw0PiowfqU0OU/ScCw50V3asn1izCJIi8tywFe/QZQWH7eg6hbinOcyNxyE1AfykKzS3MiyWZqsU7Cu5yzRWlQxouRzq8nG2I/EM6EspTnMQvtHEm1nF1gHYAxMRUmxoK28NJqbxThCT5LRC+G4lRG/gf+JQhOSUM57XwoohcxuTFsLxL6dEvLYc80aCLsYR/MMp2FudBlA4/zAaSEuUcqDfUPHlG79tWd4N5ON5a2vTrEofQg1c9L9Xzr3PebN7icROGhP+bIhKZUN7g==
ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass
 smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com;
 dkim=pass header.d=nvidia.com; arc=none
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com;
 s=selector2;
 h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck;
 bh=38noOuXBRVv1yORAes3OgK5ciZ1M5aWFJaSDbjCqkBo=;
 b=aIBUMCqSJk4Lf1HmhXUCAAcof/dzSKDqKd0HQtaYDFJlqCXBefdSN41UKYY1/fzO+YdDOgeFKFZZNIQaZd7tFklXf1mnMr8RvDyWAbELXdtU9lkDaUsxOZFitztxf9+RMm56jPjkbcEoBf9XYgU0d+6urdEHyJAsBSR9nzvjHG1DRMfaCBXgqAhthgIWFdbKimNzLXCPUXeZBvyfLwkeioXyYnYGUTw5qirNxn8FBIn+2GQRKq+7Y0EW6y0msCGNHiLzyTM0Ir66fLguVhXJdqNKaK2NoHv7+k5mgZdAcCj08iqQMdelYuKVgG2XZy0iTDdOt7paDk7gNRS7jIcNiA==
Authentication-Results: dkim=none (message not signed)
 header.d=none;dmarc=none action=none header.from=nvidia.com;
Received: from DM4PR12MB5040.namprd12.prod.outlook.com (2603:10b6:5:38b::19)
 by CH2PR12MB4101.namprd12.prod.outlook.com (2603:10b6:610:a8::22) with
 Microsoft SMTP Server (version=TLS1_2,
 cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6043.36; Mon, 30 Jan
 2023 14:34:09 +0000
Received: from DM4PR12MB5040.namprd12.prod.outlook.com
 ([fe80::9444:21e0:2a65:da95]) by DM4PR12MB5040.namprd12.prod.outlook.com
 ([fe80::9444:21e0:2a65:da95%8]) with mapi id 15.20.6043.036; Mon, 30 Jan 2023
 14:34:08 +0000
Message-ID: <e8c35d5d-d855-a38c-2f3b-6887df208360@nvidia.com>
Date:   Mon, 30 Jan 2023 16:33:59 +0200
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:102.0) Gecko/20100101
 Thunderbird/102.6.1
Subject: Re: [PATCH rdma-next 00/13] Add RDMA inline crypto support
To:     Christoph Hellwig <hch@lst.de>
Cc:     Leon Romanovsky <leon@kernel.org>,
        Jason Gunthorpe <jgg@nvidia.com>,
        Bryan Tan <bryantan@vmware.com>,
        Eric Dumazet <edumazet@google.com>,
        Israel Rukshin <israelr@nvidia.com>,
        Jakub Kicinski <kuba@kernel.org>, Jens Axboe <axboe@fb.com>,
        Keith Busch <kbusch@kernel.org>, linux-kernel@vger.kernel.org,
        linux-nvme@lists.infradead.org, linux-rdma@vger.kernel.org,
        linux-trace-kernel@vger.kernel.org,
        Masami Hiramatsu <mhiramat@kernel.org>, netdev@vger.kernel.org,
        Paolo Abeni <pabeni@redhat.com>,
        Saeed Mahameed <saeedm@nvidia.com>,
        Sagi Grimberg <sagi@grimberg.me>,
        Selvin Xavier <selvin.xavier@broadcom.com>,
        Steven Rostedt <rostedt@goodmis.org>,
        Vishnu Dasa <vdasa@vmware.com>,
        Yishai Hadas <yishaih@nvidia.com>
References: <cover.1673873422.git.leon@kernel.org>
 <20230118073654.GC27048@lst.de>
 <ac48766a-b861-4fc0-3171-7b23f175c672@nvidia.com>
 <20230130123505.GA19948@lst.de>
Content-Language: en-US
From:   Max Gurtovoy <mgurtovoy@nvidia.com>
In-Reply-To: <20230130123505.GA19948@lst.de>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit
X-ClientProxiedBy: VI1PR09CA0142.eurprd09.prod.outlook.com
 (2603:10a6:803:12c::26) To DM4PR12MB5040.namprd12.prod.outlook.com
 (2603:10b6:5:38b::19)
MIME-Version: 1.0
X-MS-PublicTrafficType: Email
X-MS-TrafficTypeDiagnostic: DM4PR12MB5040:EE_|CH2PR12MB4101:EE_
X-MS-Office365-Filtering-Correlation-Id: 382e6da5-a3ce-4b44-b22f-08db02cf0bd3
X-MS-Exchange-SenderADCheck: 1
X-MS-Exchange-AntiSpam-Relay: 0
X-Microsoft-Antispam: BCL:0;
X-Microsoft-Antispam-Message-Info:  
X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:DM4PR12MB5040.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230025)(4636009)(39860400002)(376002)(366004)(346002)(396003)(136003)(451199018)(8936002)(41300700001)(31696002)(86362001)(4744005)(5660300002)(7416002)(83380400001)(2616005)(54906003)(316002)(66556008)(66476007)(66946007)(38100700002)(6916009)(4326008)(8676002)(6666004)(6512007)(186003)(6506007)(53546011)(36756003)(6486002)(478600001)(107886003)(66899018)(31686004)(2906002)(45980500001)(43740500002);DIR:OUT;SFP:1101;
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?eHgxZDR4ZTJmamFQcENpM1NwUjlSeU1rNDJjVVppc2FLSzdGUHRnb2wrUlV6?=
 =?utf-8?B?TjVncEtBVUh3VVhQMFZ5Qm0xTlBrbVNCVnJPTmM3dm9vNVlKQ016NHlydGx3?=
 =?utf-8?B?N3ZCVTB5ZzFGRWFCakdsdUd6dGdSQ1MwYnExdE1NR0JRcklhMG5VWGlTVnNC?=
 =?utf-8?B?RDRodmRGRHZnL0pPUk5BSUdLRFY4aW5ZQm4vNjJlM1dGV0ZmQzRYVnZXVEVa?=
 =?utf-8?B?eWQyYm5YZ2pMc21ESnlBcG42R0hYc2tvbjFocWtiZkdaVmhNNUt0amNxSjBC?=
 =?utf-8?B?L3JleU1wc2FHYkdpa0k5ZlJOZlRjTFpvR05jZ2xXYW5DVkZZTVZUTTRDSHNM?=
 =?utf-8?B?OW5ISHpUVXRXOVY1Smx0WlFzSjlTV1Vxbk5CeEZoYUdYdWIrc1ZyYy9aaTBX?=
 =?utf-8?B?di9yVnArOEN5QnpRZ2dPdmZPTW9qRSszc0Q2eXYzd0MvNlIxUUhET0hOZjZZ?=
 =?utf-8?B?UE1mMWdMNkJnY3dlbXI1WFpOVXdRUzA4aHFaVUJyYlB2bnRpNTFJRkZaOGg5?=
 =?utf-8?B?Z280NkpNWXpka0FvSW94UFF5T0RXWTQwSWNXZTY1RnUyQVcyM0I1QlRXcXlk?=
 =?utf-8?B?czdvcVFYM0ErMmo2OGZtVXpOUDhNUlFUOHZENUVxem9NczBLb0hVVTNpZHFF?=
 =?utf-8?B?cWVMSmE2ZU9wT2M0YWhiOHZMcnZ1dHlPVDlhNkVYYmxLT1lPRmZiU1RNNVBP?=
 =?utf-8?B?TFE1Y3ZIMG0yTkdzanR1c282eFdTL3BmcGdwdzI1TFNWeWloM0FTNThvZnNH?=
 =?utf-8?B?aFJ2M1Z5bDZURUNNa2VVNGlhU3c2OUtJdHFMMWFnL0ZVT1pZWjJDNjJzc0lk?=
 =?utf-8?B?NjY1cS9NR1B1ejlnUTRjYUJyVXRLMDBHU1JWczFCU25LS2k3bVNEOEhsQzVG?=
 =?utf-8?B?d0gxamY2akFISms1OXNjWDJlOXNveGZIeUxkTTlyWmtVVlhMQlozaVc3RGhH?=
 =?utf-8?B?RDlTZmQzbEtQQW01QUkvZ01NS0lNWjlwRXJIUmt1aENtY1NENDQ1MkFzOFMv?=
 =?utf-8?B?MjlTSkx0c2c2bVQreVlrSlRHV29sbkpweXFQMVJkbnd4MjFsRDRLbm1aYVZQ?=
 =?utf-8?B?Q2t1U2VPbFVCcjhPZmQxZlR4czhsaUtTQ1hVSUdPclI3UUpYWThrVkpxNFJr?=
 =?utf-8?B?ajMvRFdRNlBpblRPSUFoWkpBYXZQTkdqQkZJS0k0Rk1keSt0dkFCQ1BaYWxq?=
 =?utf-8?B?WWJPemJmRHJHdmlVd0syQ1QzUXgvNlUvZVJoQ01DK29sK0wzMU9YOSt1ZzND?=
 =?utf-8?B?S2tHbzhJUStqTWFWdExMZG5DeEQ5UjVBNmUzQWZ0YzlMNzFOeGpxeEpnZy9L?=
 =?utf-8?B?WFVSTnRuQ1V5WDZ6bVhyakJEMko2VXRIMWlsRDhIL2hMeC8vVTNKODkwZHF4?=
 =?utf-8?B?Rjc2Rk1pRFJGZ0J2RHhTdVdINi9XZkxsWW9zSElFK1BweTgzNzRlWUFBanM4?=
 =?utf-8?B?ZXZ5MEo3Nk02eUYzVk04QWE4ZmdyZDZPN2lTOHlkRW1iQjdHazJvdXZ5TFJx?=
 =?utf-8?B?TmNWRmlMaUxrU0I0ZU1sVlJ1U1lBb3U4cGlPUTVWZnpCRWd5SkpOY1ovWjRO?=
 =?utf-8?B?ZkU2aHpYNGQxVytHTzcrLzZ5K3phM3FadzlyS3hBdWNBN0xyNUgrSXhERXl4?=
 =?utf-8?B?RkhHNmptbGZYdUJHdlVMZHN0ZnZhdlE2a1dleDZicVFqRGRtdjlIa3phdG54?=
 =?utf-8?B?UFdmUmVYWnpwb2pFeEtWVnJselozR3NQdmNFS1czWWJYaWJSVjhreWxEa2RK?=
 =?utf-8?B?OXpJWGViQ3NmWmdpV3FLZFViMVFRUHNKRzJxTVFTdHN6U1BZanRkYzdGSCtu?=
 =?utf-8?B?aW1HL1FtMXFDQjhobnA3ZmdkOXdwSkFGanBSeHl5WEQ0cGlhQ3UwNG5naU1m?=
 =?utf-8?B?WGZLTFNHUm90QndVUnVTclhTTGY4RWM4TmVRczllOHJtK2EzSWFMa09OZitV?=
 =?utf-8?B?c3ArWUdCSmIyaFNPOG8veUt6WlhLT2RNYVJFN1gwdmZJU1FrZzNSRVgrRHhN?=
 =?utf-8?B?SjVLYWEyakFxc1l0MndncHJVbk9ra2V2UHEvRjRRL2ZPUjlpWGpaaS9IZzBx?=
 =?utf-8?B?NHlDb3kxUkFOY1BmQUwyTzN2SmhUSG9GTzFuY3Jxa0ptR2cvK2VsOW9CR0J2?=
 =?utf-8?B?V1Y0L3NQT0Z1ckl4eWY0WDlYZGF0b3JhUGNmNEhuZitkQmJtTmZER0RhU0kv?=
 =?utf-8?Q?PrvbeuqlS5LIo9t/ytOPfHuh5yhhf32YyoZC7xa7DuJ7?=
X-OriginatorOrg: Nvidia.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 382e6da5-a3ce-4b44-b22f-08db02cf0bd3
X-MS-Exchange-CrossTenant-AuthSource: DM4PR12MB5040.namprd12.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 30 Jan 2023 14:34:08.7965
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: gSt9M8ut6pBntc1TPioguN1P8B5qI/AN9zhdNukOk3bvfj4DF6/EDc1/HWN0UrVCp7iLAYv95J3iN0zLMFfr6A==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: CH2PR12MB4101
Precedence: bulk
List-ID: <netdev.vger.kernel.org>
X-Mailing-List: netdev@vger.kernel.org


On 30/01/2023 14:35, Christoph Hellwig wrote:
> On Wed, Jan 18, 2023 at 04:20:27PM +0200, Max Gurtovoy wrote:
>> Today, if one would like to run both features using a capable device then
>> the PI will be offloaded (no SW fallback) and the Encryption will be using
>> SW fallback.
> It's not just running both - it's offering both as currently registering
> these fatures is mutally incompatible.

For sure we would like to offer both, but as mentioned before, this will 
require blk layer instrumentation and ib_core instrumentation to some 
pipeline_mr or something like that.
We are interested in doing this IO path but doing all in 1 series 
doesn't sounds possible.

First we would like to have at least the mutual exclusive solution since 
there are users that may want only one type of acceleration.

re-designing, both blk layer and ib_core for the PI + CRYPTO case in 
this series is not feasible IMO.



From mboxrd@z Thu Jan  1 00:00:00 1970
Return-Path: <netdev-owner@vger.kernel.org>
X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on
	aws-us-west-2-korg-lkml-1.web.codeaurora.org
Received: from vger.kernel.org (vger.kernel.org [23.128.96.18])
	by smtp.lore.kernel.org (Postfix) with ESMTP id 5BD23C64EC7
	for <netdev@archiver.kernel.org>; Tue, 14 Feb 2023 10:03:41 +0000 (UTC)
Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand
        id S232446AbjBNKDj (ORCPT <rfc822;netdev@archiver.kernel.org>);
        Tue, 14 Feb 2023 05:03:39 -0500
Received: from lindbergh.monkeyblade.net ([23.128.96.19]:42486 "EHLO
        lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org
        with ESMTP id S232102AbjBNKDD (ORCPT
        <rfc822;netdev@vger.kernel.org>); Tue, 14 Feb 2023 05:03:03 -0500
Received: from mail-wr1-f52.google.com (mail-wr1-f52.google.com [209.85.221.52])
        by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 04DEE25B8B;
        Tue, 14 Feb 2023 02:01:20 -0800 (PST)
Received: by mail-wr1-f52.google.com with SMTP id a2so15024551wrd.6;
        Tue, 14 Feb 2023 02:01:19 -0800 (PST)
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=1e100.net; s=20210112;
        h=content-transfer-encoding:in-reply-to:from:references:cc:to
         :content-language:subject:user-agent:mime-version:date:message-id
         :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to;
        bh=1+4q5IkASDXEOzqDRgEB04f5mJrlc9Cp/e1e4SAOKD0=;
        b=JUM0SIjv/72hQQRAQcJyDNCQ+whnUbXuUfkRtOqQWv0Wz4R6k0t1cGlgWxbhb0wn7I
         YZJxg//qBfff23n9zLIgbcyDGqGlT98jge8I6tEwphEVFPsOBBsXqJkMksVYZwP95rpf
         NaHQg3o5aXedIUOEQzCikdlnXpW1f+dGHX5w9KyMpOYOCJvc/M6j1m8gAST/lDMKeLjf
         sQP+UhiMwtYLmfL8HtZ7M2lSmbQiBCyOc6Z3ZwMZESBdCbRjAzkNoESTawrLCDkpJMwL
         gQYmTkihdlEDN3/ixLlUvepGGDov4UaWjGVJWNFieGXsN3wryy4joA1ClVDCiPCW1q0i
         EeBw==
X-Gm-Message-State: AO0yUKUx6MICgfCkstSucoRknxRwS9lB6va3ayu+rIvDfXOJ3jeKuvDI
        8FzKQAolAZaP6c/avWb2fSg=
X-Google-Smtp-Source: AK7set/Xt318zL6GpRHvNDie0q0HDZBwJM0D3FvTdggCnwo+e1rq7L2KeAk+JzfimJ/dU2UHVJJwgA==
X-Received: by 2002:a5d:6045:0:b0:2c5:60e2:ed68 with SMTP id j5-20020a5d6045000000b002c560e2ed68mr1150417wrt.0.1676368878424;
        Tue, 14 Feb 2023 02:01:18 -0800 (PST)
Received: from [192.168.64.80] (bzq-219-42-90.isdn.bezeqint.net. [62.219.42.90])
        by smtp.gmail.com with ESMTPSA id x8-20020adfdd88000000b002c5691f13eesm634637wrl.50.2023.02.14.02.01.16
        (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128);
        Tue, 14 Feb 2023 02:01:17 -0800 (PST)
Message-ID: <c3c636b4-b46b-ddab-6446-bc55349a96e3@grimberg.me>
Date:   Tue, 14 Feb 2023 12:01:15 +0200
MIME-Version: 1.0
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101
 Thunderbird/102.7.1
Subject: Re: [PATCH rdma-next 00/13] Add RDMA inline crypto support
Content-Language: en-US
To:     Max Gurtovoy <mgurtovoy@nvidia.com>, Christoph Hellwig <hch@lst.de>
Cc:     Leon Romanovsky <leon@kernel.org>,
        Jason Gunthorpe <jgg@nvidia.com>,
        Bryan Tan <bryantan@vmware.com>,
        Eric Dumazet <edumazet@google.com>,
        Israel Rukshin <israelr@nvidia.com>,
        Jakub Kicinski <kuba@kernel.org>, Jens Axboe <axboe@fb.com>,
        Keith Busch <kbusch@kernel.org>, linux-kernel@vger.kernel.org,
        linux-nvme@lists.infradead.org, linux-rdma@vger.kernel.org,
        linux-trace-kernel@vger.kernel.org,
        Masami Hiramatsu <mhiramat@kernel.org>, netdev@vger.kernel.org,
        Paolo Abeni <pabeni@redhat.com>,
        Saeed Mahameed <saeedm@nvidia.com>,
        Selvin Xavier <selvin.xavier@broadcom.com>,
        Steven Rostedt <rostedt@goodmis.org>,
        Vishnu Dasa <vdasa@vmware.com>,
        Yishai Hadas <yishaih@nvidia.com>
References: <cover.1673873422.git.leon@kernel.org>
 <20230118073654.GC27048@lst.de>
 <ac48766a-b861-4fc0-3171-7b23f175c672@nvidia.com>
 <20230130123505.GA19948@lst.de>
 <e8c35d5d-d855-a38c-2f3b-6887df208360@nvidia.com>
From:   Sagi Grimberg <sagi@grimberg.me>
In-Reply-To: <e8c35d5d-d855-a38c-2f3b-6887df208360@nvidia.com>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit
Precedence: bulk
List-ID: <netdev.vger.kernel.org>
X-Mailing-List: netdev@vger.kernel.org


>>> Today, if one would like to run both features using a capable device 
>>> then
>>> the PI will be offloaded (no SW fallback) and the Encryption will be 
>>> using
>>> SW fallback.
>> It's not just running both - it's offering both as currently registering
>> these fatures is mutally incompatible.
> 
> For sure we would like to offer both, but as mentioned before, this will 
> require blk layer instrumentation and ib_core instrumentation to some 
> pipeline_mr or something like that.

Can you explain what is needed in the block layer that prevents queueing
a request that has PI and crypto ctx?

As for the rdma portion, I don't know enough if this is something
that is supported by the HW and lacks a SW interface, or inherently
unsupported in HW...

