From: Max Gurtovoy <maxg@mellanox.com>
To: linux-nvme@lists.infradead.org, sagi@grimberg.me,
linux-rdma@vger.kernel.org, kbusch@kernel.org, hch@lst.de,
martin.petersen@oracle.com
Cc: axboe@kernel.dk, vladimirk@mellanox.com, shlomin@mellanox.com,
israelr@mellanox.com, idanb@mellanox.com, maxg@mellanox.com
Subject: [PATCH 18/19] nvmet-rdma: Implement set_mdts controller op
Date: Mon, 24 Feb 2020 18:45:43 +0200 [thread overview]
Message-ID: <20200224164544.219438-20-maxg@mellanox.com> (raw)
In-Reply-To: <20200224164544.219438-1-maxg@mellanox.com>
Set the maximal data transfer size according to device capabilities.
For example, for T10-DIF offload by supporting RDMA HCA, one uses
RDMA/rw API that limits the IO operation to use a single MR with 256
pages at the most. Limit the mdts according to RDMA/rw API and even
decrease it in order to avoid multiple splits by the local block layer
for large IOs to ease on the CPU on the target side.
Signed-off-by: Max Gurtovoy <maxg@mellanox.com>
---
drivers/nvme/target/rdma.c | 17 +++++++++++++++++
1 file changed, 17 insertions(+)
diff --git a/drivers/nvme/target/rdma.c b/drivers/nvme/target/rdma.c
index 37d262a..2227adf 100644
--- a/drivers/nvme/target/rdma.c
+++ b/drivers/nvme/target/rdma.c
@@ -30,6 +30,7 @@
#define NVMET_RDMA_DEFAULT_INLINE_DATA_SIZE PAGE_SIZE
#define NVMET_RDMA_MAX_INLINE_SGE 4
#define NVMET_RDMA_MAX_INLINE_DATA_SIZE max_t(int, SZ_16K, PAGE_SIZE)
+#define NVMET_RDMA_T10_PI_MDTS 5
struct nvmet_rdma_cmd {
struct ib_sge sge[NVMET_RDMA_MAX_INLINE_SGE + 1];
@@ -1602,6 +1603,21 @@ static void nvmet_rdma_disc_port_addr(struct nvmet_req *req,
}
}
+static u8 nvmet_rdma_set_mdts(struct nvmet_ctrl *ctrl)
+{
+ struct nvmet_port *port = ctrl->port;
+ struct rdma_cm_id *cm_id = port->priv;
+ u32 max_pages;
+
+ if (ctrl->pi_support) {
+ max_pages = rdma_rw_fr_page_list_len(cm_id->device, true);
+ /* Assume mpsmin == device_page_size == 4KB */
+ return min(ilog2(max_pages), NVMET_RDMA_T10_PI_MDTS);
+ }
+
+ return 0;
+}
+
static const struct nvmet_fabrics_ops nvmet_rdma_ops = {
.owner = THIS_MODULE,
.type = NVMF_TRTYPE_RDMA,
@@ -1612,6 +1628,7 @@ static void nvmet_rdma_disc_port_addr(struct nvmet_req *req,
.queue_response = nvmet_rdma_queue_response,
.delete_ctrl = nvmet_rdma_delete_ctrl,
.disc_traddr = nvmet_rdma_disc_port_addr,
+ .set_mdts = nvmet_rdma_set_mdts,
};
static void nvmet_rdma_remove_one(struct ib_device *ib_device, void *client_data)
--
1.8.3.1
_______________________________________________
linux-nvme mailing list
linux-nvme@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-nvme
next prev parent reply other threads:[~2020-02-24 16:47 UTC|newest]
Thread overview: 22+ messages / expand[flat|nested] mbox.gz Atom feed top
2020-02-24 16:45 [PATCH 00/19 V4] nvme-rdma/nvmet-rdma: Add metadata/T10-PI support Max Gurtovoy
2020-02-24 16:45 ` [PATCH] nvme-cli/fabrics: Add pi_enable param to connect cmd Max Gurtovoy
2020-02-24 16:45 ` [PATCH 01/19] nvme: Introduce namespace features flag Max Gurtovoy
2020-02-24 16:45 ` [PATCH 02/19] nvme: Add has_pi field to the nvme_req structure Max Gurtovoy
2020-02-24 16:45 ` [PATCH 03/19] nvme: Enforce extended LBA format for fabrics metadata Max Gurtovoy
2020-02-24 16:45 ` [PATCH 04/19] nvme: Introduce max_integrity_segments ctrl attribute Max Gurtovoy
2020-02-24 16:45 ` [PATCH 05/19] nvme-fabrics: Allow user enabling metadata/T10-PI support Max Gurtovoy
2020-02-24 16:45 ` [PATCH 06/19] nvme: Introduce NVME_INLINE_PROT_SG_CNT Max Gurtovoy
2020-02-24 16:45 ` [PATCH 07/19] nvme-rdma: Introduce nvme_rdma_sgl structure Max Gurtovoy
2020-02-24 16:45 ` [PATCH 08/19] nvme-rdma: Add metadata/T10-PI support Max Gurtovoy
2020-02-24 16:45 ` [PATCH 09/19] nvmet: Prepare metadata request Max Gurtovoy
2020-02-24 16:45 ` [PATCH 10/19] nvmet: Add metadata characteristics for a namespace Max Gurtovoy
2020-02-24 16:45 ` [PATCH 11/19] nvmet: Rename nvmet_rw_len to nvmet_rw_data_len Max Gurtovoy
2020-02-24 16:45 ` [PATCH 12/19] nvmet: Rename nvmet_check_data_len to nvmet_check_transfer_len Max Gurtovoy
2020-02-24 16:45 ` [PATCH 13/19] nvme: Add Metadata Capabilities enumerations Max Gurtovoy
2020-02-24 16:45 ` [PATCH 14/19] nvmet: Add mdts setting op for controllers Max Gurtovoy
2020-02-24 16:45 ` [PATCH 15/19] nvmet: Add metadata/T10-PI support Max Gurtovoy
2020-02-24 16:45 ` [PATCH 16/19] nvmet: Add metadata support for block devices Max Gurtovoy
2020-02-24 16:45 ` [PATCH 17/19] RDMA/rw: Expose maximal page list for a device per 1 MR Max Gurtovoy
2020-02-24 16:45 ` Max Gurtovoy [this message]
2020-02-24 16:45 ` [PATCH 19/19] nvmet-rdma: Add metadata/T10-PI support Max Gurtovoy
2020-02-24 18:13 ` [PATCH 00/19 V4] nvme-rdma/nvmet-rdma: " James Smart
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20200224164544.219438-20-maxg@mellanox.com \
--to=maxg@mellanox.com \
--cc=axboe@kernel.dk \
--cc=hch@lst.de \
--cc=idanb@mellanox.com \
--cc=israelr@mellanox.com \
--cc=kbusch@kernel.org \
--cc=linux-nvme@lists.infradead.org \
--cc=linux-rdma@vger.kernel.org \
--cc=martin.petersen@oracle.com \
--cc=sagi@grimberg.me \
--cc=shlomin@mellanox.com \
--cc=vladimirk@mellanox.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox