[PATCH] nvme-rdma: send cntlid in the RDMA_CM_REQUEST Private Data

[PATCH] nvme-rdma: send cntlid in the RDMA_CM_REQUEST Private Data

[Niklas Cassel]

When sending a RDMA_CM_REQUEST, the NVMe RDMA Transport Specification
allows you to populate the cntlid field in the RDMA_CM_REQUEST Private
Data.

The cntlid is returned by the target on completion of the first
RDMA_CM_REQUEST command (which creates the admin queue).

The cntlid field can then be populated by the host when the I/O queues
are created (using additional RDMA_CM_REQUEST commands), such that the
target can perform extra validation for additional RDMA_CM_REQUEST
commands.

This additional error code and error message is also added, such that
nvme_rdma_cm_msg() will display the proper error message if the target
fails the RDMA_CM_REQUEST command because of this extra validation.

Signed-off-by: Niklas Cassel <cassel@kernel.org>
---
 drivers/nvme/host/rdma.c  | 2 ++
 include/linux/nvme-rdma.h | 6 +++++-
 2 files changed, 7 insertions(+), 1 deletion(-)

diff --git a/drivers/nvme/host/rdma.c b/drivers/nvme/host/rdma.c
index 2eb33842f971..e3520a91147b 100644
--- a/drivers/nvme/host/rdma.c
+++ b/drivers/nvme/host/rdma.c
@@ -1876,6 +1876,8 @@ static int nvme_rdma_route_resolved(struct nvme_rdma_queue *queue)
 		 */
 		priv.hrqsize = cpu_to_le16(queue->queue_size);
 		priv.hsqsize = cpu_to_le16(queue->ctrl->ctrl.sqsize);
+		/* cntlid should only be set when creating an I/O queue */
+		priv.cntlid = cpu_to_le16(ctrl->ctrl.cntlid);
 	}
 
 	ret = rdma_connect_locked(queue->cm_id, &param);
diff --git a/include/linux/nvme-rdma.h b/include/linux/nvme-rdma.h
index eb2f04d636c8..97c5f00b9aa3 100644
--- a/include/linux/nvme-rdma.h
+++ b/include/linux/nvme-rdma.h
@@ -25,6 +25,7 @@ enum nvme_rdma_cm_status {
 	NVME_RDMA_CM_NO_RSC		= 0x06,
 	NVME_RDMA_CM_INVALID_IRD	= 0x07,
 	NVME_RDMA_CM_INVALID_ORD	= 0x08,
+	NVME_RDMA_CM_INVALID_CNTLID	= 0x09,
 };
 
 static inline const char *nvme_rdma_cm_msg(enum nvme_rdma_cm_status status)
@@ -46,6 +47,8 @@ static inline const char *nvme_rdma_cm_msg(enum nvme_rdma_cm_status status)
 		return "invalid IRD";
 	case NVME_RDMA_CM_INVALID_ORD:
 		return "Invalid ORD";
+	case NVME_RDMA_CM_INVALID_CNTLID:
+		return "invalid controller ID";
 	default:
 		return "unrecognized reason";
 	}
@@ -64,7 +67,8 @@ struct nvme_rdma_cm_req {
 	__le16		qid;
 	__le16		hrqsize;
 	__le16		hsqsize;
-	u8		rsvd[24];
+	__le16		cntlid;
+	u8		rsvd[22];
 };
 
 /**
-- 
2.46.0

Re: [PATCH] nvme-rdma: send cntlid in the RDMA_CM_REQUEST Private Data

[Christoph Hellwig]

Looks good:

Reviewed-by: Christoph Hellwig <hch@lst.de>

Re: [PATCH] nvme-rdma: send cntlid in the RDMA_CM_REQUEST Private Data

[Niklas Cassel]

On Thu, Aug 15, 2024 at 10:11:31PM +0200, Niklas Cassel wrote:
> When sending a RDMA_CM_REQUEST, the NVMe RDMA Transport Specification
> allows you to populate the cntlid field in the RDMA_CM_REQUEST Private
> Data.
> 
> The cntlid is returned by the target on completion of the first
> RDMA_CM_REQUEST command (which creates the admin queue).
> 
> The cntlid field can then be populated by the host when the I/O queues
> are created (using additional RDMA_CM_REQUEST commands), such that the
> target can perform extra validation for additional RDMA_CM_REQUEST
> commands.
> 
> This additional error code and error message is also added, such that
> nvme_rdma_cm_msg() will display the proper error message if the target
> fails the RDMA_CM_REQUEST command because of this extra validation.
> 
> Signed-off-by: Niklas Cassel <cassel@kernel.org>
> ---
>  drivers/nvme/host/rdma.c  | 2 ++
>  include/linux/nvme-rdma.h | 6 +++++-
>  2 files changed, 7 insertions(+), 1 deletion(-)

Hello NVMe maintainers,

is there any chance of getting this picked up?

Have a nice day!

Kind regards,
Niklas

Re: [PATCH] nvme-rdma: send cntlid in the RDMA_CM_REQUEST Private Data

[Sagi Grimberg]

On 26/08/2024 12:27, Niklas Cassel wrote:
> On Thu, Aug 15, 2024 at 10:11:31PM +0200, Niklas Cassel wrote:
>> When sending a RDMA_CM_REQUEST, the NVMe RDMA Transport Specification
>> allows you to populate the cntlid field in the RDMA_CM_REQUEST Private
>> Data.
>>
>> The cntlid is returned by the target on completion of the first
>> RDMA_CM_REQUEST command (which creates the admin queue).
>>
>> The cntlid field can then be populated by the host when the I/O queues
>> are created (using additional RDMA_CM_REQUEST commands), such that the
>> target can perform extra validation for additional RDMA_CM_REQUEST
>> commands.
>>
>> This additional error code and error message is also added, such that
>> nvme_rdma_cm_msg() will display the proper error message if the target
>> fails the RDMA_CM_REQUEST command because of this extra validation.
>>
>> Signed-off-by: Niklas Cassel <cassel@kernel.org>
>> ---
>>   drivers/nvme/host/rdma.c  | 2 ++
>>   include/linux/nvme-rdma.h | 6 +++++-
>>   2 files changed, 7 insertions(+), 1 deletion(-)
> Hello NVMe maintainers,
>
> is there any chance of getting this picked up?

Keith will probably pick it up for 6.12...

Re: [PATCH] nvme-rdma: send cntlid in the RDMA_CM_REQUEST Private Data

[Sagi Grimberg]

Reviewed-by: Sagi Grimberg <sagi@grimberg.me>

Re: [PATCH] nvme-rdma: send cntlid in the RDMA_CM_REQUEST Private Data

[Keith Busch]

On Thu, Aug 15, 2024 at 10:11:31PM +0200, Niklas Cassel wrote:
> When sending a RDMA_CM_REQUEST, the NVMe RDMA Transport Specification
> allows you to populate the cntlid field in the RDMA_CM_REQUEST Private
> Data.
> 
> The cntlid is returned by the target on completion of the first
> RDMA_CM_REQUEST command (which creates the admin queue).
> 
> The cntlid field can then be populated by the host when the I/O queues
> are created (using additional RDMA_CM_REQUEST commands), such that the
> target can perform extra validation for additional RDMA_CM_REQUEST
> commands.
> 
> This additional error code and error message is also added, such that
> nvme_rdma_cm_msg() will display the proper error message if the target
> fails the RDMA_CM_REQUEST command because of this extra validation.

Thanks, applied to nvme-6.12.