[PATCH RFC] nvme-ioctl: propagate PRP1 from ioctl to admin cmd

[David Epping <david.epping@missinglinkelectronics.com>]

Hello,

some NVMe admin commands (like IO queue creation) require information to
be provided via the PRP1 field of the command.
This information has not previously been passed from the ioctl to the NVMe
command structure, which prevented execution of such commands from userspace.
Likely this is not an oversight but intentional because of security concerns?

With this email I'm seeking to see if there is interest in getting support
for userspace NVMe IO queue creation in the upstream driver.
The attached patch is one way to do it and has been used for multiple years
with multiple Kernel versions now. There are additional patches required to
for example allocate sufficiently many IO queues at initialization time, but
I would like to focus on the ability to create queues for now.
A final patch would also not ignore nvme_user_cmd64().

The system setup where this patch has been used is as follows:
- P2P PCIe capable CPU (currently also IOMMU disabled)
- patched Linux in-Kernel NVMe driver for local PCIe NVMe SSDs
- FPGA accelerator implementing NVMe IO queue memory and IO queue handling,
  exposed via PCIe BAR
- vfio-pcie Kernel driver plus vfio userspace FPGA driver / application
- The userspace application creates new NVMe IO queues at the SSD using the
  patched admin ioctl and points them towards the FPGA BAR. It then informs
  the FPGA about the SSD BAR address and IO queue ID. From then on the FPGA
  can access the SSD storage entirely without software interaction.

Since the omition of PRP1 access from userspace is likely intentional, maybe
the discussion and patches by Leon Romanovsky for making dmabuf and p2pdma
available via vfio lead in the right direction:
https://lore.kernel.org/all/20251106-dmabuf-vfio-v7-0-2503bf390699@nvidia.com/

It currently seems focused on SPDK for handling the SSDs in userspace, but the
author also describes it as a general mechanism that can support other
scenarios.
How about having the in-Kernel NVMe driver accept a dmabuf as IO queue location
for creating a new IO queue? And in turn it likely has to provide a small
dmabuf to the FPGA VFIO world for access to the queue doorbells on the SSD.

Looking forward to your feedback,
David

Signed-off-by: David Epping <david.epping@missinglinkelectronics.com>
---
 drivers/nvme/host/ioctl.c | 1 +
 1 file changed, 1 insertion(+)

diff --git a/drivers/nvme/host/ioctl.c b/drivers/nvme/host/ioctl.c
index ca86d3bf7ea4..d5d740f1b554 100644
--- a/drivers/nvme/host/ioctl.c
+++ b/drivers/nvme/host/ioctl.c
@@ -306,6 +306,7 @@ static int nvme_user_cmd(struct nvme_ctrl *ctrl, struct nvme_ns *ns,
 	c.common.nsid = cpu_to_le32(cmd.nsid);
 	c.common.cdw2[0] = cpu_to_le32(cmd.cdw2);
 	c.common.cdw2[1] = cpu_to_le32(cmd.cdw3);
+	c.common.dptr.prp1 = cpu_to_le64(cmd.addr);
 	c.common.cdw10 = cpu_to_le32(cmd.cdw10);
 	c.common.cdw11 = cpu_to_le32(cmd.cdw11);
 	c.common.cdw12 = cpu_to_le32(cmd.cdw12);
-- 
2.43.0