[PATCH RFTv1 0/3] iommu/arm-smmu-v3: Add PRI support

[PATCH RFTv1 0/3] iommu/arm-smmu-v3: Add PRI support

[Nicolin Chen]

Due to a prior lack of real-world hardware support, the SMMUv3 driver does
not handles events on the PRI queue nor respond to IOPF faults.

NVIDIA has introduced a new SoC that supports PRI, and other vendors might
likely have hardware requiring this feature as well. It's time to fill the
missing pieces in the driver by leveraging the IOPF infrastructure:
 - Convert PRI events from the PRI queue into iopf_faults
 - Respond to iopf_faults by issuing CMDQ_OP_PRI_RESP

These patches were initially written on an older kernel and lightly tested
on NVIDIA SoC. I have rebased/reworked them on top of v7.0-rc2, so further
testing is required.

This is on Github:
https://github.com/nicolinc/iommufd/commits/smmuv3_pri-v1

Malak Marrid (3):
  iommu/arm-smmu-v3: Drain in-flight fault handlers
  iommu/arm-smmu-v3: Submit CMDQ_OP_PRI_RESP for IOPF event
  iommu/arm-smmu-v3: Support PRI Page Request

 drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 135 +++++++++++++-------
 1 file changed, 86 insertions(+), 49 deletions(-)

-- 
2.43.0

[PATCH RFTv1 1/3] iommu/arm-smmu-v3: Drain in-flight fault handlers

[Nicolin Chen]

From: Malak Marrid <mmarrid@nvidia.com>

When a device is switching away from a domain, either through a detach or a
replace operation, it must drain its IOPF queue that only contains the page
requests for the old domain.

Co-developed-by: Barak Biber <bbiber@nvidia.com>
Signed-off-by: Barak Biber <bbiber@nvidia.com>
Co-developed-by: Stefan Kaestle <skaestle@nvidia.com>
Signed-off-by: Stefan Kaestle <skaestle@nvidia.com>
Signed-off-by: Malak Marrid <mmarrid@nvidia.com>
Signed-off-by: Nicolin Chen <nicolinc@nvidia.com>
---
 drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 11 ++++++++++-
 1 file changed, 10 insertions(+), 1 deletion(-)

diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
index 4d00d796f0783..2176ee8bec767 100644
--- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
+++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
@@ -2843,6 +2843,12 @@ static int arm_smmu_enable_iopf(struct arm_smmu_master *master,
 	if (master->iopf_refcount) {
 		master->iopf_refcount++;
 		master_domain->using_iopf = true;
+		/*
+		 * If the device is already on the IOPF queue (domain replace),
+		 * drain in-flight fault handlers so nothing will hold the old
+		 * domain when the core switches the attach handle.
+		 */
+		iopf_queue_flush_dev(master->dev);
 		return 0;
 	}
 
@@ -2866,8 +2872,11 @@ static void arm_smmu_disable_iopf(struct arm_smmu_master *master,
 		return;
 
 	master->iopf_refcount--;
-	if (master->iopf_refcount == 0)
+	if (master->iopf_refcount == 0) {
+		/* Drain in-flight fault handlers before removing device */
+		iopf_queue_flush_dev(master->dev);
 		iopf_queue_remove_device(master->smmu->evtq.iopf, master->dev);
+	}
 }
 
 static void arm_smmu_remove_master_domain(struct arm_smmu_master *master,
-- 
2.43.0

[PATCH RFTv1 2/3] iommu/arm-smmu-v3: Submit CMDQ_OP_PRI_RESP for IOPF event

[Nicolin Chen]

From: Malak Marrid <mmarrid@nvidia.com>

To support IOMMU_FAULT_PAGE_REQ, the arm_smmu_page_response() should issue
a corresponding CMDQ_OP_PRI_RESP for SMMU HW to response the page request.

Extract the PRI info from the iopf_fault structure and fill the command.

Also, slightly rework the function flow with the existing CMDQ_OP_RESUME.

Co-developed-by: Barak Biber <bbiber@nvidia.com>
Signed-off-by: Barak Biber <bbiber@nvidia.com>
Co-developed-by: Stefan Kaestle <skaestle@nvidia.com>
Signed-off-by: Stefan Kaestle <skaestle@nvidia.com>
Signed-off-by: Malak Marrid <mmarrid@nvidia.com>
Signed-off-by: Nicolin Chen <nicolinc@nvidia.com>
---
 drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 59 +++++++++++++++------
 1 file changed, 43 insertions(+), 16 deletions(-)

diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
index 2176ee8bec767..055dde5367131 100644
--- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
+++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
@@ -992,29 +992,56 @@ static int arm_smmu_cmdq_batch_submit(struct arm_smmu_device *smmu,
 					   cmds->num, true);
 }
 
-static void arm_smmu_page_response(struct device *dev, struct iopf_fault *unused,
+static void arm_smmu_page_response(struct device *dev, struct iopf_fault *evt,
 				   struct iommu_page_response *resp)
 {
 	struct arm_smmu_cmdq_ent cmd = {0};
 	struct arm_smmu_master *master = dev_iommu_priv_get(dev);
 	int sid = master->streams[0].id;
 
-	if (WARN_ON(!master->stall_enabled))
+	/* Handle PRI responses */
+	if (evt && evt->fault.type == IOMMU_FAULT_PAGE_REQ) {
+		cmd.opcode = CMDQ_OP_PRI_RESP;
+		cmd.pri.sid = sid;
+		cmd.pri.grpid = resp->grpid;
+		if (evt->fault.prm.flags &
+		    IOMMU_FAULT_PAGE_REQUEST_PASID_VALID) {
+			cmd.substream_valid = true;
+			cmd.pri.ssid = resp->pasid;
+		}
+		switch (resp->code) {
+		case IOMMU_PAGE_RESP_SUCCESS:
+			cmd.pri.resp = PRI_RESP_SUCC;
+			break;
+		case IOMMU_PAGE_RESP_FAILURE:
+			cmd.pri.resp = PRI_RESP_FAIL;
+			break;
+		case IOMMU_PAGE_RESP_INVALID:
+			cmd.pri.resp = PRI_RESP_DENY;
+			break;
+		default:
+			WARN_ON(true);
+			return;
+		}
+	} else if (master->stall_enabled) {
+		cmd.opcode = CMDQ_OP_RESUME;
+		cmd.resume.sid = sid;
+		cmd.resume.stag = resp->grpid;
+		switch (resp->code) {
+		case IOMMU_PAGE_RESP_INVALID:
+		case IOMMU_PAGE_RESP_FAILURE:
+			cmd.resume.resp = CMDQ_RESUME_0_RESP_ABORT;
+			break;
+		case IOMMU_PAGE_RESP_SUCCESS:
+			cmd.resume.resp = CMDQ_RESUME_0_RESP_RETRY;
+			break;
+		default:
+			WARN_ON(true);
+			return;
+		}
+	} else {
+		WARN_ON(true);
 		return;
-
-	cmd.opcode		= CMDQ_OP_RESUME;
-	cmd.resume.sid		= sid;
-	cmd.resume.stag		= resp->grpid;
-	switch (resp->code) {
-	case IOMMU_PAGE_RESP_INVALID:
-	case IOMMU_PAGE_RESP_FAILURE:
-		cmd.resume.resp = CMDQ_RESUME_0_RESP_ABORT;
-		break;
-	case IOMMU_PAGE_RESP_SUCCESS:
-		cmd.resume.resp = CMDQ_RESUME_0_RESP_RETRY;
-		break;
-	default:
-		break;
 	}
 
 	arm_smmu_cmdq_issue_cmd(master->smmu, &cmd);
-- 
2.43.0

[PATCH RFTv1 3/3] iommu/arm-smmu-v3: Support PRI Page Request

[Nicolin Chen]

From: Malak Marrid <mmarrid@nvidia.com>

Now, arm_smmu_page_response() can issue CMDQ_OP_PRI_RESP for page requests
from the IOPF infrastructure. Decode the page requests from the PRI queue,
and report it to the IOPF infrastructure.

This fills the final piece for PRI support.

Co-developed-by: Barak Biber <bbiber@nvidia.com>
Signed-off-by: Barak Biber <bbiber@nvidia.com>
Co-developed-by: Stefan Kaestle <skaestle@nvidia.com>
Signed-off-by: Stefan Kaestle <skaestle@nvidia.com>
Signed-off-by: Malak Marrid <mmarrid@nvidia.com>
Signed-off-by: Nicolin Chen <nicolinc@nvidia.com>
---
 drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 65 +++++++++++----------
 1 file changed, 33 insertions(+), 32 deletions(-)

diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
index 055dde5367131..277d753c7675c 100644
--- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
+++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
@@ -2061,40 +2061,41 @@ static irqreturn_t arm_smmu_evtq_thread(int irq, void *dev)
 
 static void arm_smmu_handle_ppr(struct arm_smmu_device *smmu, u64 *evt)
 {
-	u32 sid, ssid;
-	u16 grpid;
-	bool ssv, last;
-
-	sid = FIELD_GET(PRIQ_0_SID, evt[0]);
-	ssv = FIELD_GET(PRIQ_0_SSID_V, evt[0]);
-	ssid = ssv ? FIELD_GET(PRIQ_0_SSID, evt[0]) : IOMMU_NO_PASID;
-	last = FIELD_GET(PRIQ_0_PRG_LAST, evt[0]);
-	grpid = FIELD_GET(PRIQ_1_PRG_IDX, evt[1]);
-
-	dev_info(smmu->dev, "unexpected PRI request received:\n");
-	dev_info(smmu->dev,
-		 "\tsid 0x%08x.0x%05x: [%u%s] %sprivileged %s%s%s access at iova 0x%016llx\n",
-		 sid, ssid, grpid, last ? "L" : "",
-		 evt[0] & PRIQ_0_PERM_PRIV ? "" : "un",
-		 evt[0] & PRIQ_0_PERM_READ ? "R" : "",
-		 evt[0] & PRIQ_0_PERM_WRITE ? "W" : "",
-		 evt[0] & PRIQ_0_PERM_EXEC ? "X" : "",
-		 evt[1] & PRIQ_1_ADDR_MASK);
-
-	if (last) {
-		struct arm_smmu_cmdq_ent cmd = {
-			.opcode			= CMDQ_OP_PRI_RESP,
-			.substream_valid	= ssv,
-			.pri			= {
-				.sid	= sid,
-				.ssid	= ssid,
-				.grpid	= grpid,
-				.resp	= PRI_RESP_DENY,
-			},
-		};
+	u32 sid = FIELD_GET(PRIQ_0_SID, evt[0]);
+	struct iopf_fault iopf_fault = {0};
+	struct iommu_fault *fault = &iopf_fault.fault;
+	struct arm_smmu_master *master;
+
+	INIT_LIST_HEAD(&iopf_fault.list);
 
-		arm_smmu_cmdq_issue_cmd(smmu, &cmd);
+	fault->type = IOMMU_FAULT_PAGE_REQ;
+
+	if (FIELD_GET(PRIQ_0_PRG_LAST, evt[0]))
+		fault->prm.flags |= IOMMU_FAULT_PAGE_REQUEST_LAST_PAGE;
+	if (FIELD_GET(PRIQ_0_SSID_V, evt[0])) {
+		fault->prm.flags |= IOMMU_FAULT_PAGE_REQUEST_PASID_VALID;
+		fault->prm.pasid = FIELD_GET(PRIQ_0_SSID, evt[0]);
 	}
+
+	fault->prm.grpid = FIELD_GET(PRIQ_1_PRG_IDX, evt[1]);
+
+	if (evt[0] & PRIQ_0_PERM_READ)
+		fault->prm.perm |= IOMMU_FAULT_PERM_READ;
+	if (evt[0] & PRIQ_0_PERM_WRITE)
+		fault->prm.perm |= IOMMU_FAULT_PERM_WRITE;
+	if (evt[0] & PRIQ_0_PERM_EXEC)
+		fault->prm.perm |= IOMMU_FAULT_PERM_EXEC;
+	if (evt[0] & PRIQ_0_PERM_PRIV)
+		fault->prm.perm |= IOMMU_FAULT_PERM_PRIV;
+
+	fault->prm.addr = FIELD_GET(PRIQ_1_ADDR_MASK, evt[1]) << 12;
+
+	guard(mutex)(&smmu->streams_mutex);
+	master = arm_smmu_find_master(smmu, sid);
+	if (master)
+		iommu_report_device_fault(master->dev, &iopf_fault);
+	else
+		dev_warn(smmu->dev, "PPR: unrecognized StreamID 0x%x\n", sid);
 }
 
 static irqreturn_t arm_smmu_priq_thread(int irq, void *dev)
-- 
2.43.0

Re: [PATCH RFTv1 1/3] iommu/arm-smmu-v3: Drain in-flight fault handlers

[Jason Gunthorpe]

On Tue, Mar 03, 2026 at 12:13:05PM -0800, Nicolin Chen wrote:
> From: Malak Marrid <mmarrid@nvidia.com>
> 
> When a device is switching away from a domain, either through a detach or a
> replace operation, it must drain its IOPF queue that only contains the page
> requests for the old domain.
> 
> Co-developed-by: Barak Biber <bbiber@nvidia.com>
> Signed-off-by: Barak Biber <bbiber@nvidia.com>
> Co-developed-by: Stefan Kaestle <skaestle@nvidia.com>
> Signed-off-by: Stefan Kaestle <skaestle@nvidia.com>
> Signed-off-by: Malak Marrid <mmarrid@nvidia.com>
> Signed-off-by: Nicolin Chen <nicolinc@nvidia.com>
> ---
>  drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 11 ++++++++++-
>  1 file changed, 10 insertions(+), 1 deletion(-)

Isn't this an existing bug in the stall handling?

Jason

Re: [PATCH RFTv1 1/3] iommu/arm-smmu-v3: Drain in-flight fault handlers

[Nicolin Chen]

On Fri, Mar 06, 2026 at 07:12:13PM -0400, Jason Gunthorpe wrote:
> On Tue, Mar 03, 2026 at 12:13:05PM -0800, Nicolin Chen wrote:
> > From: Malak Marrid <mmarrid@nvidia.com>
> > 
> > When a device is switching away from a domain, either through a detach or a
> > replace operation, it must drain its IOPF queue that only contains the page
> > requests for the old domain.
> > 
> > Co-developed-by: Barak Biber <bbiber@nvidia.com>
> > Signed-off-by: Barak Biber <bbiber@nvidia.com>
> > Co-developed-by: Stefan Kaestle <skaestle@nvidia.com>
> > Signed-off-by: Stefan Kaestle <skaestle@nvidia.com>
> > Signed-off-by: Malak Marrid <mmarrid@nvidia.com>
> > Signed-off-by: Nicolin Chen <nicolinc@nvidia.com>
> > ---
> >  drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 11 ++++++++++-
> >  1 file changed, 10 insertions(+), 1 deletion(-)
> 
> Isn't this an existing bug in the stall handling?

Hmm, right. I will make it a bug fix.

Thanks
Nicolin

RE: [PATCH RFTv1 3/3] iommu/arm-smmu-v3: Support PRI Page Request

[Shameer Kolothum Thodi]

> -----Original Message-----
> From: Nicolin Chen <nicolinc@nvidia.com>
> Sent: 03 March 2026 20:13
> To: will@kernel.org; robin.murphy@arm.com; Jason Gunthorpe
> <jgg@nvidia.com>
> Cc: joro@8bytes.org; praan@google.com; Malak Marrid
> <mmarrid@nvidia.com>; kees@kernel.org; Alexander.Grest@microsoft.com;
> baolu.lu@linux.intel.com; smostafa@google.com; linux-arm-
> kernel@lists.infradead.org; iommu@lists.linux.dev; linux-
> kernel@vger.kernel.org
> Subject: [PATCH RFTv1 3/3] iommu/arm-smmu-v3: Support PRI Page Request
> 
> External email: Use caution opening links or attachments
> 
> 
> From: Malak Marrid <mmarrid@nvidia.com>
> 
> Now, arm_smmu_page_response() can issue CMDQ_OP_PRI_RESP for page
> requests
> from the IOPF infrastructure. Decode the page requests from the PRI queue,
> and report it to the IOPF infrastructure.
> 
> This fills the final piece for PRI support.

Do we also need pci_reset_pri()/pci_enable_pri() to enable PRI on the
device side?

Also, is there a check that the device advertises PCI PRI capability
before calling iopf_queue_add_device()?

Thanks,
Shameer

> Co-developed-by: Barak Biber <bbiber@nvidia.com>
> Signed-off-by: Barak Biber <bbiber@nvidia.com>
> Co-developed-by: Stefan Kaestle <skaestle@nvidia.com>
> Signed-off-by: Stefan Kaestle <skaestle@nvidia.com>
> Signed-off-by: Malak Marrid <mmarrid@nvidia.com>
> Signed-off-by: Nicolin Chen <nicolinc@nvidia.com>
> ---
>  drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 65 +++++++++++------
> ----
>  1 file changed, 33 insertions(+), 32 deletions(-)
> 
> diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
> b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
> index 055dde5367131..277d753c7675c 100644
> --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
> +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
> @@ -2061,40 +2061,41 @@ static irqreturn_t arm_smmu_evtq_thread(int
> irq, void *dev)
> 
>  static void arm_smmu_handle_ppr(struct arm_smmu_device *smmu, u64
> *evt)
>  {
> -       u32 sid, ssid;
> -       u16 grpid;
> -       bool ssv, last;
> -
> -       sid = FIELD_GET(PRIQ_0_SID, evt[0]);
> -       ssv = FIELD_GET(PRIQ_0_SSID_V, evt[0]);
> -       ssid = ssv ? FIELD_GET(PRIQ_0_SSID, evt[0]) : IOMMU_NO_PASID;
> -       last = FIELD_GET(PRIQ_0_PRG_LAST, evt[0]);
> -       grpid = FIELD_GET(PRIQ_1_PRG_IDX, evt[1]);
> -
> -       dev_info(smmu->dev, "unexpected PRI request received:\n");
> -       dev_info(smmu->dev,
> -                "\tsid 0x%08x.0x%05x: [%u%s] %sprivileged %s%s%s access at iova
> 0x%016llx\n",
> -                sid, ssid, grpid, last ? "L" : "",
> -                evt[0] & PRIQ_0_PERM_PRIV ? "" : "un",
> -                evt[0] & PRIQ_0_PERM_READ ? "R" : "",
> -                evt[0] & PRIQ_0_PERM_WRITE ? "W" : "",
> -                evt[0] & PRIQ_0_PERM_EXEC ? "X" : "",
> -                evt[1] & PRIQ_1_ADDR_MASK);
> -
> -       if (last) {
> -               struct arm_smmu_cmdq_ent cmd = {
> -                       .opcode                 = CMDQ_OP_PRI_RESP,
> -                       .substream_valid        = ssv,
> -                       .pri                    = {
> -                               .sid    = sid,
> -                               .ssid   = ssid,
> -                               .grpid  = grpid,
> -                               .resp   = PRI_RESP_DENY,
> -                       },
> -               };
> +       u32 sid = FIELD_GET(PRIQ_0_SID, evt[0]);
> +       struct iopf_fault iopf_fault = {0};
> +       struct iommu_fault *fault = &iopf_fault.fault;
> +       struct arm_smmu_master *master;
> +
> +       INIT_LIST_HEAD(&iopf_fault.list);
> 
> -               arm_smmu_cmdq_issue_cmd(smmu, &cmd);
> +       fault->type = IOMMU_FAULT_PAGE_REQ;
> +
> +       if (FIELD_GET(PRIQ_0_PRG_LAST, evt[0]))
> +               fault->prm.flags |= IOMMU_FAULT_PAGE_REQUEST_LAST_PAGE;
> +       if (FIELD_GET(PRIQ_0_SSID_V, evt[0])) {
> +               fault->prm.flags |= IOMMU_FAULT_PAGE_REQUEST_PASID_VALID;
> +               fault->prm.pasid = FIELD_GET(PRIQ_0_SSID, evt[0]);
>         }
> +
> +       fault->prm.grpid = FIELD_GET(PRIQ_1_PRG_IDX, evt[1]);
> +
> +       if (evt[0] & PRIQ_0_PERM_READ)
> +               fault->prm.perm |= IOMMU_FAULT_PERM_READ;
> +       if (evt[0] & PRIQ_0_PERM_WRITE)
> +               fault->prm.perm |= IOMMU_FAULT_PERM_WRITE;
> +       if (evt[0] & PRIQ_0_PERM_EXEC)
> +               fault->prm.perm |= IOMMU_FAULT_PERM_EXEC;
> +       if (evt[0] & PRIQ_0_PERM_PRIV)
> +               fault->prm.perm |= IOMMU_FAULT_PERM_PRIV;
> +
> +       fault->prm.addr = FIELD_GET(PRIQ_1_ADDR_MASK, evt[1]) << 12;
> +
> +       guard(mutex)(&smmu->streams_mutex);
> +       master = arm_smmu_find_master(smmu, sid);
> +       if (master)
> +               iommu_report_device_fault(master->dev, &iopf_fault);
> +       else
> +               dev_warn(smmu->dev, "PPR: unrecognized StreamID 0x%x\n", sid);
>  }
> 
>  static irqreturn_t arm_smmu_priq_thread(int irq, void *dev)
> --
> 2.43.0
> 

Re: [PATCH RFTv1 3/3] iommu/arm-smmu-v3: Support PRI Page Request

[Nicolin Chen]

On Mon, Mar 09, 2026 at 04:04:31AM -0700, Shameer Kolothum Thodi wrote:
> > Now, arm_smmu_page_response() can issue CMDQ_OP_PRI_RESP for page
> > requests
> > from the IOPF infrastructure. Decode the page requests from the PRI queue,
> > and report it to the IOPF infrastructure.
> > 
> > This fills the final piece for PRI support.
> 
> Do we also need pci_reset_pri()/pci_enable_pri() to enable PRI on the
> device side?
> 
> Also, is there a check that the device advertises PCI PRI capability
> before calling iopf_queue_add_device()?

Yes and yes.. I will respin another version after a test.

Thanks
Nicolin