[PATCH v2] iommu/arm-smmu-v3-iommufd: Allow attaching nested domain for GBPA cases

[PATCH v2] iommu/arm-smmu-v3-iommufd: Allow attaching nested domain for GBPA cases

[Nicolin Chen]

A vDEVICE has been a hard requirement for attaching a nested domain to the
device. This makes sense when installing a guest STE, since a vSID must be
present and given to the kernel during the vDEVICE allocation.

But, when CR0.SMMUEN is disabled, VM doesn't really need a vSID to program
the vSMMU behavior as GBPA will take effect, in which case the vSTE in the
nested domain could have carried the bypass or abort configuration in GBPA
register. Thus, having such a hard requirement doesn't work well for GBPA.

Skip vmaster allocation in arm_smmu_attach_prepare_vmaster() for an abort
or bypass vSTE. Note that device on this attachment won't report vevents.

Update the uAPI doc accordingly.

Tested-by: Shameer Kolothum <skolothumtho@nvidia.com>
Signed-off-by: Nicolin Chen <nicolinc@nvidia.com>
---

Changelog
v2
 * Add Tested-by from Shameer
 * Skip vmaster allocation instead of bypassing vsid=0
 * Revise the uAPI doc to note a corner case when CR0.SMMUEN=1

 drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-iommufd.c | 13 ++++++++++++-
 include/uapi/linux/iommufd.h                        |  9 +++++++++
 2 files changed, 21 insertions(+), 1 deletion(-)

diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-iommufd.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-iommufd.c
index 8cd8929bbfdf8..e5fbbdbdea242 100644
--- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-iommufd.c
+++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-iommufd.c
@@ -99,6 +99,8 @@ static void arm_smmu_make_nested_domain_ste(
 int arm_smmu_attach_prepare_vmaster(struct arm_smmu_attach_state *state,
 				    struct arm_smmu_nested_domain *nested_domain)
 {
+	unsigned int cfg =
+		FIELD_GET(STRTAB_STE_0_CFG, le64_to_cpu(nested_domain->ste[0]));
 	struct arm_smmu_vmaster *vmaster;
 	unsigned long vsid;
 	int ret;
@@ -107,8 +109,17 @@ int arm_smmu_attach_prepare_vmaster(struct arm_smmu_attach_state *state,
 
 	ret = iommufd_viommu_get_vdev_id(&nested_domain->vsmmu->core,
 					 state->master->dev, &vsid);
-	if (ret)
+	/*
+	 * Attaching to a translate nested domain must allocate a vDEVICE prior,
+	 * as CD/ATS invalidations and vevents require a vSID to work properly.
+	 * A abort/bypass domain is allowed to attach w/o vmaster for GBPA case.
+	 */
+	if (ret) {
+		if (cfg == STRTAB_STE_0_CFG_ABORT ||
+		    cfg == STRTAB_STE_0_CFG_BYPASS)
+			return 0;
 		return ret;
+	}
 
 	vmaster = kzalloc(sizeof(*vmaster), GFP_KERNEL);
 	if (!vmaster)
diff --git a/include/uapi/linux/iommufd.h b/include/uapi/linux/iommufd.h
index c218c89e0e2eb..225671603ade6 100644
--- a/include/uapi/linux/iommufd.h
+++ b/include/uapi/linux/iommufd.h
@@ -450,6 +450,15 @@ struct iommu_hwpt_vtd_s1 {
  * nested domain will translate the same as the nesting parent. The S1 will
  * install a Context Descriptor Table pointing at userspace memory translated
  * by the nesting parent.
+ *
+ * It's suggested to allocate a vDEVICE object carrying vSID and then re-attach
+ * the nested domain, as soon as the vSID is available in the VMM level:
+ * - when Cfg=translate, a vDEVICE must be allocated prior to attaching to the
+ *   allocated nested domain, as CD/ATS invalidations and vevents need a vSID.
+ * - when Cfg=bypass/abort, a vDEVICE is not enforced during the nested domain
+ *   attachment, to support a GBPA case where VM sets CR0.SMMUEN=0. However, if
+ *   VM sets CR0.SMMUEN=1 while missing a vDEVICE object, kernel would fail to
+ *   report events to the VM. E.g. F_TRANSLATION when guest STE.Cfg=abort.
  */
 struct iommu_hwpt_arm_smmuv3 {
 	__aligned_le64 ste[2];
-- 
2.43.0

Re: [PATCH v2] iommu/arm-smmu-v3-iommufd: Allow attaching nested domain for GBPA cases

[Pranjal Shrivastava]

On Mon, Nov 03, 2025 at 09:27:55AM -0800, Nicolin Chen wrote:
> A vDEVICE has been a hard requirement for attaching a nested domain to the
> device. This makes sense when installing a guest STE, since a vSID must be
> present and given to the kernel during the vDEVICE allocation.
> 
> But, when CR0.SMMUEN is disabled, VM doesn't really need a vSID to program
> the vSMMU behavior as GBPA will take effect, in which case the vSTE in the
> nested domain could have carried the bypass or abort configuration in GBPA
> register. Thus, having such a hard requirement doesn't work well for GBPA.
> 
> Skip vmaster allocation in arm_smmu_attach_prepare_vmaster() for an abort
> or bypass vSTE. Note that device on this attachment won't report vevents.
> 
> Update the uAPI doc accordingly.
> 
> Tested-by: Shameer Kolothum <skolothumtho@nvidia.com>
> Signed-off-by: Nicolin Chen <nicolinc@nvidia.com>
> ---
> 
> Changelog
> v2
>  * Add Tested-by from Shameer
>  * Skip vmaster allocation instead of bypassing vsid=0
>  * Revise the uAPI doc to note a corner case when CR0.SMMUEN=1
> 
>  drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-iommufd.c | 13 ++++++++++++-
>  include/uapi/linux/iommufd.h                        |  9 +++++++++
>  2 files changed, 21 insertions(+), 1 deletion(-)
> 
> diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-iommufd.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-iommufd.c
> index 8cd8929bbfdf8..e5fbbdbdea242 100644
> --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-iommufd.c
> +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-iommufd.c
> @@ -99,6 +99,8 @@ static void arm_smmu_make_nested_domain_ste(
>  int arm_smmu_attach_prepare_vmaster(struct arm_smmu_attach_state *state,
>  				    struct arm_smmu_nested_domain *nested_domain)
>  {
> +	unsigned int cfg =
> +		FIELD_GET(STRTAB_STE_0_CFG, le64_to_cpu(nested_domain->ste[0]));
>  	struct arm_smmu_vmaster *vmaster;
>  	unsigned long vsid;
>  	int ret;
> @@ -107,8 +109,17 @@ int arm_smmu_attach_prepare_vmaster(struct arm_smmu_attach_state *state,
>  
>  	ret = iommufd_viommu_get_vdev_id(&nested_domain->vsmmu->core,
>  					 state->master->dev, &vsid);
> -	if (ret)
> +	/*
> +	 * Attaching to a translate nested domain must allocate a vDEVICE prior,
> +	 * as CD/ATS invalidations and vevents require a vSID to work properly.
> +	 * A abort/bypass domain is allowed to attach w/o vmaster for GBPA case.
> +	 */
> +	if (ret) {
> +		if (cfg == STRTAB_STE_0_CFG_ABORT ||
> +		    cfg == STRTAB_STE_0_CFG_BYPASS)
> +			return 0;
>  		return ret;
> +	}
>

Skipping the vmaster allocation entirely for the GBPA-only case (when no
vdevice is found) is much cleaner. Thanks!

>  	vmaster = kzalloc(sizeof(*vmaster), GFP_KERNEL);
>  	if (!vmaster)
> diff --git a/include/uapi/linux/iommufd.h b/include/uapi/linux/iommufd.h
> index c218c89e0e2eb..225671603ade6 100644
> --- a/include/uapi/linux/iommufd.h
> +++ b/include/uapi/linux/iommufd.h
> @@ -450,6 +450,15 @@ struct iommu_hwpt_vtd_s1 {
>   * nested domain will translate the same as the nesting parent. The S1 will
>   * install a Context Descriptor Table pointing at userspace memory translated
>   * by the nesting parent.
> + *
> + * It's suggested to allocate a vDEVICE object carrying vSID and then re-attach
> + * the nested domain, as soon as the vSID is available in the VMM level:
> + * - when Cfg=translate, a vDEVICE must be allocated prior to attaching to the
> + *   allocated nested domain, as CD/ATS invalidations and vevents need a vSID.
> + * - when Cfg=bypass/abort, a vDEVICE is not enforced during the nested domain
> + *   attachment, to support a GBPA case where VM sets CR0.SMMUEN=0. However, if
> + *   VM sets CR0.SMMUEN=1 while missing a vDEVICE object, kernel would fail to
> + *   report events to the VM. E.g. F_TRANSLATION when guest STE.Cfg=abort.
>   */

Reviewed-by: Pranjal Shrivastava <praan@google.com>

Thanks,
Praan

Re: [PATCH v2] iommu/arm-smmu-v3-iommufd: Allow attaching nested domain for GBPA cases

[Shuai Xue]

在 2025/11/4 01:27, Nicolin Chen 写道:
> A vDEVICE has been a hard requirement for attaching a nested domain to the
> device. This makes sense when installing a guest STE, since a vSID must be
> present and given to the kernel during the vDEVICE allocation.
> 
> But, when CR0.SMMUEN is disabled, VM doesn't really need a vSID to program
> the vSMMU behavior as GBPA will take effect, in which case the vSTE in the
> nested domain could have carried the bypass or abort configuration in GBPA
> register. Thus, having such a hard requirement doesn't work well for GBPA.
> 
> Skip vmaster allocation in arm_smmu_attach_prepare_vmaster() for an abort
> or bypass vSTE. Note that device on this attachment won't report vevents.
> 
> Update the uAPI doc accordingly.
> 
> Tested-by: Shameer Kolothum <skolothumtho@nvidia.com>
> Signed-off-by: Nicolin Chen <nicolinc@nvidia.com>
> ---

I've tested this patch and can confirm it resolves the GBPA hwpt
attachment issue.

Without this patch, IOMMUFD fails to attach bypass hwpt with the
following error:

   smmu_reset_exit
   qemu-system-aarch64: [iommufd=303] error attach 0008:01:00.0 (304) to id=6: No such file or directory
   Failed to attach GBPA hwpt id 6 for dev id 1

With this patch applied, the attachment succeeds:

   smmu_reset_exit
   iommufd_cdev_attach_ioas_hwpt  [iommufd=303] Successfully attached device 0008:01:00.0 (304) to id=6

Testing was performed on top of Shameer's vSMMU v5 series:

   https://patchew.org/QEMU/20251031105005.24618-1-skolothumtho@nvidia.com/

Tested-by: Shuai Xue  <xueshuai@linux.alibaba.com>

Thanks.
Shuai

Re: [PATCH v2] iommu/arm-smmu-v3-iommufd: Allow attaching nested domain for GBPA cases

[Jason Gunthorpe]

On Mon, Nov 03, 2025 at 09:27:55AM -0800, Nicolin Chen wrote:
> A vDEVICE has been a hard requirement for attaching a nested domain to the
> device. This makes sense when installing a guest STE, since a vSID must be
> present and given to the kernel during the vDEVICE allocation.
> 
> But, when CR0.SMMUEN is disabled, VM doesn't really need a vSID to program
> the vSMMU behavior as GBPA will take effect, in which case the vSTE in the
> nested domain could have carried the bypass or abort configuration in GBPA
> register. Thus, having such a hard requirement doesn't work well for GBPA.
> 
> Skip vmaster allocation in arm_smmu_attach_prepare_vmaster() for an abort
> or bypass vSTE. Note that device on this attachment won't report vevents.
> 
> Update the uAPI doc accordingly.
> 
> Tested-by: Shameer Kolothum <skolothumtho@nvidia.com>
> Signed-off-by: Nicolin Chen <nicolinc@nvidia.com>
> ---
> 
> Changelog
> v2
>  * Add Tested-by from Shameer
>  * Skip vmaster allocation instead of bypassing vsid=0
>  * Revise the uAPI doc to note a corner case when CR0.SMMUEN=1
> 
>  drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-iommufd.c | 13 ++++++++++++-
>  include/uapi/linux/iommufd.h                        |  9 +++++++++
>  2 files changed, 21 insertions(+), 1 deletion(-)

Reviewed-by: Jason Gunthorpe <jgg@nvidia.com>

Will, I will grab this in a few days unless you want it in your tree

Thanks,
Jason

Re: [PATCH v2] iommu/arm-smmu-v3-iommufd: Allow attaching nested domain for GBPA cases

[Will Deacon]

On Mon, Nov 24, 2025 at 03:34:35PM -0400, Jason Gunthorpe wrote:
> On Mon, Nov 03, 2025 at 09:27:55AM -0800, Nicolin Chen wrote:
> > A vDEVICE has been a hard requirement for attaching a nested domain to the
> > device. This makes sense when installing a guest STE, since a vSID must be
> > present and given to the kernel during the vDEVICE allocation.
> > 
> > But, when CR0.SMMUEN is disabled, VM doesn't really need a vSID to program
> > the vSMMU behavior as GBPA will take effect, in which case the vSTE in the
> > nested domain could have carried the bypass or abort configuration in GBPA
> > register. Thus, having such a hard requirement doesn't work well for GBPA.
> > 
> > Skip vmaster allocation in arm_smmu_attach_prepare_vmaster() for an abort
> > or bypass vSTE. Note that device on this attachment won't report vevents.
> > 
> > Update the uAPI doc accordingly.
> > 
> > Tested-by: Shameer Kolothum <skolothumtho@nvidia.com>
> > Signed-off-by: Nicolin Chen <nicolinc@nvidia.com>
> > ---
> > 
> > Changelog
> > v2
> >  * Add Tested-by from Shameer
> >  * Skip vmaster allocation instead of bypassing vsid=0
> >  * Revise the uAPI doc to note a corner case when CR0.SMMUEN=1
> > 
> >  drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-iommufd.c | 13 ++++++++++++-
> >  include/uapi/linux/iommufd.h                        |  9 +++++++++
> >  2 files changed, 21 insertions(+), 1 deletion(-)
> 
> Reviewed-by: Jason Gunthorpe <jgg@nvidia.com>
> 
> Will, I will grab this in a few days unless you want it in your tree

Sure, go for it.

Will

Re: [PATCH v2] iommu/arm-smmu-v3-iommufd: Allow attaching nested domain for GBPA cases

[Jason Gunthorpe]

On Mon, 03 Nov 2025 09:27:55 -0800, Nicolin Chen wrote:
> A vDEVICE has been a hard requirement for attaching a nested domain to the
> device. This makes sense when installing a guest STE, since a vSID must be
> present and given to the kernel during the vDEVICE allocation.
> 
> But, when CR0.SMMUEN is disabled, VM doesn't really need a vSID to program
> the vSMMU behavior as GBPA will take effect, in which case the vSTE in the
> nested domain could have carried the bypass or abort configuration in GBPA
> register. Thus, having such a hard requirement doesn't work well for GBPA.
> 
> [...]

Applied, thanks!

[1/1] iommu/arm-smmu-v3-iommufd: Allow attaching nested domain for GBPA cases
      https://git.kernel.org/jgg/iommufd/c/9f0b286fe40130

Best regards,
-- 
Jason Gunthorpe <jgg@nvidia.com>