Re: [PATCH v4 13/16] iommu/amd: Track host Domain ID mapping for each guest Domain ID

[Jason Gunthorpe <jgg@nvidia.com>]

On Wed, Nov 05, 2025 at 05:50:44PM +0700, Suthikulpanit, Suravee wrote:
> Jason
> 
> On 10/23/2025 3:08 AM, Jason Gunthorpe wrote:
> > On Tue, Oct 21, 2025 at 01:43:21AM +0000, Suravee Suthikulpanit wrote:
> > > Each nested domain is assigned guest domain ID (gDomID), which guest OS
> > > programs into guest Device Table Entry (gDTE). For each gDomID, the driver
> > > assigns a corresponding host domain ID (hDomID), which will be programmed
> > > into the host Device Table Entry (hDTE).
> > > 
> > > The gDTE to hDTE 1:1 mapping is stored in the nest parent domain using
> > > an xarray (struct protection_domain.gdomid_array). When invalidate the
> > > nest parent domain, the INVALIDATE_IOMMU_PAGES must be issued for each
> > > hDomID in the gdomid_array.
> > 
> > I think this should be stored in the viommu..
> > 
> > It is a small unrealistic detail but very pedantically the API allows
> > creating two VIOMMU's from the same NEST PARENT domain and if someone
> > did this then each of the VIOMMU should have its own private gDomID
> > number space and own separated xarray.
> 
> Actually, to support nested translation w/ HW-based vIOMMU support in the
> guest w/ two VFIO devices on two different physical IOMMUs, it would require
> setting up two iommufd_viommu structures (one for each IOMMU) and share the
> same parent domain (single GPA-SPA mapping). Also, AMD HW-vIOMMU use a
> single domain ID (gDomID-to-hDomID) mapping table per guest-ID. Since the
> table is indexed using gDomID, it requires single gDomID space per guest.

Yes, this is why the guest ID needs to come from the viommu object,
not the parent domain.

> In this case, it makes more sense to store the gDomID-to-hDomID mapping in
> the parent domain since:
> 
>    1. There is one gDomID-space per guest and there is one parent domain per
> guest.

No. There is one gDomID-space *per viommu*. The driver has no concept
of what a guest is, that isn't part of our API.

For AMD hardware it is the viommu object that must hold the gDomID,
and all the other related HW datastructures for a "guest".

One viommu object per what the AMD spec calls a "guest".

>    2. When host issues invalidation for a parent domain, IOMMU driver needs
> to issue an invalidate command for each hDomId used for the same parent
> domain (on each IOMMU). We can't do this if we store xarray in the viommu.
> Otherwise, we would need to store a list of vIOMMUs per parent domain.

Correct, however this is just showing that the driver invalidation
logic is incorrect. See how Nicolin is working to fix this same
problem in the ARM driver. The driver cannot assume a domain has only
a single domain id in a viommu world.

For now you should still store the xarray in the viommu and then
prevent the parent domain from being used with more than one viommu
somehow so it doesn't get into trouble with the limited invalidation
logic.

Someday maybe we can fix the invalidation logic and allow domain
sharing. But for now the code should be structured along its own
limitations..

> > Allowing two VIOMMUs to share the same hDomID could be problematic
> > because we don't know the PASID layout is consistent.
> 
> Not sure why PASID layout matters here?

I'm saying you cannot share hDomID across two VIOMMU objects, it is
not allowed.

Jason