Re: Cache Invalidation Solution for Nested IOMMU

[Nicolin Chen <nicolinc@nvidia.com>]

On Mon, Apr 03, 2023 at 08:39:02AM +0000, Tian, Kevin wrote:
> External email: Use caution opening links or attachments
> 
> 
> > From: Liu, Yi L <yi.l.liu@intel.com>
> > Sent: Monday, April 3, 2023 3:26 PM
> >
> > For VT-d, except for the device-TLB invalidation, there is no RID
> > information in the invalidation command submitted by iommu driver.
> > Device-TLB invalidation would be somehow an included operation in
> > IOTLB invalidation if the page table is used by devices that has
> > enabled ATS. So guest VT-d iommu driver may not use the Device-TLB
> > invalidation. So vRID->pRID conversion may not happen for VT-d side.
> 
> Guest can still submit a device-tlb invalidation descriptor. Just that it
> could be skipped as a nop here if the host always invalidates device-tlb
> when handling the iotlb invalidation request.

In either case, it sounds like RID isn't needed in VT-d case?

SMMU needs vSID<->pSID info to verify an ATS invalidation.

> > VT-d side requires vPASID->pPASID and vDomain_id->pDomain_id
> > conversion.
> > vPASID conversion may be needed later as we may disable guest PASID
> 
> vPASID conversion is mandatory when we enable vSVA on SIOV device.

vPASID is allocated at runtime, so the hypercall timing is a
bit different than SMMU's vSID. But I think it could go with
this uAPI too? We'd just need to turn the uAPI to a shareable
one.

> > support at this moment. For vDomain_id conversion, there is a gap.
> > Domain_id is not global today. So if there are multiple vIOMMU instance
> > exposed to guest, the vDomain_id would have conflict between the
> > invalidation commands submitted via different vIOMMU instances. Even
> > we may make its allocation global, I'm also curious when and how should
> > we build up the vDomain_id->pDomain_id relationship in kernel. Maybe a
> > new iommufd uapi just like the vRID set? or it may be part of the
> > device driver's attach_hwpt uapi (e.g. VFIO) as this relationship should
> > just exist when the hwpt and device are attached.
> >
> 
> IMHO we need a vDID->hwpt conversion i.e. the kernel wants to know
> which s1 hwpt is covered by a vDID. From this angle adding vDID at
> attach_hwpt makes more sense.

I share the same view. A "vDID" is HWPT oriented, while that
set/unset_rid_user uAPI is for passthrough devices. So, it'd
be probably better to go with a different uAPI at least.

> But honestly speaking I'm hesitating to introduce native format and those
> assistant APIs for VT-d at this point. Supporting in-kernel short path
> won't happen in short term. What we defined now may not fit the
> requirement when it comes.
>
> With that let's continue to define a customized simple format for VT-d iotlb
> invalidation, plus allowing the user to batch the request. Having extra
> packing/unpacking overhead is negligible compared to the long invalidation
> path at this moment. Then we can consider native format as a 2nd
> supported format later when in-kernel acceleration is being worked on.

It'd be okay to do it later for VT-d, so long as the uAPI we
add for SMMUv3 would potentially fit VT-d too :)

Thanks
Nicolin