Re: Cache Invalidation Solution for Nested IOMMU

[Nicolin Chen <nicolinc@nvidia.com>]

On Tue, Apr 04, 2023 at 02:42:49AM +0000, Tian, Kevin wrote:
> External email: Use caution opening links or attachments
> 
> 
> > From: Nicolin Chen <nicolinc@nvidia.com>
> > Sent: Monday, April 3, 2023 11:24 PM
> >
> > > > VT-d side requires vPASID->pPASID and vDomain_id->pDomain_id
> > > > conversion.
> > > > vPASID conversion may be needed later as we may disable guest PASID
> > >
> > > vPASID conversion is mandatory when we enable vSVA on SIOV device.
> >
> > vPASID is allocated at runtime, so the hypercall timing is a
> > bit different than SMMU's vSID. But I think it could go with
> > this uAPI too? We'd just need to turn the uAPI to a shareable
> > one.
> 
> Not necessarily. It's clearer to be a separate cmd and format.

OK. Then set/unset_rid_user can be standalone, yet it likely
needs a better naming or so.

> > > But honestly speaking I'm hesitating to introduce native format and those
> > > assistant APIs for VT-d at this point. Supporting in-kernel short path
> > > won't happen in short term. What we defined now may not fit the
> > > requirement when it comes.
> > >
> > > With that let's continue to define a customized simple format for VT-d iotlb
> > > invalidation, plus allowing the user to batch the request. Having extra
> > > packing/unpacking overhead is negligible compared to the long invalidation
> > > path at this moment. Then we can consider native format as a 2nd
> > > supported format later when in-kernel acceleration is being worked on.
> >
> > It'd be okay to do it later for VT-d, so long as the uAPI we
> > add for SMMUv3 would potentially fit VT-d too :)
> >
> 
> Yes. btw you need decide which usage is comprehended in this design:
> 
> 1) vSMMU reads cmd from guest TLBI queue when the tail register is written
>    and then submits the cmd in a user-provided buffer to the kernel.
> 
>    This is the basic path.
> 
> 2) vSMMU reads base addr of guest TLBI queue when the start register is
>    written and registers the guest queue to the kernel. In the meantime
>    establish the protocol between kvm and smmu driver so when kvm
>    traps guest write to the tail register it directly notifies the smmu driver
>    and skips the userspace. smmu driver then directly reads cmd from guest
>    queue to handle.
> 
>    This is the in-kernel short path.
> 
> 3) with VCMDQ then vSMMU needs to mmap start/head/tail/... registers
>    of VCMDQ and allows the guest to directly access. No host intervention
>    when guest submits cmd to VCMDQ.
> 
>    This is the hw acceleration path.
> 
> I'm a bit confused in some discussions whether what you implemented
> for 1) must be forward compatible with 2) and 3). This is difficult before
> we actually start working on them. Given an iommu driver will support
> multiple formats (e.g. when vhost-iommu comes) probably we should
> more focus on the minimal necessary for what 1) actually requires now?

My draft is more of an enhanced (1) with batching, meanwhile
using a mmap interface, thinking of (3). Comparing to the (2),
it simplifies the host kernel, as QEMU could load every TLBI
command into a mmap'd buffer whenever it traps one.

Maybe (2) could be a cleaner implementation. I could try it
too.

Thanks
Nicolin