Re: Cache Invalidation Solution for Nested IOMMU

[Nicolin Chen <nicolinc@nvidia.com>]

On Tue, Apr 04, 2023 at 02:15:38AM +0000, Tian, Kevin wrote:
> External email: Use caution opening links or attachments
> 
> 
> > From: Nicolin Chen <nicolinc@nvidia.com>
> > Sent: Monday, April 3, 2023 10:30 PM
> >
> > On Mon, Apr 03, 2023 at 08:00:12AM +0000, Tian, Kevin wrote:
> > > External email: Use caution opening links or attachments
> > >
> > >
> > > > From: Nicolin Chen <nicolinc@nvidia.com>
> > > > Sent: Monday, April 3, 2023 8:34 AM
> > > >
> > > > The new set_rid/unset_rid ioctls and the mmap interface would be
> > > > essential for VCMDQ support that we'd like to achieve at the end
> > > > of this journey. So, personally I'd like to see it can be used at
> > > > this stage, by the generic SMMUv3 (and potentially VT-d) too.
> > > >
> > >
> > > We talked earlier that there could be multiple VCMDQ's when the
> > > guest is assigned multiple devices behind different SMMU's. How
> > > does the mmap interface per iommufd work in that scenario?
> >
> > Trying to documenting that each IOMMUFD object can possibly have
> > a shared page, the mmap interface takes the index of an IOMMUFD
> > object ID. So, either a pt_id(S1) or a dev_id should be able to
> > identify which physical SMMU, I think.
> 
> Are all allowed cmds in VCMDQ per hwpt? If not then building the
> mmap interface per hwpt object is not correct. We may want explicit
> VCMDQ object in that case.

One VCMDQ HW per SMMU instance. So all HWPTs that are created
by devices behind the same SMMU instance share the same VCMDQ
HW. Each VCMDQ HW can also allocate multiple queues that don't
necessarily tie to any HWPT either.

> and devices behind different SMMU's may be attached to a same
> hwpt. In that case the number of VCMDQ associated to a hwpt is
> also dynamic.

Unless two HWPTs share the same S1 Context Table, how can two
devices behind different SMMUs attach to the same HWPT? And,
it doesn't sound very plausible to share the same S1 Context
Table between two devices either?

> But if just talking about batching for emulated smmu then having
> the user to pass a big buffer makes more sense.

OK. That's in align with Jason's suggestion, passing a queue
buffer via the ioctl.

> > > and looks this is different from the requirement of having a
> > > software short path in kernel to reduce the invalidation overhead
> > > for emulated vIOMMUs. In this case the invalidation queue is
> > > in guest memory then instead we want a registration cmd here.
> >
> > Yes for the first part. There are certain difficulties of doing
> > a short path, such as host kernel replacing the host queue that
> > the actual HW ran on, with a guest TLBI queue. So, my draft is
> > more about batching.
> >
> > For the last part, what's that "registration cmd" to do? In my
> > draft, the hypervisor dispatch all invalidation commands to the
> > guest TLBI queue (or call it user queue), which is transparent
> > to the guest OS.
> >
> 
> registration means the user to pass the buffer to the kernel.
> 
> If we want to support kernel short path, then we want the host
> smmu driver to directly read cmd out of guest TLBI queue.

I expect the mmap approach can go a bit further for an SMMU
that has ECMDQ capability (multi CMDQs): it could load the
guest TLBI queue without copying that buffer and inserting
into the host queue, by allocating a separate cmdq object
in the SMMU driver and mmap'ing its cmdq->q.base to QEMU.

Thanks
Nicolin