Re: Cache Invalidation Solution for Nested IOMMU

[Jason Gunthorpe <jgg@nvidia.com>]

On Tue, Apr 04, 2023 at 10:45:56PM -0700, Nicolin Chen wrote:
> On Tue, Apr 04, 2023 at 01:20:01PM -0300, Jason Gunthorpe wrote:
> > On Mon, Apr 03, 2023 at 05:02:09PM -0700, Nicolin Chen wrote:
> > 
> > > My preference is to have a mmap'd page, so the interface can
> > > be reused later by VCMDQ too. Performance-wise, it should be
> > > good enough, since it does batching, IMHO.
> > 
> > You can't reuse mmaping the queue page with vcmdq, so it doesn't seem
> > meaningful to me.
> > 
> > There should be no mmap on the SW path. If you need a half step
> > between an ioctl as a batch and a full vhost-like queue scheme then
> > using iouring with pre-registered memory would be appropriate.
> 
> I've changed to a non-mmap approach that the host kernel reads
> the guest queue directly and inserts all invalidation commands
> into the host queue.
> 
> The qsz could be as large as 128 x 64K pages. So, there has to
> be a big array of pages getting pinned in the handler.
> 
> (The handler still needs a pathway to report errors. I will add
>  tomorrow.)
> 
> Does the implementation below look fine in general?

In general get_user_pages() is really slow compared to copy_from_user

Jason