Re: Cache Invalidation Solution for Nested IOMMU

[Nicolin Chen <nicolinc@nvidia.com>]

On Thu, Apr 06, 2023 at 02:23:17PM +0800, Zhangfei Gao wrote:
> External email: Use caution opening links or attachments
> 
> 
> On Wed, 5 Apr 2023 at 00:50, Shameerali Kolothum Thodi
> <shameerali.kolothum.thodi@huawei.com> wrote:
> >
> >
> >
> > > -----Original Message-----
> > > From: Jason Gunthorpe [mailto:jgg@nvidia.com]
> > > Sent: 04 April 2023 17:20
> > > To: Nicolin Chen <nicolinc@nvidia.com>
> > > Cc: Robin Murphy <robin.murphy@arm.com>; kevin.tian@intel.com;
> > > yi.l.liu@intel.com; eric.auger@redhat.com; baolu.lu@linux.intel.com;
> > > Shameerali Kolothum Thodi <shameerali.kolothum.thodi@huawei.com>;
> > > jean-philippe@linaro.org; iommu@lists.linux.dev
> > > Subject: Re: Cache Invalidation Solution for Nested IOMMU
> > >
> > > On Mon, Apr 03, 2023 at 05:02:09PM -0700, Nicolin Chen wrote:
> > >
> > > > My preference is to have a mmap'd page, so the interface can
> > > > be reused later by VCMDQ too. Performance-wise, it should be
> > > > good enough, since it does batching, IMHO.
> > >
> > > You can't reuse mmaping the queue page with vcmdq, so it doesn't seem
> > > meaningful to me.
> > >
> > > There should be no mmap on the SW path. If you need a half step
> > > between an ioctl as a batch and a full vhost-like queue scheme then
> > > using iouring with pre-registered memory would be appropriate.
> > >
> > > I feel like this is a topic Shameerali should share some insight with
> > > since the Huawei implementation will rely on this SW path.
> >
> > So far the tests we have done are mainly using IOCTL method without
> > batching and I don't have any numbers to compare against the batched
> > method yet.
> >
> > [+Zhangfei]
> >
> > Zhangfei, do you think we can run some UADK/vSVA tests with different
> > Invalidation solutions discussed here and compare?
> 
> Hi, Nicolin
> 
> Do you have mmap branch for kernel and Qemu?
> 
> We are using ioctl method now.
> From the testing, the TLB miss impacts performance a lot, so we use
> huge page method.
> After using huge page method, guest can achieve comparable performance
> with host.

Looks like the test has been running on previous VFIO solution?

Have you tried a sanity using the latest IOMMUFD pair? I recall
that Shameer has a set of branches for linux and QEMU.

Meanwhile, I'll prepare a pair of branches for you to test the
mmap solution. Let me get back to you by the end of the week.

Thanks
Nicolin