Re: Cache Invalidation Solution for Nested IOMMU

[Nicolin Chen <nicolinc@nvidia.com>]

On Thu, Apr 06, 2023 at 08:40:04AM -0300, Jason Gunthorpe wrote:
> On Thu, Apr 06, 2023 at 02:23:17PM +0800, Zhangfei Gao wrote:
> 
> > We are using ioctl method now.
> > From the testing, the TLB miss impacts performance a lot, so we use
> > huge page method.
> > After using huge page method, guest can achieve comparable performance
> > with host.
> 
> Looks like these tests are not stressing the MM, just measuring pure
> BW of the DMA, so they don't get into the invalidation regime..
> 
> You need to measure a more real application that is actually using the
> MM (eg alloc/free memory, fork, etc) while it operates and turn on SVA.

Would an iommu map/unmap benchmark test be useful here?

I added a test program to measure map/unmap times with a
set of different sized buffers:
https://github.com/nicolinc/iommufd/commit/3eb417f2cae0234cc801c6ad74de2afb0ddbdf84
(Also thinking about sending this with a RFC series)

@Zhangfei,
In case that this could be useful, you can pull these two
branches for perf measurement with and without mmap:
# Kernel
https://github.com/nicolinc/iommufd/commits/wip/iommufd_nesting-mmap-04082023
# QEMU
https://github.com/nicolinc/qemu/commits/wip/iommufd_nesting-mmap-04072023

To test without mmap, simply revert the following commits:
# Kernel
git revert ecf602a3c8480ba7ce2c7e77c2d15ca873dbf2e4
# QEMU
git revert c726c014de70998f14b8741a6a96e18a2a7bcd0f

And, you can get the test script in the branch:
  tools/testing/selftests/iommu/iommu_benchmark.sh


On my emulation environment (very slow), with mmap, I see
improvements. I'll also try setting up a test suite on a
proper HW this week.

Thanks
Nicolin