RE: Cache Invalidation Solution for Nested IOMMU

[Shameerali Kolothum Thodi <shameerali.kolothum.thodi@huawei.com>]

> -----Original Message-----
> From: Zhangfei Gao [mailto:zhangfei.gao@linaro.org]
> Sent: 12 April 2023 03:48
> To: Nicolin Chen <nicolinc@nvidia.com>
> Cc: Jason Gunthorpe <jgg@nvidia.com>; Shameerali Kolothum Thodi
> <shameerali.kolothum.thodi@huawei.com>; Robin Murphy
> <robin.murphy@arm.com>; kevin.tian@intel.com; yi.l.liu@intel.com;
> eric.auger@redhat.com; baolu.lu@linux.intel.com; jean-philippe@linaro.org;
> iommu@lists.linux.dev; qianweili <qianweili@huawei.com>
> Subject: Re: Cache Invalidation Solution for Nested IOMMU
> 
> Hi, Nicolin
> 
> On Mon, 10 Apr 2023 at 09:08, Nicolin Chen <nicolinc@nvidia.com> wrote:
> >
> > On Thu, Apr 06, 2023 at 08:40:04AM -0300, Jason Gunthorpe wrote:
> > > On Thu, Apr 06, 2023 at 02:23:17PM +0800, Zhangfei Gao wrote:
> > >
> > > > We are using ioctl method now.
> > > > From the testing, the TLB miss impacts performance a lot, so we
> > > > use huge page method.
> > > > After using huge page method, guest can achieve comparable
> > > > performance with host.
> > >
> > > Looks like these tests are not stressing the MM, just measuring pure
> > > BW of the DMA, so they don't get into the invalidation regime..
> > >
> > > You need to measure a more real application that is actually using
> > > the MM (eg alloc/free memory, fork, etc) while it operates and turn on
> SVA.
> >
> > Would an iommu map/unmap benchmark test be useful here?
> >
> > I added a test program to measure map/unmap times with a set of
> > different sized buffers:
> >
> https://github.com/nicolinc/iommufd/commit/3eb417f2cae0234cc801c6ad
> 74d
> > e2afb0ddbdf84 (Also thinking about sending this with a RFC series)
> >
> > @Zhangfei,
> > In case that this could be useful, you can pull these two branches for
> > perf measurement with and without mmap:
> > # Kernel
> >
> https://github.com/nicolinc/iommufd/commits/wip/iommufd_nesting-mma
> p-0
> > 4082023
> > # QEMU
> >
> https://github.com/nicolinc/qemu/commits/wip/iommufd_nesting-mmap-0
> 407
> > 2023
> >
> > To test without mmap, simply revert the following commits:
> > # Kernel
> > git revert ecf602a3c8480ba7ce2c7e77c2d15ca873dbf2e4
> > # QEMU
> > git revert c726c014de70998f14b8741a6a96e18a2a7bcd0f
> >
> > And, you can get the test script in the branch:
> >   tools/testing/selftests/iommu/iommu_benchmark.sh
> >
> >
> > On my emulation environment (very slow), with mmap, I see
> > improvements. I'll also try setting up a test suite on a proper HW
> > this week.
> >
> 
> Still debugging the mmap method here,
> 
> Status 4.12
> 
> 1.  ioctl method works, based on iommufd
> 
> performance same as before, ie.6.2
> 
> Using huge page method for guest, guest app can get comparable
> performance as host
> 
> (The branches are provided by Shameer)
> https://github.com/gaozhangfei/linux-kernel-uadk/tree/private-iommufd_ne
> sting-03272023-6.3-rc4-arm
> https://github.com/gaozhangfei/qemu/tree/private-v7.2.0-iommufd-nesting
> -arm
> 
> 
> 2.  mmap method still in debug,
> (patch porting may have issues, need to port patches to the working version,
> in check)
> 
> qemu:
> 
> https://github.com/gaozhangfei/linux-kernel-uadk/tree/iommufd_nesting-m
> map-04082023
> 
> https://github.com/gaozhangfei/qemu/tree/private-v7.2.0-iommufd-nesting
> -arm-mmap

Hi Zhangfei,

I had a go with your above branches. The issue below seems to happen only
when you assign multiple devices to Guest. It looks like when you have multiple devices
attached to Guest, the host kernel receives invalid Guest cmd(0x0) and ends up issuing
that to HW triggering the below errors. 

For now, could you please try with just one device? At least in my setup it didn’t trigger
the below with just one dev.

Also as Jason pointed out, I think we might need to run an application that actually
hits the mm invalidation path in Guest kernel(arm_smmu_mm_invalidate_range()).
With test_sva_perf I couldn't see much of that happening during the test run.

Also another point is once we have BTM enabled we might not have these mm
invalidations reaching the host kernel. 

Thanks,
Shameer 
> 
> hisi sec init Kunpeng920!
> 
> qemu-system-aarch64: IOMMU_HWPT_INVALIDATE failed: No such device
> 
> qemu-system-aarch64: --------smmuv3_invalidate_cache_mmap: failed to
> invalidate TLB: prod=15fa vs cons=1000709
> 
> qemu-system-aarch64: IOMMU_HWPT_INVALIDATE failed: No such device
> 
> qemu-system-aarch64: --------smmuv3_invalidate_cache_mmap: failed to
> invalidate TLB: prod=9bb vs cons=1000709
> 
> qemu-system-aarch64: IOMMU_HWPT_INVALIDATE failed: Connection timed
> out
> 
> qemu-system-aarch64: --------smmuv3_invalidate_cache_mmap: failed to
> invalidate TLB: prod=4c0 vs cons=4c0
> 
> qemu-system-aarch64: IOMMU_HWPT_INVALIDATE failed: No such device
> 
> qemu-system-aarch64: --------smmuv3_invalidate_cache_mmap: failed to
> invalidate TLB: prod=f7a vs cons=1000709
> 
> qemu-system-aarch64: IOMMU_HWPT_INVALIDATE failed: No such device
> 
> qemu-system-aarch64: --------smmuv3_invalidate_cache_mmap: failed to
> invalidate TLB: prod=81d vs cons=1000709
> 
> host:
> 
> [218.986319] arm-smmu-v3 arm-smmu-v3.3.auto: unexpected global error
> reported (0x00000001), this could be serious
> 
> [218.996464] arm-smmu-v3 arm-smmu-v3.3.auto: CMDQ error (cons
> 0x01004805): Illegal command
> 
> [219.004606] arm-smmu-v3 arm-smmu-v3.3.auto: skipping command in
> error state:
> 
> [219.011622] arm-smmu-v3 arm-smmu-v3.3.auto:0x0060000202110687
> 
> [219.017515] arm-smmu-v3 arm-smmu-v3.3.auto:0x0000000000000000
> 
> [219.023422] arm-smmu-v3 arm-smmu-v3.3.auto: unexpected global error
> reported (0x00000001), this could be serious
> 
> [219.033550] arm-smmu-v3 arm-smmu-v3.3.auto: CMDQ error (cons
> 0x01004806): Illegal command
> 
> [219.041692] arm-smmu-v3 arm-smmu-v3.3.auto: skipping command in
> error state:
> 
> [219.048707] arm-smmu-v3 arm-smmu-v3.3.auto:0x0000000000000000
> 
> [219.054600]arm-smmu-v3 arm-smmu-v3.3.auto:0x0000000000000000