Re: Query on ARM SMMUv3 nested support

[Jason Gunthorpe <jgg@nvidia.com>]

On Wed, Mar 13, 2024 at 04:50:58PM -0700, Nicolin Chen wrote:
> On Wed, Mar 13, 2024 at 10:13:58AM +0000, Shameerali Kolothum Thodi wrote:
> > Hi Nicolin,
> > 
> > Thanks for the latest repos with basic SMMUv3 nested support enabled[1].
> > I did some  basic sanity runs on a HiSilicon platform and they seems to work as
> > expected. The only problem being we can't assign two devices to the VM if
> > they are on different physical SMMUs.
> > 
> > qemu-system-aarch64: -device
> > vfio-pci,host=0000:75:00.1,iommufd=iommufd0: [iommufd=29] error attach
> > 0000:75:00.1 (36) to id=4: Invalid argument
> > qemu-system-aarch64: -device
> > vfio-pci,host=0000:75:00.1,iommufd=iommufd0: Unable to attach dev to
> > stage-2 HW pagetable: -1
> > Segmentation fault (core dumped)
> > ...
> > 
> > I see that on the Qemu side we now allocate a single s2 hwpt and attach that into
> > all the devices . But this will only work currently if all the assigned devices
> > are under the same physical SMMUv3 as we have a check in kernel
> > whether domains are having the same SMMUv3. I remember Jason mentioning
> > that he is planning to relax that. 

Yes, I intend this to be the direction as it makes the most sense


> > So are the Qemu side changes based on that assumption? And any
> > idea how we are planning to relax that restriction? We do a check
> > for compatibility of the phys SMMUv3s and then allow/restrict in
> > kernel?  

Right, so qemu should still handle the case where the HWPT is not
compatible and create a new HWPT. Regardless of the kernel side
optimization qemu is still missing this logic apparently.

> > Do Qemu can then try allocating a separate s2 hwpt for
> > those and attach again?  Sorry if this was already discussed
> > elsewhere and I missed that.

Yes, this is the right thing.

Jason