RE: Query on ARM SMMUv3 nested support

[Shameerali Kolothum Thodi <shameerali.kolothum.thodi@huawei.com>]

> -----Original Message-----
> From: Nicolin Chen <nicolinc@nvidia.com>
> Sent: Wednesday, March 13, 2024 11:51 PM
> To: Shameerali Kolothum Thodi <shameerali.kolothum.thodi@huawei.com>
> Cc: Jason Gunthorpe <jgg@nvidia.com>; iommu@lists.linux.dev; Linuxarm
> <linuxarm@huawei.com>; Zhangfei Gao <zhangfei.gao@linaro.org>; Michael
> Shavit <mshavit@google.com>; Eric Auger <eric.auger@redhat.com>; Moritz
> Fischer <mdf@kernel.org>
> Subject: Re: Query on ARM SMMUv3 nested support
> 
> On Wed, Mar 13, 2024 at 10:13:58AM +0000, Shameerali Kolothum Thodi
> wrote:
> > Hi Nicolin,
> >
> > Thanks for the latest repos with basic SMMUv3 nested support enabled[1].
> > I did some  basic sanity runs on a HiSilicon platform and they seems to
> work as
> > expected. The only problem being we can't assign two devices to the VM if
> > they are on different physical SMMUs.
> >
> > qemu-system-aarch64: -device
> > vfio-pci,host=0000:75:00.1,iommufd=iommufd0: [iommufd=29] error
> attach
> > 0000:75:00.1 (36) to id=4: Invalid argument
> > qemu-system-aarch64: -device
> > vfio-pci,host=0000:75:00.1,iommufd=iommufd0: Unable to attach dev to
> > stage-2 HW pagetable: -1
> > Segmentation fault (core dumped)
> > ...
> >
> > I see that on the Qemu side we now allocate a single s2 hwpt and attach
> that into
> > all the devices . But this will only work currently if all the assigned devices
> > are under the same physical SMMUv3 as we have a check in kernel
> > whether domains are having the same SMMUv3. I remember Jason
> mentioning
> > that he is planning to relax that. So are the Qemu side changes based on
> that
> > assumption? And any idea how we are planning to relax that restriction?
> We
> > do a check for compatibility of the phys SMMUv3s and then allow/restrict
> in kernel?
> > Do Qemu can then try allocating a separate s2 hwpt for those and attach
> again?
> > Sorry if this was already discussed elsewhere and I missed that.
> 
> You are very right about this. I haven't thought about supporting
> that case yet. Likely I need to spare some time to refine the QEMU
> part in the coming weeks or so (have been waiting for Zhenzhong's
> full nesting patches).
> 
> Also, NVIDIA has an interest to support CMDQ-V accelerator, where
> we need more vSMMU instances, than just one. I haven't decided how
> to handle this for both single-vSMMU and multi-vSMMU versions.
> 
> Yet, some of the ongoing work might help:
> https://github.com/nicolinc/iommufd/commit/b7520901184fd9fa127abb88c
> 1f0be16b9967cff
> https://github.com/nicolinc/iommufd/commit/d969ffeac899491d3a6e54557
> bf6a46d52b95865
> So, each device should poll hw_info to get its SMMU ID, if it is
> not in the S2 list of the vSMMU's, add it and allocate a new S2.

Thanks for the reference to above CMDQ-V. Looks interesting.
I think we can have a go with retry logic first to allocate a new hwpt
if the attach fails. For now, I have a temp fix where I allocate a new
hwpt every time.

I also noticed another problem with the commit below,
6691a2f("hw/arm/smmu-common: Use sysmem for get_address _space until !!s2_hwpti")

This actually breaks a virtio-pci dev assignment when that is the only 
assigned device to the Guest without any pass-through dev in nested
scenario.

I have a temp fix for that as well here,
https://github.com/hisilicon/qemu/commit/ff0230e9593ba1a172aa7ec9162967d6ae6875b0

(I know it is ugly!). I will revisit that again. Please let me know if you have
any ideas.

FWIW, I have a working branch for vSVA here,
https://github.com/hisilicon/qemu/tree/iommufd_vsmmu-02292024-vsva-wip-v2

I still have that io_uring write issue mentioned in the other thread[1],
but for now I use io_uring API only for read and make use of normal write()
for updating kernel with page response.

I have done some basic sanity runs with our ACC devices and those seems
to work(of course needs more testing). 

Please have a look and let me know if you spot anything.

Thanks,
Shameer
1. https://lore.kernel.org/all/ad4575588dd247fa8beae60963f36404@huawei.com/