RE: Query on ARM SMMUv3 nested support

[Shameerali Kolothum Thodi <shameerali.kolothum.thodi@huawei.com>]

> -----Original Message-----
> From: Nicolin Chen <nicolinc@nvidia.com>
> Sent: Friday, March 29, 2024 7:14 AM
> To: Shameerali Kolothum Thodi <shameerali.kolothum.thodi@huawei.com>
> Cc: Jason Gunthorpe <jgg@nvidia.com>; iommu@lists.linux.dev; Linuxarm
> <linuxarm@huawei.com>; Zhangfei Gao <zhangfei.gao@linaro.org>; Michael
> Shavit <mshavit@google.com>; Eric Auger <eric.auger@redhat.com>; Moritz
> Fischer <mdf@kernel.org>; baolu.lu@linux.intel.com
> Subject: Re: Query on ARM SMMUv3 nested support
> 
> Hi Shameer,
> 
> Sorry for the late reply. Having been occupied by multiple tasks
> lately.
> 
> On Fri, Mar 22, 2024 at 03:04:42PM +0000, Shameerali Kolothum Thodi
> wrote:
> 
> > Thanks for the reference to above CMDQ-V. Looks interesting.
> > I think we can have a go with retry logic first to allocate a new hwpt
> > if the attach fails. For now, I have a temp fix where I allocate a new
> > hwpt every time.
> >
> > I also noticed another problem with the commit below,
> > 6691a2f("hw/arm/smmu-common: Use sysmem for get_address _space
> until !!s2_hwpti")
> >
> > This actually breaks a virtio-pci dev assignment when that is the only
> > assigned device to the Guest without any pass-through dev in nested
> > scenario.
> >
> > I have a temp fix for that as well here,
> > (I know it is ugly!). I will revisit that again. Please let me know if you have
> > any ideas.
> 
> I am not quite getting the case here. Is that virtio-pci device
> behind a nested-SMMU while there is no S2 hwpt? In that case, why
> does this virtio-pci device require a "nested-smmuv3"?

Hi Nicolin,

Let me try to explain with a use case,

Consider a guest boot with below options,

./qemu-system-aarch64 virt,gic-version=3,iommu=nested-smmuv3,iommufd=iommufd0 \
-enable-kvm -cpu host -m 4G -smp cpus=8,maxcpus=8 \
-object iommufd,id=iommufd0 \
-bios QEMU_EFI.fd \
-kernel Image \
-device virtio-blk-device,drive=fs \
-drive if=none,file=ubuntu.img,id=fs \
-device ioh3420,id=rp1 \
-device virtio-9p-pci,fsdev=p9fs,mount_tag=p9 \
-fsdev local,id=p9fs,path=p9root,security_model=mapped \
--append "rdinit=init console=ttyAMA0 root=/dev/vda rw earlycon=pl011,0x9000000" \
-net none \
-nographic

Here we have specified nested-smmuv3 and have  a PCIe root port(ioh3420) and
a virtio-pci dev(for virtfs). Once this VM is up and running, both the ioh3420
and virtio-9p-pci will not work as expected as we return sysmem address space
for them.

And then later when you try to hotplug a vfio dev to the PCIe root port,
device_add vfio-pci,host=0000:7d:02.1,bus=rp1,id=net1,iommufd=iommufd0

it won't work either because the address space for ioh3420 is wrong.

Taking another look at this, I think we can replace my earlier attempt to fix
this by detecting if the dev is vfio or not and add that into the check below,

--- a/hw/arm/smmu-common.c
+++ b/hw/arm/smmu-common.c
@@ -619,8 +619,15 @@ static AddressSpace *smmu_find_add_as(PCIBus
*bus, void *opaque, int devfn)
     SMMUState *s = opaque;
     SMMUPciBus *sbus = smmu_get_sbus(s, bus);
     SMMUDevice *sdev = smmu_get_sdev(s, sbus, bus, devfn);
+    bool is_vfio = false;
+    PCIDevice *pdev;

-    if (s->nested && !s->s2_hwpt) {
+    pdev = pci_find_device(bus, pci_bus_num(bus), devfn);
+    if (object_dynamic_cast(OBJECT(pdev), "vfio-pci")) {
+        is_vfio = true;
+    }
+
+    if (is_vfio && s->nested && !s->s2_hwpt) {
         return &sdev->as_sysmem;
     } else {
         return &sdev->as;

Please let me know your thoughts on this.

Thanks,
Shameer