RE: nested-smmuv3 topic for QEMU/libvirt, Nov 2024

[Shameerali Kolothum Thodi via <qemu-devel@nongnu.org>]

Hi Nicolin,

> -----Original Message-----
> From: Nicolin Chen <nicolinc@nvidia.com>
> Sent: Friday, November 1, 2024 4:09 AM
> To: Shameerali Kolothum Thodi <shameerali.kolothum.thodi@huawei.com>
> Cc: Eric Auger <eric.auger@redhat.com>; Mostafa Saleh
> <smostafa@google.com>; qemu-arm@nongnu.org; qemu-
> devel@nongnu.org; Peter Maydell <peter.maydell@linaro.org>; Jason
> Gunthorpe <jgg@nvidia.com>; Jean-Philippe Brucker <jean-
> philippe@linaro.org>; Moritz Fischer <mdf@kernel.org>; Michael Shavit
> <mshavit@google.com>; Andrea Bolognani <abologna@redhat.com>;
> Michael S. Tsirkin <mst@redhat.com>; Peter Xu <peterx@redhat.com>;
> Zhangfei Gao <zhangfei.gao@linaro.org>; nathanc@nvidia.com;
> arighi@nvidia.com; ianm@nvidia.com; jan@nvidia.com; mochs@nvidia.com
> Subject: nested-smmuv3 topic for QEMU/libvirt, Nov 2024
> 
> Hi,
> 
> This is a continued discussion following previous month's:
> https://lore.kernel.org/qemu-devel/Zvr%2Fbf7KgLN1cjOl@Asurada-Nvidia/
> 
> Kernel changes are getting closer to merge, as Jason's planning to
> take vIOMMU series and smmuv3_nesting series into the iommufd tree:
> https://lore.kernel.org/all/cover.1730313494.git.nicolinc@nvidia.com/
> https://lore.kernel.org/all/cover.1730313494.git.nicolinc@nvidia.com/
> https://lore.kernel.org/all/0-v4-9e99b76f3518+3a8-
> smmuv3_nesting_jgg@nvidia.com/
> 
> So, I think it's probably a good time to align with each others and
> talk about kicking off some QEMU upstream work in the month ahead.
> 
> @Shameer,
> Do you have some update on the pluggable smmuv3 module?

I have a bare minimum prototype code that works with a pluggable smmuv3.

...
-device pxb-pcie,id=pcie.1,bus_nr=2,bus=pcie.0 \
-device pcie-root-port,id=pcie.port1,bus=pcie.1 \
-device arm-smmuv3-nested,id=smmuv1,pci-bus=pcie.1 \
-device vfio-pci-nohotplug,host=0000:75:00.1,bus=pcie.port1,iommufd=iommufd0 \
-device pxb-pcie,id=pcie.2,bus_nr=8,bus=pcie.0 \
-device pcie-root-port,id=pcie.port2,bus=pcie.2,chassis=8 \
-device arm-smmuv3-nested,id=smmuv2,pci-bus=pcie.2 \
-device vfio-pci-nohotplug,host=0000:7d:02.1,bus=pcie.port2,iommufd=iommufd0 \
...

Something like above can now boot a Guest with the latest kernel. But I am not
sure it actually works correctly. I need a bit more time to update this and carry
out some tests. Will target that in Nov.

> 
> Updates on my side:
> 1) I have kept uAPI updated to the latest version and verified too.
>    There should be some polishing changes depending on how the basic
>    nesting infrastructure would look like from Intel/Duan's work.
> 2) I got some help from NVIDIA folks for the libvirt task. And they
>    have done some drafting and are now verifying the PCI topology
>    with "iommu=none".
> 
> Once the pluggable smmuv3 module is ready to test, we will make some
> change to libvirt for that and drop the auto-assigning patches from
> the VIRT code, so as to converge for a libvirt+QEMU test.

One query I have is, do Qemu still need to check whether the VFIO devices are
assigned correctly behind the nested vSMMUv3 w.r.t the phys SMMUv3s or not?
Or we can just trust whatever the user/libvirt specifies?

(I think even if we don't explicitly check, at present  it will eventually fail in s2
HWPT attach  for the viommu if it belongs to a different phys SMMUv3).

Thanks,
Shameer