Re: nested-smmuv3 topic for QEMU/libvirt, Nov 2024

[Nicolin Chen <nicolinc@nvidia.com>]

On Fri, Nov 01, 2024 at 06:35:23PM +0000, Shameerali Kolothum Thodi wrote:
> > @Shameer,
> > Do you have some update on the pluggable smmuv3 module?
> 
> I have a bare minimum prototype code that works with a pluggable smmuv3.
> 
> ...
> -device pxb-pcie,id=pcie.1,bus_nr=2,bus=pcie.0 \
> -device pcie-root-port,id=pcie.port1,bus=pcie.1 \
> -device arm-smmuv3-nested,id=smmuv1,pci-bus=pcie.1 \
> -device vfio-pci-nohotplug,host=0000:75:00.1,bus=pcie.port1,iommufd=iommufd0 \
> -device pxb-pcie,id=pcie.2,bus_nr=8,bus=pcie.0 \
> -device pcie-root-port,id=pcie.port2,bus=pcie.2,chassis=8 \
> -device arm-smmuv3-nested,id=smmuv2,pci-bus=pcie.2 \
> -device vfio-pci-nohotplug,host=0000:7d:02.1,bus=pcie.port2,iommufd=iommufd0 \
> ...
> 
> Something like above can now boot a Guest with the latest kernel. But I am not
> sure it actually works correctly. I need a bit more time to update this and carry
> out some tests. Will target that in Nov.

That looks nice to me! Thanks for the update.

> > Updates on my side:
> > 1) I have kept uAPI updated to the latest version and verified too.
> >    There should be some polishing changes depending on how the basic
> >    nesting infrastructure would look like from Intel/Duan's work.
> > 2) I got some help from NVIDIA folks for the libvirt task. And they
> >    have done some drafting and are now verifying the PCI topology
> >    with "iommu=none".
> >
> > Once the pluggable smmuv3 module is ready to test, we will make some
> > change to libvirt for that and drop the auto-assigning patches from
> > the VIRT code, so as to converge for a libvirt+QEMU test.
> 
> One query I have is, do Qemu still need to check whether the VFIO devices are
> assigned correctly behind the nested vSMMUv3 w.r.t the phys SMMUv3s or not?
> Or we can just trust whatever the user/libvirt specifies?

That's a good point. I assume ideally QEMU should do something,
though that would be somewhat duplicated with libvirt.

> (I think even if we don't explicitly check, at present  it will eventually fail in s2
> HWPT attach  for the viommu if it belongs to a different phys SMMUv3).

Yes. That's what we have. Perhaps simply print something like:
"is the device assigned to the correct arm-smmuv3-nested/pxb?"
?

Thanks
Nicolin