Re: nested-smmuv3 topic, Sep 2024

[Nicolin Chen <nicolinc@nvidia.com>]

On Mon, Sep 30, 2024 at 10:45:31AM +0000, Shameerali Kolothum Thodi wrote:
> > -----Original Message-----
> > From: Nicolin Chen <nicolinc@nvidia.com>
> > Sent: Thursday, September 5, 2024 9:37 PM
> > To: Shameerali Kolothum Thodi <shameerali.kolothum.thodi@huawei.com>
> > Cc: Eric Auger <eric.auger@redhat.com>; Mostafa Saleh
> > <smostafa@google.com>; qemu-arm@nongnu.org; qemu-
> > devel@nongnu.org; Peter Maydell <peter.maydell@linaro.org>; Jason
> > Gunthorpe <jgg@nvidia.com>; Jean-Philippe Brucker <jean-
> > philippe@linaro.org>; Moritz Fischer <mdf@kernel.org>; Michael Shavit
> > <mshavit@google.com>; Andrea Bolognani <abologna@redhat.com>;
> > Michael S. Tsirkin <mst@redhat.com>; Peter Xu <peterx@redhat.com>
> > Subject: Re: nested-smmuv3 topic, Sep 2024
> >
> > Hi Shameer,
> >
> > Thanks for the reply!
> >
> > On Thu, Sep 05, 2024 at 12:55:52PM +0000, Shameerali Kolothum Thodi
> > wrote:
> > > > The main takeaway from the discussion is to
> > > > 1) Turn the vSMMU module into a pluggable one, like intel-iommu
> > > > 2) Move the per-SMMU pxb bus and device auto-assign into libvirt
> > > >
> > > > Apart from the multi-vSMMU thing, there's basic nesting series:
> > > > 0) Keep updating to the latest kernel uAPIs to support nesting
> > >
> > > By this you mean the old HWPT based nested-smmuv3 support?
> >
> > HWPT + vIOMMU. The for-viommu/virq branches that I shared in my
> > kernel series have those changes. Invalidations is done via the
> > vIOMMU infrastructure.
> >
> > > >
> > > > I was trying to do all these three, but apparently too ambitious.
> > > > The kernel side of work is still taking a lot of my bandwidth. So
> > > > far I had almost-zero progress on task (1) and completely-zero on
> > > > task (2).
> > > >
> > > > <-- Help Needed --->
> > > > So, I'm wondering if anyone(s) might have some extra bandwidth in
> > > > the following months helping these two tasks, either of which can
> > > > be a standalone project I think.
> > > >
> > > > For task (0), I think I can keep updating the uAPI part, although
> > > > it'd need some help for reviews, which I was hoping to occur after
> > > > Intel sends the QEMU nesting backend patches. Once we know how big
> > > > the rework is going to be, we may need to borrow some help at that
> > > > point once again..
> > >
> > > I might have some bandwidth starting October and can take a look at
> > > task 1 above. I haven't gone through the VIOMMU API model completely
> > > yet and plan to do that soon.
> >
> 
> I had  an initial look at this and also had some discussions with Eric at KVM
> Forum(Thanks Eric!).

Wow, thank both of you!

> Going through the code, is it ok to introduce a "pci-bus" for the proposed
> nested SMMUv3 device which will create the link between the SMMUv3 dev
> and the associated root complex(pxb-pcie).
> 
> Something like below,
> 
> -device pxb-pcie,id=pcie.1,bus_nr=2,bus=pcie.0 \
> -device arm-nested-smmuv3,pci-bus=pcie.1 \
> -device pcie-root-port,id=pcie.port1,bus=pcie.1 \
> -device vfio-pci,host=0000:75:00.1, bus=pcie.port1 \
> ...
> -device pxb-pcie,id=pcie.2,bus_nr=8,bus=pcie.0 \
> -device arm-nested-smmuv3,pci-bus=pcie.2 \
> -device pcie-root-port,id=pcie.port2,bus=pcie.2 \
> -device vfio-pci,host=0000:75:00.2, bus=pcie.port2 \
> 
> This way we can invoke the pci_setup_iommu() with the
> right PCIBus during the nested SMMUv3 device realize fn.
> 
> Please let me know, if this works/scales with all the use cases we have.

That looks nice to me. Hopefully, IORT or Device Tree would be
easy to tie to the corresponding pci-bus as well..

> Also Eric mentioned that when he initially added the support for SMMUv3,
> the initial approach was -device based solution, but later changed to machine
> option instead based on review comments. I managed to find the link where
> this change was proposed(by Peter),
> 
> https://lore.kernel.org/all/CAFEAcA_H+sraWNVhEZc48eS11n6dC9CyEwTL44tPERiPBO+hbw@mail.gmail.com/
> 
> I hope the use cases we now have make it reasonable to introduce a "-device arm-nested-smmuv3" model.
> Please let me know if there are still objections to going this way.

I assume so. With multiple smmuv3 devices in the VM, we would need
this kinda flexibility to create them.

And FYI, I also found some resource in NVIDIA who will help me on
the QEMU workload, including our remaining task -- libvirt. I'll
align with them in the days ahead, and will keep all of us updated
after.

Thanks!
Nicolin