Re: [PATCH v2 1/6] hw/arm/smmuv3: Add support to associate a PCIe RC

["Daniel P. Berrangé" <berrange@redhat.com>]

On Fri, May 09, 2025 at 11:37:14AM +0100, Peter Maydell wrote:
> On Fri, 9 May 2025 at 09:00, Shameerali Kolothum Thodi
> <shameerali.kolothum.thodi@huawei.com> wrote:
> > From: Peter Maydell <peter.maydell@linaro.org>
> > > The problem here seems to me to be that in the hardware we're
> > > modelling the SMMU always exists, because it's in the SoC,
> > > but you're trying to arrange for it to be created on the
> > > command line, via -device.
> > >
> > > We don't have any of these problems with the current 'virt'
> > > board code, because we have the board code create the iommu
> > > (if the user asks for it via the iommu machine property),
> > > and it can wire it up to the PCI root complex as needed.
> >
> > Yes, currently virt creates a SMMUv3 instance and associates it with
> > pcie.0 by default. However, this setup isn't ideal once we introduce
> > support for accelerated SMMUv3, where we need to configure the
> > host SMMUv3 for nested stage translation. This is essential for VFIO-PCI
> > passthrough scenarios, where the guest manages the stage-1 page tables
> > and the host manages stage-2.
> >
> > In such cases, devices may be associated with different host SMMUv3s,
> > and having a single vSMMUv3 in the guest isn't ideal as-
> >
> > -We would need to broadcast TLBIs or perform lookups to identify the
> >  corresponding host SMMUv3.
> > -The physical SMMUv3s might differ in functionality.
> > -Some SMMUv3 implementations offer additional virtualization features (e.g., vCMDQ),
> >   where the CMDQ is is directly gets mapped to the Guest and invalidations
> >   aren't  even trapped.
> >
> > Please refer to the earlier RFC [1], which proposed a new arm-smmuv3-accel
> > device. The main feedback on that RFC was to instead convert the existing
> > arm-smmuv3 into a user-creatable device, and introduce acceleration support
> > via an optional property:
> >
> > -device arm-smmuv3,accel=on,..
> >
> > This series is the first step toward that goal—making the current arm-smmuv3 device
> > user-creatable.
> 
> (I want to start here by saying that I appreciate that I'm
> coming in without having read the previous discussion, so
> this is kind of going back over ground you've already
> been through.)
> 
> I agree that rather than having an entirely separate "SMMU with
> acceleration" it would be better to have it be a property on
> the SMMU device. But why do we need it to be user created?
> Making it user-created leads into all kinds of tricky areas
> mostly surrounding the fact that QEMU right now simply doesn't
> support having user-created sysbus devices and other kinds
> of device with complex wiring-up. -device is really intended
> for "this is a model of a device that in real hardware is
> pluggable and has basically one connection, like a PCI card
> has a PCI-slot".

In terms of "why does it need to be user created" - the goal was to expose
multiple SMMUs to the guest, each associated with a separate physical SMMU.
IIUC, each physical NUMA node would have its own SMMU.

So configuring a guest VM will require creating multiple PXBs, one for
each virtual NUMA node, and then creating SMMUs for each PXB.

Since there was a need for the user to create SMMUs for the PXBs, the
question was then raised, why shouldn't the default SMMU also be
user creatable in the same way, so that mgmt apps like libvirt have
a single way to configure the SMMUs with -device.

With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|