Re: [PATCH v2 1/6] hw/arm/smmuv3: Add support to associate a PCIe RC

["Daniel P. Berrangé" <berrange@redhat.com>]

On Fri, May 09, 2025 at 07:29:28AM +0000, Shameerali Kolothum Thodi wrote:
> 
> 
> > -----Original Message-----
> > From: Donald Dutile <ddutile@redhat.com>
> > Sent: Thursday, May 8, 2025 2:45 PM
> > To: Shameerali Kolothum Thodi
> > <shameerali.kolothum.thodi@huawei.com>; Markus Armbruster
> > <armbru@redhat.com>
> > Cc: Shameer Kolothum via <qemu-devel@nongnu.org>; qemu-
> > arm@nongnu.org; eric.auger@redhat.com; peter.maydell@linaro.org;
> > jgg@nvidia.com; nicolinc@nvidia.com; berrange@redhat.com;
> > nathanc@nvidia.com; mochs@nvidia.com; smostafa@google.com; Linuxarm
> > <linuxarm@huawei.com>; Wangzhou (B) <wangzhou1@hisilicon.com>;
> > jiangkunkun <jiangkunkun@huawei.com>; Jonathan Cameron
> > <jonathan.cameron@huawei.com>; zhangfei.gao@linaro.org
> > Subject: Re: [PATCH v2 1/6] hw/arm/smmuv3: Add support to associate a
> > PCIe RC
> > 
> > 
> > 
> > On 5/7/25 4:50 AM, Shameerali Kolothum Thodi wrote:
> > >
> > >
> > >> -----Original Message-----
> > >> From: Markus Armbruster <armbru@redhat.com>
> > >> Sent: Wednesday, May 7, 2025 8:17 AM
> > >> To: Donald Dutile <ddutile@redhat.com>
> > >> Cc: Shameer Kolothum via <qemu-devel@nongnu.org>; qemu-
> > >> arm@nongnu.org; Shameerali Kolothum Thodi
> > >> <shameerali.kolothum.thodi@huawei.com>; eric.auger@redhat.com;
> > >> peter.maydell@linaro.org; jgg@nvidia.com; nicolinc@nvidia.com;
> > >> berrange@redhat.com; nathanc@nvidia.com; mochs@nvidia.com;
> > >> smostafa@google.com; Linuxarm <linuxarm@huawei.com>; Wangzhou
> > (B)
> > >> <wangzhou1@hisilicon.com>; jiangkunkun <jiangkunkun@huawei.com>;
> > >> Jonathan Cameron <jonathan.cameron@huawei.com>;
> > >> zhangfei.gao@linaro.org
> > >> Subject: Re: [PATCH v2 1/6] hw/arm/smmuv3: Add support to associate a
> > >> PCIe RC
> > >>
> > >> Donald Dutile <ddutile@redhat.com> writes:
> > >>
> > >> [...]
> > >>
> > >>> In this series, an iommu/smmu needs to be placed -BETWEEN- a sysbus
> > >> and a PCIe-tree,
> > >>> or step-wise, plug an smmuv3 into a sysbus, and a pcie tree/domain/RC
> > >> into an SMMUv3.
> > >>
> > >> RC = root complex?
> > >
> > > Yes.
> > >
> > +1.
> > 
> > >>
> > >>> So, an smmu needs to be associated with a bus (tree), i.e., pcie.0,
> > pcie.1...
> > >>> One could model it as a PCIe device, attached at the pcie-RC ... but
> > that's
> > >> not how it's modelled in ARM hw.
> > >>
> > >> Physical ARM hardware?
> > >>
> > yes, physical hw.
> > 
> > >> Assuming the virtual devices and buses we're discussing model physical
> > >> devices and buses:
> > >>
> > >> * What are the physical devices of interest?
> > >>
> > >> * How are they wired together?  Which of the wires are buses, in
> > >>    particular PCI buses?
> > >
> > > SMMUv3 is a platform device and its placement in a system is typically as
> > below
> > > for PCI devices,
> > >
> > > +------------------+
> > > |   PCIe Devices   |
> > > +------------------+
> > >           |
> > >           v
> > >    +-------------+      +---------------+
> > >    |  PCIe RC A  |<---->| Interconnect  |
> > >    +-------------+      +---------------+
> > >                                 |
> > >                                 |
> > >                          +------v---+
> > >                          | SMMUv3.A |
> > >                          | (IOMMU)  |
> > >                          +----+-----+
> > >                               |
> > >                               v
> > >                       +-------+--------+
> > >                       |   System RAM   |
> > >                       +----------------+
> > >
> > > This patch is attempting to establish that association between the PCIe
> > RC and
> > > the SMMUv3 device so that Qemu can build the ACPI tables/DT iommu
> > mappings
> > > for the SMMUv3 device.
> > >
> > I would refer to the ARM SMMU spec, Figure 2.3 in the G.a version, where
> > it's slightly different; more like:
> 
> That's right. The spec does indeed cover all possible scenarios, whereas my earlier
> comments were focused more specifically on the common case of systems using
> SMMUv3 with PCIe devices.
> 
> Currently, QEMU doesn't support non-PCI devices with SMMUv3, neither the
> more complex distributed SMMU cases you have described below. And this series
> doesn't aim to add those supports either. If needed, we can treat those as a separate
> efforts—similar to what was attempted in [1]. That said, agree that the design
> choices we make now should not hinder adding such support in the future.
> 
> And as far as I can see, nothing in this series would prevent that and if anything,
> the new device type SMMUv3 model actually makes it easier to support those.
> 
> >   +------------------+
> >   |   PCIe Devices   | (one device, unless a PCIe switch is btwn the RC &
> > 'Devices';
> >   +------------------+   or, see more typical expansion below)
> >             |
> >      +-------------+
> >      |  PCIe RC A  |
> >      +-------------+
> >             |
> >      +------v---+    +-----------------------------------+
> >      | SMMUv3.A |    | Wide assortment of other platform |
> >      | (IOMMU)  |    |   devices not using SMMU          |
> >      +----+-----+    +-----------------------------------+
> >           |                      |   |   |
> >    +------+----------------------+---+---+-+
> >    |         System Interconnect           |
> >    +---------------------------------------+
> >                                 |
> >    +-------+--------+     +-----+-------------+
> >    |   System RAM   |<--->| CPU (NUMA socket) |
> >    +----------------+     +-------------------+
> > 
> > In fact, the PCIe can be quite complex with PCIe bridges, and multiple Root
> > Ports (RP's),
> > and multiple SMMU's:
> > 
> >      +--------------+   +--------------+   +--------------+
> >      | PCIe Device  |   | PCIe Device  |   | PCIe Device  |
> >      +--------------+   +--------------+   +--------------+
> >            |                  |                  |        <--- PCIe bus
> >       +----------+       +----------+      +----------+
> >       | PCIe RP  |       | PCIe RP  |      | PCIe RP  |  <- may be PCI Bridge, may
> > not
> >       +----------+       +----------+      +----------+
> >           |                  |                  |
> >       +----------+       +----------+       +----------+
> >       |  SMMU    |       |  SMMU    |       |  SMMU    |
> >       +----------+       +----------+       +----------+
> >           |                  |                  |   <- may be a bus, may not(hidden from OS)
> >           +------------------+------------------+
> >                              |
> >              +--------------------------+
> >              |          PCI RC A        |
> >              +--------------------------+
> > 
> > where PCIe RP's could be represented (even virtually) in -hw-
> > as a PCIe bridge, each downstream being a different PCIe bus under
> > a single PCIe RC (A, in above pic) -domain-.
> > ... or the RPs don't have to have a PCIe bridge, and look like
> > 'just an RP' that provides a PCIe (pt-to-pt, serial) bus, provided
> > by a PCIe RC. ... the PCIe architecture allows both, and I've seen
> > both implementations in hw (at least from an lspci perspective).
> > 
> > You can see the above hw implementation by doing an lspci on most
> > PCIe systems (definitely common on x86), where the RP's are represented
> > by 'PCIe bridge' elements -- and lots of them.
> > In real hw, these RP's effectively become (multiple) up & downstream
> > transaction queues
> > (which implement PCI ordering, and deadlock avoidance).
> > SMMUs are effectively 'inserted' in the (upstream) queue path(s).
> > 
> > The important take away above: the SMMU can be RP &/or device-specific -
> > - they
> > do not have to be bound to an entire PCIe domain ... the *fourth* part of
> > an lspci output for a PCIe device: Domain:Bus:Device.Function.
> > This is the case for INTEL & AMD IOMMUs ... and why the ACPI tables have
> > to describe which devices (busses often) are associated with which
> > SMMU(in IORT)
> > or IOMMU(DMAR/IVRS's for INTEL/AMD IOMMU).
> > 
> > The final take away: the (QEMU) SMMU/IOMMU must be associated with a
> > PCIe bus
> > OR, the format has to be something like:
> >    -device smmuv3, id=smmuv3.1
> >    -device <blah>, smmu=smmuv3.1
> 
> Agree. For PCie devices with SMMUv3 we need to associate it with a PCIe bus
> and for non-pci cases probably needs a per device association.
> 
> > where the device <-> SMMU (or if extended to x86, iommu) associativity is
> > set w/o bus associativity.
> > It'd be far easier to tag an entire bus with an SMMU/IOMMU, than a per-
> > device format, esp. if
> > one has lots of PCIe devices in their model; actually, even if they only have
> > one bus and 8 devices
> > (common), it'd be nice if a single iommu/smmu<->bus-num associativity
> > could be set.
> > 
> > Oh, one final note: it is possible, although I haven't seen it done this way
> > yet,
> > that an SMMU could be -in- a PCIe switch (further distributing SMMU
> > functionality
> > across a large PCIe subsystem) and it -could- be a PCIe device in the switch,
> > btwn the upstream and downstream bridges -- actually doing the SMMU
> > xlations
> > at that layer..... for QEMU & IORT, it's associated with a PCIe bus.
> > But, if done correctly, that shouldn't matter -- in the example you gave wrt
> > serial,
> > it would be a new device, using common smmu core: smmuv3-pcie.
> > [Note: AMD actually identifies it's IOMMU as a PCIe device in an RC ... but
> > still uses
> >         the ACPI tables to configure it to the OS.. so the PCIe-device is basically
> > a
> >         device w/o a PCIe driver. AMD just went through hoops dealing with
> > MS
> >         and AMD-IOMMU-identification via PCIe.]
> > 
> > So, stepping back, and looking at a broad(er) SMMU -or- IOMMU QEMU
> > perspective,
> > I would think this type of format would be best:
> > 
> > - bus pcie, id=pcie.<num>
> > - device iommu=[intel_iommu|smmuv3|amd_iommu], bus=[sysbus |
> > pcie.<num>], id=iommu.<num>
> > [Yes, I'm sticking with 'iommu' as the generic naming... everyone thinks of
> > device SMMUs as IOMMUs,
> >   and QEMU should have a more arch-agnostic naming of these system
> > functions. ]
> 
> Ok. But to circle back to what originally started this discussion—how important
> is it to rely on the default "bus" in this case? As Markus pointed out, SMMUv3
> is a platform device on the sysbus, so its default bus type can’t point to something
> like PCIe. QEMU doesn’t currently support that.
> 
> The main motivation for using the default "bus" so far has been to have better
> compatibility with libvirt. Would libvirt be flexible enough if we switched to using
> something like a "primary-bus" property instead?

Sorry if my previous comments misled you, when I previously talked about
linking via a "bus" property I was not considering the fact that "bus"
is a special property inside QEMU. From a libvirt POV we don't care what
the property is call - it was just intended to be a general illustration
of cross-referencing the iommu with the PCI bus it needed to be associated
with.

> -device arm-smmuv3,primary-bus=pcie.0
> -device virtio-net-pci,bus=pcie.0
> -device pxb-pcie,id=pcie.1,bus_nr=2
> -device arm-smmuv3,primary-bus=pcie.1
> ...


With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|