Re: [RFC PATCH 0/5] hw/arm/virt: Add support for user-creatable nested SMMUv3

["Daniel P. Berrangé" <berrange@redhat.com>]

On Thu, Feb 06, 2025 at 01:51:15PM +0000, Shameerali Kolothum Thodi wrote:
> Hmm..I don’t think just swapping the order will change the association with
> Guest SMMU here. Because, we have,
> 
> >  -device arm-smmuv3-accel,id=smmuv2,bus=pcie.2
> 
> During smmuv3-accel realize time, this will result in, 
>  pci_setup_iommu(primary_bus, ops, smmu_state);
> 
> And when the vfio dev realization happens,
>  set_iommu_device() 
>    smmu_dev_set_iommu_device(bus, smmu_state, ,)
>       --> this is where the guest smmuv3-->host smmuv3 association is first
>             established. And any further vfio dev to this Guest SMMU will
>             only succeeds if it belongs to the same phys SMMU.
> 
> ie, the Guest SMMU to pci bus association, actually make sure you have the
> same Guest SMMU for the device.

Ok, so at time of VFIO device realize, QEMU is telling the kernel
to associate a physical SMMU, and its doing this with the virtual
SMMU attached to PXB parenting the VFIO device.

> smmuv2 --> pcie.2 --> (pxb-pcie, numa_id = 1)
> 0000:dev2 -->  pcie.port2 --> pcie.2 --> smmuv2 (pxb-pcie, numa_id = 1)
> 
> Hence the association of 0000:dev2 to Guest SMMUv2 remain same.

Yes, I concur the SMMU physical <-> virtual association should
be fixed, as long as the same VFIO device is always added to
the same virtual SMMU.

> I hope this is clear. And I am not sure the association will be broken in any
> other way unless Qemu CLI specify the dev to a different PXB.

Although the ordering is at least predictable, I remain uncomfortable
about the idea of the virtual SMMU association with the physical SMMU
being a side effect of the VFIO device placement.

There is still the open door for admin mis-configuration that will not
be diagnosed. eg consider we attached VFIO device 1 from the host NUMA
node 1 to  a PXB associated with host NUMA node 0. As long as that's
the first VFIO device, the kernel will happily associate the physical
and guest SMMUs.

If we set the physical/guest SMMU relationship directly, then at the
time the VFIO device is plugged, we can diagnose the incorrectly
placed VFIO device, and better reason about behaviour.

I've another question about unplug behaviour..

 1. Plug a VFIO device for host SMMU 1 into a PXB with guest SMMU 1.
      => Kernel associates host SMMU 1 and guest SMMU 1 together
 2. Unplug this VFIO device
 3. Plug a VFIO device for host SMMU 2 into a PXB with guest SMMU 1.

Does the host/guest SMMU 1<-> 1 association remain set after step 2,
implying step 3 will fail ? Or does it get unset, allowing step 3
to succeed, and establish a new mapping host SMMU 2 to guest SMMU 1.

If step 2 does NOT break the association, do we preserve that
across a savevm+loadvm sequence of QEMU. If we don't, then step
3 would fail before the savevm, but succeed after the loadvm.

Explicitly representing the host SMMU association on the guest SMMU
config makes this behaviour unambiguous. The host / guest SMMU
relationship is fixed for the lifetime of the VM and invariant of
whatever VFIO device is (or was previously) plugged.

So I still go back to my general principle that automatic side effects
are an undesirable idea in QEMU configuration. We have a long tradition
of making everything entirely explicit to produce easily predictable
behaviour.

With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|